Skip to main content

MongoDB

The mongodb connector reads documents from MongoDB collections with find queries or aggregate pipelines. Reads never change the database. Spark job outputs can also write to a collection.

Install​

sw connectors -i mongodb

Connect​

Use one URI secret, or the connection keys. Not both.

source.yaml
kind: source
name: mongodb_demo
spec:
config:
mongo_database: { type: string, default: analytics }
secrets:
mongo_uri: { type: string }
auth:
provider: mongodb
uri: "{{ secrets.mongo_uri }}"
database: "{{ config.mongo_database }}"
KeyWhat it isRequired / default
uriA mongodb:// or mongodb+srv:// URI; its path is the database. SecretRequired, or the keys below
hostThe server's host nameRequired (with keys)
portThe port27017
databaseThe databaseRequired (with keys)
usernameThe user (set together with password)Optional
passwordThe user's password. SecretOptional
auth_sourceThe database that holds the useradmin

With the connection keys:

source.yaml
kind: source
name: mongodb_demo
spec:
secrets:
mongo_password: { type: string }
auth:
provider: mongodb
host: localhost
database: analytics
username: reader
password: "{{ secrets.mongo_password }}"

A password inside the URI is redacted (***) in logs and errors too.

Read data​

A request uses service: collection:

MethodArgumentsWhat it does
findcollection, filter, projection, sort, limit, database, batch_sizeReads the documents filter matches. sort is a list of [field, direction] pairs
aggregatecollection, pipeline, database, batch_sizeReads the documents an aggregation pipeline returns

database defaults to the connection's. batch_size (default 1000) is the number of documents per page on streamwright run. options (Spark only) are the MongoDB Spark connector's read options.

streams/events.yaml
requests:
- name: raw_events
operation:
service: collection
method: find
arguments:
collection: events
filter: { status: completed }
projection: { _id: 1, user_id: 1, amount: 1, created_at: 1 }
sort: [["created_at", 1]]
limit: 5000
transform:
mode: page
steps:
- name: events
select: |
SELECT record->>'_id' AS event_id,
record->>'user_id' AS user_id,
(record->>'amount')::DOUBLE AS amount,
record->>'created_at' AS created_at
FROM raw_events
export:
events:
step: events
primary_key: [event_id]

Each document is one record. BSON types are converted: ObjectId to text, dates to ISO 8601 text, Decimal128 to a number and binary to base64.

An aggregation pipeline runs on the server before the results come back:

streams/user_summaries.yaml
requests:
- name: summaries
operation:
service: collection
method: aggregate
arguments:
collection: events
pipeline:
- $match: { status: "completed" }
- $group:
_id: "$user_id"
total_spent: { $sum: "$amount" }
order_count: { $sum: 1 }
transform:
mode: page
steps:
- name: user_summaries
select: |
SELECT record->>'_id' AS user_id,
(record->>'total_spent')::DOUBLE AS total_spent,
(record->>'order_count')::BIGINT AS order_count
FROM summaries
export:
user_summaries:
step: user_summaries
primary_key: [user_id]

In jobs, a step uses the same operation:. A Spark job reads natively; a DuckDB job runs it as a call step:

steps:
- name: events
connection: mongo # provider: mongodb
operation:
service: collection
method: find
arguments: { collection: events, filter: { status: completed } }

Write data​

Only Spark job outputs write to a collection:

outputs:
- name: save_summaries
connection: mongo
operation:
service: collection
method: write
arguments:
collection: user_summaries
mode: merge # or append
primary_key: [user_id]
sql: SELECT user_id, SUM(amount) AS total_spent FROM events GROUP BY user_id
ArgumentWhat it doesDefault
collectionThe collection. Required
modeappend inserts a document per row; merge updates the documents that match on primary_key and inserts the restappend
primary_keymerge: the fields to match_id
databaseThe databaseThe connection's
optionsThe MongoDB Spark connector's write options

Good to know​

  • Read-only pipelines. A pipeline with a $out or $merge stage, at any depth (inside $facet or a $lookup pipeline too), is refused before it reaches the server.
  • Spark. A find runs as an aggregation pipeline. Install the Spark extra: pip install "streamwright-mongodb[spark]".
  • Vendor arguments. Other find and aggregate arguments go to pymongo unchanged, for example hint: { status: 1 } or max_time_ms: 5000.
  • Client options. Connection keys the connector does not declare go to pymongo's MongoClient unchanged, so use pymongo's option names (for example tls: true). The connector sets the server selection timeout.

See also​