MongoDB
The mongodb connector reads documents from MongoDB collections with find queries or aggregate
pipelines. Reads never change the database. Spark job outputs can also write to a collection.
Install
sw connectors -i mongodb
Connect
Use one URI secret, or the connection keys. Not both.
kind: source
name: mongodb_demo
spec:
config:
mongo_database: { type: string, default: analytics }
secrets:
mongo_uri: { type: string }
auth:
provider: mongodb
uri: "{{ secrets.mongo_uri }}"
database: "{{ config.mongo_database }}"
| Key | What it is | Required / default |
|---|---|---|
uri | A mongodb:// or mongodb+srv:// URI; its path is the database. Secret | Required, or the keys below |
host | The server's host name | Required (with keys) |
port | The port | 27017 |
database | The database | Required (with keys) |
username | The user (set together with password) | Optional |
password | The user's password. Secret | Optional |
auth_source | The database that holds the user | admin |
With the connection keys:
kind: source
name: mongodb_demo
spec:
secrets:
mongo_password: { type: string }
auth:
provider: mongodb
host: localhost
database: analytics
username: reader
password: "{{ secrets.mongo_password }}"
A password inside the URI is redacted (***) in logs and errors too.
Read data
A request uses service: collection:
| Method | Arguments | What it does |
|---|---|---|
find | collection, filter, projection, sort, limit, database, batch_size | Reads the documents filter matches. sort is a list of [field, direction] pairs |
aggregate | collection, pipeline, database, batch_size | Reads the documents an aggregation pipeline returns |
database defaults to the connection's. batch_size (default 1000) is the number of documents per page on
streamwright run. options (Spark only) are the MongoDB Spark connector's read options.
requests:
- name: raw_events
operation:
service: collection
method: find
arguments:
collection: events
filter: { status: completed }
projection: { _id: 1, user_id: 1, amount: 1, created_at: 1 }
sort: [["created_at", 1]]
limit: 5000
transform:
mode: page
steps:
- name: events
select: |
SELECT record->>'_id' AS event_id,
record->>'user_id' AS user_id,
(record->>'amount')::DOUBLE AS amount,
record->>'created_at' AS created_at
FROM raw_events
export:
events:
step: events
primary_key: [event_id]
Each document is one record. BSON types are converted: ObjectId to text, dates to ISO 8601 text, Decimal128 to a
number and binary to base64.
An aggregation pipeline runs on the server before the results come back:
requests:
- name: summaries
operation:
service: collection
method: aggregate
arguments:
collection: events
pipeline:
- $match: { status: "completed" }
- $group:
_id: "$user_id"
total_spent: { $sum: "$amount" }
order_count: { $sum: 1 }
transform:
mode: page
steps:
- name: user_summaries
select: |
SELECT record->>'_id' AS user_id,
(record->>'total_spent')::DOUBLE AS total_spent,
(record->>'order_count')::BIGINT AS order_count
FROM summaries
export:
user_summaries:
step: user_summaries
primary_key: [user_id]
In jobs, a step uses the same operation:. A Spark job reads natively; a DuckDB job runs it as a
call step:
steps:
- name: events
connection: mongo # provider: mongodb
operation:
service: collection
method: find
arguments: { collection: events, filter: { status: completed } }
Write data
Only Spark job outputs write to a collection:
outputs:
- name: save_summaries
connection: mongo
operation:
service: collection
method: write
arguments:
collection: user_summaries
mode: merge # or append
primary_key: [user_id]
sql: SELECT user_id, SUM(amount) AS total_spent FROM events GROUP BY user_id
| Argument | What it does | Default |
|---|---|---|
collection | The collection. Required | |
mode | append inserts a document per row; merge updates the documents that match on primary_key and inserts the rest | append |
primary_key | merge: the fields to match | _id |
database | The database | The connection's |
options | The MongoDB Spark connector's write options |
Good to know
- Read-only pipelines. A pipeline with a
$outor$mergestage, at any depth (inside$facetor a$lookuppipeline too), is refused before it reaches the server. - Spark. A
findruns as an aggregation pipeline. Install the Spark extra:pip install "streamwright-mongodb[spark]". - Vendor arguments. Other
findandaggregatearguments go to pymongo unchanged, for examplehint: { status: 1 }ormax_time_ms: 5000. - Client options. Connection keys the connector does not declare go to pymongo's
MongoClientunchanged, so use pymongo's option names (for exampletls: true). The connector sets the server selection timeout.
See also
- DuckDB jobs: call steps.
- Starter template:
examples/sources/readers/mongodb_demo.