Orchestration
Orchestration runs streamwright run for many users, accounts and ad networks, as Dagster
jobs. Use it when you run the same sources for many accounts across several networks. It lives in its own
repository, streamwright-orchestration.
You write a pipeline once (which streams, in which order). Each network says how to run it on that network.
For each account, every node of the pipeline becomes one streamwright run command.
Orchestration is a prototype. Its network list (config/networks.yaml) and its account and secrets files are
stand-ins: in production they come from a registry service or API.
The model
Four ideas, each in one place:
| Idea | What it is |
|---|---|
| Pipeline | A named order of nodes, one file per pipeline (config/pipelines/NAME.yaml) |
| Network | For one ad network: the source folder, its inputs and its stream names (config/networks.yaml) |
| Account | A (user, account, network) row with its own token; see Accounts & secrets |
| Execution mode | Where each streamwright run runs: a local process, Docker or Kubernetes |
Orchestration never imports StreamWright. Every node shells out to the streamwright CLI that STREAMWRIGHT_BIN
points to.
Pipelines
A pipeline lists its nodes and the order they run in. after names the nodes that must finish first.
name: metadata # must match the file name
nodes:
campaigns: {}
ad_groups: {after: [campaigns]}
ad_group_hierarchy: {after: [campaigns]}
location_targets: {after: [campaigns]}
keywords: {after: [ad_groups]}
audience_targets: {after: [ad_groups]}
- Node names are canonical names, the same on every network. The shared vocabulary maps them to each network's streams.
- An unknown node in
after, a node that runs after itself, and a cycle are errors. - To add a pipeline, add a file. No code change.
Networks
A network entry tells orchestration how to run a pipeline on that network:
networks:
google_ads:
source: sources/ads/google_ads # the source folder, from the repository root
timezone: America/New_York # default --timezone; an account's timezone wins
connectors: [google_ads, gaql] # passed as --allow-connector
image: streamwright-pipeline:local # the container image for docker and k8s modes
inputs:
customer_ids: "{{ row.account_id }}"
start_date: "-7d"
developer_token: "secret:{{ app.developer_token }}"
refresh_token: "secret:{{ row.token }}"
streams:
campaigns: campaigns
ad_groups: ad_groups
# ... one entry per canonical node
inputsfill the inputs the source declares. See Accounts & secrets.streamsmaps every canonical node to the network's stream. See Shared vocabulary.
One job per pipeline and network
- Each (pipeline, network) pair becomes one Dagster job, named
ads_PIPELINE__NETWORK. - When you trigger a pipeline for a user, orchestration finds the user's accounts and runs the job of each account's network.
- Each node builds its command from the source's declared inputs, then runs it after its upstream nodes. In the default mode the command looks like this:
streamwright run SOURCE --stream STREAM --set NAME=VALUE ... --timezone TZ --output duckdb:WAREHOUSE:SCHEMA --allow-connector ...
Secrets are not on this command line: they reach the run as STREAMWRIGHT_SECRET_<NAME> variables.
Where the command runs
STREAMWRIGHT_EXECUTION picks the execution mode. The jobs, the order and the inputs are the same in every mode.
| Mode | Where each node runs | Where the data goes |
|---|---|---|
subprocess (default) | The local streamwright CLI | A local DuckDB file |
docker | A docker run --rm of the network's image | A mounted DuckDB file |
k8s | A Kubernetes Job (one pod) of that image | A DuckLake: Parquet files on S3, catalog in PostgreSQL |
STREAMWRIGHT_EXECUTION=docker .venv/bin/python -m streamwright.orchestration.definitions metadata u1
The unit of work is always streamwright run, so moving from your laptop to Kubernetes is a setting. The source and
its streams stay the same.
Try it
git clone https://github.com/karthick-jaganathan/streamwright-orchestration
cd streamwright-orchestration
python3 -m venv .venv && .venv/bin/pip install -e .
export STREAMWRIGHT_BIN=/path/to/venv/bin/streamwright # a streamwright CLI with the networks' connectors
.venv/bin/python -m streamwright.orchestration.definitions metadata u1 # run the metadata pipeline for user u1
.venv/bin/dagster dev -m streamwright.orchestration.definitions # the jobs in the Dagster UI
Runs started from the command line show in the Dagster UI only when both use the same DAGSTER_HOME.
See also
- Shared vocabulary: one pipeline across networks with different stream names.
- Accounts & secrets: where each run's accounts and credentials come from.