Skip to main content

Orchestration

Orchestration runs streamwright run for many users, accounts and ad networks, as Dagster jobs. Use it when you run the same sources for many accounts across several networks. It lives in its own repository, streamwright-orchestration.

In short

You write a pipeline once (which streams, in which order). Each network says how to run it on that network. For each account, every node of the pipeline becomes one streamwright run command.

note

Orchestration is a prototype. Its network list (config/networks.yaml) and its account and secrets files are stand-ins: in production they come from a registry service or API.

The model​

Four ideas, each in one place:

IdeaWhat it is
PipelineA named order of nodes, one file per pipeline (config/pipelines/NAME.yaml)
NetworkFor one ad network: the source folder, its inputs and its stream names (config/networks.yaml)
AccountA (user, account, network) row with its own token; see Accounts & secrets
Execution modeWhere each streamwright run runs: a local process, Docker or Kubernetes

Orchestration never imports StreamWright. Every node shells out to the streamwright CLI that STREAMWRIGHT_BIN points to.

Pipelines​

A pipeline lists its nodes and the order they run in. after names the nodes that must finish first.

config/pipelines/metadata.yaml
name: metadata # must match the file name
nodes:
campaigns: {}
ad_groups: {after: [campaigns]}
ad_group_hierarchy: {after: [campaigns]}
location_targets: {after: [campaigns]}
keywords: {after: [ad_groups]}
audience_targets: {after: [ad_groups]}
  • Node names are canonical names, the same on every network. The shared vocabulary maps them to each network's streams.
  • An unknown node in after, a node that runs after itself, and a cycle are errors.
  • To add a pipeline, add a file. No code change.

Networks​

A network entry tells orchestration how to run a pipeline on that network:

config/networks.yaml (excerpt)
networks:
google_ads:
source: sources/ads/google_ads # the source folder, from the repository root
timezone: America/New_York # default --timezone; an account's timezone wins
connectors: [google_ads, gaql] # passed as --allow-connector
image: streamwright-pipeline:local # the container image for docker and k8s modes
inputs:
customer_ids: "{{ row.account_id }}"
start_date: "-7d"
developer_token: "secret:{{ app.developer_token }}"
refresh_token: "secret:{{ row.token }}"
streams:
campaigns: campaigns
ad_groups: ad_groups
# ... one entry per canonical node

One job per pipeline and network​

  1. Each (pipeline, network) pair becomes one Dagster job, named ads_PIPELINE__NETWORK.
  2. When you trigger a pipeline for a user, orchestration finds the user's accounts and runs the job of each account's network.
  3. Each node builds its command from the source's declared inputs, then runs it after its upstream nodes. In the default mode the command looks like this:
streamwright run SOURCE --stream STREAM --set NAME=VALUE ... --timezone TZ --output duckdb:WAREHOUSE:SCHEMA --allow-connector ...

Secrets are not on this command line: they reach the run as STREAMWRIGHT_SECRET_<NAME> variables.

Where the command runs​

STREAMWRIGHT_EXECUTION picks the execution mode. The jobs, the order and the inputs are the same in every mode.

ModeWhere each node runsWhere the data goes
subprocess (default)The local streamwright CLIA local DuckDB file
dockerA docker run --rm of the network's imageA mounted DuckDB file
k8sA Kubernetes Job (one pod) of that imageA DuckLake: Parquet files on S3, catalog in PostgreSQL
STREAMWRIGHT_EXECUTION=docker .venv/bin/python -m streamwright.orchestration.definitions metadata u1

The unit of work is always streamwright run, so moving from your laptop to Kubernetes is a setting. The source and its streams stay the same.

Try it​

git clone https://github.com/karthick-jaganathan/streamwright-orchestration
cd streamwright-orchestration
python3 -m venv .venv && .venv/bin/pip install -e .
export STREAMWRIGHT_BIN=/path/to/venv/bin/streamwright # a streamwright CLI with the networks' connectors

.venv/bin/python -m streamwright.orchestration.definitions metadata u1 # run the metadata pipeline for user u1
.venv/bin/dagster dev -m streamwright.orchestration.definitions # the jobs in the Dagster UI

Runs started from the command line show in the Dagster UI only when both use the same DAGSTER_HOME.

See also​