Shared vocabulary
A shared vocabulary lets orchestration run one pipeline on many networks. Networks name their
streams differently: one calls it accounts, another customers, and a third has no campaigns stream at all. Instead
of a pipeline per network, you write one pipeline with canonical node names, and each network maps those names
to its own streams.
Each network's streams: map says, for every node: "run this stream" or false ("skip it here").
| Canonical node | network_a | network_b | network_c |
|---|---|---|---|
accounts | accounts | customers | accounts |
campaigns | campaigns | campaigns | unsupported |
metrics | daily_metrics | stats | insights |
One pipeline uses the canonical names on the left. Each network's streams: map provides its own stream for each node, or marks it unsupported — so the same pipeline runs across networks that name (or lack) streams differently.
Example
The metadata pipeline uses the nodes campaigns, ad_groups, ad_group_hierarchy, keywords and more. Meta
names the ad-group tier ad_sets and has no hierarchy or keyword streams:
networks:
meta_ads:
streams:
campaigns: campaigns
ad_groups: ad_sets # same idea, different name (an alias)
ad_group_hierarchy: false # meta has no such stream: skip the node
location_targets: false
keywords: false
audience_targets: false
campaign_performance: campaign_insights
after:
ad_groups: [] # meta fetches ad sets per account, not after campaigns
ad_groupsruns Meta'sad_setsstream.ad_group_hierarchy,keywordsand the otherfalsenodes are skipped on Meta, and drop out of other nodes' dependencies.afterreplaces the pipeline's order for a node on this network only.
Rules
- Map every node. Only an explicit
falseskips a node. A node with no entry is an error, so a typo is caught before anything runs. - Map to real streams. A node mapped to a name must be a stream of the network's source.
- Change the order per network.
after:gives a node new upstream nodes on this network, replacing the pipeline'safterfor it. - One job per network. Each (pipeline, network) pair becomes its own Dagster job, so each network runs only the nodes it has.
To add a network, you write a small entry like this one, not a new pipeline.
See also
- Orchestration overview
- Accounts & secrets: how a network's
inputsreach the run.