Skip to main content

Shared vocabulary

A shared vocabulary lets orchestration run one pipeline on many networks. Networks name their streams differently: one calls it accounts, another customers, and a third has no campaigns stream at all. Instead of a pipeline per network, you write one pipeline with canonical node names, and each network maps those names to its own streams.

In short

Each network's streams: map says, for every node: "run this stream" or false ("skip it here").

Canonical nodenetwork_anetwork_bnetwork_c
accountsaccountscustomersaccounts
campaignscampaignscampaignsunsupported
metricsdaily_metricsstatsinsights

One pipeline uses the canonical names on the left. Each network's streams: map provides its own stream for each node, or marks it unsupported — so the same pipeline runs across networks that name (or lack) streams differently.

Example​

The metadata pipeline uses the nodes campaigns, ad_groups, ad_group_hierarchy, keywords and more. Meta names the ad-group tier ad_sets and has no hierarchy or keyword streams:

config/networks.yaml (excerpt)
networks:
meta_ads:
streams:
campaigns: campaigns
ad_groups: ad_sets # same idea, different name (an alias)
ad_group_hierarchy: false # meta has no such stream: skip the node
location_targets: false
keywords: false
audience_targets: false
campaign_performance: campaign_insights
after:
ad_groups: [] # meta fetches ad sets per account, not after campaigns
  • ad_groups runs Meta's ad_sets stream.
  • ad_group_hierarchy, keywords and the other false nodes are skipped on Meta, and drop out of other nodes' dependencies.
  • after replaces the pipeline's order for a node on this network only.

Rules​

  • Map every node. Only an explicit false skips a node. A node with no entry is an error, so a typo is caught before anything runs.
  • Map to real streams. A node mapped to a name must be a stream of the network's source.
  • Change the order per network. after: gives a node new upstream nodes on this network, replacing the pipeline's after for it.
  • One job per network. Each (pipeline, network) pair becomes its own Dagster job, so each network runs only the nodes it has.

To add a network, you write a small entry like this one, not a new pipeline.

See also​