Skip to main content

Sources

A source pulls data from one system, such as an ad platform, a REST API, a file store or a database. It is a folder of YAML, not code: one source.yaml and one file per stream in streams/. Use a source when you want to pull data and write it to files or a warehouse.

In short

source.yaml says who you are and how to connect. Each streams/*.yaml file says what to fetch, shape and write. Each run brings its own settings and secrets.

my_source/
├── source.yaml
└── streams/
├── accounts.yaml
├── campaigns.yaml
└── metrics.yaml

Run it with streamwright run my_source. Check it first with streamwright validate my_source.

source.yaml​

source.yaml holds what every stream shares: the source's inputs, how to sign in, and shared defaults.

source.yaml
kind: source
name: my_source
spec:
config: # non-secret settings, given per run
account_id: { type: string }
region: { type: string, default: us }
secrets: # credentials, never written in the source
token: { type: string }
auth:
provider: restapi # the generic REST connector, or an SDK connector: postgres, s3, google_ads, ...
type: bearer # bearer, oauth2_refresh_token, api_key, basic
token: "{{ secrets.token }}"
http: # shared HTTP defaults for REST sources
base_url: https://api.example.com
paginator: { type: cursor, token_path: next_token, param: cursor }
records: { path: data }
policy:
retry: { codes: [429, 500, 502, 503], max_attempts: 5 }

This source:

  • takes one required setting (account_id), one optional setting (region, default us) and one secret (token);
  • signs in to a REST API with a bearer token (this is the source's connection);
  • follows cursor pages and reads records from each response's data field;
  • retries every request on rate limits and server errors, up to 5 attempts (this is the source's default policy).
KeyWhat it does
spec.configNon-secret inputs. A run gives them with --set, a --config file, or the input's default. Use them as {{ config.NAME }}. Types: string, integer, number, boolean, date, or list (with items).
spec.secretsCredentials. A run gives them with STREAMWRIGHT_SECRET_<NAME> environment variables or a --secrets file. Use them as {{ secrets.NAME }}. They are redacted in every log line.
authWhere: the connector (provider) and how to sign in. A REST source picks a type such as bearer or oauth2_refresh_token; an SDK connector such as google_ads, postgres or s3 takes its own keys.
httpShared HTTP defaults for REST sources: base_url, paginator, records, headers. See the REST API connector.
policyHow: the default retry and rate_limit of every request. A stream's own policy: replaces them.
exportOptional. Defaults for every export, such as a file_name template. See Export.

One source, many clients​

Nothing client-specific lives in the source. Each run brings its own settings and secrets, so one source serves every client.

clients/example.yaml
config: # values for spec.config
account_id: "1234567890"
streams: [campaigns, metrics] # optional: which streams to run (default: all)
streamwright run my_source \
--config clients/example.yaml \
--secrets clients/example.secrets.yaml
OptionWhat it does
--config FILEOne client's config values, and optionally the streams to run
--set NAME=VALUEOne config value; it overrides the same value from --config
--secrets FILEValues for spec.secrets; they override STREAMWRIGHT_SECRET_<NAME> variables
--stream NAMERun only this stream or export; it replaces the file's streams list

The same split lets one pipeline run across many ad networks. See shared vocabulary.

See also​

  • Streams: what goes in each streams/*.yaml file.
  • Outputs: where a run writes.
  • CLI: every run option.