Sources
A source pulls data from one system, such as an ad platform, a REST API, a file store or a database. It is a
folder of YAML, not code: one source.yaml and one file per stream in streams/. Use a source when you want to pull
data and write it to files or a warehouse.
source.yaml says who you are and how to connect. Each streams/*.yaml file says what to fetch, shape and
write. Each run brings its own settings and secrets.
my_source/
├── source.yaml
└── streams/
├── accounts.yaml
├── campaigns.yaml
└── metrics.yaml
Run it with streamwright run my_source. Check it first with streamwright validate my_source.
source.yaml
source.yaml holds what every stream shares: the source's inputs, how to sign in, and shared defaults.
kind: source
name: my_source
spec:
config: # non-secret settings, given per run
account_id: { type: string }
region: { type: string, default: us }
secrets: # credentials, never written in the source
token: { type: string }
auth:
provider: restapi # the generic REST connector, or an SDK connector: postgres, s3, google_ads, ...
type: bearer # bearer, oauth2_refresh_token, api_key, basic
token: "{{ secrets.token }}"
http: # shared HTTP defaults for REST sources
base_url: https://api.example.com
paginator: { type: cursor, token_path: next_token, param: cursor }
records: { path: data }
policy:
retry: { codes: [429, 500, 502, 503], max_attempts: 5 }
This source:
- takes one required setting (
account_id), one optional setting (region, defaultus) and one secret (token); - signs in to a REST API with a bearer token (this is the source's connection);
- follows cursor pages and reads records from each response's
datafield; - retries every request on rate limits and server errors, up to 5 attempts (this is the source's default policy).
| Key | What it does |
|---|---|
spec.config | Non-secret inputs. A run gives them with --set, a --config file, or the input's default. Use them as {{ config.NAME }}. Types: string, integer, number, boolean, date, or list (with items). |
spec.secrets | Credentials. A run gives them with STREAMWRIGHT_SECRET_<NAME> environment variables or a --secrets file. Use them as {{ secrets.NAME }}. They are redacted in every log line. |
auth | Where: the connector (provider) and how to sign in. A REST source picks a type such as bearer or oauth2_refresh_token; an SDK connector such as google_ads, postgres or s3 takes its own keys. |
http | Shared HTTP defaults for REST sources: base_url, paginator, records, headers. See the REST API connector. |
policy | How: the default retry and rate_limit of every request. A stream's own policy: replaces them. |
export | Optional. Defaults for every export, such as a file_name template. See Export. |
One source, many clients
Nothing client-specific lives in the source. Each run brings its own settings and secrets, so one source serves every client.
config: # values for spec.config
account_id: "1234567890"
streams: [campaigns, metrics] # optional: which streams to run (default: all)
streamwright run my_source \
--config clients/example.yaml \
--secrets clients/example.secrets.yaml
| Option | What it does |
|---|---|
--config FILE | One client's config values, and optionally the streams to run |
--set NAME=VALUE | One config value; it overrides the same value from --config |
--secrets FILE | Values for spec.secrets; they override STREAMWRIGHT_SECRET_<NAME> variables |
--stream NAME | Run only this stream or export; it replaces the file's streams list |
The same split lets one pipeline run across many ad networks. See shared vocabulary.