Skip to main content

Accounts & secrets

To run a pipeline for a user, orchestration needs two things it never hard-codes: which accounts to run, and which credentials to use. Both come from providers that are read at run time: an HTTP API in production, or a local file in development.

In short

The token lives on the account. The app credentials live in the secrets provider, per network and region. A network's inputs hand both to streamwright run as secret environment variables.

Two providers​

Each provider uses its URL when set, otherwise its file.

ProviderRead fromHolds
AccountsSTREAMWRIGHT_ACCOUNTS_URL, else STREAMWRIGHT_ACCOUNTS_FILE (default ~/.streamwright/accounts.yaml)The (user, account, network) rows, each with its own token
SecretsSTREAMWRIGHT_SECRETS_URL, else STREAMWRIGHT_SECRETS_FILE (default ~/.streamwright/secrets.yaml)Each network's app credentials, by region. No tokens.
Files are for development

Use the local files for development and testing. In production, STREAMWRIGHT_ACCOUNTS_URL and STREAMWRIGHT_SECRETS_URL return the same documents over HTTPS.

Accounts​

A clients map groups accounts under a client; the client's key is the user_id you trigger a pipeline for.

~/.streamwright/accounts.yaml
clients:
client_a:
name: Example Client
region: region_a # picks the secrets of this region
accounts:
- network: network_a
account_id: "1234567890" # customer_id also works
token: "<account-token>" # this account's own token
timezone: UTC # overrides the network's default timezone
  • Each account carries its own token, so tokens never sit in the secrets provider.
  • Any other key on the account (for example login_customer_id) is available to the network's inputs as {{ row.NAME }}.
  • An account without a region uses STREAMWRIGHT_REGION.
  • For quick development, a flat accounts: list of rows also works.

Secrets​

App-level credentials per network: an optional default, plus values per region that override it. No tokens here.

~/.streamwright/secrets.yaml
secrets:
network_a:
default:
api_key: "<api-key>"
api_secret: "<api-secret>"
regions:
region_b:
api_key: "<region-b-api-key>" # overrides default for region_b

A network entry with no default or regions is used as it is, for every region.

How a run gets its credentials​

For each account row, the orchestrator:

  1. Looks up the network's app credentials for the account's region (the region's values over default).
  2. Takes the token from the account row itself.
  3. Maps both onto the source's inputs, using the network's inputs (below).

Into streamwright run​

A network's inputs give a value to each input the source declares. A value is a literal or a small template:

config/networks.yaml (excerpt)
inputs:
account_ids: "{{ row.account_id }}"
start_date: "-7d"
api_key: "secret:{{ app.api_key }}"
token: "secret:{{ row.token }}" # the token comes from the account
ReferenceValue
{{ row.NAME }}A key of the account row
{{ app.NAME }}The network's app credentials, resolved for the account's region
{{ ctx.NAME }}user_id, account_id, network, timezone or schema
  • A value starting with secret: reaches streamwright only as a STREAMWRIGHT_SECRET_<NAME> environment variable: never on the command line, in the run's config, or in logs (see the CLI). Every spec.secrets input is passed this way.
  • Every other value is passed as a plain --set. A secret: value for a config input is refused, because it would end up on the command line.
  • A reference that is not set leaves the input unset, so the source's default applies. A required input with no value, or an inputs key the source does not declare, is an error.

So the streamwright CLI knows nothing about networks or regions: orchestration resolves accounts and regions, then hands it a flat set of inputs.

Good to know​

  • In Docker mode, each secret is passed to docker run by name only (-e STREAMWRIGHT_SECRET_<NAME>), so no value is in the command or the image.
  • In Kubernetes mode, the pods read the secrets from a Kubernetes Secret; the Job definition holds only its name.
  • streamwright output is redacted (***) for the run's secret values before it reaches the Dagster log.