Accounts & secrets
To run a pipeline for a user, orchestration needs two things it never hard-codes: which accounts to run, and which credentials to use. Both come from providers that are read at run time: an HTTP API in production, or a local file in development.
The token lives on the account. The app credentials live in the secrets provider, per network and region. A
network's inputs hand both to streamwright run as secret environment variables.
Two providers
Each provider uses its URL when set, otherwise its file.
| Provider | Read from | Holds |
|---|---|---|
| Accounts | STREAMWRIGHT_ACCOUNTS_URL, else STREAMWRIGHT_ACCOUNTS_FILE (default ~/.streamwright/accounts.yaml) | The (user, account, network) rows, each with its own token |
| Secrets | STREAMWRIGHT_SECRETS_URL, else STREAMWRIGHT_SECRETS_FILE (default ~/.streamwright/secrets.yaml) | Each network's app credentials, by region. No tokens. |
Use the local files for development and testing. In production, STREAMWRIGHT_ACCOUNTS_URL and
STREAMWRIGHT_SECRETS_URL return the same documents over HTTPS.
Accounts
A clients map groups accounts under a client; the client's key is the user_id you trigger a pipeline for.
clients:
client_a:
name: Example Client
region: region_a # picks the secrets of this region
accounts:
- network: network_a
account_id: "1234567890" # customer_id also works
token: "<account-token>" # this account's own token
timezone: UTC # overrides the network's default timezone
- Each account carries its own
token, so tokens never sit in the secrets provider. - Any other key on the account (for example
login_customer_id) is available to the network'sinputsas{{ row.NAME }}. - An account without a region uses
STREAMWRIGHT_REGION. - For quick development, a flat
accounts:list of rows also works.
Secrets
App-level credentials per network: an optional default, plus values per region that override it. No tokens
here.
secrets:
network_a:
default:
api_key: "<api-key>"
api_secret: "<api-secret>"
regions:
region_b:
api_key: "<region-b-api-key>" # overrides default for region_b
A network entry with no default or regions is used as it is, for every region.
How a run gets its credentials
For each account row, the orchestrator:
- Looks up the network's app credentials for the account's
region(the region's values overdefault). - Takes the token from the account row itself.
- Maps both onto the source's inputs, using the network's
inputs(below).
Into streamwright run
A network's inputs give a value to each input the source declares. A value is a literal or a small template:
inputs:
account_ids: "{{ row.account_id }}"
start_date: "-7d"
api_key: "secret:{{ app.api_key }}"
token: "secret:{{ row.token }}" # the token comes from the account
| Reference | Value |
|---|---|
{{ row.NAME }} | A key of the account row |
{{ app.NAME }} | The network's app credentials, resolved for the account's region |
{{ ctx.NAME }} | user_id, account_id, network, timezone or schema |
- A value starting with
secret:reachesstreamwrightonly as aSTREAMWRIGHT_SECRET_<NAME>environment variable: never on the command line, in the run's config, or in logs (see the CLI). Everyspec.secretsinput is passed this way. - Every other value is passed as a plain
--set. Asecret:value for a config input is refused, because it would end up on the command line. - A reference that is not set leaves the input unset, so the source's default applies. A required input with no
value, or an
inputskey the source does not declare, is an error.
So the streamwright CLI knows nothing about networks or regions: orchestration resolves accounts and regions, then
hands it a flat set of inputs.
Good to know
- In Docker mode, each secret is passed to
docker runby name only (-e STREAMWRIGHT_SECRET_<NAME>), so no value is in the command or the image. - In Kubernetes mode, the pods read the secrets from a Kubernetes Secret; the Job definition holds only its name.
streamwrightoutput is redacted (***) for the run's secret values before it reaches the Dagster log.