CLI
The streamwright command runs sources and jobs, checks them, installs connectors and serves jobs over HTTP. The
short alias sw works everywhere (sw run, sw validate). Run sw <command> --help for the full help of any
command.
sw validate checks, sw plan shows the order, sw run runs. The manifest's kind picks the engine, so the same
three commands work for sources and jobs.
Commands at a glance
| Command | What it does |
|---|---|
run | Run a source and write its records, or run a job manifest |
validate | Check sources, jobs or a serve.yaml without running them |
plan | Print a job's execution plan (no secrets needed) |
duckdb | Validate, plan and run DuckDBJob manifests, and manage their batches |
spark | Validate, plan, lock and run SparkJob manifests |
state | Create, upgrade and check the state tables in a database |
connectors | Show installed connectors; list, search and install connectors from the hub |
serve | Serve jobs over HTTP as a REST API |
worker | Run the jobs serve queued |
queue | Purge and migrate the run queue of serve |
systemd install | Install systemd units for serve, worker and a daily purge |
mcp | Run the StreamWright MCP server for AI assistants |
Most commands also take the shared logging options.
Exit status of run and the job commands:
| Status | Meaning |
|---|---|
0 | Success |
1 | Invalid job, or the run failed |
2 | Usage or input error: invalid source, inputs, stream names or options, or a missing or disallowed connector (nothing was fetched) |
130 | Interrupted |
Run and check
streamwright run
streamwright run SOURCE [options]
streamwright run JOB.yaml [job options] # or: streamwright run --job JOB.yaml
Runs a source and writes its records: by default Singer messages on stdout, with logs on stderr. Given a job
manifest, it runs the job on its engine (duckdb or spark) with that
engine's options. Options for sources only (--output, --stream, --state, ...) are refused for a job.
| Option | What it does | Default |
|---|---|---|
SOURCE | A source folder (source.yaml plus streams/), a source file, or a job manifest | |
--job JOB, -j | A job manifest, as an option instead of SOURCE | |
--config FILE | A client's settings: config values and the streams (or exports) to run | |
--set NAME=VALUE | One spec.config value (repeatable; lists as a,b,c); overrides --config | |
--secrets FILE | Values for spec.secrets; overrides STREAMWRIGHT_SECRET_<NAME> variables | |
--state FILE | State from a previous run (JSON, or a Singer STATE message), for incremental streams | |
--timezone TZ | The client's IANA time zone (such as America/New_York) for today and incremental windows | The machine's local date |
--stream NAME | Run only this stream, or the stream of this export (repeatable; replaces the --config file's streams). The streams its from_stream partitions come from run too. | All streams |
--output TARGET | Where records go: singer, jsonl:DIR, csv:DIR, tsv:DIR, parquet:DIR, dlt:DESTINATION[:DATASET], duckdb:PATH[:SCHEMA], ducklake:CATALOG[:SCHEMA]. See Outputs. | singer |
--file-name TEMPLATE | File outputs only: each export's path inside DIR. Uses {{ export }}, {{ source }}, {{ today }}, {{ timestamp }}, {{ format }}, {{ config.NAME }}. | The YAML's file_name, else <export>.<date>.<time>.<unique>.<ext> |
--allow-connector NAME | Allow only this connector (repeatable) | Any installed connector |
--summary [FILE] | Also write the run summary as JSON to FILE, whatever the outcome. Bare --summary logs it as one INFO line (event: run_summary). |
sw run sources/google_ads --config clients/example.yaml --secrets secrets.yaml \
--output parquet:out --file-name "{{ config.client }}/{{ export }}_{{ today }}.{{ format }}"
streamwright validate
streamwright validate [PATH ...] [--strict] [--format text|json|github] [--export-schema DIR]
Checks sources (schema, references, SQL) without fetching anything. It walks folders recursively. It also checks job
manifests (as the path, or with --job) and a serve.yaml with its jobs, without needing secrets. A job is checked
by its engine, with that engine's options (sw duckdb validate -h, sw spark validate -h).
| Option | What it does | Default |
|---|---|---|
PATH ... | Files or folders to check, recursively; a source folder is checked as one source | $STREAMWRIGHT_CONFIGS |
--strict | Exit with status 1 on warnings too | |
--format text|json|github | Output format; github prints GitHub Actions annotations | text |
--export-schema DIR | Write the JSON Schemas of sources and stream files (for editors and CI), then exit | |
--kind source | The kind to assume for files that do not declare kind | |
--allow-connector NAME | Sources: only allow these connectors in auth.provider (repeatable) |
sw validate sources/ --strict --format github
streamwright plan
streamwright plan JOB.yaml # or: streamwright plan --job JOB.yaml
Prints a job's steps, outputs and the order they run in. It runs nothing and needs no secrets. The manifest's kind
picks the engine, and its options are that engine's (sw duckdb plan -h, sw spark plan -h).
| Option | What it does | Default |
|---|---|---|
JOB | The job manifest: a YAML file, or its folder (job.yaml) | |
--job JOB, -j | The job manifest, as an option instead of JOB |
Jobs and state
streamwright duckdb
streamwright duckdb validate JOB.yaml [--strict] [--graph] [--input FILE|-]
streamwright duckdb plan JOB.yaml
streamwright duckdb run JOB.yaml [--dry-run] [--secrets FILE] [--input FILE|-]
streamwright duckdb run JOB.yaml [--once | --drain | --available-now | --follow [--interval 10s]] [--max-messages N]
streamwright duckdb state ACTION JOB.yaml [JOB_ID] [options]
Runs kind: DuckDBJob manifests in an in-process DuckDB (see DuckDB jobs).
streamwright validate | plan | run JOB send a DuckDBJob here automatically. JOB is a YAML file, or a folder
that holds job.yaml.
Job options
| Option | What it does | Default |
|---|---|---|
--job JOB, -j | The job manifest, as an option instead of JOB | |
--config FILE · --set NAME=VALUE | spec.config values (--set is repeatable and overrides --config) | |
--overlay FILE | A YAML overlay merged onto the job (repeatable, in order; null deletes a key, a list replaces the job's) | |
--allow-connector NAME | Use only this connector (repeatable) | Any installed connector |
--secrets FILE | run: spec.secrets values (or STREAMWRIGHT_SECRET_<NAME> variables) | |
--input FILE | validate, run: rows for a job with spec.input, as a JSON array or NDJSON (- reads stdin); see Input payloads | |
--dry-run | run: validate, resolve the inputs and print the plan; connect to nothing, write nothing | |
--summary [FILE] | run: also write the run's summary as JSON, whatever the outcome; bare: one INFO log line | |
--strict | Treat warnings as errors | |
--graph | validate: also print the plan of a valid job |
Run overrides
These run options override runtime keys for one run. See
Runtime and state and Lookups and fan-out.
| Option | What it does | Default |
|---|---|---|
--job-key KEY | A batch with this key runs once per scope; a later run with it does nothing (needs a persistent runtime.state) | The manifest's runtime.state.job_key |
--failed-job-policy abandon|retry|block | What the run first does with a failed batch: give it up, resume it, or stop | abandon |
--max-job-attempts N | With retry: how many times a failed batch may start, the first run included | 3 |
--state-lock-policy fail|wait|prune | When a lock token is held: stop, wait, or run only the free items | fail |
--state-lock-timeout DURATION | With wait: how long to wait (30, 30s, 5m) | The manifest's runtime.state.lock.timeout |
--apply-state-migrations | Create or upgrade the state tables before the run; without it a pending migration fails the run | Off |
--fan-out-concurrency N | runtime.fan_out: how many items (accounts) run at once | The manifest's fan_out.concurrency |
--fan-out-on-failure continue|stop | runtime.fan_out: after an item fails, go on with the others or start no new item | continue |
--allow-lookup-exec | Let spec.lookups run their commands and Python functions; without it such a lookup stops the run (exit 2) | Off |
Broker modes
For jobs that consume a broker. See Brokers: batch modes.
| Option | What it does | Default |
|---|---|---|
--once | One batch: unfinished batches resume, then one new batch runs | Yes |
--drain | Batches until a poll finds no message | |
--available-now | Batches until the messages there at the start are processed | |
--follow | Keep polling until interrupted, pausing --interval when a poll finds nothing | |
--interval DURATION | With --follow: the pause after an empty poll (30s, 5m, 1h) | 10s |
--max-messages N | The most messages one batch takes (the consume step's max_messages wins); 0 only resumes unfinished batches and relays the outbox | 500 |
sw duckdb run jobs/orders.yaml --secrets secrets.yaml --set client=example
sw duckdb run jobs/order_events.yaml --drain --max-messages 1000 --summary
Managing batches: duckdb state
Unfinished batches resume on the next run. Use state to look at a job's batches and fix stuck ones.
| Action | What it does |
|---|---|
list JOB [--scope TEXT] [--limit N] [--json] | The newest batches (default 20) with status, scope and error |
status JOB | The newest batch of the job's scope (selected by --config/--set) |
show JOB JOB_ID | One batch: its pinned inputs, lock tokens, metrics and output receipts |
retry JOB JOB_ID [--no-run] | Resume a failed batch, then run the job (--no-run: only mark it) |
abandon JOB JOB_ID | Give a batch up; what it wrote stays |
resend JOB JOB_ID [--output NAME] | Send an apply output's held rows again at the next run (check the vendor first) |
skip JOB JOB_ID [--output NAME] | Mark an apply output's held rows failed (UNKNOWN_OUTCOME) |
leases JOB [--json] | Every lease token of the job: owner, epoch, expiry |
purge JOB [--older-than 30d] [--scope TEXT] | Delete completed, abandoned and skipped batches older than the age (default 30 days) |
migrate JOB [--dry-run] | Create or upgrade the job's state tables (--dry-run prints the SQL) |
check JOB | Exit 0 when the state tables are up to date, 1 when a migration is pending |
Every action takes the job with its --config/--set values and --secrets, but needs only the state
connection's secrets. retry runs the job, so it needs them all.
streamwright spark
streamwright spark validate JOB.yaml [--strict] [--analyze]
streamwright spark plan JOB.yaml
streamwright spark run JOB.yaml [--once | --drain | --available-now | --follow]
streamwright spark lock JOB.yaml --spark 3.5.3 # or --check
streamwright spark state ACTION JOB.yaml
Validates, plans and runs kind: SparkJob manifests on Apache Spark (local, standalone or AWS EMR on EC2), and
manages their batches. Requires pip install streamwright-spark. See the Spark CLI reference for
every option.
streamwright state
streamwright state migrate [--url URL] [--schema SCHEMA] [--dry-run]
streamwright state show|check [--url URL] [--schema SCHEMA]
Works on the state tables (sw_*) of a database you name by URL, so a deployment can create and check them ahead of
time. A job run fails while a migration is pending, unless it runs with --apply-state-migrations.
| Action or option | What it does | Default |
|---|---|---|
migrate | Apply the pending migrations (safe to repeat) | |
show | List every migration, applied or pending | |
check | Exit 0 when up to date, 1 when a migration is pending, 2 when the URL is missing or invalid | |
--url URL | The database, such as sqlite:///state.db, duckdb:///state.duckdb, postgresql://...?schema=NAME or mysql://... | $STREAMWRIGHT_STATE_URL |
--schema SCHEMA | The tables' schema, overriding the URL's ?schema= (MySQL uses the URL's database; SQLite has no schema) | streamwright |
--dry-run | migrate: print the SQL; change nothing |
sw state check --url "$STREAMWRIGHT_STATE_URL" || exit 1
Connectors
streamwright connectors
streamwright connectors [NAME ...] # installed connectors (or only these), like pip show
streamwright connectors ls # the hub's catalog, with install status
streamwright connectors find WORDS # search the catalog
streamwright connectors -i KEY [KEY ...] [--yes] # install from the hub
With no arguments it shows each installed connector like pip show: its package and version, the libraries and
DuckDB extensions it uses, and the logger names to pass to --log.
| Option | What it does | Default |
|---|---|---|
list, ls, -ls, --list | List the catalog and what is installed | |
find WORDS, search, -f, -s, --find, --search | Search the catalog by keyword | |
install KEY ..., i, -i, --install | Install connectors with one pip install; an unknown key installs nothing | |
--yes | install: skip the prompt for connectors that are not official | Prompt |
--hub-url URL | The catalog to read for list, find and install | $STREAMWRIGHT_HUB_URL, else the live hub |
The catalog comes from the StreamWright hub, or a cached copy when the hub cannot be reached.
Serve jobs over HTTP
streamwright serve
streamwright serve SERVE_YAML [logging options]
Serves the jobs in SERVE_YAML as a REST API. Each request is checked and queued; a worker
runs it. Exit status 2 means serve.yaml or one of its jobs cannot be served (each problem is logged).
| Option | What it does | Default |
|---|---|---|
SERVE_YAML | The service: its jobs, callers and limits | |
| Logging options | See Logging options |
streamwright serve serve.yaml
curl -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"set": {"client_id": "1001"}}' http://127.0.0.1:8080/v1/jobs/client_feed_upload/runs
Requires pip install "streamwright[serve]". See Serving jobs over HTTP.
streamwright worker
streamwright worker SERVE_YAML [--processes N] [logging options]
Runs the runs serve queued, until stopped. SIGTERM or Ctrl-C lets runs under way finish first. See
Workers.
| Option | What it does | Default |
|---|---|---|
SERVE_YAML | The service whose queued runs to run | |
--processes N | How many runs at once, each in its own process | 1 |
streamwright queue
streamwright queue purge SERVE_YAML [--older-than AGE] [--dry-run]
streamwright queue migrate SERVE_YAML [--dry-run]
| Action or option | What it does | Default |
|---|---|---|
purge | Delete ended runs older than --older-than; queued and running runs are kept | |
--older-than AGE | purge: days (30) or a duration (30d, 12h) | 30 days |
migrate | Create or upgrade the queue's tables (safe to repeat) | |
--dry-run | purge: count the runs; migrate: print the SQL. Changes nothing. |
streamwright systemd install
sudo /opt/streamwright/venv/bin/streamwright systemd install [--unit-dir DIR] [--config-dir DIR] [--force] [--dry-run]
Writes the systemd units for serve, worker and a daily queue purge. Run it as root. The environment files in
--config-dir are created with mode 0600 when missing, and never replaced.
| Option | What it does | Default |
|---|---|---|
--unit-dir DIR | Where systemd reads the units | /etc/systemd/system |
--config-dir DIR | The folder of serve.yaml and the environment files | /etc/streamwright |
--force | Replace units that exist and differ | |
--dry-run | Print the units; write nothing |
AI assistants
streamwright mcp
streamwright mcp [--root DIR ...] [--transport stdio|sse|streamable-http [--allow-http] [--host HOST] [--port PORT]]
Runs the StreamWright Model Context Protocol (MCP) server. AI coding assistants use it to discover connectors, read their contracts and validate sources and jobs.
| Option | What it does | Default |
|---|---|---|
--transport | stdio (no network), sse or streamable-http | stdio |
--allow-http | Required for the network transports; callers send the token in STREAMWRIGHT_MCP_TOKEN | |
--host HOST · --port PORT | Network transports: interface and port | 127.0.0.1 · 8000 |
--root DIR | A folder the tools may read manifests from (repeatable) | $STREAMWRIGHT_MCP_ROOTS, else the working directory |
Requires pip install streamwright-mcp. See the MCP guide.
Logging options
run, validate, serve, worker, the duckdb commands and sw spark run take these options. Logs go to
stderr.
| Option | What it does | Default |
|---|---|---|
--log-level LEVEL | The level of StreamWright's loggers (DEBUG to CRITICAL) | INFO |
--log NAME=LEVEL | The level of any logger by name (repeatable; root is the root logger) | |
--log-format text|json | Plain lines, or one JSON object per line | text |
--log-config FILE | Your own Python logging configuration (dictConfig, YAML or JSON) | |
--log-max-chars N | Cut longer messages (0: never) | 20000 |
To log API calls:
- HTTP connectors:
--log streamwright.network=INFOlogs one line per call;DEBUGadds headers and bodies. Otherwisestreamwright.networkstays atWARNING. - SDK connectors: use the SDK's logger, such as
--log google.ads.googleads.client=DEBUG.sw connectorslists each connector's loggers.
Secret values are redacted from every log line, including SDK lines and your own --log-config handlers.
Job log tags and labels
Inside a job run, each text line is tagged with the job, its labels, the fan-out item and the step or output at
work. JSON lines carry the same as keys (job, labels, item, scope, job_id, step, output, lookup).
[...] INFO streamwright.duckdb [ads_campaigns tenant=t1 region=eu account=a1 output=save]: ...
Add labels in the job manifest (at most 8; lowercase names; values use {{ config.* }} only, never secrets):
runtime:
log:
labels:
tenant: "{{ config.tenant_id }}"
region: "{{ config.region }}"