Skip to main content

CLI

The streamwright command runs sources and jobs, checks them, installs connectors and serves jobs over HTTP. The short alias sw works everywhere (sw run, sw validate). Run sw <command> --help for the full help of any command.

In short

sw validate checks, sw plan shows the order, sw run runs. The manifest's kind picks the engine, so the same three commands work for sources and jobs.

Commands at a glance​

CommandWhat it does
runRun a source and write its records, or run a job manifest
validateCheck sources, jobs or a serve.yaml without running them
planPrint a job's execution plan (no secrets needed)
duckdbValidate, plan and run DuckDBJob manifests, and manage their batches
sparkValidate, plan, lock and run SparkJob manifests
stateCreate, upgrade and check the state tables in a database
connectorsShow installed connectors; list, search and install connectors from the hub
serveServe jobs over HTTP as a REST API
workerRun the jobs serve queued
queuePurge and migrate the run queue of serve
systemd installInstall systemd units for serve, worker and a daily purge
mcpRun the StreamWright MCP server for AI assistants

Most commands also take the shared logging options.

Exit status of run and the job commands:

StatusMeaning
0Success
1Invalid job, or the run failed
2Usage or input error: invalid source, inputs, stream names or options, or a missing or disallowed connector (nothing was fetched)
130Interrupted

Run and check​

streamwright run​

streamwright run SOURCE [options]
streamwright run JOB.yaml [job options] # or: streamwright run --job JOB.yaml

Runs a source and writes its records: by default Singer messages on stdout, with logs on stderr. Given a job manifest, it runs the job on its engine (duckdb or spark) with that engine's options. Options for sources only (--output, --stream, --state, ...) are refused for a job.

OptionWhat it doesDefault
SOURCEA source folder (source.yaml plus streams/), a source file, or a job manifest
--job JOB, -jA job manifest, as an option instead of SOURCE
--config FILEA client's settings: config values and the streams (or exports) to run
--set NAME=VALUEOne spec.config value (repeatable; lists as a,b,c); overrides --config
--secrets FILEValues for spec.secrets; overrides STREAMWRIGHT_SECRET_<NAME> variables
--state FILEState from a previous run (JSON, or a Singer STATE message), for incremental streams
--timezone TZThe client's IANA time zone (such as America/New_York) for today and incremental windowsThe machine's local date
--stream NAMERun only this stream, or the stream of this export (repeatable; replaces the --config file's streams). The streams its from_stream partitions come from run too.All streams
--output TARGETWhere records go: singer, jsonl:DIR, csv:DIR, tsv:DIR, parquet:DIR, dlt:DESTINATION[:DATASET], duckdb:PATH[:SCHEMA], ducklake:CATALOG[:SCHEMA]. See Outputs.singer
--file-name TEMPLATEFile outputs only: each export's path inside DIR. Uses {{ export }}, {{ source }}, {{ today }}, {{ timestamp }}, {{ format }}, {{ config.NAME }}.The YAML's file_name, else <export>.<date>.<time>.<unique>.<ext>
--allow-connector NAMEAllow only this connector (repeatable)Any installed connector
--summary [FILE]Also write the run summary as JSON to FILE, whatever the outcome. Bare --summary logs it as one INFO line (event: run_summary).
sw run sources/google_ads --config clients/example.yaml --secrets secrets.yaml \
--output parquet:out --file-name "{{ config.client }}/{{ export }}_{{ today }}.{{ format }}"

streamwright validate​

streamwright validate [PATH ...] [--strict] [--format text|json|github] [--export-schema DIR]

Checks sources (schema, references, SQL) without fetching anything. It walks folders recursively. It also checks job manifests (as the path, or with --job) and a serve.yaml with its jobs, without needing secrets. A job is checked by its engine, with that engine's options (sw duckdb validate -h, sw spark validate -h).

OptionWhat it doesDefault
PATH ...Files or folders to check, recursively; a source folder is checked as one source$STREAMWRIGHT_CONFIGS
--strictExit with status 1 on warnings too
--format text|json|githubOutput format; github prints GitHub Actions annotationstext
--export-schema DIRWrite the JSON Schemas of sources and stream files (for editors and CI), then exit
--kind sourceThe kind to assume for files that do not declare kind
--allow-connector NAMESources: only allow these connectors in auth.provider (repeatable)
sw validate sources/ --strict --format github

streamwright plan​

streamwright plan JOB.yaml # or: streamwright plan --job JOB.yaml

Prints a job's steps, outputs and the order they run in. It runs nothing and needs no secrets. The manifest's kind picks the engine, and its options are that engine's (sw duckdb plan -h, sw spark plan -h).

OptionWhat it doesDefault
JOBThe job manifest: a YAML file, or its folder (job.yaml)
--job JOB, -jThe job manifest, as an option instead of JOB

Jobs and state​

streamwright duckdb​

streamwright duckdb validate JOB.yaml [--strict] [--graph] [--input FILE|-]
streamwright duckdb plan JOB.yaml
streamwright duckdb run JOB.yaml [--dry-run] [--secrets FILE] [--input FILE|-]
streamwright duckdb run JOB.yaml [--once | --drain | --available-now | --follow [--interval 10s]] [--max-messages N]
streamwright duckdb state ACTION JOB.yaml [JOB_ID] [options]

Runs kind: DuckDBJob manifests in an in-process DuckDB (see DuckDB jobs). streamwright validate | plan | run JOB send a DuckDBJob here automatically. JOB is a YAML file, or a folder that holds job.yaml.

Job options​

OptionWhat it doesDefault
--job JOB, -jThe job manifest, as an option instead of JOB
--config FILE · --set NAME=VALUEspec.config values (--set is repeatable and overrides --config)
--overlay FILEA YAML overlay merged onto the job (repeatable, in order; null deletes a key, a list replaces the job's)
--allow-connector NAMEUse only this connector (repeatable)Any installed connector
--secrets FILErun: spec.secrets values (or STREAMWRIGHT_SECRET_<NAME> variables)
--input FILEvalidate, run: rows for a job with spec.input, as a JSON array or NDJSON (- reads stdin); see Input payloads
--dry-runrun: validate, resolve the inputs and print the plan; connect to nothing, write nothing
--summary [FILE]run: also write the run's summary as JSON, whatever the outcome; bare: one INFO log line
--strictTreat warnings as errors
--graphvalidate: also print the plan of a valid job

Run overrides​

These run options override runtime keys for one run. See Runtime and state and Lookups and fan-out.

OptionWhat it doesDefault
--job-key KEYA batch with this key runs once per scope; a later run with it does nothing (needs a persistent runtime.state)The manifest's runtime.state.job_key
--failed-job-policy abandon|retry|blockWhat the run first does with a failed batch: give it up, resume it, or stopabandon
--max-job-attempts NWith retry: how many times a failed batch may start, the first run included3
--state-lock-policy fail|wait|pruneWhen a lock token is held: stop, wait, or run only the free itemsfail
--state-lock-timeout DURATIONWith wait: how long to wait (30, 30s, 5m)The manifest's runtime.state.lock.timeout
--apply-state-migrationsCreate or upgrade the state tables before the run; without it a pending migration fails the runOff
--fan-out-concurrency Nruntime.fan_out: how many items (accounts) run at onceThe manifest's fan_out.concurrency
--fan-out-on-failure continue|stopruntime.fan_out: after an item fails, go on with the others or start no new itemcontinue
--allow-lookup-execLet spec.lookups run their commands and Python functions; without it such a lookup stops the run (exit 2)Off

Broker modes​

For jobs that consume a broker. See Brokers: batch modes.

OptionWhat it doesDefault
--onceOne batch: unfinished batches resume, then one new batch runsYes
--drainBatches until a poll finds no message
--available-nowBatches until the messages there at the start are processed
--followKeep polling until interrupted, pausing --interval when a poll finds nothing
--interval DURATIONWith --follow: the pause after an empty poll (30s, 5m, 1h)10s
--max-messages NThe most messages one batch takes (the consume step's max_messages wins); 0 only resumes unfinished batches and relays the outbox500
sw duckdb run jobs/orders.yaml --secrets secrets.yaml --set client=example
sw duckdb run jobs/order_events.yaml --drain --max-messages 1000 --summary

Managing batches: duckdb state​

Unfinished batches resume on the next run. Use state to look at a job's batches and fix stuck ones.

ActionWhat it does
list JOB [--scope TEXT] [--limit N] [--json]The newest batches (default 20) with status, scope and error
status JOBThe newest batch of the job's scope (selected by --config/--set)
show JOB JOB_IDOne batch: its pinned inputs, lock tokens, metrics and output receipts
retry JOB JOB_ID [--no-run]Resume a failed batch, then run the job (--no-run: only mark it)
abandon JOB JOB_IDGive a batch up; what it wrote stays
resend JOB JOB_ID [--output NAME]Send an apply output's held rows again at the next run (check the vendor first)
skip JOB JOB_ID [--output NAME]Mark an apply output's held rows failed (UNKNOWN_OUTCOME)
leases JOB [--json]Every lease token of the job: owner, epoch, expiry
purge JOB [--older-than 30d] [--scope TEXT]Delete completed, abandoned and skipped batches older than the age (default 30 days)
migrate JOB [--dry-run]Create or upgrade the job's state tables (--dry-run prints the SQL)
check JOBExit 0 when the state tables are up to date, 1 when a migration is pending

Every action takes the job with its --config/--set values and --secrets, but needs only the state connection's secrets. retry runs the job, so it needs them all.

streamwright spark​

streamwright spark validate JOB.yaml [--strict] [--analyze]
streamwright spark plan JOB.yaml
streamwright spark run JOB.yaml [--once | --drain | --available-now | --follow]
streamwright spark lock JOB.yaml --spark 3.5.3 # or --check
streamwright spark state ACTION JOB.yaml

Validates, plans and runs kind: SparkJob manifests on Apache Spark (local, standalone or AWS EMR on EC2), and manages their batches. Requires pip install streamwright-spark. See the Spark CLI reference for every option.

streamwright state​

streamwright state migrate [--url URL] [--schema SCHEMA] [--dry-run]
streamwright state show|check [--url URL] [--schema SCHEMA]

Works on the state tables (sw_*) of a database you name by URL, so a deployment can create and check them ahead of time. A job run fails while a migration is pending, unless it runs with --apply-state-migrations.

Action or optionWhat it doesDefault
migrateApply the pending migrations (safe to repeat)
showList every migration, applied or pending
checkExit 0 when up to date, 1 when a migration is pending, 2 when the URL is missing or invalid
--url URLThe database, such as sqlite:///state.db, duckdb:///state.duckdb, postgresql://...?schema=NAME or mysql://...$STREAMWRIGHT_STATE_URL
--schema SCHEMAThe tables' schema, overriding the URL's ?schema= (MySQL uses the URL's database; SQLite has no schema)streamwright
--dry-runmigrate: print the SQL; change nothing
sw state check --url "$STREAMWRIGHT_STATE_URL" || exit 1

Connectors​

streamwright connectors​

streamwright connectors [NAME ...] # installed connectors (or only these), like pip show
streamwright connectors ls # the hub's catalog, with install status
streamwright connectors find WORDS # search the catalog
streamwright connectors -i KEY [KEY ...] [--yes] # install from the hub

With no arguments it shows each installed connector like pip show: its package and version, the libraries and DuckDB extensions it uses, and the logger names to pass to --log.

OptionWhat it doesDefault
list, ls, -ls, --listList the catalog and what is installed
find WORDS, search, -f, -s, --find, --searchSearch the catalog by keyword
install KEY ..., i, -i, --installInstall connectors with one pip install; an unknown key installs nothing
--yesinstall: skip the prompt for connectors that are not officialPrompt
--hub-url URLThe catalog to read for list, find and install$STREAMWRIGHT_HUB_URL, else the live hub

The catalog comes from the StreamWright hub, or a cached copy when the hub cannot be reached.

Serve jobs over HTTP​

streamwright serve​

streamwright serve SERVE_YAML [logging options]

Serves the jobs in SERVE_YAML as a REST API. Each request is checked and queued; a worker runs it. Exit status 2 means serve.yaml or one of its jobs cannot be served (each problem is logged).

OptionWhat it doesDefault
SERVE_YAMLThe service: its jobs, callers and limits
Logging optionsSee Logging options
streamwright serve serve.yaml
curl -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"set": {"client_id": "1001"}}' http://127.0.0.1:8080/v1/jobs/client_feed_upload/runs

Requires pip install "streamwright[serve]". See Serving jobs over HTTP.

streamwright worker​

streamwright worker SERVE_YAML [--processes N] [logging options]

Runs the runs serve queued, until stopped. SIGTERM or Ctrl-C lets runs under way finish first. See Workers.

OptionWhat it doesDefault
SERVE_YAMLThe service whose queued runs to run
--processes NHow many runs at once, each in its own process1

streamwright queue​

streamwright queue purge SERVE_YAML [--older-than AGE] [--dry-run]
streamwright queue migrate SERVE_YAML [--dry-run]
Action or optionWhat it doesDefault
purgeDelete ended runs older than --older-than; queued and running runs are kept
--older-than AGEpurge: days (30) or a duration (30d, 12h)30 days
migrateCreate or upgrade the queue's tables (safe to repeat)
--dry-runpurge: count the runs; migrate: print the SQL. Changes nothing.

streamwright systemd install​

sudo /opt/streamwright/venv/bin/streamwright systemd install [--unit-dir DIR] [--config-dir DIR] [--force] [--dry-run]

Writes the systemd units for serve, worker and a daily queue purge. Run it as root. The environment files in --config-dir are created with mode 0600 when missing, and never replaced.

OptionWhat it doesDefault
--unit-dir DIRWhere systemd reads the units/etc/systemd/system
--config-dir DIRThe folder of serve.yaml and the environment files/etc/streamwright
--forceReplace units that exist and differ
--dry-runPrint the units; write nothing

See Deploying with systemd.

AI assistants​

streamwright mcp​

streamwright mcp [--root DIR ...] [--transport stdio|sse|streamable-http [--allow-http] [--host HOST] [--port PORT]]

Runs the StreamWright Model Context Protocol (MCP) server. AI coding assistants use it to discover connectors, read their contracts and validate sources and jobs.

OptionWhat it doesDefault
--transportstdio (no network), sse or streamable-httpstdio
--allow-httpRequired for the network transports; callers send the token in STREAMWRIGHT_MCP_TOKEN
--host HOST · --port PORTNetwork transports: interface and port127.0.0.1 · 8000
--root DIRA folder the tools may read manifests from (repeatable)$STREAMWRIGHT_MCP_ROOTS, else the working directory

Requires pip install streamwright-mcp. See the MCP guide.

Logging options​

run, validate, serve, worker, the duckdb commands and sw spark run take these options. Logs go to stderr.

OptionWhat it doesDefault
--log-level LEVELThe level of StreamWright's loggers (DEBUG to CRITICAL)INFO
--log NAME=LEVELThe level of any logger by name (repeatable; root is the root logger)
--log-format text|jsonPlain lines, or one JSON object per linetext
--log-config FILEYour own Python logging configuration (dictConfig, YAML or JSON)
--log-max-chars NCut longer messages (0: never)20000

To log API calls:

  • HTTP connectors: --log streamwright.network=INFO logs one line per call; DEBUG adds headers and bodies. Otherwise streamwright.network stays at WARNING.
  • SDK connectors: use the SDK's logger, such as --log google.ads.googleads.client=DEBUG. sw connectors lists each connector's loggers.
note

Secret values are redacted from every log line, including SDK lines and your own --log-config handlers.

Job log tags and labels​

Inside a job run, each text line is tagged with the job, its labels, the fan-out item and the step or output at work. JSON lines carry the same as keys (job, labels, item, scope, job_id, step, output, lookup).

[...] INFO streamwright.duckdb [ads_campaigns tenant=t1 region=eu account=a1 output=save]: ...

Add labels in the job manifest (at most 8; lowercase names; values use {{ config.* }} only, never secrets):

runtime:
log:
labels:
tenant: "{{ config.tenant_id }}"
region: "{{ config.region }}"