Spark CLI reference
sw spark checks, plans, pins and runs kind: SparkJob files, and manages their batches. It is also available as
streamwright spark, streamwright-spark and sw-spark.
You rarely need to type spark: sw validate, sw plan and sw run send a SparkJob here on their own. Use
sw spark for lock, state and the Spark-only options.
sw spark validate JOB [options]
sw spark plan JOB [options]
sw spark run JOB [options]
sw spark lock JOB [options]
sw spark state ACTION JOB [JOB_ID] [options]
JOB is a YAML file, or a folder that holds job.yaml. You can also pass it as an option: --job/-j.
Exit status: 0 success, 1 invalid job or failed run, 2 usage or input error, 130 interrupted.
Options every command takes
validate, plan, run and lock take these. They work as on a DuckDBJob
(streamwright duckdb).
| Option | What it does |
|---|---|
--job FILE, -j | The job manifest, as an option |
--overlay FILE | A YAML overlay merged onto the job, for example overlays/prod.yaml (repeatable) |
--config FILE · --set NAME=VALUE | spec.config values (--set is repeatable). Not taken by lock, which reads only the job's connections |
--allow-connector NAME | Use only these connectors (repeatable) |
sw spark run
Runs a SparkJob where runtime.platform says: in this process, or submitted to a cluster (spark-submit, or an
EMR step). Unfinished batches of the job's runtime.state scope resume first, then new batches run.
sw spark run jobs/orders.yaml --secrets secrets.yaml --set client=acme
sw spark run jobs/order_events.yaml --available-now --max-messages 5000 --summary
sw spark run jobs/orders.yaml --lock orders.lock.json --lock-required --detach
sw spark run jobs/orders.yaml --dry-run
Execution modes
| Mode | What it does | Good for |
|---|---|---|
--once (default) | Resumes unfinished batches, then runs one new batch (up to --max-messages) and exits | Tests, local checks, small syncs |
--available-now | Notes the broker's end offsets at launch, runs batches until it reaches them, then exits. Messages that arrive later wait for the next run. A broker without end offsets runs as --drain | Scheduled jobs (Airflow, cron, EventBridge) |
--drain | Runs batches until a poll finds no message | Emptying a queue or topic, migrations |
--follow | Keeps polling until interrupted, pausing --interval (default 10s) after an empty poll | Long-running worker containers |
--drain, --available-now and --follow are for jobs that consume a broker (a step with method: consume).
Run options
These behave as on sw duckdb run; see streamwright duckdb for more detail.
| Option | What it does |
|---|---|
--secrets FILE | spec.secrets values (or STREAMWRIGHT_SECRET_<NAME> variables) |
--max-messages N | The most messages one batch takes (default 500; the consume step's max_messages wins). 0 only resumes unfinished batches and relays the outbox |
--interval DURATION | With --follow: the pause after an empty poll (default 10s; 30s, 5m, 1h) |
--job-key KEY | Jobs without a consume step: a batch with this key runs once per runtime.state.scope (overrides runtime.state.job_key) |
--state-lock-policy fail|wait|prune | When a runtime.state.lock token is held: stop, wait up to --state-lock-timeout, or run only the free lock.each items (a run left with none is recorded skipped, exit 0) |
--state-lock-timeout DURATION | Policy wait: how long to wait (30, 30s, 5m) |
--failed-job-policy abandon|retry|block | What to do first with a failed batch of the scope: give it up (default), resume it while it has started fewer than --max-job-attempts times, or stop until state retry or state abandon |
--max-job-attempts N | Policy retry: how many times a failed batch may start in all, the first run included (default 3) |
--allow-lookup-exec | Let spec.lookups run commands and Python functions on the driver (exit 2 without it). See Lookups and fan-out |
--fan-out-concurrency N · --fan-out-on-failure continue|stop | runtime.fan_out: how many items run at once; after an item fails, run the others (default) or start no new one |
--apply-state-migrations | Create or upgrade the state tables before the run. Without it, a pending migration fails the run (exit 1): apply it with sw spark state migrate JOB |
--summary [FILE] | Also write the run's summary as JSON to FILE on the driver (status ok, skipped or failed, times, job ids, each output's rows and files, the error). A bare --summary logs it as one INFO line |
--dry-run | Validate and resolve the job, then print its plan and the submission it would launch (the spark-submit command or the EMR step). Runs nothing |
The state lock, failed-job, lookup and migration options are forwarded to the driver when the job runs on a cluster.
Spark-only run options
| Option | What it does |
|---|---|
--conf KEY=VALUE | A Spark setting over runtime.spark.conf (repeatable), for example --conf spark.executor.memory=4g. Keys need the spark. prefix. The launcher's own keys (spark.master, spark.submit.deployMode, spark.jars*, ...) are refused (exit 2) |
--spark-log-level LEVEL | The Spark (JVM) log level: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN. Default: spark.log.level in runtime.spark.conf, else Spark's |
--detach | Remote platforms only: submit the job and exit without waiting |
--lock FILE | The dependency lock file (from sw spark lock) to check the installed versions against. Not the same as runtime.state.lock |
--lock-required | Refuse to run without a valid dependency lock file, on every platform |
The logging options (--log-level, --log, --log-format, --log-config,
--log-max-chars) set StreamWright's loggers. They are forwarded to a cluster driver, except --log-config, which is
local only.
sw spark validate
Checks SparkJob manifests offline, without Spark, brokers or secret values: structure, inputs, connections (against the installed connectors' contracts), steps, outputs, references, runtime and platform. It accepts several jobs.
sw spark validate jobs/orders.yaml --set client=acme --analyze
sw spark validate jobs/*.yaml --strict --lock orders.lock.json
| Option | What it does |
|---|---|
--strict | Treat warnings as errors (for example unused steps or unneeded persistence) |
--analyze | Also print the Spark actions and recomputation warnings of a valid job |
--graph | Also print each step's and output's dependencies |
--lock FILE | The dependency lock file to check the installed versions against |
sw spark plan
Prints a valid job's Spark actions, recomputation warnings and step dependencies. It is the same as
validate --analyze --graph, so it does not take those two flags. It does take --strict and --lock.
sw spark plan jobs/orders.yaml
sw spark lock
Writes the job's dependency lock file, so a cluster runs the versions you tested with. It records the Spark, Python
and package versions installed here (including streamwright-spark and the connectors), the JARs the job's
connectors need, and optionally the EMR release and a packaged Python environment. run --lock FILE checks it.
sw spark lock jobs/orders.yaml --spark 3.5.3 --emr-release emr-7.2.0 --out orders.lock.json
sw spark lock jobs/orders.yaml --out orders.lock.json --check
| Option | What it does |
|---|---|
--spark VERSION | The Spark version the job runs on, for example 3.5.3. Required to write a lock. With --check it defaults to the installed pyspark |
--out FILE | The lock file to write (with --check: to read). Default: runtime.artifacts.lock, else <name>.lock.json |
--emr-release LABEL | The EMR release label to record (with --check: to compare), for example emr-7.2.0 |
--env-archive FILE | A packaged virtual environment (.tar.gz) for remote executors, recorded with its sha256. Not with --check |
--jar GROUP:ARTIFACT=VERSION | A JAR's version (repeatable): for a JAR a connector needs without a default version, or to replace its default. A malformed value is refused (exit 2) |
--check | Read the lock instead of writing it. Fails (exit 2) when the installed Spark, Python or package versions or the EMR release differ, or when the job's connectors now need packages or JARs the lock does not pin (or no longer need ones it pins) |
sw spark state
Manages a SparkJob's batches, leases and state tables. These are the same sw_* tables and actions as
streamwright duckdb state. Batches are kept per runtime.state.scope; a
failed batch stops only its scope.
sw spark state migrate job.yaml [--dry-run] # create or upgrade the tables
sw spark state check job.yaml # exit 1 while a migration is pending
sw spark state status job.yaml # the newest batch of the job's scope
sw spark state list job.yaml [--scope TEXT] [--limit N] [--json]
sw spark state show job.yaml JOB_ID # pinned manifest (inputs), lock tokens, metrics, receipts
sw spark state retry job.yaml JOB_ID # resume it at the next run, with its pinned values and tokens
sw spark state abandon job.yaml JOB_ID # give it up: its scope runs new batches again
sw spark state resend job.yaml JOB_ID [--output NAME] # send an apply output's held rows again at the next run
sw spark state skip job.yaml JOB_ID [--output NAME] # mark held rows failed (UNKNOWN_OUTCOME)
sw spark state purge job.yaml --older-than 30d # delete old finished batches (sequences stay)
sw spark state leases job.yaml # every lease token: owner, epoch, expiry
Good to know:
resendis for rows whose outcome is unknown and whose operation is not idempotent: check the vendor first, as they may be applied twice.- Every action takes the job (
JOBor--job,--overlay,--config,--set) and--secrets; only the state connection's secrets are needed. Pass the job's required--set/--configvalues, as the run did. - Unlike
duckdb state retry,sw spark state retrydoes not run the job: the batch resumes at the job's next run. retryandabandontake the batch's lock tokens, so they fail while a run of its scope holds them.