Skip to main content

Spark CLI reference

sw spark checks, plans, pins and runs kind: SparkJob files, and manages their batches. It is also available as streamwright spark, streamwright-spark and sw-spark.

In short

You rarely need to type spark: sw validate, sw plan and sw run send a SparkJob here on their own. Use sw spark for lock, state and the Spark-only options.

sw spark validate JOB [options]
sw spark plan JOB [options]
sw spark run JOB [options]
sw spark lock JOB [options]
sw spark state ACTION JOB [JOB_ID] [options]

JOB is a YAML file, or a folder that holds job.yaml. You can also pass it as an option: --job/-j.

Exit status: 0 success, 1 invalid job or failed run, 2 usage or input error, 130 interrupted.

Options every command takes​

validate, plan, run and lock take these. They work as on a DuckDBJob (streamwright duckdb).

OptionWhat it does
--job FILE, -jThe job manifest, as an option
--overlay FILEA YAML overlay merged onto the job, for example overlays/prod.yaml (repeatable)
--config FILE · --set NAME=VALUEspec.config values (--set is repeatable). Not taken by lock, which reads only the job's connections
--allow-connector NAMEUse only these connectors (repeatable)

sw spark run​

Runs a SparkJob where runtime.platform says: in this process, or submitted to a cluster (spark-submit, or an EMR step). Unfinished batches of the job's runtime.state scope resume first, then new batches run.

sw spark run jobs/orders.yaml --secrets secrets.yaml --set client=acme
sw spark run jobs/order_events.yaml --available-now --max-messages 5000 --summary
sw spark run jobs/orders.yaml --lock orders.lock.json --lock-required --detach
sw spark run jobs/orders.yaml --dry-run

Execution modes​

ModeWhat it doesGood for
--once (default)Resumes unfinished batches, then runs one new batch (up to --max-messages) and exitsTests, local checks, small syncs
--available-nowNotes the broker's end offsets at launch, runs batches until it reaches them, then exits. Messages that arrive later wait for the next run. A broker without end offsets runs as --drainScheduled jobs (Airflow, cron, EventBridge)
--drainRuns batches until a poll finds no messageEmptying a queue or topic, migrations
--followKeeps polling until interrupted, pausing --interval (default 10s) after an empty pollLong-running worker containers

--drain, --available-now and --follow are for jobs that consume a broker (a step with method: consume).

Run options​

These behave as on sw duckdb run; see streamwright duckdb for more detail.

OptionWhat it does
--secrets FILEspec.secrets values (or STREAMWRIGHT_SECRET_<NAME> variables)
--max-messages NThe most messages one batch takes (default 500; the consume step's max_messages wins). 0 only resumes unfinished batches and relays the outbox
--interval DURATIONWith --follow: the pause after an empty poll (default 10s; 30s, 5m, 1h)
--job-key KEYJobs without a consume step: a batch with this key runs once per runtime.state.scope (overrides runtime.state.job_key)
--state-lock-policy fail|wait|pruneWhen a runtime.state.lock token is held: stop, wait up to --state-lock-timeout, or run only the free lock.each items (a run left with none is recorded skipped, exit 0)
--state-lock-timeout DURATIONPolicy wait: how long to wait (30, 30s, 5m)
--failed-job-policy abandon|retry|blockWhat to do first with a failed batch of the scope: give it up (default), resume it while it has started fewer than --max-job-attempts times, or stop until state retry or state abandon
--max-job-attempts NPolicy retry: how many times a failed batch may start in all, the first run included (default 3)
--allow-lookup-execLet spec.lookups run commands and Python functions on the driver (exit 2 without it). See Lookups and fan-out
--fan-out-concurrency N · --fan-out-on-failure continue|stopruntime.fan_out: how many items run at once; after an item fails, run the others (default) or start no new one
--apply-state-migrationsCreate or upgrade the state tables before the run. Without it, a pending migration fails the run (exit 1): apply it with sw spark state migrate JOB
--summary [FILE]Also write the run's summary as JSON to FILE on the driver (status ok, skipped or failed, times, job ids, each output's rows and files, the error). A bare --summary logs it as one INFO line
--dry-runValidate and resolve the job, then print its plan and the submission it would launch (the spark-submit command or the EMR step). Runs nothing

The state lock, failed-job, lookup and migration options are forwarded to the driver when the job runs on a cluster.

Spark-only run options​

OptionWhat it does
--conf KEY=VALUEA Spark setting over runtime.spark.conf (repeatable), for example --conf spark.executor.memory=4g. Keys need the spark. prefix. The launcher's own keys (spark.master, spark.submit.deployMode, spark.jars*, ...) are refused (exit 2)
--spark-log-level LEVELThe Spark (JVM) log level: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN. Default: spark.log.level in runtime.spark.conf, else Spark's
--detachRemote platforms only: submit the job and exit without waiting
--lock FILEThe dependency lock file (from sw spark lock) to check the installed versions against. Not the same as runtime.state.lock
--lock-requiredRefuse to run without a valid dependency lock file, on every platform

The logging options (--log-level, --log, --log-format, --log-config, --log-max-chars) set StreamWright's loggers. They are forwarded to a cluster driver, except --log-config, which is local only.

sw spark validate​

Checks SparkJob manifests offline, without Spark, brokers or secret values: structure, inputs, connections (against the installed connectors' contracts), steps, outputs, references, runtime and platform. It accepts several jobs.

sw spark validate jobs/orders.yaml --set client=acme --analyze
sw spark validate jobs/*.yaml --strict --lock orders.lock.json
OptionWhat it does
--strictTreat warnings as errors (for example unused steps or unneeded persistence)
--analyzeAlso print the Spark actions and recomputation warnings of a valid job
--graphAlso print each step's and output's dependencies
--lock FILEThe dependency lock file to check the installed versions against

sw spark plan​

Prints a valid job's Spark actions, recomputation warnings and step dependencies. It is the same as validate --analyze --graph, so it does not take those two flags. It does take --strict and --lock.

sw spark plan jobs/orders.yaml

sw spark lock​

Writes the job's dependency lock file, so a cluster runs the versions you tested with. It records the Spark, Python and package versions installed here (including streamwright-spark and the connectors), the JARs the job's connectors need, and optionally the EMR release and a packaged Python environment. run --lock FILE checks it.

sw spark lock jobs/orders.yaml --spark 3.5.3 --emr-release emr-7.2.0 --out orders.lock.json
sw spark lock jobs/orders.yaml --out orders.lock.json --check
OptionWhat it does
--spark VERSIONThe Spark version the job runs on, for example 3.5.3. Required to write a lock. With --check it defaults to the installed pyspark
--out FILEThe lock file to write (with --check: to read). Default: runtime.artifacts.lock, else <name>.lock.json
--emr-release LABELThe EMR release label to record (with --check: to compare), for example emr-7.2.0
--env-archive FILEA packaged virtual environment (.tar.gz) for remote executors, recorded with its sha256. Not with --check
--jar GROUP:ARTIFACT=VERSIONA JAR's version (repeatable): for a JAR a connector needs without a default version, or to replace its default. A malformed value is refused (exit 2)
--checkRead the lock instead of writing it. Fails (exit 2) when the installed Spark, Python or package versions or the EMR release differ, or when the job's connectors now need packages or JARs the lock does not pin (or no longer need ones it pins)

sw spark state​

Manages a SparkJob's batches, leases and state tables. These are the same sw_* tables and actions as streamwright duckdb state. Batches are kept per runtime.state.scope; a failed batch stops only its scope.

sw spark state migrate job.yaml [--dry-run] # create or upgrade the tables
sw spark state check job.yaml # exit 1 while a migration is pending
sw spark state status job.yaml # the newest batch of the job's scope
sw spark state list job.yaml [--scope TEXT] [--limit N] [--json]
sw spark state show job.yaml JOB_ID # pinned manifest (inputs), lock tokens, metrics, receipts
sw spark state retry job.yaml JOB_ID # resume it at the next run, with its pinned values and tokens
sw spark state abandon job.yaml JOB_ID # give it up: its scope runs new batches again
sw spark state resend job.yaml JOB_ID [--output NAME] # send an apply output's held rows again at the next run
sw spark state skip job.yaml JOB_ID [--output NAME] # mark held rows failed (UNKNOWN_OUTCOME)
sw spark state purge job.yaml --older-than 30d # delete old finished batches (sequences stay)
sw spark state leases job.yaml # every lease token: owner, epoch, expiry

Good to know:

  • resend is for rows whose outcome is unknown and whose operation is not idempotent: check the vendor first, as they may be applied twice.
  • Every action takes the job (JOB or --job, --overlay, --config, --set) and --secrets; only the state connection's secrets are needed. Pass the job's required --set/--config values, as the run did.
  • Unlike duckdb state retry, sw spark state retry does not run the job: the batch resumes at the job's next run.
  • retry and abandon take the batch's lock tokens, so they fail while a run of its scope holds them.