Spark jobs (kind: SparkJob)
A SparkJob is a job that runs on Apache Spark instead of an in-process DuckDB. Use it when your data is too large for one machine, or when you already run Spark. It runs on your machine, on a Spark standalone cluster or on AWS EMR on EC2.
A SparkJob has the same parts as a DuckDBJob: connections, steps and outputs. Every read and write is an
operation:, the SQL is Spark SQL, and runtime.platform says where the job runs.
Install it with pip install streamwright-spark. This adds sw spark to the CLI, and lets sw validate, sw plan
and sw run accept kind: SparkJob files.
Spark or DuckDB?
| Use a DuckDBJob when | Use a SparkJob when |
|---|---|
| The data fits on one machine | The data is too large for one machine, or you already run Spark |
| You want no cluster and a fast start | You need a Spark cluster (standalone or EMR) |
| You write to DuckDB, files, databases, brokers or APIs | You also need Spark features: Delta Lake catalog registration, global views and caching, multi-schema file reads |
What is the same, what differs
Both engines read the same keys, with the same meaning: spec.config, spec.secrets, spec.lookups,
connections, steps, outputs, policy, log, runtime.state, runtime.limits and runtime.fan_out. The
DuckDB jobs page explains them. This section covers only what is different on Spark.
| On a SparkJob | Instead of (DuckDBJob) |
|---|---|
Every read is an operation:, for example {service: object, method: read} | read: and path: on a step |
Every output has an operation: and its own sql | step:, table: and mode: directly on the output |
persist, checkpoint and view on a step (Views & caching) | materialize (view or table) |
runtime.platform (required) and runtime.spark (Launchers & runtime) | runtime.threads, runtime.memory_limit |
Two more Spark rules:
- In
sql, use the:batch_id,:batch_tsand:driver_hostparameters.{{ batch.id }},{{ batch.ts }}and the like work in paths, file names and predicates, but are an error insql. - A
policykey Spark cannot honour in a place (for examplerate_limiton an output) is an error.
How it works
A SparkJob keeps the data and the bookkeeping apart:
- Data goes through Spark. Spark executors read and write cloud storage (S3, GCS, Azure Blob, Delta Lake) and brokers (Kafka, SQS) directly. Your rows never pass through the state database.
- Bookkeeping goes to the state database. A small database (PostgreSQL, MySQL or a SQLite file) tracks leases,
batch history and write receipts. These are the same
sw_*tables a DuckDBJob uses. An output cannot write through the state connection. See Runtime, state & delivery guarantees. - No duplicate writes. Each batch has a fixed id, and each finished output records a write receipt. If a node dies or the job restarts midway, the outputs that already finished are skipped. A Delta Lake append also checks the table's transaction log first, so a replayed batch never appends twice.
Call any connector (call steps)
A SparkJob reads and writes object stores, Delta Lake, databases and brokers natively. To reach a connector that has
no Spark reader, such as a REST API or an ads platform, use a call step. It works as on a DuckDBJob (see
Call steps): a connection, an operation:, and the same policy keys.
steps:
- name: raw_items
connection: api # provider: restapi, or an ads connector
operation:
path: /v1/items
method: GET
policy:
page_size: 1000
retry: { max_attempts: 5, backoff: exponential, max_delay: 30s }
rate_limit: { requests: 2, per: 1s }
timeout: 60s
on_error: fail
sql: SELECT get_json_object(record, '$.id') AS id FROM records
- name: item_details # one call per id of raw_items
connection: api
partitions: [{ from: raw_items, field: id, name: item_id }]
operation:
path: /v1/items/{{ partition.item_id }}
method: GET
-
The Spark driver runs the connector and loads its records into a DataFrame,
page_sizerecords at a time (default 5000). -
The step's
sqlreads the viewrecords: one row per record, with onerecordcolumn holding the record's JSON text. Read fields withget_json_objectorfrom_json. Withoutsql, the step has the records' own columns. -
partitionscalls once per distinct, non-null value of a column of an earlier step, read as{{ partition.NAME }}(namedefaults tofield). -
A call step can also take
async_job:instead ofoperation:to run a report job, as on a DuckDBJob (see Async reports).
A call step's records are collected on the driver, so they must fit in the driver's memory. A step stops with an
error once it passes runtime.limits.call_records records in a batch (default 1,000,000).
Run a source stream (source steps)
A source: + stream: step runs one stream of a StreamWright source folder on the Spark driver,
as on a DuckDBJob. Use it when a report or sync is already a source stream: incremental windows, lookback and
bookmarks come with it.
steps:
- name: performance
source: ../sources/microsoft_ads # the source folder
stream: campaign_performance
config: { account_ids: ["{{ config.account_id }}"], customer_id: "{{ config.customer_id }}" }
sql: SELECT get_json_object(record, '$.campaign_id') AS campaign_id FROM records
- The step's rows are the stream's export rows. With
sql, read them fromrecordsas above; without it, the step has the export's columns, even when no rows came. - The rows are collected on the driver, with the same
runtime.limits.call_recordscap as call steps. Bookmarks are kept per partition, so each one resumes where it stopped. sourceis a local folder (a relative path starts at the job file's folder) or ans3://folder. When the job file itself is on S3, a relative path is read from beside it. The driver downloads the folder's YAML files (source.yamlandstreams/*.yaml), up to 5 MiB.
Apply outputs (writing rows back to a vendor API) also work on Spark, with the same keys. Their
rows are collected on the driver, at most runtime.limits.apply_rows (default 100,000).
Run a job
sw spark run events_to_delta.yaml # one batch (the default)
sw spark run -j events_to_delta.yaml --available-now # work through the backlog, then stop
Jobs that consume a broker can also run with --drain or --follow. See Execution modes
for each mode, and Launchers, runtime & catalogs to run on a cluster.
Example: event files to Delta Lake
This job reads JSON event files from S3, keeps the purchases with SQL, and appends them to a Delta Lake table.
Sample input (incoming/events/events.json)
{"event_id": "evt_101", "user_id": "usr_99", "event_type": "view", "amount": 0.0, "event_time": "2026-10-19T10:00:00Z"}
{"event_id": "evt_102", "user_id": "usr_42", "event_type": "purchase", "amount": 49.99, "event_time": "2026-10-19T10:05:00Z"}
{"event_id": "evt_103", "user_id": "usr_17", "event_type": "purchase", "amount": 120.50, "event_time": "2026-10-19T10:12:00Z"}
The job (events_to_delta.yaml)
kind: SparkJob
version: 1
name: events_to_delta
description: "Ingest event files and write paid purchases to Delta Lake"
connections:
cloud_storage:
provider: s3
roots: ["s3://example-lakehouse/"]
region: us-east-1
lake_delta:
provider: deltalake
roots: ["s3://example-lakehouse/"]
storage: cloud_storage
steps:
# Fetch: read the incoming files. Shape: keep the paid events with Spark SQL
- name: filtered_events
connection: cloud_storage
operation:
service: object
method: read
arguments:
path: "incoming/events/"
format: auto
options: { recursive: true }
sql: |
SELECT event_id, user_id, event_type, amount, event_time
FROM input
WHERE amount > 0
outputs:
# Write: append the rows to a Delta Lake table
- name: save_events
connection: lake_delta
operation:
service: table
method: write
arguments:
path: "s3://example-lakehouse/delta/purchases"
table_name: purchases
mode: append
sql: "SELECT * FROM filtered_events"
runtime:
state:
sqlite: /var/lib/streamwright/events_to_delta.db
platform:
type: local
spark:
app_name: events_to_delta
A read step's sql reads the rows it just read as input.
Run it
sw spark run --job events_to_delta.yaml --once
Result
Rows with amount > 0 are appended to the purchases Delta table. The view event evt_101 is filtered out:
event_id | user_id | event_type | amount | event_time |
|---|---|---|---|---|
evt_102 | usr_42 | purchase | 49.99 | 2026-10-19 10:05:00 |
evt_103 | usr_17 | purchase | 120.50 | 2026-10-19 10:12:00 |
See also
- Spark CLI reference
- Launchers, runtime & catalogs
- DuckDB jobs: the keys both engines share