Skip to main content

Spark jobs (kind: SparkJob)

A SparkJob is a job that runs on Apache Spark instead of an in-process DuckDB. Use it when your data is too large for one machine, or when you already run Spark. It runs on your machine, on a Spark standalone cluster or on AWS EMR on EC2.

In short

A SparkJob has the same parts as a DuckDBJob: connections, steps and outputs. Every read and write is an operation:, the SQL is Spark SQL, and runtime.platform says where the job runs.

Install it with pip install streamwright-spark. This adds sw spark to the CLI, and lets sw validate, sw plan and sw run accept kind: SparkJob files.

Spark or DuckDB?​

Use a DuckDBJob whenUse a SparkJob when
The data fits on one machineThe data is too large for one machine, or you already run Spark
You want no cluster and a fast startYou need a Spark cluster (standalone or EMR)
You write to DuckDB, files, databases, brokers or APIsYou also need Spark features: Delta Lake catalog registration, global views and caching, multi-schema file reads
INPUT SOURCESCloud StorageS3 · GCS · Azure BlobJSON · Parquet · CSVPartitioned directoriesEvent StreamsApache Kafka · AWS SQSString / JSON payloadsTopic & queue backlogsSTREAMWRIGHT SPARK ENGINELocal · Standalone · EMRCONTROL PLANE · Driver & Statejob.yamlDeclarative SQLDriver RunnerEpoch Leases & ManifestState StorePostgres / SQLiteSubmits Catalyst DAGDATA PLANE · Distributed Spark Execution1. Ingest & AlignSchema groupingRegex path extraction2. Catalyst SQLIn-memory viewsLineage & caching3. Merge / WritePartition pruningWrite receiptsLAKEHOUSE DESTINATIONSDelta Lake & IcebergACID TransactionsUpsert / Merge / AppendDeterministic idempotencyParquet TablesPartitioned file datasetsS3 / GCS / Hive MetastoreCatalog registration

What is the same, what differs​

Both engines read the same keys, with the same meaning: spec.config, spec.secrets, spec.lookups, connections, steps, outputs, policy, log, runtime.state, runtime.limits and runtime.fan_out. The DuckDB jobs page explains them. This section covers only what is different on Spark.

On a SparkJobInstead of (DuckDBJob)
Every read is an operation:, for example {service: object, method: read}read: and path: on a step
Every output has an operation: and its own sqlstep:, table: and mode: directly on the output
persist, checkpoint and view on a step (Views & caching)materialize (view or table)
runtime.platform (required) and runtime.spark (Launchers & runtime)runtime.threads, runtime.memory_limit

Two more Spark rules:

  • In sql, use the :batch_id, :batch_ts and :driver_host parameters. {{ batch.id }}, {{ batch.ts }} and the like work in paths, file names and predicates, but are an error in sql.
  • A policy key Spark cannot honour in a place (for example rate_limit on an output) is an error.

How it works​

A SparkJob keeps the data and the bookkeeping apart:

  1. Data goes through Spark. Spark executors read and write cloud storage (S3, GCS, Azure Blob, Delta Lake) and brokers (Kafka, SQS) directly. Your rows never pass through the state database.
  2. Bookkeeping goes to the state database. A small database (PostgreSQL, MySQL or a SQLite file) tracks leases, batch history and write receipts. These are the same sw_* tables a DuckDBJob uses. An output cannot write through the state connection. See Runtime, state & delivery guarantees.
  3. No duplicate writes. Each batch has a fixed id, and each finished output records a write receipt. If a node dies or the job restarts midway, the outputs that already finished are skipped. A Delta Lake append also checks the table's transaction log first, so a replayed batch never appends twice.

Call any connector (call steps)​

A SparkJob reads and writes object stores, Delta Lake, databases and brokers natively. To reach a connector that has no Spark reader, such as a REST API or an ads platform, use a call step. It works as on a DuckDBJob (see Call steps): a connection, an operation:, and the same policy keys.

steps:
- name: raw_items
connection: api # provider: restapi, or an ads connector
operation:
path: /v1/items
method: GET
policy:
page_size: 1000
retry: { max_attempts: 5, backoff: exponential, max_delay: 30s }
rate_limit: { requests: 2, per: 1s }
timeout: 60s
on_error: fail
sql: SELECT get_json_object(record, '$.id') AS id FROM records

- name: item_details # one call per id of raw_items
connection: api
partitions: [{ from: raw_items, field: id, name: item_id }]
operation:
path: /v1/items/{{ partition.item_id }}
method: GET
  • The Spark driver runs the connector and loads its records into a DataFrame, page_size records at a time (default 5000).

  • The step's sql reads the view records: one row per record, with one record column holding the record's JSON text. Read fields with get_json_object or from_json. Without sql, the step has the records' own columns.

  • partitions calls once per distinct, non-null value of a column of an earlier step, read as {{ partition.NAME }} (name defaults to field).

  • A call step can also take async_job: instead of operation: to run a report job, as on a DuckDBJob (see Async reports).

note

A call step's records are collected on the driver, so they must fit in the driver's memory. A step stops with an error once it passes runtime.limits.call_records records in a batch (default 1,000,000).

Run a source stream (source steps)​

A source: + stream: step runs one stream of a StreamWright source folder on the Spark driver, as on a DuckDBJob. Use it when a report or sync is already a source stream: incremental windows, lookback and bookmarks come with it.

steps:
- name: performance
source: ../sources/microsoft_ads # the source folder
stream: campaign_performance
config: { account_ids: ["{{ config.account_id }}"], customer_id: "{{ config.customer_id }}" }
sql: SELECT get_json_object(record, '$.campaign_id') AS campaign_id FROM records
  • The step's rows are the stream's export rows. With sql, read them from records as above; without it, the step has the export's columns, even when no rows came.
  • The rows are collected on the driver, with the same runtime.limits.call_records cap as call steps. Bookmarks are kept per partition, so each one resumes where it stopped.
  • source is a local folder (a relative path starts at the job file's folder) or an s3:// folder. When the job file itself is on S3, a relative path is read from beside it. The driver downloads the folder's YAML files (source.yaml and streams/*.yaml), up to 5 MiB.

Apply outputs (writing rows back to a vendor API) also work on Spark, with the same keys. Their rows are collected on the driver, at most runtime.limits.apply_rows (default 100,000).

Run a job​

sw spark run events_to_delta.yaml # one batch (the default)
sw spark run -j events_to_delta.yaml --available-now # work through the backlog, then stop

Jobs that consume a broker can also run with --drain or --follow. See Execution modes for each mode, and Launchers, runtime & catalogs to run on a cluster.

Example: event files to Delta Lake​

This job reads JSON event files from S3, keeps the purchases with SQL, and appends them to a Delta Lake table.

Sample input (incoming/events/events.json)​

{"event_id": "evt_101", "user_id": "usr_99", "event_type": "view", "amount": 0.0, "event_time": "2026-10-19T10:00:00Z"}
{"event_id": "evt_102", "user_id": "usr_42", "event_type": "purchase", "amount": 49.99, "event_time": "2026-10-19T10:05:00Z"}
{"event_id": "evt_103", "user_id": "usr_17", "event_type": "purchase", "amount": 120.50, "event_time": "2026-10-19T10:12:00Z"}

The job (events_to_delta.yaml)​

events_to_delta.yaml
kind: SparkJob
version: 1
name: events_to_delta
description: "Ingest event files and write paid purchases to Delta Lake"

connections:
cloud_storage:
provider: s3
roots: ["s3://example-lakehouse/"]
region: us-east-1
lake_delta:
provider: deltalake
roots: ["s3://example-lakehouse/"]
storage: cloud_storage

steps:
# Fetch: read the incoming files. Shape: keep the paid events with Spark SQL
- name: filtered_events
connection: cloud_storage
operation:
service: object
method: read
arguments:
path: "incoming/events/"
format: auto
options: { recursive: true }
sql: |
SELECT event_id, user_id, event_type, amount, event_time
FROM input
WHERE amount > 0

outputs:
# Write: append the rows to a Delta Lake table
- name: save_events
connection: lake_delta
operation:
service: table
method: write
arguments:
path: "s3://example-lakehouse/delta/purchases"
table_name: purchases
mode: append
sql: "SELECT * FROM filtered_events"

runtime:
state:
sqlite: /var/lib/streamwright/events_to_delta.db
platform:
type: local
spark:
app_name: events_to_delta

A read step's sql reads the rows it just read as input.

Run it​

sw spark run --job events_to_delta.yaml --once

Result​

Rows with amount > 0 are appended to the purchases Delta table. The view event evt_101 is filtered out:

event_iduser_idevent_typeamountevent_time
evt_102usr_42purchase49.992026-10-19 10:05:00
evt_103usr_17purchase120.502026-10-19 10:12:00

See also​