Skip to main content

Views, caching & lineage

Each SparkJob step becomes a Spark temporary view that later steps and outputs read by its name. Steps are lazy DataFrames: the outputs' writes trigger the work. Four step keys control how a step's view is shared, cached and logged.

KeyWhat it does
view: globalAlso share the view with other Spark sessions and tools
persistCache the step's result so it is computed once
checkpoint: trueSave the result and cut its lineage (long query plans)
logLog a line per row of the step

Global and local views (view)​

By default (view: local) a step is a temporary view, visible only inside this job. With view: global it is also registered in Spark's global_temp database, where other sessions and tools (Thrift Server, notebooks, BI tools) can query it.

steps:
- name: active_records # local view (default)
sql: "SELECT * FROM input WHERE status = 'ACTIVE'"

- name: daily_summary # local and global view
view: global
sql: "SELECT category, count(*) AS total FROM input GROUP BY category"

A global step is registered twice:

  1. Locally as daily_summary, so later steps in the job use SELECT * FROM daily_summary as usual.

  2. Globally as global_temp.daily_summary, for other sessions:

    SELECT * FROM global_temp.daily_summary WHERE total > 100;

Caching (persist)​

Cache a step when several later steps or outputs read it, so Spark computes it once:

steps:
- name: joined_data
persist: memory_and_disk
sql: |
SELECT o.*, c.name
FROM orders o
JOIN customers c ON o.customer_id = c.customer_id
persist valueSpark storage levelWhat it does
memory_and_diskMEMORY_AND_DISKKeeps it in memory and spills to disk under memory pressure (recommended)
memoryMEMORY_ONLYKeeps it in memory only
diskDISK_ONLYKeeps it on executor disk

persist also takes any Spark storage level name, such as MEMORY_AND_DISK_2. Cached steps are released (unpersisted) at the end of each batch, so memory does not build up across batches.

Cutting lineage (checkpoint)​

Deeply nested or iterative steps can build very large Spark query plans. checkpoint: true saves the step's result to the checkpoint folder and cuts its lineage, so later steps start from the saved data.

runtime:
spark:
conf:
spark.checkpoint.dir: s3://example-lakehouse/checkpoints/

steps:
- name: truncated_step
checkpoint: true
sql: "SELECT * FROM complex_step"
  • A step with checkpoint: true needs spark.checkpoint.dir in runtime.spark.conf. Use a folder the whole cluster can reach.
  • When the step also has persist, it is cached first, then checkpointed from the cache.

Logging a step's rows (log)​

log works as on a DuckDBJob (see Logging a step's rows): one step NAME: MESSAGE line per row, up to max_rows (default 20, at most 1000), on the streamwright.steps logger, then step NAME: ... and N more row(s). Nothing is logged for a step with no rows.

steps:
- name: skipped_actions
persist: memory_and_disk # the log's Spark actions then read the cache
sql: "SELECT name, resource_name FROM input JOIN existing_actions USING (name)"
log: {message: "conversion action {name} already exists ({resource_name}): not created", max_rows: 50}
Logging costs Spark actions

Logging runs one Spark action to collect max_rows + 1 rows, and a second one (count()) only when there are more than max_rows. Each recomputes the step unless it is persisted, so add persist to a costly step you log.

See also​