Views, caching & lineage
Each SparkJob step becomes a Spark temporary view that later steps and outputs read by its name. Steps are lazy DataFrames: the outputs' writes trigger the work. Four step keys control how a step's view is shared, cached and logged.
| Key | What it does |
|---|---|
view: global | Also share the view with other Spark sessions and tools |
persist | Cache the step's result so it is computed once |
checkpoint: true | Save the result and cut its lineage (long query plans) |
log | Log a line per row of the step |
Global and local views (view)
By default (view: local) a step is a temporary view, visible only inside this job. With view: global it is also
registered in Spark's global_temp database, where other sessions and tools (Thrift Server, notebooks, BI tools)
can query it.
steps:
- name: active_records # local view (default)
sql: "SELECT * FROM input WHERE status = 'ACTIVE'"
- name: daily_summary # local and global view
view: global
sql: "SELECT category, count(*) AS total FROM input GROUP BY category"
A global step is registered twice:
-
Locally as
daily_summary, so later steps in the job useSELECT * FROM daily_summaryas usual. -
Globally as
global_temp.daily_summary, for other sessions:SELECT * FROM global_temp.daily_summary WHERE total > 100;
Caching (persist)
Cache a step when several later steps or outputs read it, so Spark computes it once:
steps:
- name: joined_data
persist: memory_and_disk
sql: |
SELECT o.*, c.name
FROM orders o
JOIN customers c ON o.customer_id = c.customer_id
persist value | Spark storage level | What it does |
|---|---|---|
memory_and_disk | MEMORY_AND_DISK | Keeps it in memory and spills to disk under memory pressure (recommended) |
memory | MEMORY_ONLY | Keeps it in memory only |
disk | DISK_ONLY | Keeps it on executor disk |
persist also takes any Spark storage level name, such as MEMORY_AND_DISK_2. Cached steps are released (unpersisted) at the end of each batch, so memory does not build up across batches.
Cutting lineage (checkpoint)
Deeply nested or iterative steps can build very large Spark query plans. checkpoint: true saves the step's
result to the checkpoint folder and cuts its lineage, so later steps start from the saved data.
runtime:
spark:
conf:
spark.checkpoint.dir: s3://example-lakehouse/checkpoints/
steps:
- name: truncated_step
checkpoint: true
sql: "SELECT * FROM complex_step"
- A step with
checkpoint: trueneedsspark.checkpoint.dirinruntime.spark.conf. Use a folder the whole cluster can reach. - When the step also has
persist, it is cached first, then checkpointed from the cache.
Logging a step's rows (log)
log works as on a DuckDBJob (see Logging a step's rows): one
step NAME: MESSAGE line per row, up to max_rows (default 20, at most 1000), on the streamwright.steps logger,
then step NAME: ... and N more row(s). Nothing is logged for a step with no rows.
steps:
- name: skipped_actions
persist: memory_and_disk # the log's Spark actions then read the cache
sql: "SELECT name, resource_name FROM input JOIN existing_actions USING (name)"
log: {message: "conversion action {name} already exists ({resource_name}): not created", max_rows: 50}
Logging runs one Spark action to collect max_rows + 1 rows, and a second one (count()) only when there are more
than max_rows. Each recomputes the step unless it is persisted, so add persist to a costly step you log.
See also
- Spark jobs overview
sw spark planprints a job's Spark actions and recomputation warnings.