Launchers, runtime & catalogs
Every SparkJob says where it runs in runtime.platform. sw spark run then runs it in the current process, or
submits it to your cluster for you. This page covers the platforms, Spark settings, environment variables, secrets on
a cluster, and registering Delta Lake tables in a catalog.
Develop with type: local. To go to a cluster, change runtime.platform; the steps and outputs stay the same.
Platforms (runtime.platform)
runtime:
platform:
type: emr_ec2 # local | standalone | emr_ec2
region: us-east-1
staging: s3://example-artifacts/streamwright/
cluster:
id: j-2ABCDEF123456 # or stack_name: my-emr-stack
type | Keys | What it does |
|---|---|---|
local | none | Runs the driver in the current process (Spark local[*]). Use it for development and tests |
standalone | master: a spark://host:7077 URL | Submits the job with spark-submit to a Spark standalone cluster |
emr_ec2 | region, staging (an s3:// location), and cluster with id, stack_name or stack_arn | Stages the job's files in staging and adds a spark-submit step to an existing EMR on EC2 cluster |
Good to know about EMR:
- Install the AWS extra:
pip install "streamwright-spark[emr]". - With
cluster.stack_name(orstack_arn), the launcher looks up the cluster id in CloudFormation, so submissions keep working when the stack recreates the cluster. If the stack has more than one cluster,logical_resource_idpicks one. - The job must target an existing cluster. Creating a cluster per job is not supported.
Use sw spark run JOB --dry-run to see the spark-submit command or EMR step a run would launch, and --detach
to submit without waiting. Pin versions for a cluster with sw spark lock.
On a remote platform, a SQLite state store needs an absolute path (runtime.state.sqlite: /var/lib/...).
Kubernetes is not a Spark platform. Sources (streamwright run) run on Kubernetes through
orchestration, with one Kubernetes Job per pipeline node.
Spark settings (runtime.spark)
runtime:
spark:
app_name: daily_orders
conf:
spark.sql.shuffle.partitions: "64"
spark.checkpoint.dir: s3://example-lakehouse/checkpoints/
| Key | What it does |
|---|---|
app_name | The Spark application name |
conf | Spark settings. Keys need the spark. prefix. Keys the launcher sets itself (spark.master, spark.submit.deployMode, spark.jars*, ...) are refused |
sw spark run --conf KEY=VALUE overrides a conf value for one run.
Environment variables (runtime.env)
Variables to set for the job's driver and executors. Names must be valid variable names, and values must be plain values (not lists or maps).
runtime:
env:
APP_ENV: "production"
LOG_LEVEL: "INFO"
Secrets on a cluster (runtime.secrets)
Locally, you pass secrets with --secrets FILE or STREAMWRIGHT_SECRET_<NAME> variables. On a cluster, the driver
fetches them itself from AWS Secrets Manager, the one supported secrets manager:
runtime:
secrets:
source: aws_secretsmanager
mapping:
db_password: "prod/database/password" # secret name -> Secrets Manager id
Use them in a connection's credential fields with {{ secrets.NAME }} (only there):
connections:
database:
provider: postgres
host: "db.internal"
user: "app_user"
password: "{{ secrets.db_password }}"
- Secret values are never put in the submitted step's arguments, its environment or the lock file; the driver fetches them itself.
- On a remote platform,
runtime.envvalues that referencesecrets.*needruntime.secretswithsource: aws_secretsmanager.
Registering Delta tables in a catalog
A Delta Lake output can register its table in the Spark catalog when the table is not there yet. Give it
table_name, and optionally schema and comment. The catalog is the one your Spark session uses, such as a Hive
metastore, AWS Glue or Databricks Unity Catalog, set up through runtime.spark.conf.
connections:
lake_storage:
provider: s3
roots: ["s3://example-lakehouse/"]
region: us-east-1
lake_delta:
provider: deltalake
roots: ["s3://example-lakehouse/"]
storage: lake_storage # the object-store connection that holds the tables
outputs:
- name: save_sales
connection: lake_delta
operation:
service: table
method: write
arguments:
mode: append
path: "s3://example-lakehouse/delta/sales"
table_name: sales
schema: curated_db # created if missing (CREATE SCHEMA IF NOT EXISTS)
comment: "Cleaned daily sales"
sql: "SELECT * FROM sales_view"
| Argument | What it does |
|---|---|
table_name | Register the table under this name (schema.table also works) |
schema | The catalog schema of table_name. Default: the deltalake connection's schema |
schema_location · schema_comment | The location and comment of a schema the write creates |
comment · table_properties | The table's comment and TBLPROPERTIES |
The engine caches catalog lookups, so it does not repeat the same metadata queries in every batch.