Skip to main content

Launchers, runtime & catalogs

Every SparkJob says where it runs in runtime.platform. sw spark run then runs it in the current process, or submits it to your cluster for you. This page covers the platforms, Spark settings, environment variables, secrets on a cluster, and registering Delta Lake tables in a catalog.

In short

Develop with type: local. To go to a cluster, change runtime.platform; the steps and outputs stay the same.

Platforms (runtime.platform)​

runtime:
platform:
type: emr_ec2 # local | standalone | emr_ec2
region: us-east-1
staging: s3://example-artifacts/streamwright/
cluster:
id: j-2ABCDEF123456 # or stack_name: my-emr-stack
typeKeysWhat it does
localnoneRuns the driver in the current process (Spark local[*]). Use it for development and tests
standalonemaster: a spark://host:7077 URLSubmits the job with spark-submit to a Spark standalone cluster
emr_ec2region, staging (an s3:// location), and cluster with id, stack_name or stack_arnStages the job's files in staging and adds a spark-submit step to an existing EMR on EC2 cluster

Good to know about EMR:

  • Install the AWS extra: pip install "streamwright-spark[emr]".
  • With cluster.stack_name (or stack_arn), the launcher looks up the cluster id in CloudFormation, so submissions keep working when the stack recreates the cluster. If the stack has more than one cluster, logical_resource_id picks one.
  • The job must target an existing cluster. Creating a cluster per job is not supported.

Use sw spark run JOB --dry-run to see the spark-submit command or EMR step a run would launch, and --detach to submit without waiting. Pin versions for a cluster with sw spark lock.

note

On a remote platform, a SQLite state store needs an absolute path (runtime.state.sqlite: /var/lib/...).

Kubernetes is not a Spark platform. Sources (streamwright run) run on Kubernetes through orchestration, with one Kubernetes Job per pipeline node.

Spark settings (runtime.spark)​

runtime:
spark:
app_name: daily_orders
conf:
spark.sql.shuffle.partitions: "64"
spark.checkpoint.dir: s3://example-lakehouse/checkpoints/
KeyWhat it does
app_nameThe Spark application name
confSpark settings. Keys need the spark. prefix. Keys the launcher sets itself (spark.master, spark.submit.deployMode, spark.jars*, ...) are refused

sw spark run --conf KEY=VALUE overrides a conf value for one run.

Environment variables (runtime.env)​

Variables to set for the job's driver and executors. Names must be valid variable names, and values must be plain values (not lists or maps).

runtime:
env:
APP_ENV: "production"
LOG_LEVEL: "INFO"

Secrets on a cluster (runtime.secrets)​

Locally, you pass secrets with --secrets FILE or STREAMWRIGHT_SECRET_<NAME> variables. On a cluster, the driver fetches them itself from AWS Secrets Manager, the one supported secrets manager:

runtime:
secrets:
source: aws_secretsmanager
mapping:
db_password: "prod/database/password" # secret name -> Secrets Manager id

Use them in a connection's credential fields with {{ secrets.NAME }} (only there):

connections:
database:
provider: postgres
host: "db.internal"
user: "app_user"
password: "{{ secrets.db_password }}"
  • Secret values are never put in the submitted step's arguments, its environment or the lock file; the driver fetches them itself.
  • On a remote platform, runtime.env values that reference secrets.* need runtime.secrets with source: aws_secretsmanager.

Registering Delta tables in a catalog​

A Delta Lake output can register its table in the Spark catalog when the table is not there yet. Give it table_name, and optionally schema and comment. The catalog is the one your Spark session uses, such as a Hive metastore, AWS Glue or Databricks Unity Catalog, set up through runtime.spark.conf.

connections:
lake_storage:
provider: s3
roots: ["s3://example-lakehouse/"]
region: us-east-1
lake_delta:
provider: deltalake
roots: ["s3://example-lakehouse/"]
storage: lake_storage # the object-store connection that holds the tables

outputs:
- name: save_sales
connection: lake_delta
operation:
service: table
method: write
arguments:
mode: append
path: "s3://example-lakehouse/delta/sales"
table_name: sales
schema: curated_db # created if missing (CREATE SCHEMA IF NOT EXISTS)
comment: "Cleaned daily sales"
sql: "SELECT * FROM sales_view"
ArgumentWhat it does
table_nameRegister the table under this name (schema.table also works)
schemaThe catalog schema of table_name. Default: the deltalake connection's schema
schema_location · schema_commentThe location and comment of a schema the write creates
comment · table_propertiesThe table's comment and TBLPROPERTIES

The engine caches catalog lookups, so it does not repeat the same metadata queries in every batch.

See also​