Skip to main content

Architecture

StreamWright is a small core with parts you can swap. This page shows how the parts connect, so you can predict where a change belongs.

In short

A source or job says what to fetch, shape and write. A connector knows how to talk to one system. An output decides where records go. Orchestration decides when runs happen.

The parts​

PartWhat it decidesWhere it lives
SourceWhat to read and how to shape itA folder: source.yaml and streams/
JobSeveral reads, SQL steps and writes in one run, with state between runsOne manifest: kind: DuckDBJob or kind: SparkJob
ConnectorHow to talk to a system: sign-in, paging, retriesAn installed package (sw connectors -i NAME)
OutputWhere records goA run-time choice (--output), or a job's outputs
OrchestrationWhen and where runs happen, per user and networkA separate Dagster project

Each part is independent. You can add a connector, switch an output or change the schedule without touching the others.

The YAML never holds connector code. It names a connection (where), an operation (what to invoke) and a policy (how to behave). The connector turns these into real calls.

How the parts connect​

The diagram shows a source run. A job runs on its own engine (DuckDB or Spark) in the same way: its steps call connectors or run a source's stream, and its outputs write tables, files, brokers or vendor APIs.

A source run, step by step​

  1. Load. The CLI loads the source and the run's inputs (--set, --config, --secrets). It checks them before any request is made.
  2. Fetch. Each stream's requests fetch records through the connector.
  3. Shape. The transform steps shape them with DuckDB SQL.
  4. Write. The export datasets go to the output, with the run's state.

A job run, step by step​

  1. Load. The engine loads the manifest and its inputs (--set, --config, --secrets, --input).
  2. Resume. Batches a previous run left unfinished resume first; by default a failed one is given up (abandoned). Then a new batch runs.
  3. Fetch and shape. Steps run in dependency order. streamwright plan JOB.yaml prints that order.
  4. Write. Outputs write tables or files, publish to a broker, or apply rows to a vendor. The batch is recorded in the job's state.

See DuckDB jobs for state and delivery guarantees.

Other ways to run​

  • streamwright serve exposes jobs as a REST API; streamwright worker runs the queued runs. See Serving jobs.
  • sw spark run runs kind: SparkJob manifests on Apache Spark. See Spark jobs.
  • streamwright mcp runs an MCP server so AI assistants can look up connectors and check sources and jobs. See MCP.
  • Orchestration runs streamwright run per user and network, and uses a shared vocabulary so one pipeline fits many networks. See Orchestration.