Architecture
StreamWright is a small core with parts you can swap. This page shows how the parts connect, so you can predict where a change belongs.
A source or job says what to fetch, shape and write. A connector knows how to talk to one system. An output decides where records go. Orchestration decides when runs happen.
The parts
| Part | What it decides | Where it lives |
|---|---|---|
| Source | What to read and how to shape it | A folder: source.yaml and streams/ |
| Job | Several reads, SQL steps and writes in one run, with state between runs | One manifest: kind: DuckDBJob or kind: SparkJob |
| Connector | How to talk to a system: sign-in, paging, retries | An installed package (sw connectors -i NAME) |
| Output | Where records go | A run-time choice (--output), or a job's outputs |
| Orchestration | When and where runs happen, per user and network | A separate Dagster project |
Each part is independent. You can add a connector, switch an output or change the schedule without touching the others.
The YAML never holds connector code. It names a connection (where), an operation (what to invoke) and a policy (how to behave). The connector turns these into real calls.
How the parts connect
The diagram shows a source run. A job runs on its own engine (DuckDB or Spark) in the same way: its steps call connectors or run a source's stream, and its outputs write tables, files, brokers or vendor APIs.
A source run, step by step
- Load. The CLI loads the source and the run's inputs (
--set,--config,--secrets). It checks them before any request is made. - Fetch. Each stream's
requestsfetch records through the connector. - Shape. The
transformsteps shape them with DuckDB SQL. - Write. The
exportdatasets go to the output, with the run's state.
A job run, step by step
- Load. The engine loads the manifest and its inputs (
--set,--config,--secrets,--input). - Resume. Batches a previous run left unfinished resume first; by default a failed one is given up (abandoned). Then a new batch runs.
- Fetch and shape. Steps run in dependency order.
streamwright plan JOB.yamlprints that order. - Write. Outputs write tables or files, publish to a broker, or apply rows to a vendor. The batch is recorded in the job's state.
See DuckDB jobs for state and delivery guarantees.
Other ways to run
streamwright serveexposes jobs as a REST API;streamwright workerruns the queued runs. See Serving jobs.sw spark runrunskind: SparkJobmanifests on Apache Spark. See Spark jobs.streamwright mcpruns an MCP server so AI assistants can look up connectors and check sources and jobs. See MCP.- Orchestration runs
streamwright runper user and network, and uses a shared vocabulary so one pipeline fits many networks. See Orchestration.