What is StreamWright?
StreamWright is a command-line tool that turns a few YAML files into a data pipeline. You say what to read (an API, a file store or a database), how to shape it with SQL, and where to write it. StreamWright handles sign-in, paging, retries and state. You write no connector code.
Every pipeline does three things: fetch, shape, write. Every call to the outside world says where
(connection), what (operation) and how (policy).
- 1. Fetch
requestsRead records from an API, files or a database. The connector signs in, pages and retries.
- 2. Shape
transformReshape records with DuckDB SQL: filter, join, cast types, compute columns.
- 3. Write
exportWrite files, DuckDB or DuckLake, Singer or dlt, or send rows back to an ad platform or REST API.
Two things you build
| You build a... | It is | Run it with |
|---|---|---|
Source (kind: source) | A folder that pulls data from one system: a source.yaml and one file per stream | streamwright run SOURCE |
Job (kind: DuckDBJob or kind: SparkJob) | One file with connections, steps and outputs; it keeps state between runs | streamwright run JOB.yaml |
Start with a source when you only need to pull data. Use a job when one run must combine several reads, SQL steps and writes, or write rows back to a vendor.
A source looks like this:
my_source/
├── source.yaml # inputs, sign-in, shared HTTP or connection settings
└── streams/
├── campaigns.yaml # one file per stream
└── metrics.yaml
A job looks like this:
kind: DuckDBJob
version: 1
name: campaigns_sync
spec:
config:
account_id: { type: string }
secrets:
token: { type: string }
connections:
api: # where
provider: restapi
type: bearer
token: "{{ secrets.token }}"
http:
base_url: https://api.example.com
warehouse:
provider: duckdb
path: warehouse.duckdb
steps:
- name: raw_campaigns # fetch
connection: api
operation: # what
path: /v1/accounts/{{ config.account_id }}/campaigns
method: GET
policy: # how
retry: { codes: [429, 503], max_attempts: 5 }
timeout: 30s
- name: campaigns # shape
sql: |
SELECT id, name FROM raw_campaigns WHERE status = 'ACTIVE'
outputs:
- name: save_campaigns # write
connection: warehouse
step: campaigns
table: campaigns
mode: overwrite
Fetch, shape, write
Every stream and every job does the same three things, in this order.
| Stage | In a source stream | In a job |
|---|---|---|
| Fetch | requests | Steps that read files, call a connector, run a source's stream or consume a broker |
| Shape | transform steps (DuckDB SQL) | sql steps |
| Write | export, to the output you pick at run time | outputs: write a table or files, publish to a broker, or apply rows to a vendor |
Where, what, how
Every call to the outside world is built from the same parts.
| Part | Answers | What you write |
|---|---|---|
connection | Where | A named connection in a job; auth.provider and its keys in a source |
operation | What to invoke | SDK: service, method, arguments. HTTP: path, method, params, ... |
policy | How to behave | retry, rate_limit, timeout, on_error, ... |
apply | How rows are sent when writing back (reverse ETL) | An apply output of a job. See Apply outputs |
Run it your way
You write the YAML once. How you run it is a separate choice.
| Way | Command | Read |
|---|---|---|
| CLI | streamwright run (short alias: sw run) | Quickstart, CLI |
| Service | streamwright serve takes runs over HTTP; streamwright worker runs them | Serving jobs |
| Spark | sw spark run for kind: SparkJob, locally or on a cluster | Spark jobs |
| Orchestration | Dagster runs streamwright run per user and network | Orchestration |
An AI assistant can also look up connectors and check your YAML through streamwright mcp. See
MCP server.
Why use it
- Configuration, not code. A new source is a few YAML files. You do not write or deploy connector code.
- Plain SQL to shape data. Filter, join, cast and compute columns with SQL.
- One pipeline, many networks. A shared vocabulary maps one pipeline onto ad networks that name their streams differently.
- Runs anywhere. The same source runs on your laptop, in Docker or on Kubernetes.
- Open outputs. Singer messages, JSONL, CSV, TSV or Parquet files, DuckDB or DuckLake, or any dlt destination.
Where to go next
| You want to... | Start with |
|---|---|
| Run something in five minutes, offline | Quickstart |
| Pull data from an API, files or a database | Sources, Streams |
| Combine reads, SQL and writes in one run, with exactly-once state | DuckDB jobs, Spark jobs |
| Write rows back to an ad platform or REST API | Apply outputs |
| Copy a working example | Examples |
Quickstart →
Install the CLI and run an example offline in a few minutes.
Concepts →
Sources, streams and outputs.
Examples →
Runnable sources, jobs and a REST service.
Architecture →
How the pieces fit together.
Key terms
| Term | Meaning |
|---|---|
| Source | A folder (source.yaml and streams/) that pulls data from one system |
| Stream | One dataset of a source, in one streams/*.yaml file: requests, transform, export |
| Job | One manifest (kind: DuckDBJob or kind: SparkJob) with connections, steps and outputs, and state between runs |
| Step | One unit of a job: a read, a connector call, a source's stream, or SQL over earlier steps |
| Output | Where results are written: --output for a source run, the outputs list of a job |
| Connection | Where a call goes: a named system and how to sign in to it |
| Operation | What a call invokes: an SDK call or an HTTP request |
| Policy | How a call behaves: retries, rate limits, timeouts, errors |
| Connector | The installable package for one vendor or system (sw connectors -i NAME) |
| Output target | The value of --output for a source run, such as jsonl:DIR or duckdb:PATH |