Skip to main content

What is StreamWright?

StreamWright is a command-line tool that turns a few YAML files into a data pipeline. You say what to read (an API, a file store or a database), how to shape it with SQL, and where to write it. StreamWright handles sign-in, paging, retries and state. You write no connector code.

In short

Every pipeline does three things: fetch, shape, write. Every call to the outside world says where (connection), what (operation) and how (policy).

  1. 1. Fetchrequests

    Read records from an API, files or a database. The connector signs in, pages and retries.

  2. 2. Shapetransform

    Reshape records with DuckDB SQL: filter, join, cast types, compute columns.

  3. 3. Writeexport

    Write files, DuckDB or DuckLake, Singer or dlt, or send rows back to an ad platform or REST API.

Two things you build​

You build a...It isRun it with
Source (kind: source)A folder that pulls data from one system: a source.yaml and one file per streamstreamwright run SOURCE
Job (kind: DuckDBJob or kind: SparkJob)One file with connections, steps and outputs; it keeps state between runsstreamwright run JOB.yaml

Start with a source when you only need to pull data. Use a job when one run must combine several reads, SQL steps and writes, or write rows back to a vendor.

A source looks like this:

my_source/
├── source.yaml # inputs, sign-in, shared HTTP or connection settings
└── streams/
├── campaigns.yaml # one file per stream
└── metrics.yaml

A job looks like this:

campaigns_sync.yaml
kind: DuckDBJob
version: 1
name: campaigns_sync
spec:
config:
account_id: { type: string }
secrets:
token: { type: string }
connections:
api: # where
provider: restapi
type: bearer
token: "{{ secrets.token }}"
http:
base_url: https://api.example.com
warehouse:
provider: duckdb
path: warehouse.duckdb
steps:
- name: raw_campaigns # fetch
connection: api
operation: # what
path: /v1/accounts/{{ config.account_id }}/campaigns
method: GET
policy: # how
retry: { codes: [429, 503], max_attempts: 5 }
timeout: 30s
- name: campaigns # shape
sql: |
SELECT id, name FROM raw_campaigns WHERE status = 'ACTIVE'
outputs:
- name: save_campaigns # write
connection: warehouse
step: campaigns
table: campaigns
mode: overwrite

Fetch, shape, write​

Every stream and every job does the same three things, in this order.

StageIn a source streamIn a job
FetchrequestsSteps that read files, call a connector, run a source's stream or consume a broker
Shapetransform steps (DuckDB SQL)sql steps
Writeexport, to the output you pick at run timeoutputs: write a table or files, publish to a broker, or apply rows to a vendor

Where, what, how​

Every call to the outside world is built from the same parts.

PartAnswersWhat you write
connectionWhereA named connection in a job; auth.provider and its keys in a source
operationWhat to invokeSDK: service, method, arguments. HTTP: path, method, params, ...
policyHow to behaveretry, rate_limit, timeout, on_error, ...
applyHow rows are sent when writing back (reverse ETL)An apply output of a job. See Apply outputs

Run it your way​

You write the YAML once. How you run it is a separate choice.

WayCommandRead
CLIstreamwright run (short alias: sw run)Quickstart, CLI
Servicestreamwright serve takes runs over HTTP; streamwright worker runs themServing jobs
Sparksw spark run for kind: SparkJob, locally or on a clusterSpark jobs
OrchestrationDagster runs streamwright run per user and networkOrchestration

An AI assistant can also look up connectors and check your YAML through streamwright mcp. See MCP server.

Why use it​

  • Configuration, not code. A new source is a few YAML files. You do not write or deploy connector code.
  • Plain SQL to shape data. Filter, join, cast and compute columns with SQL.
  • One pipeline, many networks. A shared vocabulary maps one pipeline onto ad networks that name their streams differently.
  • Runs anywhere. The same source runs on your laptop, in Docker or on Kubernetes.
  • Open outputs. Singer messages, JSONL, CSV, TSV or Parquet files, DuckDB or DuckLake, or any dlt destination.

Where to go next​

You want to...Start with
Run something in five minutes, offlineQuickstart
Pull data from an API, files or a databaseSources, Streams
Combine reads, SQL and writes in one run, with exactly-once stateDuckDB jobs, Spark jobs
Write rows back to an ad platform or REST APIApply outputs
Copy a working exampleExamples

Key terms​

TermMeaning
SourceA folder (source.yaml and streams/) that pulls data from one system
StreamOne dataset of a source, in one streams/*.yaml file: requests, transform, export
JobOne manifest (kind: DuckDBJob or kind: SparkJob) with connections, steps and outputs, and state between runs
StepOne unit of a job: a read, a connector call, a source's stream, or SQL over earlier steps
OutputWhere results are written: --output for a source run, the outputs list of a job
ConnectionWhere a call goes: a named system and how to sign in to it
OperationWhat a call invokes: an SDK call or an HTTP request
PolicyHow a call behaves: retries, rate limits, timeouts, errors
ConnectorThe installable package for one vendor or system (sw connectors -i NAME)
Output targetThe value of --output for a source run, such as jsonl:DIR or duckdb:PATH