Data pipelines,
written in YAML.
Fetch from APIs, files and databases, shape with SQL, and write to files, warehouses or back to ad platforms. StreamWright handles sign-in, paging, retries and state.
- connectors
- 19
- to shape data
- SQL
- ways to run
- 3
# streams/orders.yaml: fetch -> shape -> write
requests: # fetch
- name: raw_orders
operation:
path: /v1/orders
params:
since: "{{ config.start_date }}"
transform: # shape
mode: page
steps:
- name: orders
select: |
SELECT record->>'id' AS order_id,
record->>'customer' AS customer,
(record->>'amount')::DOUBLE AS amount
FROM raw_orders
WHERE record->>'status' = 'active'
export: # write
orders:
step: orders
primary_key: [order_id]
file_name: "{{ source }}/orders/{{ today }}.{{ format }}"
Reads from and writes to
- Google Ads
- Meta Ads
- Microsoft Ads
- LinkedIn Ads
- Amazon Ads
- Apple Ads
- OpenAI Ads
- PostgreSQL
- MySQL
- MongoDB
- Amazon S3
- Cloud Storage
- Azure Blob
- Delta Lake
- Google Sheets
- Kafka
- Amazon SQS
- DuckDB
- Spark
- Parquet
How it works
Fetch, Shape, Write: One Stream
Every pipeline does the same three things. Here is one stream doing them, with the data at each step.
streams/orders.yaml
requests:
- name: raw_orders
operation:
path: /v1/orders
params:
since: "{{ config.start_date }}"
# base URL, sign-in, paging and retries: source.yaml
GET /v1/orders?since=2026-01-01 · 2 pages · 3 records
# page 1 · meta.next_cursor: "c2"{"id": "A-1001", "customer": "acme", "amount": "129.90", "status": "active"}{"id": "A-1002", "customer": "globex", "amount": "42.00", "status": "cancelled"}# page 2 · ?cursor=c2 · no next cursor: done{"id": "A-1003", "customer": "initech", "amount": "310.50", "status": "active"}
Connectors
Many connectors. One YAML grammar.
A connector is a small package that knows one system. Install the ones you need; your YAML looks the same for every one of them.
- Google AdsAds API
- PostgreSQLSQL Database
- Amazon S3Object Store
- Meta AdsAds API
- MySQLSQL Database
- REST APIsHTTP API
- Microsoft AdsAds API
- LinkedIn AdsAds API
- Amazon AdsAds API
- Apple AdsAds API
- OpenAI AdsAds API
- MongoDBDocument DB
- Google Cloud StorageObject Store
- Azure Blob StorageObject Store
- Delta LakeLakehouse
- Local FilesFile Reader
- Google SheetsSpreadsheet
- Apache KafkaEvent Broker
- Amazon SQSMessage Queue
Write to · run on
- Apache Spark
- DuckDB / DuckLake
- Apache Parquet
- Singer Targets
- dlt destinations
- Dagster
- Docker Container
- Kubernetes (K8s)
Need another API? The REST connector reads most JSON APIs from YAML alone.All connectors
Reverse ETL
Send data back to ad platforms
The job that reads your data can also write it back. You write a query; StreamWright sends each row to the platform's API and records what happened to it.
Rows come from SQL
Join your payload, files or warehouse tables; look up IDs on the platform first if you need to.
Sent in batches
Rows go out in calls of the size you choose. Each call is logged with its row count and time.
Every row has an outcome
Rejected rows keep the platform's reason, and you can query the outcomes with SQL.
Applied rows are not resent
A re-run skips rows that were already applied, and calls that may have landed wait for your decision.
Run it from the CLI, on Spark, or behind streamwright serve so other systems can post rows over HTTP. Works with Google, Microsoft, Meta, LinkedIn, Amazon, Apple and OpenAI ads, and REST APIs.
outputs:
- name: upload
connection: ads # provider: google_ads
step: conversions # one row = one conversion
operation:
service: ConversionUploadService
method: upload_click_conversions
arguments: {customer_id: "{{ config.customer_id }}"}
apply:
on_row_error: continue # a bad row fails alone
log: # say why rows failed
sql: |
SELECT gclid, outcome.error_code AS code
FROM upload WHERE outcome.status = 'failed'
message: "gclid {gclid} rejected: {code}"
output upload: call 1/3: 10 row(s), 10 applied, 0 failed, 0.9 soutput upload: call 2/3: 10 row(s), 9 applied, 1 failed, 0.8 soutput upload: call 3/3: 8 row(s), 8 applied, 0 failed, 0.7 soutput upload: gclid Cj0KCQ... rejected: EXPIRED_EVENT
Run it your way
Write it once. Run it anywhere.
Start on your laptop. Move the same file to a service, a cluster or an orchestrator when you need to.
Command line
Run a source or a job from your laptop, a cron entry or CI. Records go to stdout as Singer messages, or to the files and tables you choose.
$ sw run files_demo --set data_root=files_demo/data --output parquet:out/INFO stream 'orders': 5 record(s) written (orders: 5), 2 request(s), 0 retriesINFO run finished in 0.5 s: 3 stream(s), 11 record(s) written
REST service
sw serve · sw worker
Spark
kind: SparkJob
Orchestration
Dagster · Docker · Kubernetes
AI assistants
sw mcp
Your assistant looks up connectors, validates sources and jobs, and tries SQL before you run anything. It never runs a pipeline or reads a secret.
Why StreamWright
Less code to write, less to run
How StreamWright compares with connector frameworks and with scripts you write yourself.
| StreamWright | Connector frameworks | Custom scripts | |
|---|---|---|---|
| Adding an API | A YAML file, checked against a schema | A connector class per API, or a UI builder | A script per API |
| Transforming data | DuckDB SQL while records move | Usually later, in the warehouse | Hand-written Pandas or dict code |
| Writing back to ad APIs | Built in, with an outcome for every row | A separate reverse ETL tool | Hand-written batching and retries |
| What it takes to run | One Python process | Often a JVM, a Docker stack or a server | Little to run, a lot to maintain |
| Where data goes | Files, DuckDB, Delta, Singer, dlt, ad platforms | The tool's own destinations | Whatever the script was written for |
| AI assistants | An MCP server: sw mcp | Rarely built in | None |
Get started
Your first pipeline, in minutes.
Three steps to run the offline example on your machine. Then point a source at your own API.
- 1Install the CLI and a connector
$ pip install streamwright$ sw connectors -i files
- 2Get the examples
$ git clone https://github.com/karthick-jaganathan/streamwright.git$ cd streamwright
- 3Run the offline example
$ sw run examples/sources/readers/files_demo \ --set data_root=examples/sources/readers/files_demo/data \ --output jsonl:out
- run finished in 0.5 s: 3 stream(s), 11 record(s) written