Quickstart
In about five minutes you install StreamWright, run an example source that reads local files, and look at what it
wrote. You need Python, pip and git. You need no accounts or credentials.
sw validate checks a source without reading data. sw run fetches, shapes and writes it.
1. Install the CLI and the files connector
pip install streamwright
sw connectors -i files
- The first command installs the
streamwrightCLI and its short aliassw, with no connectors. - The second installs the
filesconnector, which reads local CSV, JSON lines and Parquet files.
You should see the connector when you run sw connectors files:
Name: files
Title: Local files
Family: files
...
2. Get the examples
git clone https://github.com/karthick-jaganathan/streamwright.git
cd streamwright
You will run examples/sources/readers/files_demo. It is a source with three streams (customers, orders and
products) that read sample files from its data/ folder.
3. Check the source
sw validate examples/sources/readers/files_demo
validate checks the source, its inputs, stream names and SQL. It reads no data.
You should see:
4 file(s) checked: 0 error(s), 0 warning(s)
4. Run it
sw run examples/sources/readers/files_demo \
--set data_root=examples/sources/readers/files_demo/data \
--output jsonl:out
--set data_root=...gives the source its one required input: the folder to read files from.--output jsonl:outwrites one JSON lines file per export intoout/. Without--output, records go to stdout as Singer messages.
You should see a log that ends with a summary like this:
... run finished in 0.5 s: 3 stream(s), 11 record(s) written (customers: 3, orders: 5, products: 3), ...
Warnings such as no file matches 'customers/...jsonl'; skipping it are expected. The sample data has customer
files for only a few days.
5. Look at the output
ls -R out
head -3 out/customers.*.jsonl
You should see something like this. Dates and the unique part of the names will differ.
out/
├── customers.2026-11-19.095648984539.9_foe3bq.jsonl
├── products.2026-11-19.095649022031.q250j72x.jsonl
├── files_demo/orders/2026-11-19.jsonl
└── state.json
{"city": "London", "customer_id": "cu_1", "customer_name": "Ada", "signed_up": "2026-10-01"}
- Each export gets its own file, named
<export>.<date>.<time>.<unique>.jsonlby default. - The
ordersexport sets its own path withfile_name, so it lands inout/files_demo/orders/. state.jsonholds the bookmarks of incremental streams (here, the last day ofcustomersread). Pass it back with--state out/state.jsonand the next run reads only new days.
You have run a source end to end: fetch (requests), shape (transform) and write (export).
Write your own source
A source is a folder with a source.yaml and one file per stream. This one reads campaigns from a REST API.
kind: source
name: my_source
spec:
config:
account_id: { type: string }
secrets:
token: { type: string }
auth:
provider: restapi
type: bearer
token: "{{ secrets.token }}"
http:
base_url: https://api.example.com
requests:
- name: raw_campaigns
operation:
path: /v1/accounts/{{ config.account_id }}/campaigns
method: GET
transform:
mode: page
steps:
- name: campaigns
select: |
SELECT record->>'id' AS id,
record->>'name' AS name,
record->>'status' AS status
FROM raw_campaigns
export:
campaigns:
step: campaigns
primary_key: [id]
Install the REST connector, then check and run the source. Secrets come from STREAMWRIGHT_SECRET_<NAME>
environment variables or a --secrets FILE.
sw connectors -i restapi
sw validate my_source
STREAMWRIGHT_SECRET_TOKEN='...' sw run my_source --set account_id=1234567890 --output jsonl:out
Other ways to install
| You want to... | Run |
|---|---|
| Browse the connector catalog | sw connectors -ls |
| Search the catalog | sw connectors find ads |
| Install several connectors at once | sw connectors -i google_ads meta_ads restapi |
| Install a connector with pip, from its repo folder | pip install "git+https://github.com/karthick-jaganathan/streamwright.git#subdirectory=connectors/readers/postgres" |
| Install the core and every connector from a clone | make install (in the cloned repo) |
API connectors such as google_ads also install the vendor's SDK.