Skip to main content

Quickstart

In about five minutes you install StreamWright, run an example source that reads local files, and look at what it wrote. You need Python, pip and git. You need no accounts or credentials.

In short

sw validate checks a source without reading data. sw run fetches, shapes and writes it.

1. Install the CLI and the files connector​

pip install streamwright
sw connectors -i files
  • The first command installs the streamwright CLI and its short alias sw, with no connectors.
  • The second installs the files connector, which reads local CSV, JSON lines and Parquet files.

You should see the connector when you run sw connectors files:

Name: files
Title: Local files
Family: files
...

2. Get the examples​

git clone https://github.com/karthick-jaganathan/streamwright.git
cd streamwright

You will run examples/sources/readers/files_demo. It is a source with three streams (customers, orders and products) that read sample files from its data/ folder.

3. Check the source​

sw validate examples/sources/readers/files_demo

validate checks the source, its inputs, stream names and SQL. It reads no data.

You should see:

4 file(s) checked: 0 error(s), 0 warning(s)

4. Run it​

sw run examples/sources/readers/files_demo \
--set data_root=examples/sources/readers/files_demo/data \
--output jsonl:out
  • --set data_root=... gives the source its one required input: the folder to read files from.
  • --output jsonl:out writes one JSON lines file per export into out/. Without --output, records go to stdout as Singer messages.

You should see a log that ends with a summary like this:

... run finished in 0.5 s: 3 stream(s), 11 record(s) written (customers: 3, orders: 5, products: 3), ...

Warnings such as no file matches 'customers/...jsonl'; skipping it are expected. The sample data has customer files for only a few days.

5. Look at the output​

ls -R out
head -3 out/customers.*.jsonl

You should see something like this. Dates and the unique part of the names will differ.

out/
├── customers.2026-11-19.095648984539.9_foe3bq.jsonl
├── products.2026-11-19.095649022031.q250j72x.jsonl
├── files_demo/orders/2026-11-19.jsonl
└── state.json
{"city": "London", "customer_id": "cu_1", "customer_name": "Ada", "signed_up": "2026-10-01"}
  • Each export gets its own file, named <export>.<date>.<time>.<unique>.jsonl by default.
  • The orders export sets its own path with file_name, so it lands in out/files_demo/orders/.
  • state.json holds the bookmarks of incremental streams (here, the last day of customers read). Pass it back with --state out/state.json and the next run reads only new days.

You have run a source end to end: fetch (requests), shape (transform) and write (export).

Write your own source​

A source is a folder with a source.yaml and one file per stream. This one reads campaigns from a REST API.

my_source/source.yaml
kind: source
name: my_source
spec:
config:
account_id: { type: string }
secrets:
token: { type: string }
auth:
provider: restapi
type: bearer
token: "{{ secrets.token }}"
http:
base_url: https://api.example.com
my_source/streams/campaigns.yaml
requests:
- name: raw_campaigns
operation:
path: /v1/accounts/{{ config.account_id }}/campaigns
method: GET
transform:
mode: page
steps:
- name: campaigns
select: |
SELECT record->>'id' AS id,
record->>'name' AS name,
record->>'status' AS status
FROM raw_campaigns
export:
campaigns:
step: campaigns
primary_key: [id]

Install the REST connector, then check and run the source. Secrets come from STREAMWRIGHT_SECRET_<NAME> environment variables or a --secrets FILE.

sw connectors -i restapi
sw validate my_source
STREAMWRIGHT_SECRET_TOKEN='...' sw run my_source --set account_id=1234567890 --output jsonl:out

Other ways to install​

You want to...Run
Browse the connector catalogsw connectors -ls
Search the catalogsw connectors find ads
Install several connectors at oncesw connectors -i google_ads meta_ads restapi
Install a connector with pip, from its repo folderpip install "git+https://github.com/karthick-jaganathan/streamwright.git#subdirectory=connectors/readers/postgres"
Install the core and every connector from a clonemake install (in the cloned repo)

API connectors such as google_ads also install the vendor's SDK.

Next steps​

  • Sources and Streams: the parts of a source.
  • Outputs: files, DuckDB, DuckLake and dlt destinations.
  • DuckDB jobs: several reads, SQL steps and writes in one run.
  • Examples: more sources, jobs and a REST service.
  • CLI: every command and option.