Examples
The StreamWright repo has runnable examples under
examples/: sources, jobs and a REST
service. Copy the one closest to what you need. Start with the offline ones; they need no credentials.
Sources are in examples/sources/. Jobs are in examples/<system>/jobs/<resource>/<action>.yaml.
sw validate examples checks all of them.
Run an example offline
Clone the repo, install the files connector, then run from the repo root:
git clone https://github.com/karthick-jaganathan/streamwright.git && cd streamwright
sw connectors -i files
# a source: reads sample CSV, JSON lines and Parquet files, writes JSONL to out/
sw run examples/sources/readers/files_demo \
--set data_root=examples/sources/readers/files_demo/data \
--output jsonl:out
# a job: check it and see its plan (no secrets needed)
sw validate examples/files/jobs/orders/sync.yaml
sw plan examples/files/jobs/orders/sync.yaml
The Quickstart walks through the source run and its output step by step.
Run an example with credentials
API and database examples need credentials. Give them with --secrets FILE or STREAMWRIGHT_SECRET_<NAME>
environment variables. Give the source's inputs with --set or --config:
sw connectors -i openai_ads
sw run examples/sources/ads/openai_ads \
--secrets secrets.yaml \
--set account_ids=1234567890,1234567891 \
--output jsonl:out
- Each source's
source.yamllists the inputs (spec.config) and secrets (spec.secrets) it needs. Here,secrets.yamlholdsopenai_ads_key. - List inputs take comma-separated values, as in
--set account_ids=1234567890,1234567891. - Secrets are redacted in every log line.
--outputpicks where records go, for examplejsonl:out,parquet:outorduckdb:analytics.duckdb.
Sources: readers
| Example | What it shows |
|---|---|
readers/files_demo | Runs offline on local JSONL, CSV and Parquet files: the full fetch, shape and write flow with no credentials |
readers/s3_demo | Reading objects from Amazon S3 |
readers/gcs_demo | Reading objects from Google Cloud Storage |
readers/azure_blob_demo | Reading objects from Azure Blob Storage |
readers/google_sheets_demo | Reading tabular data from Google Sheets |
readers/postgres_demo | Read-only SELECT queries on PostgreSQL |
readers/mysql_demo | Read-only SELECT queries on MySQL |
readers/mongodb_demo | Reading collections and aggregations from MongoDB |
readers/deltalake_demo | Querying Delta Lake tables through DuckDB |
Sources: Ads APIs
Every ads source maps to the shared vocabulary, so one cross-network pipeline covers all of them.
| Example | Transport | What it shows |
|---|---|---|
ads/google_ads | SDK | Google Ads GAQL queries, search_stream batches, and performance reporting |
ads/microsoft_ads | SDK | Microsoft Advertising SOAP services and CampaignPerformance reports |
ads/meta_ads | SDK | Meta Marketing API campaigns, ad sets, and daily campaign insights |
ads/linkedin_ads | HTTP | LinkedIn Marketing Rest.li endpoints, dotted query parameter encoding, and offset pagination |
ads/amazon_ads | HTTP | Amazon Sponsored Products and profiles, with an OAuth2 refresh token and regional endpoints |
ads/apple_ads | HTTP | Apple Ads campaigns and reporting, with pagination in the POST body and an account header |
ads/openai_ads | HTTP | OpenAI Ads bearer auth, cursor pagination (last_id → after), and delivery insights |
Jobs
Job manifests (kind: DuckDBJob) are organized by system: examples/<system>/jobs/<resource>/<action>.yaml.
- The job's
nameis<system>_<resource>_<action>. payloads/<resource>/holds sample input rows for--input.config/holds templates of the values you supply.- An overlay sits next to its job as
<action>.<environment>.yaml, such asfiles/jobs/orders/sync.local.yaml.
The Google Ads jobs take their accounts from --set customer_id=... --set login_customer_id=...; their gclids,
budgets and ids are placeholders. Run the jobs that write with --set validate_only=true first: Google checks the
request and nothing changes.
Read, shape and load
| Example | What it shows |
|---|---|
files/jobs/orders/sync.yaml | Local file reads, SQL steps, a table merge and a split_by export, offline |
deltalake/jobs/orders/merge.yaml | A Delta Lake merge with schema evolution |
kafka/jobs/order_files/ingest.yaml | A consume step, dynamic file paths, row indexes, a Delta merge |
google_ads/jobs/campaigns/sync.yaml, ad_groups/sync.yaml | Connector call steps; partitions (one call per campaign) |
openai_ads/jobs/ad_groups/sync.yaml | An HTTP connector's call steps |
openai_ads/jobs/ad_group_metrics/sync.yaml | A step that runs a source's stream, SQL metrics, and a merge into a DuckDB warehouse |
Write back to a vendor (apply outputs)
These jobs use apply outputs to send rows to an ad platform (reverse ETL).
| Example | What it shows |
|---|---|
google_ads/jobs/conversion_actions/create.yaml | Creates items; existing items skipped and logged (a step's log:); results logged (the output's log:) |
google_ads/jobs/conversions/upload.yaml | Rows from --input; one search for the named actions (partitions + GAQL IN); unknown ones skipped; chunked calls; rejected rows logged |
google_ads/jobs/conversions/upload_folder.yaml | A folder, file by file, 5 at once (runtime.fan_out over a files lookup); each file uploaded once (runtime.state.job_key) |
google_ads/jobs/campaigns/create.yaml, update.yaml | mutate_campaigns: create and update, with partial failures |
meta_ads/jobs/conversions/send_events.yaml | The Conversions API: events built and hashed in SQL (sha256), a call per chunk, rejected rows logged |
meta_ads/jobs/campaigns/create.yaml, custom_conversions/create.yaml | Graph API create_* edges: a request per row; a rejected row fails alone |
microsoft_ads/jobs/campaigns/create.yaml, conversion_goals/create.yaml | SOAP Add* operations: existing names skipped, dotted columns for nested fields, new ids logged |
microsoft_ads/jobs/conversions/upload.yaml | ApplyOfflineConversions from --input rows: unknown goals skipped, chunked calls, rejected rows logged |
linkedin_ads/jobs/conversions/send_events.yaml | A REST API connector's apply output: a chunk of rows per request, emails hashed in SQL |
A REST service
examples/serve/serve.yaml serves
the Google Ads jobs over HTTP with streamwright serve:
- Callers use API keys; only the keys' SHA-256 digests are in the environment.
- The manager account comes from the key's claims, not from the request.
- Swagger UI is at
/docsand Prometheus metrics at/metrics. streamwright workerruns the queued runs.
streamwright validate examples/serve/serve.yaml checks it without the service's environment.