Skip to main content

Examples

The StreamWright repo has runnable examples under examples/: sources, jobs and a REST service. Copy the one closest to what you need. Start with the offline ones; they need no credentials.

In short

Sources are in examples/sources/. Jobs are in examples/<system>/jobs/<resource>/<action>.yaml. sw validate examples checks all of them.

Run an example offline​

Clone the repo, install the files connector, then run from the repo root:

git clone https://github.com/karthick-jaganathan/streamwright.git && cd streamwright
sw connectors -i files

# a source: reads sample CSV, JSON lines and Parquet files, writes JSONL to out/
sw run examples/sources/readers/files_demo \
--set data_root=examples/sources/readers/files_demo/data \
--output jsonl:out

# a job: check it and see its plan (no secrets needed)
sw validate examples/files/jobs/orders/sync.yaml
sw plan examples/files/jobs/orders/sync.yaml

The Quickstart walks through the source run and its output step by step.

Run an example with credentials​

API and database examples need credentials. Give them with --secrets FILE or STREAMWRIGHT_SECRET_<NAME> environment variables. Give the source's inputs with --set or --config:

sw connectors -i openai_ads
sw run examples/sources/ads/openai_ads \
--secrets secrets.yaml \
--set account_ids=1234567890,1234567891 \
--output jsonl:out
  • Each source's source.yaml lists the inputs (spec.config) and secrets (spec.secrets) it needs. Here, secrets.yaml holds openai_ads_key.
  • List inputs take comma-separated values, as in --set account_ids=1234567890,1234567891.
  • Secrets are redacted in every log line.
  • --output picks where records go, for example jsonl:out, parquet:out or duckdb:analytics.duckdb.

Sources: readers​

ExampleWhat it shows
readers/files_demoRuns offline on local JSONL, CSV and Parquet files: the full fetch, shape and write flow with no credentials
readers/s3_demoReading objects from Amazon S3
readers/gcs_demoReading objects from Google Cloud Storage
readers/azure_blob_demoReading objects from Azure Blob Storage
readers/google_sheets_demoReading tabular data from Google Sheets
readers/postgres_demoRead-only SELECT queries on PostgreSQL
readers/mysql_demoRead-only SELECT queries on MySQL
readers/mongodb_demoReading collections and aggregations from MongoDB
readers/deltalake_demoQuerying Delta Lake tables through DuckDB

Sources: Ads APIs​

Every ads source maps to the shared vocabulary, so one cross-network pipeline covers all of them.

ExampleTransportWhat it shows
ads/google_adsSDKGoogle Ads GAQL queries, search_stream batches, and performance reporting
ads/microsoft_adsSDKMicrosoft Advertising SOAP services and CampaignPerformance reports
ads/meta_adsSDKMeta Marketing API campaigns, ad sets, and daily campaign insights
ads/linkedin_adsHTTPLinkedIn Marketing Rest.li endpoints, dotted query parameter encoding, and offset pagination
ads/amazon_adsHTTPAmazon Sponsored Products and profiles, with an OAuth2 refresh token and regional endpoints
ads/apple_adsHTTPApple Ads campaigns and reporting, with pagination in the POST body and an account header
ads/openai_adsHTTPOpenAI Ads bearer auth, cursor pagination (last_id → after), and delivery insights

Jobs​

Job manifests (kind: DuckDBJob) are organized by system: examples/<system>/jobs/<resource>/<action>.yaml.

  • The job's name is <system>_<resource>_<action>.
  • payloads/<resource>/ holds sample input rows for --input.
  • config/ holds templates of the values you supply.
  • An overlay sits next to its job as <action>.<environment>.yaml, such as files/jobs/orders/sync.local.yaml.
Google Ads jobs that write

The Google Ads jobs take their accounts from --set customer_id=... --set login_customer_id=...; their gclids, budgets and ids are placeholders. Run the jobs that write with --set validate_only=true first: Google checks the request and nothing changes.

Read, shape and load​

ExampleWhat it shows
files/jobs/orders/sync.yamlLocal file reads, SQL steps, a table merge and a split_by export, offline
deltalake/jobs/orders/merge.yamlA Delta Lake merge with schema evolution
kafka/jobs/order_files/ingest.yamlA consume step, dynamic file paths, row indexes, a Delta merge
google_ads/jobs/campaigns/sync.yaml, ad_groups/sync.yamlConnector call steps; partitions (one call per campaign)
openai_ads/jobs/ad_groups/sync.yamlAn HTTP connector's call steps
openai_ads/jobs/ad_group_metrics/sync.yamlA step that runs a source's stream, SQL metrics, and a merge into a DuckDB warehouse

Write back to a vendor (apply outputs)​

These jobs use apply outputs to send rows to an ad platform (reverse ETL).

ExampleWhat it shows
google_ads/jobs/conversion_actions/create.yamlCreates items; existing items skipped and logged (a step's log:); results logged (the output's log:)
google_ads/jobs/conversions/upload.yamlRows from --input; one search for the named actions (partitions + GAQL IN); unknown ones skipped; chunked calls; rejected rows logged
google_ads/jobs/conversions/upload_folder.yamlA folder, file by file, 5 at once (runtime.fan_out over a files lookup); each file uploaded once (runtime.state.job_key)
google_ads/jobs/campaigns/create.yaml, update.yamlmutate_campaigns: create and update, with partial failures
meta_ads/jobs/conversions/send_events.yamlThe Conversions API: events built and hashed in SQL (sha256), a call per chunk, rejected rows logged
meta_ads/jobs/campaigns/create.yaml, custom_conversions/create.yamlGraph API create_* edges: a request per row; a rejected row fails alone
microsoft_ads/jobs/campaigns/create.yaml, conversion_goals/create.yamlSOAP Add* operations: existing names skipped, dotted columns for nested fields, new ids logged
microsoft_ads/jobs/conversions/upload.yamlApplyOfflineConversions from --input rows: unknown goals skipped, chunked calls, rejected rows logged
linkedin_ads/jobs/conversions/send_events.yamlA REST API connector's apply output: a chunk of rows per request, emails hashed in SQL

A REST service​

examples/serve/serve.yaml serves the Google Ads jobs over HTTP with streamwright serve:

  • Callers use API keys; only the keys' SHA-256 digests are in the environment.
  • The manager account comes from the key's claims, not from the request.
  • Swagger UI is at /docs and Prometheus metrics at /metrics.
  • streamwright worker runs the queued runs.

streamwright validate examples/serve/serve.yaml checks it without the service's environment.