Connectors
A connector is an installable package that knows one vendor or system: how to sign in, which calls it can make,
and how to page and retry them. A source names its connector in auth.provider; a
job names it in a connection's provider. You install only the connectors you need, one command each.
sw connectors -i NAME installs a connector. Every connector is then used the same way: a connection (where), an
operation (what) and an optional policy (how).
Install a connector
Connectors install from the StreamWright hub:
sw connectors ls # the catalog, and what is installed
sw connectors find ads # search the catalog
sw connectors -i postgres # install one connector
sw connectors -i google_ads meta_ads --yes # install several; --yes skips the confirmation prompt
sw connectors # the installed connectors: version, libraries, loggers
sw connectors postgres # one installed connector, in detail
Each connector's package is streamwright-<name> (for example streamwright-google-ads). See
streamwright connectors for every option.
All connectors
Engines says where a connector runs: streamwright run (source), DuckDB jobs and
Spark jobs. "Call step" means the job runs the connector's source side to fetch records; "apply
output" means a job output sends rows to the vendor's API (reverse ETL).
| Connector | Reads / writes | Engines |
|---|---|---|
restapi | Any REST API, declared in YAML. Writes: apply outputs | source; DuckDB and Spark (call step, apply output) |
google_ads | Reports and resources through GAQL. Writes: apply outputs | source; DuckDB and Spark (call step, apply output) |
microsoft_ads | Entities and reports (SOAP API). Writes: apply outputs | source; DuckDB and Spark (call step, apply output) |
meta_ads | Marketing API objects and insights. Writes: apply outputs | source; DuckDB and Spark (call step, apply output) |
linkedin_ads | Ad accounts, campaigns, creatives, analytics. Writes: apply outputs | source; DuckDB and Spark (call step, apply output) |
amazon_ads | Profiles and Sponsored Products entities. Writes: apply outputs | source; DuckDB and Spark (call step, apply output) |
apple_ads | Campaigns, ad groups, keywords, reports. Writes: apply outputs | source; DuckDB and Spark (call step, apply output) |
openai_ads | Accounts, campaigns, ad groups, ads, performance. Writes: apply outputs | source; DuckDB and Spark (call step, apply output) |
files | Local CSV, TSV, JSON, JSONL, Parquet (runs offline). Writes: files | source, DuckDB, Spark |
s3 | Objects in Amazon S3 or an S3-compatible store. Writes: objects | source, DuckDB, Spark |
gcs | Objects in Google Cloud Storage. Writes: objects | source, DuckDB, Spark |
azure_blob | Blobs in Azure Blob Storage. Writes: blobs (Spark only) | source, DuckDB, Spark |
postgres | Read-only SELECT queries and table reads. Writes: tables | source, DuckDB, Spark |
mysql | Read-only SELECT queries and table reads. Writes: tables | source, DuckDB, Spark |
mongodb | find and aggregate reads. Writes: collections (Spark only) | source, Spark; DuckDB (call step) |
deltalake | Delta Lake tables, local or in object storage. Writes: append, overwrite, merge | source, DuckDB, Spark |
google_sheets | Cell values as rows. Read-only | source, Spark; DuckDB (call step) |
kafka | Peek at or consume topics. Writes: publish events | source, DuckDB, Spark |
sqs | Peek at or consume queues. Writes: publish messages | source, DuckDB, Spark |
How a source uses a connector
Every connector is used the same way. Only the auth keys and the operation differ.
kind: source
name: example
auth:
provider: postgres # the connector
# sign-in keys: settings from config, credentials from secrets
requests:
- name: raw_example
operation: # what to call: a method and its arguments, or an HTTP request
# ...
transform:
mode: page
steps:
- name: example
select: SELECT ... FROM raw_example
export:
example:
step: example
primary_key: [id]
- Fetch:
requestsfetch records. Each record is one JSON value,record, in a table named after the request. - Shape and write:
transformshapes them with DuckDB SQL, andexportwrites the result. See Streams.
Same on every connector
The connector pages do not repeat these:
- Policy.
policy: {retry, rate_limit, page_size, timeout, on_error}on a source, stream or job step. - Vendor arguments. An argument the connector does not declare goes to the vendor API or library as is. You can use any field the vendor supports without waiting for a connector update.
- Secrets. Credentials must be
{{ secrets.NAME }}references. Their values show as***in every log line and error.
Restrict which connectors a run may use with --allow-connector NAME (repeatable). By default any installed
connector can be used.