Skip to main content

Connectors

A connector is an installable package that knows one vendor or system: how to sign in, which calls it can make, and how to page and retry them. A source names its connector in auth.provider; a job names it in a connection's provider. You install only the connectors you need, one command each.

In short

sw connectors -i NAME installs a connector. Every connector is then used the same way: a connection (where), an operation (what) and an optional policy (how).

Install a connector​

Connectors install from the StreamWright hub:

sw connectors ls # the catalog, and what is installed
sw connectors find ads # search the catalog
sw connectors -i postgres # install one connector
sw connectors -i google_ads meta_ads --yes # install several; --yes skips the confirmation prompt
sw connectors # the installed connectors: version, libraries, loggers
sw connectors postgres # one installed connector, in detail

Each connector's package is streamwright-<name> (for example streamwright-google-ads). See streamwright connectors for every option.

All connectors​

Engines says where a connector runs: streamwright run (source), DuckDB jobs and Spark jobs. "Call step" means the job runs the connector's source side to fetch records; "apply output" means a job output sends rows to the vendor's API (reverse ETL).

ConnectorReads / writesEngines
restapiAny REST API, declared in YAML. Writes: apply outputssource; DuckDB and Spark (call step, apply output)
google_adsReports and resources through GAQL. Writes: apply outputssource; DuckDB and Spark (call step, apply output)
microsoft_adsEntities and reports (SOAP API). Writes: apply outputssource; DuckDB and Spark (call step, apply output)
meta_adsMarketing API objects and insights. Writes: apply outputssource; DuckDB and Spark (call step, apply output)
linkedin_adsAd accounts, campaigns, creatives, analytics. Writes: apply outputssource; DuckDB and Spark (call step, apply output)
amazon_adsProfiles and Sponsored Products entities. Writes: apply outputssource; DuckDB and Spark (call step, apply output)
apple_adsCampaigns, ad groups, keywords, reports. Writes: apply outputssource; DuckDB and Spark (call step, apply output)
openai_adsAccounts, campaigns, ad groups, ads, performance. Writes: apply outputssource; DuckDB and Spark (call step, apply output)
filesLocal CSV, TSV, JSON, JSONL, Parquet (runs offline). Writes: filessource, DuckDB, Spark
s3Objects in Amazon S3 or an S3-compatible store. Writes: objectssource, DuckDB, Spark
gcsObjects in Google Cloud Storage. Writes: objectssource, DuckDB, Spark
azure_blobBlobs in Azure Blob Storage. Writes: blobs (Spark only)source, DuckDB, Spark
postgresRead-only SELECT queries and table reads. Writes: tablessource, DuckDB, Spark
mysqlRead-only SELECT queries and table reads. Writes: tablessource, DuckDB, Spark
mongodbfind and aggregate reads. Writes: collections (Spark only)source, Spark; DuckDB (call step)
deltalakeDelta Lake tables, local or in object storage. Writes: append, overwrite, mergesource, DuckDB, Spark
google_sheetsCell values as rows. Read-onlysource, Spark; DuckDB (call step)
kafkaPeek at or consume topics. Writes: publish eventssource, DuckDB, Spark
sqsPeek at or consume queues. Writes: publish messagessource, DuckDB, Spark

How a source uses a connector​

Every connector is used the same way. Only the auth keys and the operation differ.

source.yaml
kind: source
name: example
auth:
provider: postgres # the connector
# sign-in keys: settings from config, credentials from secrets
streams/example.yaml
requests:
- name: raw_example
operation: # what to call: a method and its arguments, or an HTTP request
# ...
transform:
mode: page
steps:
- name: example
select: SELECT ... FROM raw_example
export:
example:
step: example
primary_key: [id]
  • Fetch: requests fetch records. Each record is one JSON value, record, in a table named after the request.
  • Shape and write: transform shapes them with DuckDB SQL, and export writes the result. See Streams.

Same on every connector​

The connector pages do not repeat these:

  • Policy. policy: {retry, rate_limit, page_size, timeout, on_error} on a source, stream or job step.
  • Vendor arguments. An argument the connector does not declare goes to the vendor API or library as is. You can use any field the vendor supports without waiting for a connector update.
  • Secrets. Credentials must be {{ secrets.NAME }} references. Their values show as *** in every log line and error.
Allow-list connectors

Restrict which connectors a run may use with --allow-connector NAME (repeatable). By default any installed connector can be used.