Skip to main content

Amazon S3

The s3 connector reads and writes objects (CSV, TSV, JSON, JSONL, Parquet) in Amazon S3 or an S3-compatible store such as MinIO, Ceph or LocalStack. It takes the same file options as the files connector; this page covers only what is specific to S3.

Install​

sw connectors -i s3

Connect​

source.yaml
kind: source
name: s3_demo
spec:
secrets:
aws_key_id: { type: string }
aws_secret: { type: string }
auth:
provider: s3
roots: ["s3://example-bucket/exports/"] # the only prefixes the connector may use
key_id: "{{ secrets.aws_key_id }}"
secret: "{{ secrets.aws_secret }}"
region: us-east-1
KeyWhat it isRequired / default
rootsThe s3://bucket/prefix/ URLs the connector may read and, in jobs, write. A path without a scheme is relative to the first rootRequired
key_idAccess key id. Secret. Leave out key_id and secret for anonymous access to public bucketsOptional
secretSecret access key. SecretOptional
session_tokenSession token of temporary credentials. SecretOptional
regionThe buckets' region, for example us-east-1Optional
endpointAn S3-compatible endpoint as host[:port], without a schemeOptional
url_stylevhost or path (most S3-compatible stores use path)Optional
use_sslfalse: plain HTTP to the endpointtrue

For MinIO or LocalStack, add for example endpoint: localhost:9000, url_style: path and use_ssl: false. roots, region, endpoint, url_style and use_ssl may also be {{ config.* }} references.

Read data​

A request uses service: object, method: read, with the file-reading options and format options of the files connector:

streams/events.yaml
requests:
- name: raw_events
operation:
service: object
method: read
arguments:
path: "s3://example-bucket/exports/events/*.parquet"
format: parquet

transform:
mode: page
steps:
- name: events
select: |
SELECT
record->>'event_id' AS event_id,
(record->>'timestamp')::TIMESTAMPTZ AS event_time
FROM raw_events

export:
events:
step: events
primary_key: [event_id]

In a DuckDB job, use a read step (read: parquet, path: events/*.parquet); in a Spark job, the same operation: as above. See Files: read data.

Write data​

DuckDB and Spark job outputs write objects inside the roots, with the file-writing options:

outputs:
- name: save_events
connection: lake # provider: s3
step: events
path: daily/events.parquet # relative to the first root
format: parquet
mode: overwrite

Good to know​

  • Spark credentials. Spark never takes credentials from the connection. It uses the platform's credentials (instance profile, environment), and a Spark connection that sets them is refused. Install the Spark extra: pip install "streamwright-s3[spark]".
  • Roots are exact. s3://example-bucket/ does not match s3://example-bucket-evil/....
  • No ? in paths. A ?, %, # or backslash in a root, a path or a listed key is refused before anything is read, so a URL cannot carry extra settings.
  • Credentials stay scoped. Each root gets its own credentials in DuckDB, so two connections never share them.
  • Vendor arguments. Reader and writer options pass through to the engine, for example options: { hive_partitioning: true }.

See also​