Amazon S3
The s3 connector reads and writes objects (CSV, TSV, JSON, JSONL, Parquet) in Amazon S3 or an S3-compatible
store such as MinIO, Ceph or LocalStack. It takes the same file options as the files connector;
this page covers only what is specific to S3.
Install
sw connectors -i s3
Connect
kind: source
name: s3_demo
spec:
secrets:
aws_key_id: { type: string }
aws_secret: { type: string }
auth:
provider: s3
roots: ["s3://example-bucket/exports/"] # the only prefixes the connector may use
key_id: "{{ secrets.aws_key_id }}"
secret: "{{ secrets.aws_secret }}"
region: us-east-1
| Key | What it is | Required / default |
|---|---|---|
roots | The s3://bucket/prefix/ URLs the connector may read and, in jobs, write. A path without a scheme is relative to the first root | Required |
key_id | Access key id. Secret. Leave out key_id and secret for anonymous access to public buckets | Optional |
secret | Secret access key. Secret | Optional |
session_token | Session token of temporary credentials. Secret | Optional |
region | The buckets' region, for example us-east-1 | Optional |
endpoint | An S3-compatible endpoint as host[:port], without a scheme | Optional |
url_style | vhost or path (most S3-compatible stores use path) | Optional |
use_ssl | false: plain HTTP to the endpoint | true |
For MinIO or LocalStack, add for example endpoint: localhost:9000, url_style: path and use_ssl: false.
roots, region, endpoint, url_style and use_ssl may also be {{ config.* }} references.
Read data
A request uses service: object, method: read, with the
file-reading options and
format options of the files connector:
requests:
- name: raw_events
operation:
service: object
method: read
arguments:
path: "s3://example-bucket/exports/events/*.parquet"
format: parquet
transform:
mode: page
steps:
- name: events
select: |
SELECT
record->>'event_id' AS event_id,
(record->>'timestamp')::TIMESTAMPTZ AS event_time
FROM raw_events
export:
events:
step: events
primary_key: [event_id]
In a DuckDB job, use a read step (read: parquet, path: events/*.parquet); in a Spark job, the same
operation: as above. See Files: read data.
Write data
DuckDB and Spark job outputs write objects inside the roots, with the file-writing options:
outputs:
- name: save_events
connection: lake # provider: s3
step: events
path: daily/events.parquet # relative to the first root
format: parquet
mode: overwrite
Good to know
- Spark credentials. Spark never takes credentials from the connection. It uses the platform's credentials
(instance profile, environment), and a Spark connection that sets them is refused. Install the Spark extra:
pip install "streamwright-s3[spark]". - Roots are exact.
s3://example-bucket/does not matchs3://example-bucket-evil/.... - No
?in paths. A?,%,#or backslash in a root, a path or a listed key is refused before anything is read, so a URL cannot carry extra settings. - Credentials stay scoped. Each root gets its own credentials in DuckDB, so two connections never share them.
- Vendor arguments. Reader and writer options pass through to the engine, for example
options: { hive_partitioning: true }.
See also
- Files: every file-reading and file-writing option.
- Starter template:
examples/sources/readers/s3_demo.