Google Cloud Storage
The gcs connector reads and writes objects (CSV, TSV, JSON, JSONL, Parquet) in Google Cloud Storage buckets,
using gs:// roots. It takes the same file options as the files connector; this page
covers only what is specific to GCS.
Install
sw connectors -i gcs
Connect
source.yaml
kind: source
name: gcs_demo
spec:
secrets:
gcs_key_id: { type: string }
gcs_secret: { type: string }
auth:
provider: gcs
roots: ["gs://example-bucket/exports/"] # gs:// only
key_id: "{{ secrets.gcs_key_id }}"
secret: "{{ secrets.gcs_secret }}"
| Key | What it is | Required / default |
|---|---|---|
roots | The gs://bucket/prefix/ URLs the connector may read and, in jobs, write. A path without a scheme is relative to the first root | Required |
key_id | HMAC access id. Secret. Leave out key_id and secret for anonymous access to public buckets | Optional |
secret | HMAC secret. Secret | Optional |
Create an HMAC key in the Cloud Storage console under Settings > Interoperability. roots may also be
{{ config.* }} references.
Read data
A request uses service: object, method: read, with the
file-reading options and
format options of the files connector:
streams/events.yaml
requests:
- name: raw_events
operation:
service: object
method: read
arguments:
path: "gs://example-bucket/exports/events/*.jsonl"
format: jsonl
transform:
mode: page
steps:
- name: events
select: |
SELECT
record->>'event_id' AS event_id,
(record->>'timestamp')::TIMESTAMPTZ AS event_time
FROM raw_events
export:
events:
step: events
primary_key: [event_id]
In a DuckDB job, use a read step (read: jsonl, path: events/*.jsonl); in a Spark job, the same operation: as
above. See Files: read data.
Write data
DuckDB and Spark job outputs write objects inside the roots, with the file-writing options:
outputs:
- name: save_events
connection: lake # provider: gcs
step: events
path: daily/events.parquet # relative to the first root
format: parquet
mode: overwrite
Good to know
- Spark credentials. Spark never takes a key from the connection. It uses the platform's credentials, and a Spark
connection that sets one is refused. Spark reaches GCS through its S3-compatible endpoint. Install the Spark extra:
pip install "streamwright-gcs[spark]". - Roots are exact, as for
s3: a root matches only its own bucket and prefix. - No
?in paths. A?,%,#or backslash in a root or path is refused before anything is read. - Vendor arguments. Reader and writer options pass through to the engine, for example
options: { hive_partitioning: true }.
See also
- Files: every file-reading and file-writing option.
- Starter template:
examples/sources/readers/gcs_demo.