Skip to main content

Google Cloud Storage

The gcs connector reads and writes objects (CSV, TSV, JSON, JSONL, Parquet) in Google Cloud Storage buckets, using gs:// roots. It takes the same file options as the files connector; this page covers only what is specific to GCS.

Install​

sw connectors -i gcs

Connect​

source.yaml
kind: source
name: gcs_demo
spec:
secrets:
gcs_key_id: { type: string }
gcs_secret: { type: string }
auth:
provider: gcs
roots: ["gs://example-bucket/exports/"] # gs:// only
key_id: "{{ secrets.gcs_key_id }}"
secret: "{{ secrets.gcs_secret }}"
KeyWhat it isRequired / default
rootsThe gs://bucket/prefix/ URLs the connector may read and, in jobs, write. A path without a scheme is relative to the first rootRequired
key_idHMAC access id. Secret. Leave out key_id and secret for anonymous access to public bucketsOptional
secretHMAC secret. SecretOptional

Create an HMAC key in the Cloud Storage console under Settings > Interoperability. roots may also be {{ config.* }} references.

Read data​

A request uses service: object, method: read, with the file-reading options and format options of the files connector:

streams/events.yaml
requests:
- name: raw_events
operation:
service: object
method: read
arguments:
path: "gs://example-bucket/exports/events/*.jsonl"
format: jsonl

transform:
mode: page
steps:
- name: events
select: |
SELECT
record->>'event_id' AS event_id,
(record->>'timestamp')::TIMESTAMPTZ AS event_time
FROM raw_events

export:
events:
step: events
primary_key: [event_id]

In a DuckDB job, use a read step (read: jsonl, path: events/*.jsonl); in a Spark job, the same operation: as above. See Files: read data.

Write data​

DuckDB and Spark job outputs write objects inside the roots, with the file-writing options:

outputs:
- name: save_events
connection: lake # provider: gcs
step: events
path: daily/events.parquet # relative to the first root
format: parquet
mode: overwrite

Good to know​

  • Spark credentials. Spark never takes a key from the connection. It uses the platform's credentials, and a Spark connection that sets one is refused. Spark reaches GCS through its S3-compatible endpoint. Install the Spark extra: pip install "streamwright-gcs[spark]".
  • Roots are exact, as for s3: a root matches only its own bucket and prefix.
  • No ? in paths. A ?, %, # or backslash in a root or path is refused before anything is read.
  • Vendor arguments. Reader and writer options pass through to the engine, for example options: { hive_partitioning: true }.

See also​