Skip to main content

Azure Blob Storage

The azure_blob connector reads blobs (CSV, TSV, JSON, JSONL, Parquet) in Azure Blob Storage containers, Data Lake Storage Gen2 included, and writes them from Spark jobs. It takes the same file options as the files connector; this page covers only what is specific to Azure.

Install​

sw connectors -i azure_blob

Connect​

source.yaml
kind: source
name: azure_blob_demo
spec:
secrets:
azure_connection_string: { type: string }
auth:
provider: azure_blob
roots: ["azure://my-container/exports/"]
connection_string: "{{ secrets.azure_connection_string }}"

Sign in one way:

MethodKeys
Connection stringconnection_string
Account keyaccount_name and account_key
Service principalaccount_name, tenant_id, client_id and client_secret
Platform credentials (managed identity, az login)account_name alone
KeyWhat it isRequired / default
rootsThe azure://container/prefix/ or abfss://container/prefix/ URLs the connector may use. A path without a scheme is relative to the first rootRequired
connection_stringThe storage account's connection string. SecretOptional
account_nameThe storage accountOptional
account_keyThe account key (needs account_name). SecretOptional
tenant_idThe service principal's tenantOptional
client_idThe service principal's application idOptional
client_secretThe service principal's secret (needs account_name). SecretOptional
endpointThe blob endpoint for an emulator such as Azurite: host[:port] or an http(s) URLOptional
use_sslfalse: plain HTTP to the endpointtrue

For example, with an account key:

source.yaml
kind: source
name: azure_blob_demo
spec:
secrets:
azure_account_key: { type: string }
auth:
provider: azure_blob
roots: ["azure://my-container/exports/"]
account_name: "myaccount"
account_key: "{{ secrets.azure_account_key }}"

The account key inside a connection string is redacted (***) in logs and errors too.

Read data​

A request uses service: object, method: read, with the file-reading options and format options of the files connector:

streams/events.yaml
requests:
- name: raw_events
operation:
service: object
method: read
arguments:
path: "azure://my-container/exports/events/*.parquet"
format: parquet

transform:
mode: page
steps:
- name: events
select: |
SELECT
record->>'event_id' AS event_id,
(record->>'timestamp')::TIMESTAMPTZ AS event_time
FROM raw_events

export:
events:
step: events
primary_key: [event_id]

In a DuckDB job, use a read step (read: parquet, path: events/*.parquet); in a Spark job, the same operation: as above. See Files: read data.

Write data​

Only Spark job outputs write blobs, with the file-writing options. DuckDB's Azure extension cannot write to Azure storage, so a DuckDB job output to azure_blob is refused.

outputs:
- name: save_events
connection: lake # provider: azure_blob
operation:
service: object
method: write
arguments:
path: daily/events # relative to the first root
format: parquet
mode: overwrite
sql: SELECT * FROM events

Good to know​

  • Spark credentials. Spark needs account_name and uses the platform's credentials. A Spark connection that sets a key, a connection string or a service principal is refused. Install the Spark extra: pip install "streamwright-azure_blob[spark]".
  • Roots are exact. Reads stay inside the root's container and prefix.
  • No ? in paths. A ?, %, # or backslash in a root or path is refused before anything is read.
  • Vendor arguments. Reader and writer options pass through to the engine, for example options: { recursive: true }.

See also​