Azure Blob Storage
The azure_blob connector reads blobs (CSV, TSV, JSON, JSONL, Parquet) in Azure Blob Storage containers, Data
Lake Storage Gen2 included, and writes them from Spark jobs. It takes the same file options as the
files connector; this page covers only what is specific to Azure.
Install
sw connectors -i azure_blob
Connect
kind: source
name: azure_blob_demo
spec:
secrets:
azure_connection_string: { type: string }
auth:
provider: azure_blob
roots: ["azure://my-container/exports/"]
connection_string: "{{ secrets.azure_connection_string }}"
Sign in one way:
| Method | Keys |
|---|---|
| Connection string | connection_string |
| Account key | account_name and account_key |
| Service principal | account_name, tenant_id, client_id and client_secret |
Platform credentials (managed identity, az login) | account_name alone |
| Key | What it is | Required / default |
|---|---|---|
roots | The azure://container/prefix/ or abfss://container/prefix/ URLs the connector may use. A path without a scheme is relative to the first root | Required |
connection_string | The storage account's connection string. Secret | Optional |
account_name | The storage account | Optional |
account_key | The account key (needs account_name). Secret | Optional |
tenant_id | The service principal's tenant | Optional |
client_id | The service principal's application id | Optional |
client_secret | The service principal's secret (needs account_name). Secret | Optional |
endpoint | The blob endpoint for an emulator such as Azurite: host[:port] or an http(s) URL | Optional |
use_ssl | false: plain HTTP to the endpoint | true |
For example, with an account key:
kind: source
name: azure_blob_demo
spec:
secrets:
azure_account_key: { type: string }
auth:
provider: azure_blob
roots: ["azure://my-container/exports/"]
account_name: "myaccount"
account_key: "{{ secrets.azure_account_key }}"
The account key inside a connection string is redacted (***) in logs and errors too.
Read data
A request uses service: object, method: read, with the
file-reading options and
format options of the files connector:
requests:
- name: raw_events
operation:
service: object
method: read
arguments:
path: "azure://my-container/exports/events/*.parquet"
format: parquet
transform:
mode: page
steps:
- name: events
select: |
SELECT
record->>'event_id' AS event_id,
(record->>'timestamp')::TIMESTAMPTZ AS event_time
FROM raw_events
export:
events:
step: events
primary_key: [event_id]
In a DuckDB job, use a read step (read: parquet, path: events/*.parquet); in a Spark job, the same
operation: as above. See Files: read data.
Write data
Only Spark job outputs write blobs, with the
file-writing options. DuckDB's Azure extension cannot write to
Azure storage, so a DuckDB job output to azure_blob is refused.
outputs:
- name: save_events
connection: lake # provider: azure_blob
operation:
service: object
method: write
arguments:
path: daily/events # relative to the first root
format: parquet
mode: overwrite
sql: SELECT * FROM events
Good to know
- Spark credentials. Spark needs
account_nameand uses the platform's credentials. A Spark connection that sets a key, a connection string or a service principal is refused. Install the Spark extra:pip install "streamwright-azure_blob[spark]". - Roots are exact. Reads stay inside the root's container and prefix.
- No
?in paths. A?,%,#or backslash in a root or path is refused before anything is read. - Vendor arguments. Reader and writer options pass through to the engine, for example
options: { recursive: true }.
See also
- Files: every file-reading and file-writing option.
- Starter template:
examples/sources/readers/azure_blob_demo.