Apache Kafka
The kafka connector reads messages from, and publishes events to, Apache Kafka topics. It works with Kafka,
Confluent Cloud, Redpanda and Amazon MSK. Most of the time you use it from DuckDB jobs and
Spark jobs: a consume step reads a batch, and a publish output sends events.
Offsets are committed only after the batch's outputs succeed. If a job fails, nothing is committed and the messages are read again.
Install
sw connectors -i kafka
For Amazon MSK with IAM, install the msk-iam extra: pip install "streamwright-kafka[msk-iam]".
Connect
connections:
kafka_bus:
provider: kafka
bootstrap_servers: "broker:9092"
security_protocol: SASL_SSL
sasl_mechanism: SCRAM-SHA-512
username: "{{ secrets.kafka_user }}"
password: "{{ secrets.kafka_password }}"
| Key | What it is | Required / default |
|---|---|---|
bootstrap_servers | Comma-separated broker addresses | Required |
security_protocol | PLAINTEXT, SSL, SASL_PLAINTEXT or SASL_SSL | PLAINTEXT |
sasl_mechanism | PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or AWS_MSK_IAM | Optional |
username | SASL user name. Secret | Optional |
password | SASL password. Secret | Optional |
region | The AWS region, required for AWS_MSK_IAM | Optional |
ca_location | A CA certificate file for TLS | Optional |
- Amazon MSK with IAM: set
sasl_mechanism: AWS_MSK_IAM, a SASLsecurity_protocolandregion. - Aliases:
security_protocolcan also be writtensecurity.protocol,sasl_mechanismassasl.mechanism, andca_locationasssl.ca.location. Setting a key and its alias together is an error. - Do not put credentials in
optionsor in Spark configuration.
Read data
Consume a topic (jobs)
A consume step reads one batch of messages into the table input. A job has at most one.
steps:
- name: raw_events
connection: kafka_bus
operation:
service: message
method: consume
arguments:
destination: events # the topic
group_id: events-consumer
sql: |
SELECT message_id, key, body, headers['source'] AS source
FROM input
Each run reads one batch, runs your steps and outputs, then commits the offsets.
| Argument | What it does | Default |
|---|---|---|
destination | The topic. Required | |
group_id | The consumer group. Required | |
from | Where a group with no committed offsets starts: earliest, latest, or error (fail) | earliest |
max_messages | Messages per batch | The run's --max-messages |
max_wait | How long one poll waits, for example 10s | |
options | Extra Kafka client settings, for example {session.timeout.ms: 30000} |
The input table:
| Column | What it holds |
|---|---|
message_id | topic:partition:offset |
key | The message key, if set |
body | The payload as UTF-8 text. Parse JSON with json_extract_string or ->> in DuckDB, or from_json in Spark |
headers | The message headers, as a map of text |
timestamp | The broker timestamp (epoch milliseconds, as text) |
To run more than one batch per run, use --drain, --available-now (up to the end offsets at the start of the run)
or --follow. See DuckDB batch modes and the Spark CLI.
Peek at a topic (sources)
method: read reads messages without a consumer group and without committing anything. Use it on
streamwright run, or in a DuckDB job as a call step.
requests:
- name: raw_events
operation:
service: message
method: read
arguments:
destination: events
from: earliest # default: latest
max_messages: 500 # default: 100
max_wait: 5s # default: 10s
transform:
mode: page
steps:
- name: events
select: |
SELECT record->>'key' AS event_key,
record->>'value' AS payload
FROM raw_events
export:
events:
step: events
Each record has topic, partition, offset, key, value, headers and timestamp.
Write data
A publish output sends each row of its sql (or step) as one JSON event. It takes no mode.
outputs:
- name: send_notifications
connection: kafka_bus
operation:
service: message
method: publish
arguments:
destination: processed-events
key: event_id # the column that keys each event
sql: |
SELECT event_id, status, current_timestamp AS processed_at
FROM raw_events
- Events go through the batch's outbox and are delivered at least once, after the batch's writes commit.
- Each event has a
dedupe-keyheader,job:batch:output:N(N is its position), so consumers can drop duplicates. - Publish also takes
options(Kafka client settings).
Good to know
- Engines. Consume steps and publish outputs run in DuckDB and Spark jobs.
method: readruns onstreamwright run, and in a DuckDB job as a call step. - Vendor arguments.
optionspasses any Kafka client setting through unchanged.
See also
- DuckDB jobs: consume steps and publish outputs.
- Amazon SQS: the same model for queues.