Skip to main content

Apache Kafka

The kafka connector reads messages from, and publishes events to, Apache Kafka topics. It works with Kafka, Confluent Cloud, Redpanda and Amazon MSK. Most of the time you use it from DuckDB jobs and Spark jobs: a consume step reads a batch, and a publish output sends events.

In short

Offsets are committed only after the batch's outputs succeed. If a job fails, nothing is committed and the messages are read again.

Install​

sw connectors -i kafka

For Amazon MSK with IAM, install the msk-iam extra: pip install "streamwright-kafka[msk-iam]".

Connect​

connections:
kafka_bus:
provider: kafka
bootstrap_servers: "broker:9092"
security_protocol: SASL_SSL
sasl_mechanism: SCRAM-SHA-512
username: "{{ secrets.kafka_user }}"
password: "{{ secrets.kafka_password }}"
KeyWhat it isRequired / default
bootstrap_serversComma-separated broker addressesRequired
security_protocolPLAINTEXT, SSL, SASL_PLAINTEXT or SASL_SSLPLAINTEXT
sasl_mechanismPLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or AWS_MSK_IAMOptional
usernameSASL user name. SecretOptional
passwordSASL password. SecretOptional
regionThe AWS region, required for AWS_MSK_IAMOptional
ca_locationA CA certificate file for TLSOptional
  • Amazon MSK with IAM: set sasl_mechanism: AWS_MSK_IAM, a SASL security_protocol and region.
  • Aliases: security_protocol can also be written security.protocol, sasl_mechanism as sasl.mechanism, and ca_location as ssl.ca.location. Setting a key and its alias together is an error.
  • Do not put credentials in options or in Spark configuration.

Read data​

Consume a topic (jobs)​

A consume step reads one batch of messages into the table input. A job has at most one.

steps:
- name: raw_events
connection: kafka_bus
operation:
service: message
method: consume
arguments:
destination: events # the topic
group_id: events-consumer
sql: |
SELECT message_id, key, body, headers['source'] AS source
FROM input

Each run reads one batch, runs your steps and outputs, then commits the offsets.

ArgumentWhat it doesDefault
destinationThe topic. Required
group_idThe consumer group. Required
fromWhere a group with no committed offsets starts: earliest, latest, or error (fail)earliest
max_messagesMessages per batchThe run's --max-messages
max_waitHow long one poll waits, for example 10s
optionsExtra Kafka client settings, for example {session.timeout.ms: 30000}

The input table:

ColumnWhat it holds
message_idtopic:partition:offset
keyThe message key, if set
bodyThe payload as UTF-8 text. Parse JSON with json_extract_string or ->> in DuckDB, or from_json in Spark
headersThe message headers, as a map of text
timestampThe broker timestamp (epoch milliseconds, as text)

To run more than one batch per run, use --drain, --available-now (up to the end offsets at the start of the run) or --follow. See DuckDB batch modes and the Spark CLI.

Peek at a topic (sources)​

method: read reads messages without a consumer group and without committing anything. Use it on streamwright run, or in a DuckDB job as a call step.

streams/events.yaml
requests:
- name: raw_events
operation:
service: message
method: read
arguments:
destination: events
from: earliest # default: latest
max_messages: 500 # default: 100
max_wait: 5s # default: 10s
transform:
mode: page
steps:
- name: events
select: |
SELECT record->>'key' AS event_key,
record->>'value' AS payload
FROM raw_events
export:
events:
step: events

Each record has topic, partition, offset, key, value, headers and timestamp.

Write data​

A publish output sends each row of its sql (or step) as one JSON event. It takes no mode.

outputs:
- name: send_notifications
connection: kafka_bus
operation:
service: message
method: publish
arguments:
destination: processed-events
key: event_id # the column that keys each event
sql: |
SELECT event_id, status, current_timestamp AS processed_at
FROM raw_events
  • Events go through the batch's outbox and are delivered at least once, after the batch's writes commit.
  • Each event has a dedupe-key header, job:batch:output:N (N is its position), so consumers can drop duplicates.
  • Publish also takes options (Kafka client settings).

Good to know​

  • Engines. Consume steps and publish outputs run in DuckDB and Spark jobs. method: read runs on streamwright run, and in a DuckDB job as a call step.
  • Vendor arguments. options passes any Kafka client setting through unchanged.

See also​