Amazon SQS
The sqs connector reads messages from, and publishes messages to, Amazon SQS queues, standard and FIFO. Most of
the time you use it from DuckDB jobs and Spark jobs: a consume step reads a batch,
and a publish output sends messages. It works like the Kafka connector.
Messages are deleted from the queue only after the batch's outputs succeed. If a job fails, they return to the queue when their visibility timeout ends.
Install
sw connectors -i sqs
Connect
connections:
sqs_bus:
provider: sqs
region: "us-east-1"
| Key | What it is | Required / default |
|---|---|---|
region | The queue's AWS region | Required |
key_id | Access key id. Secret | Optional |
secret | Secret access key. Secret | Optional |
session_token | Session token of temporary credentials. Secret | Optional |
endpoint | An http(s):// endpoint, for example LocalStack's http://localhost:4566 | Optional |
Without key_id and secret, the connector uses the platform's credentials (IAM role, environment). If you set them,
set both:
connections:
sqs_bus:
provider: sqs
region: "us-east-1"
key_id: "{{ secrets.aws_key }}"
secret: "{{ secrets.aws_secret }}"
Read data
Consume a queue (jobs)
A consume step reads one batch of messages into the table input. A job has at most one.
steps:
- name: queue_messages
connection: sqs_bus
operation:
service: message
method: consume
arguments:
destination: "https://sqs.us-east-1.amazonaws.com/123456789012/events-queue" # the queue URL
max_messages: 100 # per batch (received 10 at a time)
max_wait: 20s # long polling
visibility_timeout: 300 # seconds
sql: |
SELECT message_id, body, headers['tenant'] AS tenant
FROM input
| Argument | What it does |
|---|---|
destination | The queue URL. Required |
max_messages | Messages per batch. SQS returns at most 10 per receive call, so the connector calls it repeatedly |
max_wait | Long-poll wait, for example 20s |
visibility_timeout | Seconds a received message stays hidden from other consumers |
options | Additional SQS request parameters |
The input table:
| Column | What it holds |
|---|---|
message_id | The SQS message id |
key | The message group id (FIFO queues) |
body | The payload as text |
headers | The message attributes (string values), as a map of text |
timestamp | SentTimestamp (epoch milliseconds, as text) |
To run more than one batch per run, use --drain or --follow. --available-now runs as --drain, because SQS has
no end offsets. See DuckDB batch modes and the Spark CLI.
Peek at a queue (sources)
method: read receives messages with a zero visibility timeout, so it never hides or deletes them. Use it on
streamwright run, or in a DuckDB job as a call step.
requests:
- name: raw_messages
operation:
service: message
method: read
arguments:
destination: "https://sqs.us-east-1.amazonaws.com/123456789012/events-queue"
max_messages: 50
max_wait: 5s
transform:
mode: page
steps:
- name: messages
select: |
SELECT record->>'message_id' AS message_id,
record->>'body' AS body
FROM raw_messages
export:
messages:
step: messages
primary_key: [message_id]
Each record has message_id, body, attributes and message_attributes.
Write data
A publish output sends each row of its sql (or step) as one JSON message, in batches of ten. It takes no mode.
outputs:
- name: send_notifications
connection: sqs_bus
operation:
service: message
method: publish
arguments:
destination: "https://sqs.us-east-1.amazonaws.com/123456789012/completed-queue"
sql: |
SELECT event_id, status, current_timestamp AS processed_at
FROM queue_messages
- Messages go through the batch's outbox and are delivered at least once, after the batch's writes commit.
- For FIFO queues, set
group_id(the message group). Each message'sMessageDeduplicationIdcomes from the dedupe keyjob:batch:outputand its position, so SQS drops duplicates. - Publish also takes
options.
Good to know
- Engines. Consume steps and publish outputs run in DuckDB and Spark jobs.
method: readruns onstreamwright run, and in a DuckDB job as a call step. - Aliases.
max_messagescan also be writtenMaxNumberOfMessages,max_waitasWaitTimeSeconds, andvisibility_timeoutasVisibilityTimeout. Setting a key and its alias together is an error. - Vendor arguments. boto3 receive and send parameters pass through unchanged, for example
MessageAttributeNames: [All].
See also
- DuckDB jobs: consume steps and publish outputs.
- Apache Kafka: the same model for topics.