Serving jobs over HTTP
streamwright serve turns your DuckDB jobs into a REST API, so other systems can start a run with an HTTP request
instead of a shell command. Use it when an application, a scheduler or a partner needs to trigger jobs, and you want
to control who may run what. It is the job CLI over HTTP: the same jobs, inputs and checks.
servenever runs a job. It checks the caller and the request, then queues the run.workerprocesses take runs from the queue and run them asstreamwright duckdb runwould.serve.yamllists the jobs, who may call each one, and which inputs a caller may set. Nothing else is reachable.
| On the command line | Over HTTP |
|---|---|
streamwright duckdb run JOB | POST /v1/jobs/JOB/runs |
--set NAME=VALUE | "set": {"NAME": VALUE} in the body |
--input FILE | "input": [ ... ] in the body (jobs with spec.input) |
--job-key KEY | the Idempotency-Key header |
--secrets, --allow-lookup-exec, --fan-out-concurrency, ... | never in a request: fixed by the operator in serve.yaml and the server's environment |
A caller is not the operator, so a request carries only set, input, an idempotency key and options.async.
This page has three parts:
- Use it: quick start, calling a job and caller inputs.
- Configure it: the serve.yaml file, the queue database and authentication.
- Run it in production: workers, multiple hosts, systemd, metrics and queue maintenance.
Quick start
Take any DuckDB job, for example orders.yaml from DuckDB jobs, and put this serve.yaml next to
it:
bind: 127.0.0.1:8080
docs: {enabled: true}
jobs:
orders:
manifest: orders.yaml
pip install "streamwright[serve]"
streamwright validate serve.yaml # check serve.yaml, each job entry and each manifest
streamwright serve serve.yaml # terminal 1: the API
streamwright worker serve.yaml # terminal 2: runs the queued runs
curl -H "Content-Type: application/json" -d '{"set": {"min_amount": 100}}' \
http://127.0.0.1:8080/v1/jobs/orders/runs
On a loopback address you can leave out state and auth: the queue is queue.db next to serve.yaml (its tables
are created at start) and there is one anonymous caller. Open http://127.0.0.1:8080/docs to try the API in
Swagger UI.
Calling a job
POST /v1/jobs/client_feed_upload/runs
Authorization: Bearer <API key or JWT>
Idempotency-Key: 2026-11-16-client-1001
Content-Type: application/json
{"set": {"client_id": "1001", "feed_date": "2026-11-16"}, "options": {"async": false}}
- Waiting. The request waits up to
wait_seconds. It answers200with the run if the run ends in time, else202withLocation: /v1/runs/RUN_IDto poll. Withoptions.async: truethe answer is202at once. - Idempotency. An
Idempotency-Key(1-128 printable characters) belongs to the caller and the job. The same key and body return the same run; the same key with another body is422. Jobs with apply outputs require a key. The key is also the run's job key, so a run taken over by another worker resumes its batches. - One run per scope. The job's
runtime.state.scopeis rendered with the request's values. If a run with that scope is active,on_busydecides:rejectanswers409,queuewaits behind it.
Responses
| Code | When |
|---|---|
200 | The run ended within wait_seconds |
202 | The run is queued or running; poll the Location |
400 | Query strings are not accepted |
401 | Missing or invalid API key or JWT |
403 | The caller is not in the job's principals, or lacks a claim an input is bound to |
404 | No such job, or a run that is not the caller's |
409 | A run with the same scope is active (on_busy: reject); for the caller's own run, with its Location |
413 · 415 | The body is too large · not JSON |
422 | The body is invalid (an unknown key, an input it may not set, a value that fails its checks), or the same Idempotency-Key with another body |
428 | A job with apply outputs, called without an Idempotency-Key |
429 | The rate window is full; see Retry-After |
503 | The queue is full (1,000 queued runs); see Retry-After |
Errors look like {"status": "error", "code", "message", "request_id"}. A failed run tells the caller only a code
(run-failed, invalid-input, invalid-job or timeout); the cause goes to the worker's log. Every log line of a
run carries the request's X-Request-ID (req=). Request bodies and row values are never logged.
Endpoints
| Endpoint | Returns |
|---|---|
GET /v1/jobs | The caller's jobs, what each lets it set, and whether it takes rows |
POST /v1/jobs/JOB/runs | Starts a run |
POST /v1/jobs/JOB/validate | {"valid": ..., "issues": [...]} for the request's values |
GET /v1/jobs/JOB/plan | The job's plan |
GET /v1/jobs/JOB/status | The caller's latest runs of the job |
GET /v1/runs/RUN_ID | One run: queued, running, succeeded or failed, with its summary |
GET /healthz · GET /readyz | Liveness (the process answers) · readiness (the state store answers) |
GET /metrics | Prometheus metrics (below) |
GET /openapi.json · GET /docs | OpenAPI 3.1 for the caller's jobs · Swagger UI over it |
Caller inputs
A caller may set every spec.config input, checked against its type and limits (enum, pattern, minimum,
maximum, max_length). A caller can never set:
- an input that serve.yaml binds (
set), - an input with a
lookup:(a caller's value would skip the lookup and could name any account), - anything in
spec.secrets.
serve refuses to start when a settable input is used in a sensitive place without a tight enough type:
| Where the input is used | It must be |
|---|---|
SQL: sql, a read query, a call's arguments.query, a lookup's query | a number, boolean or date, or text whose enum or pattern allows only letters, digits and _ - . : @ + (e.g. ^[A-Za-z0-9_-]{1,64}$) |
What is read or written: a connection, path, file_name, layout, table, destination, a lookup's command or secret_id | a number or date, or text limited to letters, digits and _ - : @ + = , |
Patterns with ., negated classes, \s or case-insensitive matching never count as tight.
Bind the tenant part of a path from a claim, and keep the connection's roots to the tenant's prefix: the roots
stay the hard boundary.
Configuration
This part and the next are for the operator: what the service reads at start, and who may call it.
The serve.yaml file
bind: 0.0.0.0:8080 # a non-loopback address needs an `auth` section
state: # the run queue (see below)
url: postgresql://sw:{{ secrets.state_password }}@pg.example.com:5432/streamwright?schema=sw
auth:
api_keys: # only SHA-256 digests (hex), read from the environment
- id: crm_prod
sha256_env: SW_KEY_CRM_PROD_SHA256
claims: {tenant: t1}
jwt: {issuer: https://idp.example.com, audience: streamwright, jwks_url: https://idp.example.com/jwks}
docs: {enabled: true} # Swagger UI at /docs
metrics: {enabled: true} # Prometheus metrics at /metrics
jobs:
client_feed_upload:
manifest: jobs/client_feed_upload.yaml # relative to serve.yaml
principals: [crm_prod] # an api_keys id, or a JWT `sub`
set: # inputs the caller may NOT set
tenant_id: {from_claim: tenant} # taken from the caller's claim
region: {value: eu} # fixed
payload: {max_bytes: 5MB} # jobs with spec.input
server_options: {fan_out_concurrency: 4, allow_lookup_exec: false}
limits: {concurrency: 4, rate: {requests: 10, per: 1m}, timeout: 2h, wait_seconds: 30}
on_busy: reject
Top-level keys
| Key | What it does | Default |
|---|---|---|
bind | host:port to listen on; a non-loopback address needs auth | 127.0.0.1:8080 |
state | The run queue's database (below) | on loopback: queue.db next to serve.yaml (under systemd: $STATE_DIRECTORY/queue.db); otherwise required |
auth.api_keys | API keys: id, sha256_env (the variable holding the key's SHA-256 in hex), optional claims | none |
auth.jwt | JWT callers: issuer, audience, jwks_url (https:// only) | none |
docs.enabled | Serve Swagger UI at /docs | false |
metrics.enabled | Serve Prometheus metrics at /metrics | false |
jobs | The jobs to serve (at least one) | required |
Job entries (jobs.NAME)
The name is the job's URL segment (/v1/jobs/NAME): 1-128 letters, digits and _ . -.
| Key | What it does | Default |
|---|---|---|
manifest | The job manifest, relative to serve.yaml | required |
principals | Who may call: an api_keys id, or else a JWT sub | required with auth |
set.INPUT | Bind an input so callers cannot set it: {from_claim: CLAIM} or {value: V} | other spec.config inputs are settable |
payload.max_bytes | Largest body for a job with spec.input (512KB, 5MB); can only lower the job's limit | the job's spec.input limit |
server_options | duckdb run options the operator fixes (below) | the duckdb run defaults |
limits.concurrency | Most runs of this job at once, across all workers | no limit |
limits.rate | {requests: N, per: DURATION} per caller, across all hosts | no limit |
limits.timeout | Kill a run that takes longer (30s, 10m, 2h, 1d) | 1h |
limits.wait_seconds | How long a request waits for its run before answering 202 | 30 |
on_busy | When a run with the same scope is active: reject (409) or queue (wait behind it) | reject |
server_options takes allow_lookup_exec, fan_out_concurrency, fan_out_on_failure, failed_job_policy,
max_job_attempts, state_lock_policy, state_lock_timeout, allowed_connectors and apply_state_migrations.
The jobs' secrets come from the workers' environment (STREAMWRIGHT_SECRET_<NAME>), as for the CLI.
Queue database
state takes a url (sqlite:///queue.db, postgresql://..., mysql://...; the schema as ?schema=NAME) or a
connection plus schema, like a job's runtime.state:
state:
connection:
provider: postgres
host: pg.example.com
port: 6432 # e.g. PgBouncer
database: streamwright
user: sw_serve
password: "{{ secrets.state_password }}"
sslmode: require
schema: sw # default: streamwright
{{ secrets.NAME }}readsSTREAMWRIGHT_SECRET_<NAME>; it is the only referencestateaccepts.- DuckDB cannot hold the queue, since only one process can write a DuckDB file. Several hosts share one Postgres or MySQL database.
- PgBouncer in transaction pooling mode works.
Authentication
- API keys. serve.yaml holds only each key's SHA-256 digest, read from the variable named in
sha256_env. Make one withecho -n "$KEY" | sha256sum. To rotate, add the new digest next to the old one, then remove the old one. - JWT. The token must be signed with an asymmetric algorithm (RS*, PS*, ES*, EdDSA) by a key from the
issuer's JWKS.
iss,aud,expandsubare required. The caller is thesub, and its claims feedfrom_claim. - Callers are kept apart. Each caller's runs, Idempotency-Keys and rate limits are its own. A
subthat equals an API key's id cannot be listed: rename the key.
serve does not do TLS. Put a reverse proxy or load balancer in front of it. A service without auth only binds a
loopback address.
Running in production
How the service runs: its workers, its hosts, its process manager, its metrics and its upkeep.
Workers
streamwright worker serve.yaml --processes 8 --log-format json
A worker claims the oldest run it may take (one per scope, limits.concurrency per job), runs it in a child
process, and kills it after limits.timeout. --processes N (default 1) sets how many runs it works on at once. A
host needs roughly --processes times one run's memory. To serve more runs, raise --processes or add hosts.
- Stopping.
SIGTERMorCtrl-Clets runs under way finish;SIGKILLstops everything at once. - When a worker dies, its claim expires after a minute and another worker resumes the run from the job's state. A run never runs twice at once.
- Logs. Each ended run logs its timings:
run RUN_ID succeeded (attempt 1, queued 287 ms, ran 3048 ms).
serve and worker take the same logging options as other commands (--log-level, --log, --log-format,
--log-config, --log-max-chars); see the CLI reference.
Multiple hosts
serve keeps nothing between requests: run it on several hosts behind a load balancer, with the same serve.yaml and
one network database as state. Set the load balancer's idle timeout above the jobs' wait_seconds.
systemd
streamwright systemd install writes the units streamwright-serve, streamwright-worker and a daily
streamwright-purge.timer, with your installation's streamwright in ExecStart:
python3 -m venv /opt/streamwright/venv
/opt/streamwright/venv/bin/pip install "streamwright[serve]" streamwright-google-ads # and the jobs' connectors
sudo useradd --system --no-create-home --home-dir /var/lib/streamwright --shell /usr/sbin/nologin streamwright
sudo /opt/streamwright/venv/bin/streamwright systemd install # --dry-run prints the units first
sudo cp serve.yaml /etc/streamwright/ # then fill in common.env and worker.env there
sudo systemctl daemon-reload
sudo systemctl enable --now streamwright-serve streamwright-worker streamwright-purge.timer
systemd install option | What it does | Default |
|---|---|---|
--unit-dir DIR | Where systemd reads the units | /etc/systemd/system |
--config-dir DIR | The folder of serve.yaml and the environment files | /etc/streamwright |
--force | Replace units that exist and differ | |
--dry-run | Print the units; write nothing |
It also creates two environment files (mode 0600, never replaced) for the secrets:
# /etc/streamwright/common.env: serve, worker and purge (state password, API key digests)
STREAMWRIGHT_SECRET_STATE_PASSWORD=...
SW_KEY_CRM_PROD_SHA256=...
# /etc/streamwright/worker.env: the jobs' secrets, for the workers only
STREAMWRIGHT_SECRET_DEVELOPER_TOKEN=...
What the units do
- The units run as the system user
streamwrightwith home/var/lib/streamwright, where DuckDB keeps its extensions. Other folders the jobs write to needReadWritePaths=. - The worker runs with
--processes 8and logs JSON to the journal. - The purge timer runs
streamwright queue purgewith--older-than 30donce a day. TimeoutStopSec=2hcuts runs short on stop: keep it above the jobs'limits.timeout.- Change settings with a drop-in (
sudo systemctl edit streamwright-worker) so--forcecan still update the unit.
Metrics
With metrics: {enabled: true}, GET /metrics returns Prometheus text to an authenticated caller (a scraper sends
an API key), for the jobs that caller may call.
| Metric | Meaning |
|---|---|
sw_queue_runs{job,status} | Runs queued or running now |
sw_queue_oldest_seconds{job} | Age of the oldest queued run |
sw_recent_runs{job,status} | Runs that ended in the last 5 minutes |
sw_recent_queue_wait_seconds_avg, _max{job} | Their wait from queued to first claimed |
sw_recent_run_seconds_avg, _max{job} | Their run time |
sw_http_requests_total{route,method,status} | Requests this host answered |
sw_http_request_seconds_sum, _count{route,method} | Time taken to answer them |
Queue maintenance
streamwright queue purge serve.yaml --older-than 30d [--dry-run] # e.g. daily, from cron
streamwright queue migrate serve.yaml [--dry-run] # after an upgrade that changes the tables
| Command | What it does |
|---|---|
queue purge | Deletes runs that ended (succeeded or failed) longer ago than --older-than (days, or 30d, 12h; default 30 days). Queued and running runs are kept. --dry-run only counts them |
queue migrate | Creates or upgrades the queue's tables. An existing queue that needs an upgrade waits for it: run it once, then restart the services. --dry-run prints the SQL |
Both commands need only serve.yaml's state. A job's own history is purged separately, with
streamwright duckdb state purge JOB.
See also
- DuckDB jobs: job manifests, inputs, state and scopes.
- Input payloads: jobs that take rows (
spec.input). - Apply outputs: delivery rules when a run is taken over.
- Lookups and fan-out: why inputs with a
lookup:are never settable. - CLI reference:
serve,worker,queueandsystemd install.