Skip to main content

Serving jobs over HTTP

streamwright serve turns your DuckDB jobs into a REST API, so other systems can start a run with an HTTP request instead of a shell command. Use it when an application, a scheduler or a partner needs to trigger jobs, and you want to control who may run what. It is the job CLI over HTTP: the same jobs, inputs and checks.

  • serve never runs a job. It checks the caller and the request, then queues the run.
  • worker processes take runs from the queue and run them as streamwright duckdb run would.
  • serve.yaml lists the jobs, who may call each one, and which inputs a caller may set. Nothing else is reachable.
On the command lineOver HTTP
streamwright duckdb run JOBPOST /v1/jobs/JOB/runs
--set NAME=VALUE"set": {"NAME": VALUE} in the body
--input FILE"input": [ ... ] in the body (jobs with spec.input)
--job-key KEYthe Idempotency-Key header
--secrets, --allow-lookup-exec, --fan-out-concurrency, ...never in a request: fixed by the operator in serve.yaml and the server's environment

A caller is not the operator, so a request carries only set, input, an idempotency key and options.async.

This page has three parts:

Quick start​

Take any DuckDB job, for example orders.yaml from DuckDB jobs, and put this serve.yaml next to it:

serve.yaml
bind: 127.0.0.1:8080
docs: {enabled: true}
jobs:
orders:
manifest: orders.yaml
pip install "streamwright[serve]"
streamwright validate serve.yaml # check serve.yaml, each job entry and each manifest
streamwright serve serve.yaml # terminal 1: the API
streamwright worker serve.yaml # terminal 2: runs the queued runs

curl -H "Content-Type: application/json" -d '{"set": {"min_amount": 100}}' \
http://127.0.0.1:8080/v1/jobs/orders/runs

On a loopback address you can leave out state and auth: the queue is queue.db next to serve.yaml (its tables are created at start) and there is one anonymous caller. Open http://127.0.0.1:8080/docs to try the API in Swagger UI.

Calling a job​

POST /v1/jobs/client_feed_upload/runs
Authorization: Bearer <API key or JWT>
Idempotency-Key: 2026-11-16-client-1001
Content-Type: application/json

{"set": {"client_id": "1001", "feed_date": "2026-11-16"}, "options": {"async": false}}
  • Waiting. The request waits up to wait_seconds. It answers 200 with the run if the run ends in time, else 202 with Location: /v1/runs/RUN_ID to poll. With options.async: true the answer is 202 at once.
  • Idempotency. An Idempotency-Key (1-128 printable characters) belongs to the caller and the job. The same key and body return the same run; the same key with another body is 422. Jobs with apply outputs require a key. The key is also the run's job key, so a run taken over by another worker resumes its batches.
  • One run per scope. The job's runtime.state.scope is rendered with the request's values. If a run with that scope is active, on_busy decides: reject answers 409, queue waits behind it.

Responses​

CodeWhen
200The run ended within wait_seconds
202The run is queued or running; poll the Location
400Query strings are not accepted
401Missing or invalid API key or JWT
403The caller is not in the job's principals, or lacks a claim an input is bound to
404No such job, or a run that is not the caller's
409A run with the same scope is active (on_busy: reject); for the caller's own run, with its Location
413 · 415The body is too large · not JSON
422The body is invalid (an unknown key, an input it may not set, a value that fails its checks), or the same Idempotency-Key with another body
428A job with apply outputs, called without an Idempotency-Key
429The rate window is full; see Retry-After
503The queue is full (1,000 queued runs); see Retry-After

Errors look like {"status": "error", "code", "message", "request_id"}. A failed run tells the caller only a code (run-failed, invalid-input, invalid-job or timeout); the cause goes to the worker's log. Every log line of a run carries the request's X-Request-ID (req=). Request bodies and row values are never logged.

Endpoints​

EndpointReturns
GET /v1/jobsThe caller's jobs, what each lets it set, and whether it takes rows
POST /v1/jobs/JOB/runsStarts a run
POST /v1/jobs/JOB/validate{"valid": ..., "issues": [...]} for the request's values
GET /v1/jobs/JOB/planThe job's plan
GET /v1/jobs/JOB/statusThe caller's latest runs of the job
GET /v1/runs/RUN_IDOne run: queued, running, succeeded or failed, with its summary
GET /healthz · GET /readyzLiveness (the process answers) · readiness (the state store answers)
GET /metricsPrometheus metrics (below)
GET /openapi.json · GET /docsOpenAPI 3.1 for the caller's jobs · Swagger UI over it

Caller inputs​

A caller may set every spec.config input, checked against its type and limits (enum, pattern, minimum, maximum, max_length). A caller can never set:

  • an input that serve.yaml binds (set),
  • an input with a lookup: (a caller's value would skip the lookup and could name any account),
  • anything in spec.secrets.

serve refuses to start when a settable input is used in a sensitive place without a tight enough type:

Where the input is usedIt must be
SQL: sql, a read query, a call's arguments.query, a lookup's querya number, boolean or date, or text whose enum or pattern allows only letters, digits and _ - . : @ + (e.g. ^[A-Za-z0-9_-]{1,64}$)
What is read or written: a connection, path, file_name, layout, table, destination, a lookup's command or secret_ida number or date, or text limited to letters, digits and _ - : @ + = ,

Patterns with ., negated classes, \s or case-insensitive matching never count as tight.

warning

Bind the tenant part of a path from a claim, and keep the connection's roots to the tenant's prefix: the roots stay the hard boundary.

Configuration​

This part and the next are for the operator: what the service reads at start, and who may call it.

The serve.yaml file​

serve.yaml
bind: 0.0.0.0:8080 # a non-loopback address needs an `auth` section
state: # the run queue (see below)
url: postgresql://sw:{{ secrets.state_password }}@pg.example.com:5432/streamwright?schema=sw
auth:
api_keys: # only SHA-256 digests (hex), read from the environment
- id: crm_prod
sha256_env: SW_KEY_CRM_PROD_SHA256
claims: {tenant: t1}
jwt: {issuer: https://idp.example.com, audience: streamwright, jwks_url: https://idp.example.com/jwks}
docs: {enabled: true} # Swagger UI at /docs
metrics: {enabled: true} # Prometheus metrics at /metrics
jobs:
client_feed_upload:
manifest: jobs/client_feed_upload.yaml # relative to serve.yaml
principals: [crm_prod] # an api_keys id, or a JWT `sub`
set: # inputs the caller may NOT set
tenant_id: {from_claim: tenant} # taken from the caller's claim
region: {value: eu} # fixed
payload: {max_bytes: 5MB} # jobs with spec.input
server_options: {fan_out_concurrency: 4, allow_lookup_exec: false}
limits: {concurrency: 4, rate: {requests: 10, per: 1m}, timeout: 2h, wait_seconds: 30}
on_busy: reject

Top-level keys

KeyWhat it doesDefault
bindhost:port to listen on; a non-loopback address needs auth127.0.0.1:8080
stateThe run queue's database (below)on loopback: queue.db next to serve.yaml (under systemd: $STATE_DIRECTORY/queue.db); otherwise required
auth.api_keysAPI keys: id, sha256_env (the variable holding the key's SHA-256 in hex), optional claimsnone
auth.jwtJWT callers: issuer, audience, jwks_url (https:// only)none
docs.enabledServe Swagger UI at /docsfalse
metrics.enabledServe Prometheus metrics at /metricsfalse
jobsThe jobs to serve (at least one)required

Job entries (jobs.NAME)

The name is the job's URL segment (/v1/jobs/NAME): 1-128 letters, digits and _ . -.

KeyWhat it doesDefault
manifestThe job manifest, relative to serve.yamlrequired
principalsWho may call: an api_keys id, or else a JWT subrequired with auth
set.INPUTBind an input so callers cannot set it: {from_claim: CLAIM} or {value: V}other spec.config inputs are settable
payload.max_bytesLargest body for a job with spec.input (512KB, 5MB); can only lower the job's limitthe job's spec.input limit
server_optionsduckdb run options the operator fixes (below)the duckdb run defaults
limits.concurrencyMost runs of this job at once, across all workersno limit
limits.rate{requests: N, per: DURATION} per caller, across all hostsno limit
limits.timeoutKill a run that takes longer (30s, 10m, 2h, 1d)1h
limits.wait_secondsHow long a request waits for its run before answering 20230
on_busyWhen a run with the same scope is active: reject (409) or queue (wait behind it)reject

server_options takes allow_lookup_exec, fan_out_concurrency, fan_out_on_failure, failed_job_policy, max_job_attempts, state_lock_policy, state_lock_timeout, allowed_connectors and apply_state_migrations. The jobs' secrets come from the workers' environment (STREAMWRIGHT_SECRET_<NAME>), as for the CLI.

Queue database​

state takes a url (sqlite:///queue.db, postgresql://..., mysql://...; the schema as ?schema=NAME) or a connection plus schema, like a job's runtime.state:

state:
connection:
provider: postgres
host: pg.example.com
port: 6432 # e.g. PgBouncer
database: streamwright
user: sw_serve
password: "{{ secrets.state_password }}"
sslmode: require
schema: sw # default: streamwright
  • {{ secrets.NAME }} reads STREAMWRIGHT_SECRET_<NAME>; it is the only reference state accepts.
  • DuckDB cannot hold the queue, since only one process can write a DuckDB file. Several hosts share one Postgres or MySQL database.
  • PgBouncer in transaction pooling mode works.

Authentication​

  • API keys. serve.yaml holds only each key's SHA-256 digest, read from the variable named in sha256_env. Make one with echo -n "$KEY" | sha256sum. To rotate, add the new digest next to the old one, then remove the old one.
  • JWT. The token must be signed with an asymmetric algorithm (RS*, PS*, ES*, EdDSA) by a key from the issuer's JWKS. iss, aud, exp and sub are required. The caller is the sub, and its claims feed from_claim.
  • Callers are kept apart. Each caller's runs, Idempotency-Keys and rate limits are its own. A sub that equals an API key's id cannot be listed: rename the key.
warning

serve does not do TLS. Put a reverse proxy or load balancer in front of it. A service without auth only binds a loopback address.

Running in production​

How the service runs: its workers, its hosts, its process manager, its metrics and its upkeep.

Workers​

streamwright worker serve.yaml --processes 8 --log-format json

A worker claims the oldest run it may take (one per scope, limits.concurrency per job), runs it in a child process, and kills it after limits.timeout. --processes N (default 1) sets how many runs it works on at once. A host needs roughly --processes times one run's memory. To serve more runs, raise --processes or add hosts.

  • Stopping. SIGTERM or Ctrl-C lets runs under way finish; SIGKILL stops everything at once.
  • When a worker dies, its claim expires after a minute and another worker resumes the run from the job's state. A run never runs twice at once.
  • Logs. Each ended run logs its timings: run RUN_ID succeeded (attempt 1, queued 287 ms, ran 3048 ms).

serve and worker take the same logging options as other commands (--log-level, --log, --log-format, --log-config, --log-max-chars); see the CLI reference.

Multiple hosts​

serve keeps nothing between requests: run it on several hosts behind a load balancer, with the same serve.yaml and one network database as state. Set the load balancer's idle timeout above the jobs' wait_seconds.

systemd​

streamwright systemd install writes the units streamwright-serve, streamwright-worker and a daily streamwright-purge.timer, with your installation's streamwright in ExecStart:

python3 -m venv /opt/streamwright/venv
/opt/streamwright/venv/bin/pip install "streamwright[serve]" streamwright-google-ads # and the jobs' connectors
sudo useradd --system --no-create-home --home-dir /var/lib/streamwright --shell /usr/sbin/nologin streamwright
sudo /opt/streamwright/venv/bin/streamwright systemd install # --dry-run prints the units first
sudo cp serve.yaml /etc/streamwright/ # then fill in common.env and worker.env there
sudo systemctl daemon-reload
sudo systemctl enable --now streamwright-serve streamwright-worker streamwright-purge.timer
systemd install optionWhat it doesDefault
--unit-dir DIRWhere systemd reads the units/etc/systemd/system
--config-dir DIRThe folder of serve.yaml and the environment files/etc/streamwright
--forceReplace units that exist and differ
--dry-runPrint the units; write nothing

It also creates two environment files (mode 0600, never replaced) for the secrets:

# /etc/streamwright/common.env: serve, worker and purge (state password, API key digests)
STREAMWRIGHT_SECRET_STATE_PASSWORD=...
SW_KEY_CRM_PROD_SHA256=...
# /etc/streamwright/worker.env: the jobs' secrets, for the workers only
STREAMWRIGHT_SECRET_DEVELOPER_TOKEN=...
What the units do
  • The units run as the system user streamwright with home /var/lib/streamwright, where DuckDB keeps its extensions. Other folders the jobs write to need ReadWritePaths=.
  • The worker runs with --processes 8 and logs JSON to the journal.
  • The purge timer runs streamwright queue purge with --older-than 30d once a day.
  • TimeoutStopSec=2h cuts runs short on stop: keep it above the jobs' limits.timeout.
  • Change settings with a drop-in (sudo systemctl edit streamwright-worker) so --force can still update the unit.

Metrics​

With metrics: {enabled: true}, GET /metrics returns Prometheus text to an authenticated caller (a scraper sends an API key), for the jobs that caller may call.

MetricMeaning
sw_queue_runs{job,status}Runs queued or running now
sw_queue_oldest_seconds{job}Age of the oldest queued run
sw_recent_runs{job,status}Runs that ended in the last 5 minutes
sw_recent_queue_wait_seconds_avg, _max{job}Their wait from queued to first claimed
sw_recent_run_seconds_avg, _max{job}Their run time
sw_http_requests_total{route,method,status}Requests this host answered
sw_http_request_seconds_sum, _count{route,method}Time taken to answer them

Queue maintenance​

streamwright queue purge serve.yaml --older-than 30d [--dry-run] # e.g. daily, from cron
streamwright queue migrate serve.yaml [--dry-run] # after an upgrade that changes the tables
CommandWhat it does
queue purgeDeletes runs that ended (succeeded or failed) longer ago than --older-than (days, or 30d, 12h; default 30 days). Queued and running runs are kept. --dry-run only counts them
queue migrateCreates or upgrades the queue's tables. An existing queue that needs an upgrade waits for it: run it once, then restart the services. --dry-run prints the SQL

Both commands need only serve.yaml's state. A job's own history is purged separately, with streamwright duckdb state purge JOB.

See also​