Input payloads
An input payload lets the caller hand a DuckDBJob its rows, instead of the job reading them from somewhere. Use it when another system decides what to process, for example a CRM that sends today's conversions to upload.
In short
spec.input declares the columns, --input gives the rows, and the steps read them as the table input.
Example
spec:
input:
columns: # typed like spec.config inputs; optional unless `required: true`
gclid: {type: string, required: true, max_length: 200}
conversion_action: {type: string, required: true}
conversion_value: {type: number, minimum: 0}
currency_code: {type: string, enum: [USD, EUR, GBP]}
max_rows: 10000
max_bytes: 104857600
unknown_columns: reject
retain: until_complete
steps:
- name: valid_conversions
sql: SELECT * FROM input WHERE conversion_value > 0
conversions.json
[
{"gclid": "EXAMPLE_GCLID_1", "conversion_action": "purchase", "conversion_value": 25.0, "currency_code": "USD"},
{"gclid": "EXAMPLE_GCLID_2", "conversion_action": "purchase", "conversion_value": 0}
]
streamwright duckdb validate conversions.yaml --input conversions.json # checks the payload too
streamwright duckdb run conversions.yaml --input conversions.json
cat conversions.ndjson | streamwright duckdb run conversions.yaml --input -
The payload is a JSON array of objects, or NDJSON (one object per line), from a file or from stdin (--input -).
Over HTTP, streamwright serve takes the same rows as "input": [ ... ] in the request body.
How it works
- Checked before anything runs. A row that does not fit fails the run (exit 2) before a batch exists, with every problem listed. That covers an unknown or missing column, a wrong type, a broken constraint, a repeated key, and nesting deeper than 32 levels.
- Typed, not guessed. The columns load in memory with their declared types: string
VARCHAR, integerBIGINT, numberDOUBLE, booleanBOOLEAN, dateDATE, listT[]. - Pinned for retries. The rows are pinned in the batch, so a retry reads the same rows.
- Removed when done. When the batch completes, the rows are removed from the state store; only their SHA-256 and row count stay. Row values are never logged.
Options
| Key | What it does | Default |
|---|---|---|
columns | The columns, each typed like a spec.config input (type, required, enum, pattern, minimum, maximum, max_length). A column is optional unless required: true. | |
max_rows | Most rows a payload may have | 10000 |
max_bytes | Largest payload, checked before it is parsed | 104857600 (100 MB) |
unknown_columns | A column not in columns: reject the payload, or drop the column | reject |
retain | until_complete keeps the rows until the batch completes, so a retry can read them; none never stores them | until_complete |
Good to know
- One
inputtable.inputis also the table a consume step fills from a broker, so a job has eitherspec.inputor a consume step, not both. - New work needs rows. A job with
spec.inputneeds--inputto start new work (--max-messages 0only resumes unfinished batches). - Run a payload once.
--job-keymakes the work for a given payload run once. retain: nonecannot resume. A failed batch cannot be resumed then: abandon it.- The same checks guard
spec.config. For example,--set client_id=...againstpattern: "^[0-9]{1,12}$"is refused before the run. This matters wherever an input reaches SQL text.
See also
- Apply outputs: send the rows to a vendor API.
- DuckDB jobs.