Skip to main content

Input payloads

An input payload lets the caller hand a DuckDBJob its rows, instead of the job reading them from somewhere. Use it when another system decides what to process, for example a CRM that sends today's conversions to upload.

In short

spec.input declares the columns, --input gives the rows, and the steps read them as the table input.

Example​

spec:
input:
columns: # typed like spec.config inputs; optional unless `required: true`
gclid: {type: string, required: true, max_length: 200}
conversion_action: {type: string, required: true}
conversion_value: {type: number, minimum: 0}
currency_code: {type: string, enum: [USD, EUR, GBP]}
max_rows: 10000
max_bytes: 104857600
unknown_columns: reject
retain: until_complete
steps:
- name: valid_conversions
sql: SELECT * FROM input WHERE conversion_value > 0
conversions.json
[
{"gclid": "EXAMPLE_GCLID_1", "conversion_action": "purchase", "conversion_value": 25.0, "currency_code": "USD"},
{"gclid": "EXAMPLE_GCLID_2", "conversion_action": "purchase", "conversion_value": 0}
]
streamwright duckdb validate conversions.yaml --input conversions.json # checks the payload too
streamwright duckdb run conversions.yaml --input conversions.json
cat conversions.ndjson | streamwright duckdb run conversions.yaml --input -

The payload is a JSON array of objects, or NDJSON (one object per line), from a file or from stdin (--input -). Over HTTP, streamwright serve takes the same rows as "input": [ ... ] in the request body.

How it works​

  1. Checked before anything runs. A row that does not fit fails the run (exit 2) before a batch exists, with every problem listed. That covers an unknown or missing column, a wrong type, a broken constraint, a repeated key, and nesting deeper than 32 levels.
  2. Typed, not guessed. The columns load in memory with their declared types: string VARCHAR, integer BIGINT, number DOUBLE, boolean BOOLEAN, date DATE, list T[].
  3. Pinned for retries. The rows are pinned in the batch, so a retry reads the same rows.
  4. Removed when done. When the batch completes, the rows are removed from the state store; only their SHA-256 and row count stay. Row values are never logged.

Options​

KeyWhat it doesDefault
columnsThe columns, each typed like a spec.config input (type, required, enum, pattern, minimum, maximum, max_length). A column is optional unless required: true.
max_rowsMost rows a payload may have10000
max_bytesLargest payload, checked before it is parsed104857600 (100 MB)
unknown_columnsA column not in columns: reject the payload, or drop the columnreject
retainuntil_complete keeps the rows until the batch completes, so a retry can read them; none never stores themuntil_complete

Good to know​

  • One input table. input is also the table a consume step fills from a broker, so a job has either spec.input or a consume step, not both.
  • New work needs rows. A job with spec.input needs --input to start new work (--max-messages 0 only resumes unfinished batches).
  • Run a payload once. --job-key makes the work for a given payload run once.
  • retain: none cannot resume. A failed batch cannot be resumed then: abandon it.
  • The same checks guard spec.config. For example, --set client_id=... against pattern: "^[0-9]{1,12}$" is refused before the run. This matters wherever an input reaches SQL text.

See also​