Skip to main content

Outputs

An output is where a source run writes its exports. You pick it at run time with --output; it is not part of the source. The same source can write files on your laptop and load a warehouse in production, with no change.

In short

The source says what to write (its exports). --output says where.

streamwright run my_source --output parquet:out

This writes one Parquet file per export into out/, plus a state.json with the run's bookmarks.

Pick an output​

--output valueResult
singer (default)Singer SCHEMA / RECORD / STATE messages on stdout
jsonl:DIR · csv:DIR · tsv:DIR · parquet:DIROne file per export plus state.json, written atomically
duckdb:PATH[:SCHEMA] · ducklake:CATALOG[:SCHEMA]Load into a DuckDB file or a DuckLake catalog
dlt:DESTINATION[:DATASET]Load into any dlt destination, such as dlt:duckdb or dlt:bigquery:marketing

Logs always go to stderr, so singer output on stdout can be piped to another tool.

Files​

FormatEach export's file
jsonlOne JSON object per line
csv / tsvA header row, then one row per record
parquetOne row per record, with the export step's column types, zstd-compressed

Files are named <export>.<date>.<time>.<unique>.<ext> unless you name them. --file-name sets the path of each export's file inside DIR:

streamwright run my_source --output parquet:out --file-name "{{ export }}/{{ today }}.parquet"
  • The template can use {{ export }}, {{ source }}, {{ today }}, {{ timestamp }}, {{ format }} and {{ config.NAME }}.
  • Subfolders are created. Every export needs its own path. A file with the same name is replaced.
  • --file-name replaces any file_name set in the YAML (see Export).
  • --file-name works with file outputs only; with another output it is an error.

Warehouses​

OutputUse it to...
duckdb:PATH[:SCHEMA]Load straight into a local DuckDB file
ducklake:CATALOG[:SCHEMA]Load into a DuckLake catalog: a file, or postgres:DSN for a Postgres catalog. Put data files on S3 with STREAMWRIGHT_DUCKLAKE_DATA_PATH.
dlt:DESTINATION[:DATASET]Use dlt's destinations (BigQuery, Snowflake, Postgres and more) and its schema evolution. dlt also keeps the run's state.
note

These outputs are for sources. A job (kind: DuckDBJob, kind: SparkJob) declares its writes in the manifest's outputs. See DuckDB jobs.

See also​

  • Streams: how exports are defined.
  • CLI: every run option.