What do you want to do?
Pull data
Read an API, files or a database
A source folder: sign-in, requests, SQL and exports. Run it with streamwright run.
Sources →Build a pipelineCombine reads, SQL and writes in one job
A DuckDB or Spark job with steps, outputs and state, so each output is written once.
Jobs →Send data backWrite rows to ad platforms and REST APIs
Apply outputs send each row and keep its outcome. Re-runs skip applied rows.
Reverse ETL →Three ideas to remember
- 1
Fetch → Shape → Write
Everything reads records, shapes them with SQL, then writes them. In a source:
requests→transform→export. In a job: read steps →sqlsteps →outputs. - 2
connection · operation · policy
Every call to the outside world says where (the connection), what (the operation) and how (retries, rate limits, timeouts).
- 3
Same YAML, many runtimes
Run it from the CLI, behind a REST service, on Spark, or per user and network with Dagster.