Skip to main content

REST API

The restapi connector reads from any REST API: GitHub, Stripe, Jira, webhook feeds or your own services. You describe the API in YAML (base URL, sign-in, paging) and each stream's requests; you do not write a connector package. From a job, it can also send rows to any endpoint.

The ad connectors LinkedIn Ads, Amazon Ads, Apple Ads and OpenAI Ads work the same way. This page is the full reference for all of them.

Install​

sw connectors -i restapi

Connect​

A connection has two parts: the sign-in (auth: in a source, with provider: restapi) and the http block that describes the API.

Sign-in​

Set type to one of the built-in methods:

typeKeys
bearertoken
api_keyname (the header or query parameter), value, in (header or query; default header)
basicusername, password
oauth2_refresh_tokentoken_url, client_id, client_secret, refresh_token, scopes (optional). StreamWright gets and refreshes the access token for you.

The credentials (token, value, password, client_secret, refresh_token) must be secret references ({{ secrets.NAME }}). They are redacted (***) in every log line.

The http block​

In a source it sits next to auth:; in a job it goes inside the connection. It holds the defaults every request shares.

KeyWhat it does
base_urlPrefix for every request's path
headersHeaders sent with every request
params_encodingplain or dotted. dotted turns nested params into a.b.c=value (Rest.li-style APIs)
paginatorHow to page (below)
recordsWhere the records are in each response: path (for example data.items) or explode
source.yaml
kind: source
name: github_demo

spec:
config:
org: { type: string, description: "GitHub organization name" }
secrets:
github_token: { type: string, description: "Personal access token" }

auth:
provider: restapi
type: bearer
token: "{{ secrets.github_token }}"

http:
base_url: https://api.github.com
headers:
Accept: application/vnd.github+json
X-GitHub-Api-Version: "2022-11-28"
paginator:
type: link_header
policy:
retry:
codes: [429, 500, 502, 503, 504]
max_attempts: 5
backoff: exponential
  • policy.retry sets how failed requests are retried for every stream of the source.
  • link_header paging follows the Link: rel="next" header until there is no next page.

Read data​

In a source​

streams/repositories.yaml
description: List repositories in an organization.

requests:
- name: raw_repos
operation:
path: "/orgs/{{ config.org }}/repos"
method: GET
params:
per_page: 100
sort: updated

transform:
mode: page
steps:
- name: repos
select: |
SELECT record->>'id' AS id,
record->>'name' AS name,
record->>'full_name' AS full_name,
(record->>'private')::BOOLEAN AS is_private,
(record->>'stargazers_count')::INTEGER AS stars
FROM raw_repos

export:
repositories:
step: repos
primary_key: [id]
  • Each request's operation is one HTTP call. Its records reach the stream's SQL as the table raw_repos.
  • Each record is one JSON record. Read fields with record->>'name'.

Request keys​

A request's operation (in a stream, a job step or an output) takes:

KeyWhat it doesDefault
pathRelative to http.base_url, or a full https:// URLRequired
methodGET, POST, PUT, PATCH or DELETEGET
paramsQuery string parameters
headersHeaders added to the source's (an empty value removes one)
jsonJSON request body
params_encodingOverrides the source's
paginatorOverrides the source's
recordsOverrides the source's

Values can use templates: {{ config.* }}, {{ secrets.* }}, {{ partition.* }} and {{ window.* }}.

Paginators​

typeTypical keysPages by
noneOne request only
offsetoffset_param (default offset), limit_param (default limit), page_size (default 100), total_path, in: bodyAn offset and a limit
page_numberpage_param (default page), size_param, page_size, total_pages_pathA page number
cursorparam, token_path, next_url_path, has_more_pathA token (or next URL) from the previous response
link_headerThe Link response header

Paging stops when there is no next page or a page comes back empty. With in: body, the offset paginator writes its parameters into the JSON body instead of the query string (see Apple Ads).

In a job​

The connection holds the sign-in, the http defaults and a default policy. A step's operation is the same request as in a stream, and its sql reads the table records (call steps).

connections:
rest_api:
provider: restapi
type: bearer
token: "{{ secrets.api_token }}"
http:
base_url: "{{ config.base_url }}"
headers: {Accept: application/json}
policy:
rate_limit: {requests: 10, per: 1s}

steps:
- name: items
connection: rest_api
operation:
path: /items
method: GET
params: {status: active}
records: {path: data.items}
paginator: {type: cursor, param: starting_after, token_path: meta.next_cursor}
sql: |
SELECT record->>'id' AS item_id,
record->>'name' AS name,
CAST(record->>'price' AS DECIMAL(10, 2)) AS price
FROM records
note

In a job, {{ secrets.* }} is allowed only in the connection's credential keys (such as token), not in headers or request values.

Write data back​

A job's apply output (reverse ETL) sends rows to any endpoint of the API as HTTP requests:

outputs:
- name: send
connection: rest_api
step: rows # each row becomes one request's JSON body
operation:
path: /items
method: POST
apply:
on_row_error: continue
  • Without json, each row is sent as one request, with the row as its JSON body. When json holds {{ rows }} (for example json: {elements: "{{ rows }}"}), the whole chunk goes in one request.
  • {{ row.<column> }} puts a row's value in the path, parameters, headers or body (/items/{{ row.id }}).
  • method is POST (default), PUT, PATCH or DELETE. PUT and DELETE are retried by policy.retry; POST and PATCH are sent once.
  • A row the API rejects fails alone, with the HTTP status as its code.

Good to know​

  • Engines: sources (streamwright run), and DuckDB and Spark jobs through call steps. Writes run only in job apply outputs.
  • Vendor arguments: any query parameter or body field goes to the API unchanged, for example params: {page_size: 100}. Paging parameters set by the paginator take precedence.
  • Retries: by default a request is retried on HTTP 429, 500, 502, 503 and 504 (3 attempts, exponential backoff). Set policy.retry (codes, max_attempts, backoff, max_delay) to change this. A stream's policy.retry replaces the source's.

See also​

  • Streams: requests, transforms and exports.
  • Call steps: reading from a connector inside a job.
  • Apply outputs: how rows are sent, and what happens when a call is cut short.