REST API
The restapi connector reads from any REST API: GitHub, Stripe, Jira, webhook feeds or your own services. You
describe the API in YAML (base URL, sign-in, paging) and each stream's requests; you do not write a connector
package. From a job, it can also send rows to any endpoint.
The ad connectors LinkedIn Ads, Amazon Ads, Apple Ads and OpenAI Ads work the same way. This page is the full reference for all of them.
Install
sw connectors -i restapi
Connect
A connection has two parts: the sign-in (auth: in a source, with provider: restapi) and the http block that
describes the API.
Sign-in
Set type to one of the built-in methods:
type | Keys |
|---|---|
bearer | token |
api_key | name (the header or query parameter), value, in (header or query; default header) |
basic | username, password |
oauth2_refresh_token | token_url, client_id, client_secret, refresh_token, scopes (optional). StreamWright gets and refreshes the access token for you. |
The credentials (token, value, password, client_secret, refresh_token) must be secret references
({{ secrets.NAME }}). They are redacted (***) in every log line.
The http block
In a source it sits next to auth:; in a job it goes inside the connection. It holds the defaults every request
shares.
| Key | What it does |
|---|---|
base_url | Prefix for every request's path |
headers | Headers sent with every request |
params_encoding | plain or dotted. dotted turns nested params into a.b.c=value (Rest.li-style APIs) |
paginator | How to page (below) |
records | Where the records are in each response: path (for example data.items) or explode |
kind: source
name: github_demo
spec:
config:
org: { type: string, description: "GitHub organization name" }
secrets:
github_token: { type: string, description: "Personal access token" }
auth:
provider: restapi
type: bearer
token: "{{ secrets.github_token }}"
http:
base_url: https://api.github.com
headers:
Accept: application/vnd.github+json
X-GitHub-Api-Version: "2022-11-28"
paginator:
type: link_header
policy:
retry:
codes: [429, 500, 502, 503, 504]
max_attempts: 5
backoff: exponential
policy.retrysets how failed requests are retried for every stream of the source.link_headerpaging follows theLink: rel="next"header until there is no next page.
Read data
In a source
description: List repositories in an organization.
requests:
- name: raw_repos
operation:
path: "/orgs/{{ config.org }}/repos"
method: GET
params:
per_page: 100
sort: updated
transform:
mode: page
steps:
- name: repos
select: |
SELECT record->>'id' AS id,
record->>'name' AS name,
record->>'full_name' AS full_name,
(record->>'private')::BOOLEAN AS is_private,
(record->>'stargazers_count')::INTEGER AS stars
FROM raw_repos
export:
repositories:
step: repos
primary_key: [id]
- Each request's
operationis one HTTP call. Its records reach the stream's SQL as the tableraw_repos. - Each record is one JSON
record. Read fields withrecord->>'name'.
Request keys
A request's operation (in a stream, a job step or an output) takes:
| Key | What it does | Default |
|---|---|---|
path | Relative to http.base_url, or a full https:// URL | Required |
method | GET, POST, PUT, PATCH or DELETE | GET |
params | Query string parameters | |
headers | Headers added to the source's (an empty value removes one) | |
json | JSON request body | |
params_encoding | Overrides the source's | |
paginator | Overrides the source's | |
records | Overrides the source's |
Values can use templates: {{ config.* }}, {{ secrets.* }}, {{ partition.* }} and {{ window.* }}.
Paginators
type | Typical keys | Pages by |
|---|---|---|
none | One request only | |
offset | offset_param (default offset), limit_param (default limit), page_size (default 100), total_path, in: body | An offset and a limit |
page_number | page_param (default page), size_param, page_size, total_pages_path | A page number |
cursor | param, token_path, next_url_path, has_more_path | A token (or next URL) from the previous response |
link_header | The Link response header |
Paging stops when there is no next page or a page comes back empty. With in: body, the offset paginator writes its
parameters into the JSON body instead of the query string (see Apple Ads).
In a job
The connection holds the sign-in, the http defaults and a default policy. A step's operation is the same request
as in a stream, and its sql reads the table records (call steps).
connections:
rest_api:
provider: restapi
type: bearer
token: "{{ secrets.api_token }}"
http:
base_url: "{{ config.base_url }}"
headers: {Accept: application/json}
policy:
rate_limit: {requests: 10, per: 1s}
steps:
- name: items
connection: rest_api
operation:
path: /items
method: GET
params: {status: active}
records: {path: data.items}
paginator: {type: cursor, param: starting_after, token_path: meta.next_cursor}
sql: |
SELECT record->>'id' AS item_id,
record->>'name' AS name,
CAST(record->>'price' AS DECIMAL(10, 2)) AS price
FROM records
In a job, {{ secrets.* }} is allowed only in the connection's credential keys (such as token), not in headers
or request values.
Write data back
A job's apply output (reverse ETL) sends rows to any endpoint of the API as HTTP requests:
outputs:
- name: send
connection: rest_api
step: rows # each row becomes one request's JSON body
operation:
path: /items
method: POST
apply:
on_row_error: continue
- Without
json, each row is sent as one request, with the row as its JSON body. Whenjsonholds{{ rows }}(for examplejson: {elements: "{{ rows }}"}), the whole chunk goes in one request. {{ row.<column> }}puts a row's value in the path, parameters, headers or body (/items/{{ row.id }}).methodisPOST(default),PUT,PATCHorDELETE.PUTandDELETEare retried bypolicy.retry;POSTandPATCHare sent once.- A row the API rejects fails alone, with the HTTP status as its code.
Good to know
- Engines: sources (
streamwright run), and DuckDB and Spark jobs through call steps. Writes run only in job apply outputs. - Vendor arguments: any query parameter or body field goes to the API unchanged, for example
params: {page_size: 100}. Paging parameters set by the paginator take precedence. - Retries: by default a request is retried on HTTP 429, 500, 502, 503 and 504 (3 attempts, exponential
backoff). Set
policy.retry(codes,max_attempts,backoff,max_delay) to change this. A stream'spolicy.retryreplaces the source's.
See also
- Streams: requests, transforms and exports.
- Call steps: reading from a connector inside a job.
- Apply outputs: how rows are sent, and what happens when a call is cut short.