Skip to content

Pagination

Most endpoints that return lists support pagination via the limit and offset parameters.

Parameters

Parameter Type Default Description
limit integer varies Maximum number of results per page
offset integer 0 Number of records to skip

Limits per endpoint

Endpoint Min Max Default
GET /dags/{dag_id}/runs 1 100 20
GET /phases/{phase}/logs 1 500 50
GET /phases/{phase}/data 1 500 50
GET /phases/{phase}/errors 1 500 50

Paginated response structure

All paginated endpoints include these fields:

{
  "items": [...],
  "total": 1500,
  "limit": 50,
  "offset": 0,
  "has_more": true
}
Field Description
total Total records matching the filters, regardless of the current page
has_more true if more results exist beyond offset + limit

Example: iterate all pages

# Page 1
curl "http://<host>:<port>/api/v1/dags/my-dag/runs/2026-05-01/run-id/phases/extract/logs?limit=50&offset=0" \
  -H "Authorization: Bearer <token>"

# Page 2
curl "http://<host>:<port>/api/v1/dags/my-dag/runs/2026-05-01/run-id/phases/extract/logs?limit=50&offset=50" \
  -H "Authorization: Bearer <token>"

import requests

headers = {"Authorization": f"Bearer {token}"}
base_url = "http://<host>:<port>/api/v1/dags/my-dag/runs/2026-05-01/run-id/phases/extract/logs"
all_logs = []
offset = 0
limit = 50

while True:
    response = requests.get(
        base_url,
        params={"limit": limit, "offset": offset},
        headers=headers,
    )
    data = response.json()
    all_logs.extend(data["items"])

    if not data["has_more"]:
        break
    offset += limit

const headers = { Authorization: `Bearer ${token}` };
const allLogs = [];
let offset = 0;
const limit = 50;

while (true) {
  const res = await fetch(
    `/api/v1/dags/my-dag/runs/2026-05-01/run-id/phases/extract/logs?limit=${limit}&offset=${offset}`,
    { headers }
  );
  const data = await res.json();
  allLogs.push(...data.items);

  if (!data.has_more) break;
  offset += limit;
}

Alternative: stream

To retrieve all logs for a phase without managing pages, the GET /phases/{phase}/logs endpoint supports streaming in NDJSON (JSON Lines) format. Send the Accept: application/x-ndjson header in the request.

Large datasets

The stream applies no result limit. Consume it incrementally to avoid memory issues.

curl "http://<host>:<port>/api/v1/dags/my-dag/runs/2026-05-01/run-id/phases/extract/logs?level=ERROR" \
  -H "Authorization: Bearer <token>" \
  -H "Accept: application/x-ndjson" \
  | while IFS= read -r line; do
      echo "$line"
    done

import json, requests

with requests.get(
    "http://<host>:<port>/api/v1/dags/my-dag/runs/2026-05-01/run-id/phases/extract/logs",
    params={"level": "ERROR"},
    headers={
        "Authorization": f"Bearer {token}",
        "Accept": "application/x-ndjson",
    },
    stream=True,
) as r:
    for line in r.iter_lines():
        if line:
            log = json.loads(line)
            print(log)