Pagination

Historical endpoints return data in pages. Each response includes a next_token field — pass it back on your next request to get the following page. This guide explains how pagination works and shows a complete example for fetching historical headlines.

How it works

Paginated endpoints return a response envelope with four fields:

When has_more is false and next_token is null, you have reached the end of the dataset.

Endpoints that support pagination

News feeds

• GET /v1/headlines/feed/historical/ticker/{ticker} — historical asset headlines
• GET /v1/headlines/feed/historical/macro/{model_id} — historical macro headlines
• GET /v1/events/historical/ticker/{ticker} — historical events for a ticker
• GET /v1/event_headlines/feed/historical/ticker/{ticker} — historical event headlines for a ticker

Indices

• GET /v1/headlines/index/ticker/historical/{ticker} — historical headline sentiment index for a ticker
• GET /v1/headlines/index/macro/historical/global/{model_id} — historical global macro headline sentiment index
• GET /v1/headlines/index/macro/historical/regional/{model_id} — historical regional macro headline sentiment index
• POST /v1/headlines/index/macro/query/global — custom query for global macro headline sentiment index
• POST /v1/headlines/index/macro/query/regional — custom query for regional macro headline sentiment index
• GET /v1/events/index/historical/ticker/{ticker} — historical event sentiment metrics for a ticker
• POST /v1/events/index/query — custom query for event sentiment metrics

Fetching all pages

The pattern is the same for every paginated endpoint: loop until has_more is false, passing the next_token from each response into the next request.

import os
import requests
import time

API_KEY = os.environ["PERMUTABLE_API_KEY"]
HEADERS = {"x-api-key": API_KEY}

all_headlines = []
next_token = None

while True:
    params = {
        "start_date": "2024-01-01",
        "limit": 500,
    }
    if next_token:
        params["next_token"] = next_token

    resp = requests.get(
        "https://copilot-api.permutable.ai/v1/headlines/feed/historical/ticker/BZ_COM",
        headers=HEADERS,
        params=params,
    )
    resp.raise_for_status()
    page = resp.json()

    all_headlines.extend(page["data"])

    if not page["has_more"]:
        break

    next_token = page["next_token"]
    time.sleep(0.1)  # stay within rate limits between pages

print(f"Fetched {len(all_headlines)} headlines")

Choosing a page size

The limit parameter controls how many records are returned per page (maximum 1,000). Larger pages mean fewer requests and less quota usage. A limit of 500 is a good default for most use cases.

Avoid parallelised requests. Do not fire multiple page requests simultaneously. The next_token encodes a cursor position — pages must be fetched sequentially. Parallel requests will not speed up retrieval and may trigger rate limiting.

Quota considerations

Each page request counts as one API call. Fetching a large historical dataset with many pages will consume quota. To minimise request count:

• Use the largest limit that fits your memory constraints (up to 1,000)
• Narrow the date range with start_date and end_date to fetch only what you need
• Cache results locally — avoid re-fetching the same date range repeatedly

For bulk historical data, a CSV download via the Permutable Platform may be more efficient than paginating through the API.

Next steps

• Rate Limits — understand your monthly quota and per-second limits
• API Reference — explore all paginated endpoints interactively