Chapter 26 · Ingestion

Pulling from a third-party API

Rate limits you do not control, pagination you cannot change, and a backfill that takes four days. How to make the first run finish and the nightly run cheap.

11 min read

The third adapter shape. A document store is slow but yours; a database is fast and yours; a third-party API is neither, and every constraint in it is somebody else's decision.

Atlas's support tickets come through one: 100 requests per minute, shared with the rest of the company's integrations, no bulk export, and a list endpoint that returns 50 records per page. The resolution notes from the previous chapter came from a database replica; here the same data arrives the hard way, which is the common case.

The arithmetic, first

Do this before writing the adapter. It frequently changes the plan.

// Back-of-envelope, and worth being explicit about.
const totalRecords = 41_000
const pageSize = 50
const listRequests = Math.ceil(totalRecords / pageSize)     // 820
const detailRequests = totalRecords                         // if detail is separate
const requestsPerMinute = 100 * 0.5   // ⚠️ half the budget: it is shared

const minutes = (listRequests + detailRequests) / requestsPerMinute
console.log(`backfill: ${Math.round(minutes / 60)} hours`)  // ~14 hours

Fourteen hours for the initial backfill, and that is with a detail call per record. If the list endpoint returns everything you need, it is twenty minutes — so the first question to ask of the API is whether a field mask or an expansion parameter removes the detail call entirely.

// Almost every API has one, and it is the single biggest lever here.
await api.listTickets({ fields: 'id,subject,category,closed_at,resolution_notes', limit: 50 })

Taking half the published rate limit as your budget is deliberate. The limit is usually per-account rather than per-integration, and an ingest that consumes all of it breaks whatever else the company has connected — which is a support ticket about your project on day one.

One limiter for the whole process

// src/ingest/sources/api/limiter.ts
// A token bucket, shared by every call to this API from this process. The
// important word is "shared": three code paths with their own limiters is
// three times the configured rate.
export class RateLimiter {
  private tokens: number
  private last = Date.now()

  constructor(private readonly perMinute: number, private readonly burst = 10) {
    this.tokens = burst
  }

  async take(): Promise<void> {
    for (;;) {
      const now = Date.now()
      this.tokens = Math.min(
        this.burst,
        this.tokens + ((now - this.last) / 60_000) * this.perMinute,
      )
      this.last = now
      if (this.tokens >= 1) { this.tokens -= 1; return }
      await sleep(Math.ceil(((1 - this.tokens) / this.perMinute) * 60_000))
    }
  }

  /** The server's own answer overrides the local estimate. A 429 means the
   *  bucket was wrong, so stop guessing and wait as instructed. */
  async penalise(retryAfterSeconds: number) {
    this.tokens = 0
    await sleep(retryAfterSeconds * 1000)
  }
}

export const ticketApi = new RateLimiter(50)   // half of 100

async function call<T>(fn: () => Promise<Response>): Promise<T> {
  for (let attempt = 0; attempt < 5; attempt++) {
    await ticketApi.take()
    const response = await fn()

    if (response.status === 429) {
      const wait = Number(response.headers.get('retry-after') ?? 2 ** attempt)
      logger.warn('api.rate_limited', { attempt, waitSeconds: wait })
      await ticketApi.penalise(wait)
      continue
    }
    if (response.status >= 500) { await sleep(2 ** attempt * 1000); continue }
    if (!response.ok) {
      // 4xx other than 429 will not change on retry. Fail the document, not the run.
      throw new SkipDocument('fetch', `${response.status} ${await response.text()}`)
    }
    return response.json() as Promise<T>
  }
  throw new Error('exhausted retries')
}

The backfill is a different program from the nightly run

Conflating them is the mistake that makes this hard. They have opposite requirements.

	Backfill	Incremental
Volume	41,000 records	~200 per night
Duration	Hours to days	Minutes
Interruption	Certain	Unlikely
Ordering	Oldest first, so partial is useful	Newest first
Failure handling	Must resume exactly	Can re-run whole

// The backfill keeps its own durable cursor, because "run it again" is not an
// acceptable recovery strategy for a fourteen-hour job.
interface BackfillState {
  sourceId: string
  cursor: string | null
  recordsDone: number
  startedAt: Date
  completedAt: Date | null
}

export async function backfill(sourceId: string) {
  const state = await loadOrCreateState(sourceId)

  while (!state.completedAt) {
    const page = await call<Page>(() => listPage(state.cursor))

    for (const record of page.items) {
      await ingestRecord(record)      // idempotent, per chapter 23
    }

    // Persist AFTER the page is fully ingested, never before. A crash then
    // re-does one page — fifty idempotent records — rather than skipping it.
    state.cursor = page.nextCursor
    state.recordsDone += page.items.length
    if (!page.nextCursor) state.completedAt = new Date()
    await saveState(state)

    logger.info('backfill.progress', {
      sourceId, done: state.recordsDone,
      rate: state.recordsDone / ((Date.now() - +state.startedAt) / 60_000),
    })
  }
}

Saving the cursor after the page rather than before is the whole of the resumability design, and it is worth stating the invariant plainly:

Advance the cursor only past work that is durably done. Re-doing idempotent work is free; skipping work is silent.

Logging a rate rather than only a count is the other detail worth keeping. A backfill that is going to take fourteen hours should say so in the first five minutes, while there is still time to ask for a higher rate limit or a bulk export.

Ordering the backfill so partial progress is useful

Oldest first is a deliberate choice and the opposite of what feels natural.

Newest-first is more useful while incomplete and far more dangerous, because the missing data is the data nobody thinks to check. Oldest-first makes an incomplete backfill fail visibly against the question set, which is what you want from a state you will be in for fourteen hours.

If you do run newest-first — and for a very long backfill that is a defensible trade — record the coverage boundary and have retrieval declare it:

// The honest version of a partial corpus.
const coverage = await getCoverage('ticket_notes')
if (coverage.oldestIndexed > query.impliedPeriodStart) {
  answer.caveats.push(
    `Ticket history is indexed back to ${fmtDate(coverage.oldestIndexed)}. ` +
    `Earlier tickets are not yet searchable.`,
  )
}

Incremental, once the backfill is done

// Minutes, not hours, because the API's filter does the work.
export async function* discoverTicketsSince(since: Date) {
  const from = new Date(+since - 5 * 60_000)    // the same rewind as chapter 25
  let cursor: string | undefined

  do {
    const page = await call<Page>(() =>
      listPage(cursor, { updatedSince: from.toISOString(), limit: 50 }),
    )
    for (const item of page.items) yield toDiscovered(item)
    cursor = page.nextCursor
  } while (cursor)
}

⚠️ Verify that the API's updatedSince filter means what you expect, by the same reasoning as chapter 24's warning about modifiedAt. The cheap test: pick a record, change a field through the UI, and see whether it appears in a filtered list a minute later. Two minutes of work, and it is the difference between a working incremental sync and one that has silently stopped seeing edits.

The webhook that does not replace polling

Many APIs offer webhooks, and they are worth taking — they turn a nightly delay into seconds for the records that matter.

They are not a replacement for the scheduled run, for three reasons: deliveries are lost, your endpoint has downtime, and a webhook tells you a record changed but not that one was deleted outside the event model.

// Webhooks reduce latency. The scheduled reconciliation guarantees correctness.
export async function onTicketWebhook(event: TicketEvent) {
  await ingestRecord(await fetchTicket(event.ticketId))
  // Do NOT advance the watermark here. The scheduled run still covers this
  // window, and an advanced watermark would make it skip records whose
  // webhooks were lost — which is exactly the case the schedule exists for.
}

That comment is the trap in one line. A webhook handler that advances the sync watermark converts the scheduled run from a safety net into a no-op, and the system then depends entirely on a delivery mechanism that is explicitly best-effort.

The failure mode: the backfill that never finishes

The backfill runs. It is slow, so it is left running. Each morning it has made progress, so nobody looks closely.

What is actually happening: the API returns 429 for a burst at 09:00 when the company's other integrations wake up, the limiter penalises correctly, and throughput for those hours is near zero. The job is making progress at 40% of the projected rate, and the fourteen-hour estimate is really four days.

Nothing is broken, so nothing alerts.

// Project completion on every page, and fail the job when the projection
// drifts past what was agreed. A slow job that is five times over its estimate
// is a decision to re-take, not a job to keep waiting on.
const elapsed = Date.now() - +state.startedAt
const rate = state.recordsDone / elapsed
const projected = (state.total - state.recordsDone) / rate

if (projected > state.budgetMs * 1.5) {
  throw new IngestAnomaly(
    `backfill projection ${fmtDuration(projected)} exceeds the ` +
    `${fmtDuration(state.budgetMs)} budget. Rate-limited ${state.rateLimitHits} times. ` +
    `Ask for a higher limit or a bulk export.`,
  )
}

The error message naming the remedy matters. "Backfill is slow" gets watched; "ask for a bulk export or a higher rate limit" gets actioned, and in most companies that request is granted in a day by someone who had no idea it was needed.

Key points

Do the arithmetic before writing the adapter. A field mask that removes the detail call can turn fourteen hours into twenty minutes.
Budget half the published rate limit. It is usually per-account and shared.
One shared limiter per API per process. Several limiters is several times the rate.
Let a 429's Retry-After override your local estimate; your bucket was wrong.
The backfill and the nightly run are different programs with opposite requirements. Give the backfill a durable cursor.
Advance the cursor only past durably finished work.
Backfill oldest first, so an incomplete corpus fails visibly rather than hiding its gap in the recent data.
Webhooks reduce latency; the scheduled run guarantees correctness. A webhook must never advance the watermark.
Project completion continuously and fail when the projection blows the budget, with the remedy in the message.

Next: parsing PDFs — the stage that quietly destroys more retrieval quality than any other.