A large API import should save its next-page checkpoint only after the current page is safely committed. Pair that checkpoint with stable record keys and replay-safe writes, so a timeout or restart can repeat work without silently skipping records or creating duplicates.
A checkpoint is a promise about what is already safe
A large API import should save a checkpoint only after the records covered by that checkpoint are safely stored. If the process stops earlier, it should retry that work; if it saves progress too soon, a restart can skip records that never reached the destination.
A checkpoint is the source’s continuation state: perhaps a page number, an offset, or an opaque cursor returned by the API. It is not necessarily a record count. Treat it as a boundary the source provides, and advance it only when your application can prove the corresponding page has been handled.
Find out what the source guarantees before choosing a design
Read the API’s pagination behavior and test it with real responses from a non-production environment where possible. Determine whether the next request uses a cursor, a page number, or a link; whether a cursor expires; and whether the source can add or change records while an import is underway.
Offset pagination can be vulnerable to shifting results: if records are inserted near the beginning between requests, later offsets may no longer point to the same items. A cursor or stable sort key can reduce that risk when the source supports it, but no pagination method should be assumed to provide a consistent snapshot unless the API documents that behavior.
Make replay safe with a stable record key
Most reliable imports are designed for at-least-once processing: a page might be requested or processed again after a timeout, even if the previous attempt partly succeeded. Make that repeat harmless. Use a stable source identifier, such as the source system’s record ID, to update or insert the corresponding destination record rather than blindly creating a second copy.
Define what an update means. For example, decide whether a later source value replaces a locally edited field, whether blank source values erase existing data, and how deletions or inactive records are represented. If multiple systems can edit the same information, establish field ownership instead of allowing import order to decide.
Commit records and checkpoint as one controlled step
A practical flow is to fetch one page, validate its structure and records, write the page, and then persist the returned continuation state. Where the database supports transactions for the relevant writes, commit the records and checkpoint together. That prevents a crash between the two operations from making progress appear more complete than it is.
If a single transaction is impractical, make record writes idempotent and store enough import state to reconcile a restart. Keep the checkpoint, import run identifier, timestamps, and page-level outcome in a durable place—not only in process memory. An expired cursor should trigger a deliberate recovery path, such as restarting from a safe boundary and deduplicating by source key, rather than silently continuing with guessed state.
Retry temporary failures, but stop on problems that need attention
Network timeouts, temporary service errors, and throttling may be recoverable. Retry them with a bounded delay strategy and a maximum attempt or time limit. Honor any retry guidance the API provides, and avoid launching overlapping workers for the same import unless the design coordinates their checkpoints and writes.
Invalid authentication, a changed response shape, malformed records, or a permission failure is different: repeating the same request may not help. Record the failure with enough context to diagnose it, retain the last confirmed checkpoint, and route the issue for correction. Avoid logging secrets or unnecessary personal data.
Verify recovery under the conditions that cause data loss
Test a failure after some records are written but before the checkpoint advances. Restart the job and confirm that repeats do not create duplicates or overwrite protected fields. Also test a failure after checkpoint persistence, repeated page responses, an empty final page, a throttled request, and a cursor that is no longer valid.
Compare source and destination using stable identifiers, not just total row counts; totals can match while one record is missing and another is duplicated. Track pages attempted, records accepted or rejected, retries, last successful checkpoint, and time since the last completed run. A clear mismatch or stalled run should produce an actionable alert.
When a checkpointing system is unnecessary
For a small import that reliably completes in one request, or a task that can safely rebuild a disposable destination from scratch, durable page checkpoints may add needless complexity. A simpler scheduled job with clear failure reporting may be enough.
For a long-running or business-critical import, first document the source’s pagination guarantees, the destination’s duplicate policy, and the recovery procedure. That short design check is a concrete next step: it reveals whether you need cursors, transactions, a reconciliation process, or simply a restartable import.
Before you run a resumable import
- Confirm how the source defines the next page or continuation point, and whether that value can expire.
- Choose a stable record key and decide how the destination handles an already-imported record.
- Persist the checkpoint only after the page’s records are safely committed.
- Set bounded retries and distinguish transient failures from invalid data or authorization errors.
- Test interruption, restart, repeated pages, and changing source data before relying on the import.
Questions about this approach
Are cursors always safer than page numbers or offsets?
A stable cursor or source-defined continuation token is often safer than a simple offset when the source supports it, because changing records can shift offset-based pages. Check the API’s documented ordering and consistency guarantees; a cursor alone does not prove the import is a snapshot.
How can an import avoid duplicate records after a restart?
Usually, the destination’s stable source identifier is the best basis for repeat-safe writes. Decide explicitly whether an existing record is updated, left unchanged, or flagged for review, and protect fields the import is not allowed to own.
What should happen if the API cursor expires?
Keep the last confirmed checkpoint and follow a defined recovery policy. If restarting from an earlier boundary, deduplicate by source identifier and reconcile the result. Do not invent a new cursor or advance past data that has not been verified.
For help applying this approach, explore our API Integration & Custom API Development services.
Discuss your project with Full Blown Studio
Tell us what needs to work better and what you have tried. Discuss your project, email bob@fullblown.com, or call (661) 429-0940.
