Independent web product
AnimeHalo
A live discovery catalog built from sources that disagree on names, use different identifiers, and go offline mid-import. I built the data pipeline and the web experience for more than 17,000 anime titles.
Recover a catalog release.
Take over a stalled import. Inspect the provider response, recover a checkpoint, and choose when readers see the new title data. The console uses twelve fictional records and a simulated clock.
Preparing the release workspace…
External integration
Query the real provider.
Request one public media record from AniList’s GraphQL endpoint. This sends a request from your browser and displays the returned title and episode count.
Ready to send a GraphQL request.
Implementation
Publishing while readers stay online
AnimeHalo’s packer checks that imported enrichments were built against the same Wikidata snapshot. Its story files are partitioned by title key and named from their content. The downloadable example combines these mechanisms with conditional checkpoint and release writes.
if (joinHash !== snapshot.hash ||
point.snapshotHash !== snapshot.hash) {
throw new SnapshotMismatch("Mixed source generations");
}
const chunks = await Promise.all(parts.map(async records => {
const body = JSON.stringify(canonical({ schema, records }));
const hash = await digest(body);
if (!store.blobs.has(hash)) store.blobs.set(hash, body);
return hash;
}));
const id = await digest(JSON.stringify({
schema, partitions, chunks, source: snapshot.hash
}));From provider IDs to a ranked catalog
I wrote importers for AniList, Wikidata, TMDB, and TVmaze. The AniList importer prefers a numeric ID from a matching source URL, falls back to Wikidata’s non-deprecated P8729 identifiers, and remembers titles missing from a successful response. It respects provider rate limits and periodically replaces its checkpoint file with a completed write.

Source explorer.
Follow the public implementation from integration logic to failure tests.
animehalo.mjsCheckpoint a page
A rate limit persists its retry time. A successful response advances the cursor only after every row has been validated.
export async function ingestNext({ store, snapshot, readPage, now = Date.now, pageSize = 3, beforeCommit }) { if (!Number.isSafeInteger(pageSize) || pageSize < 1) throw new RangeError("Invalid page size."); const saved = store.readCheckpoint(); if (saved.snapshotHash !== snapshot.hash) throw new SnapshotMismatch("The cursor belongs to a different input snapshot."); if (now() < saved.blockedUntil) return { kind: "held", until: saved.blockedUntil }; const page = snapshot.titles.slice(saved.cursor, saved.cursor + pageSize); if (!page.length) return { kind: "complete" }; const ids = page.map(item => item.id); const reply = await readPage(ids); if (reply.status === 429) { const until = now() + retryDelay(reply.retryAfter, now()); store.writeCheckpoint(saved.revision, { ...saved, blockedUntil: until }); return { kind: "throttled", until, ids }; } if (reply.status !== 200) throw new Error("Provider rejected the page (HTTP " + reply.status + ")."); const media = validatePage(ids, reply.rows); if (beforeCommit) await beforeCommit(); store.writeCheckpoint(saved.revision, { ...saved, cursor: saved.cursor + page.length, media: { ...saved.media, ...media }, blockedUntil: 0 }); return { kind: "committed", ids, missing: ids.filter(id => media[id] === null) };}