One index that
reads the whole web.
Answers are only as good as what the engine can retrieve. Ipnops runs its own live-web index — pages crawled, chunked into passages, embedded, and kept fresh — so /search and /answer ground on a real corpus, not a cold model. It grows on a schedule, learns from every query it serves, and reaches further when a topic is new. This is the other side of discovery: how the web gets into the index, and how your own site gets fetched by the AI crawlers writing today's answers.
# retrieval runs on the corpus Ipnops holds
{ "query": "generative engine optimization",
"mode": "bm25+vector", "reranked": true,
"hits": [{ "url": "https://…",
"passage": "…the cited evidence…", "score": 0.81 }],
"source": "index" # not the open web
}01 / 05
A model can only answer from what it can reach.
Every grounded answer starts with retrieval. If the corpus is thin, stale, or off-topic, the best model still guesses — or leans on whatever the open web happened to surface. Ipnops treats the index as the product: a corpus it crawls, embeds, and refreshes itself, so retrieval runs over pages it actually holds. When a query outruns the corpus, answering falls through to the live web and those pages are indexed on the way out. The index gets better every time it's used.
02 / 05
How the web becomes an index that retrieval can trust.
Three loops keep the corpus growing, current, and shaped by real demand — no manual curation, no one-time snapshot that rots.
Coverage crawl
A scheduled pass drains the discovery frontier, walking outward across the web graph one page at a time. Each page is fetched, stripped to readable content, chunked into passages, embedded, and stored — with thin pages and near-duplicates filtered out so the corpus stays dense, not padded.
Freshness recrawl
A scheduled pass re-fetches the stalest in-demand pages so answers stay current. Pages that get retrieved often are recrawled sooner; pages nobody asks about drift to the back. Freshness follows attention.
Learn as you serve
When a query falls through to the live web, the pages that ground the answer are indexed on the way out. The corpus learns from real demand — the questions people actually ask shape what it holds next.
03 / 05
Lexical precision and semantic recall — in one ranked pass.
Retrieval runs on ParadeDB: BM25 full-text over Postgres for exact-term precision, plus vector similarity over passage embeddings for meaning the keywords miss. The two are fused and reranked into one ordered set of passages. When the embedding sidecar is unavailable, retrieval degrades to lexical-only rather than failing — an answer still lands.
documents fetched, chunked into passages, and embedded — the same store that BM25 and vector search both run over. No second pipeline to keep in sync.
Every result carries its source URL and passage, so /answer can cite the exact evidence behind a claim. Retrieval isn't a black box — it's a ranked list of real pages you can open.
-- BM25 full-text + vector, one query
SELECT url, passage,
paradedb.score(id) AS bm25,
1 - (embedding <=> $q) AS vec
FROM chunks
WHERE passage @@@ $text
ORDER BY bm25 + vec DESC;
-- no sidecar? falls back to bm25 alone04 / 05
Point it at a domain or a question, and coverage grows.
New topic areas are seeded two ways: crawl a domain to pull it in wholesale, or fan a query across the discovery ladder to gather the best sources for a subject. Bulk seeding runs as a background job, isolated from live search, so growing the index never slows down the answers it's serving.
# pull a domain in wholesale
POST /index/seed { "domain": "example.com" }
# or fan a query across the ladder
POST /index/seed { "query": "answer engine optimization" }
# bulk seeds run as a background job,
# isolated from live /search- Coverage crawl
- Freshness recrawl
- Learn as you serve
05 / 05
And the AI crawlers indexing your site.
Indexing runs both directions. The Ipnops index reads the web; the AI answer engines read you. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest fetch your pages to write their answers — and because crawlers don't run JavaScript, that traffic only shows up in your server logs. Ipnops Discovery observes those hits, scores each page for retrieval readiness across nine engines, and applies the levers that make your site citable.
Ground your answers on a real index.
One project included, free. Query the live-web index through /search and /answer, or install the tag to see which AI crawlers are indexing your own site.