The model didn't get worse. The ground under it moved.
Most AI workflow failures get framed as a model problem or a prompt problem. A quieter failure mode sits one layer down: the documents a workflow retrieves from drift out of date, and nothing in the system is built to notice.
If a workflow answers pricing questions, applies a policy, or drafts a recommendation from retrieved context, its accuracy is bounded by the freshness of that context. A perfect model reasoning faithfully over an outdated document still produces a wrong answer, and it will do so with the same confident tone as a correct one.
Why retrieval doesn't know a document is stale
Embedding-based retrieval ranks candidates by similarity to the query. Similarity has no built-in concept of recency: a well-written, thoroughly-indexed old policy page can score higher than a thin, recently-added correction simply because it matches the query's wording more closely.
Deleted or superseded source documents are a related gap. Unless a pipeline explicitly removes or flags their vectors when the source is retired, those embeddings stay in the index and keep competing for retrieval slots indefinitely.
A broader review of retrieval-augmented generation research frames RAG as a way to offset a model's outdated internal knowledge with external sources, while flagging retrieval quality and integration overhead as open, ongoing challenges rather than solved problems.
Standard checks don't catch this failure
A workflow pulling from a stale document is often faithfully summarizing exactly what it retrieved. Output-quality checks that score whether the answer matches the retrieved context will still pass, because the model did its job correctly on the wrong input.
This is why the failure tends to surface downstream, in a customer complaint about a changed fee, a compliance question about a retired policy, or an operator asking why the system cited last year's terms, rather than in a dashboard.
- Faithfulness and groundedness scores check the answer against what was retrieved, not whether what was retrieved is current.
- High-stakes categories (pricing, compliance, eligibility) are the ones most worth a scheduled spot check against the live source.
Treat the knowledge base like a production dependency
The fix is operational, not a smarter prompt. Decide how source changes reach the index, how deletions are handled, and who is accountable when a document goes stale between syncs.
Cloudflare's managed retrieval product is one concrete example of this being treated as infrastructure rather than an afterthought: its documentation describes scheduled sync jobs that process new, modified, and deleted files on a configurable interval, with manual or webhook-triggered syncs available for changes that can't wait for the next cycle.
A workflow doesn't need that specific product to apply the same pattern: pick a sync interval appropriate to how often the underlying documents actually change, process deletions as a first-class event rather than an edge case, and run a periodic full reconciliation to catch anything the incremental path missed.
- Match the sync interval to how fast the source actually changes, not to a default nobody revisited.
- Handle deletions explicitly; an orphaned vector for a retired document doesn't remove itself.
- Run a periodic full re-index as a backstop against missed or failed incremental updates.
Freshness needs an owner
A knowledge base without an assigned owner degrades the same way an unmonitored database does: slowly, invisibly, and in a way nobody notices until a downstream decision is visibly wrong.
Name who is accountable for the source documents, how changes get flagged to them, and what the recovery path is when a sync fails. That ownership line is cheap to put in place before launch and expensive to reconstruct after a bad answer has already reached a customer.