By Greg Nowak. Last updated 2026-11-12.
You remove an outdated policy from the shared document store. Later, someone asks the internal AI a question. The retired rule comes back, complete with a convincing citation.
In a retrieval-augmented generation system, or RAG, removing the file is only part of withdrawing its content. You also need to know whether the change reached the search index, the passages retrieved for questions and any stored answers.
Before accepting a fix, test three actions separately: updating a document, replacing it and deleting it. Each test should show that the answer follows what happened to the source.
Trace the document through the system
Start with a map of your actual ingestion pipeline: the source document, the connector or ingestion job, the search index, the retrieved passages and the response cache, if there is one. Include the step that splits documents into smaller searchable chunks.
For your test document, record its source identifier, version marker and indexed passage identifiers. Ask the person maintaining the integration to show how those records connect. If an obsolete sentence survives, you then have something specific to trace.
Agree on the expected result before testing. An update should produce answers that reflect the revised rule. A replacement should make the successor available and withdraw the predecessor. After deletion, the removed material should no longer support an answer.
Update: look for both versions of the passage
Azure Storage indexers illustrate why these tests need separate attention. Microsoft documents automatic change detection through storage timestamps, while deletion detection needs a deliberately configured strategy. The same indexer can handle both, but the behaviours are distinct. See Microsoft’s change and delete detection documentation.
Use a harmless test document and change one distinctive sentence. Record when you changed the source and when ingestion finished. Then inspect the retrieved passage before judging the generated answer.
You should find the revised sentence with the right source identifier and version marker. Search for the old wording too. Finding the new text proves it arrived; you still need evidence that the superseded passage is no longer searchable.
Replace: check that the old version has retired
A replacement deserves its own test, particularly if uploading the successor changes the filename, path or document identifier. Document what the integration is supposed to do with both versions.
Microsoft warns that changes to document keys or certain directory paths can break indexer tracking. Agree on how your integration recognises a replacement before depending on the upload workflow.
Use two clearly marked versions and ask a question whose answer differs between them. Inspect the retrieved passages, then search for the old identifier separately. The test needs to show that the successor is available and the predecessor has left active retrieval.
The document owner should identify which version is authoritative. That decision needs to be clear before anyone can verify that the integration is serving the right one.
Delete: find out what actually removes indexed content
Azure’s documentation describes orphaned search documents left behind when source deletions do not reach the index. It also warns that adding a deletion policy later will not clear documents deleted before that policy existed. Its suggested remedy is a new index and indexer with the policy configured from the outset.
Elastic documents a different mechanism. A full connector sync removes indexed documents that no longer exist in the source. Incremental syncs process changes since the previous sync and require a successful initial full sync. The details are in Elastic’s content sync documentation.
For your deployment, ask to see the deletion mechanism’s configuration, its last successful run and a deletion test result. A job labelled “sync” does not establish which records it removes.
| Source action | Evidence from retrieval | Evidence from the answer |
|---|---|---|
| Update a document | The revised passage has the expected source and version markers. The old wording is absent. | The answer uses the revised rule. |
| Replace a document | The successor is searchable. A separate check of the predecessor’s identifier returns no active passages. | The citation points to the authoritative version. |
| Delete a document | No passages remain for the deleted source identifier. | Both fresh and repeated questions stop quoting the removed material. |
Test cached answers separately
A clean retrieval result still leaves the cache to check. OWASP recommends invalidating cache entries when source documents are updated, deleted or have their permissions changed. Its deletion guidance also covers associated chunks, embeddings, derived indexes and cached responses. See the OWASP RAG Security Cheat Sheet.
Test a fresh retrieval and repeat a question that previously produced an answer. Record whether each request used a cache. Ask the implementation owner how stored responses are linked to their supporting sources and how those links trigger invalidation.
If retrieval returns the right material but the repeated answer still contains the retired wording, investigate the cache path next. Keep the test open until both paths meet the agreed result.
Agree how quickly changes must reach users
Set a freshness target with the content owner: how long may a source change take to appear in the answers people receive? The consequences of outdated guidance should inform that decision. Record who responds when the deadline is missed and what users should be told while freshness is uncertain.
Elastic’s connector interface supports recurring full syncs and, where supported, incremental syncs. It also shows connector health, ingestion status, errors and recent sync history. Those records help establish what ran and whether it completed, as described in Elastic’s Connectors UI reference.
Build alerts around failed jobs and missed completion deadlines. Also schedule reconciliation suited to your platform: compare authoritative sources with indexed records, then investigate mismatches. For Elastic connectors, account for the documented role of full syncs in maintaining consistency when choosing that schedule.
Keep a small set of repeatable update, replacement and deletion checks. Run them after ingestion configuration changes and save the results alongside the relevant job records. Measure freshness from the source change through to verified retrieval and answer results. A completed ingestion job is only one point in that sequence.
Give the investigation a clear scope
A focused GrN.dk engagement could map the ingestion pipeline, identify where deletion propagation breaks and establish a reconciliation schedule. Where appropriate, Python automation could compare source and index records, run the agreed checks and raise freshness alerts.
The deliverables should be practical: a pipeline map, documented behaviour for updates, replacements and deletions, evidence from all three tests and a named person responsible for failures. That gives the business a basis for deciding when its internal AI is ready to answer questions about current policy.
If your assistant still quotes a withdrawn document, bring one reproducible example. Greg can help scope the investigation across the source, retrieved passages and cached answer, and agree on checks that show whether the repair works.
Related on GrN.dk
- Apache 2.4.68: What to Check Before You Call It Patched
- Scanned PDFs and AI Search: Fix the Text Before the Chatbot
- Who Can Read That Answer? Permissions for Internal AI Search
Need help with this kind of work?
Discuss your internal AI freshness checks Get in touch with Greg.
