Before Your Support Bot Learns the Help Center, Test Whether It Can Forget
By Greg Nowak. Updated 23 August 2026.
Connecting a support bot to the help center sounds straightforward: index the existing articles, retrieve the relevant passages, and use them to answer customer questions. The harder test comes later. What happens when a policy changes, a product is retired, or an inaccurate article is unpublished?
In a retrieval-augmented generation system, adding a page usually does not train the underlying model. It creates searchable copies: extracted text, chunks, embeddings, metadata, index records, and sometimes cached answers. Removing the public page therefore does not prove the bot has stopped using it.
Before an old help center becomes the bot’s knowledge base, make deletion part of the acceptance criteria.
The important test is whether retrieval can go backwards
Most project teams test positive retrieval: can the bot find the refund policy, installation guide, or account instructions? Production readiness also requires negative retrieval: after a source is withdrawn, can the system reliably stop finding it?
This is an operational issue, not something a stronger prompt will solve. Current Azure AI Search documentation, for example, distinguishes ordinary change detection from deletion detection. Its blob indexer can detect deleted material through configured native soft delete or custom metadata. Pinecone likewise exposes explicit deletion operations using record IDs or metadata filters.
The lesson is platform-independent: removal must travel through the same content pipeline as publication, and the result must be observable.
| Checkpoint | Passing evidence | Typical failure |
|---|---|---|
| Help center | The selected article is unpublished or marked inactive | The page disappears publicly but remains in an export feed |
| Ingestion | The sync records the stable source ID as removed | The connector only detects additions and updates |
| Search or vector store | No active chunks remain for that source ID | One document produced several chunks, but only one was deleted |
| Bot response | Fresh sessions no longer use or cite the withdrawn source | A stale cache or secondary index still supplies the answer |
| Regression tests | The deletion scenario passes before every release | The team fixes one incident without keeping a repeatable test |
Run the test through the real production path
Choose one low-risk article with a distinctive answer and record its canonical URL, stable source ID, locale, owner, publication status, and last-modified value. Ask a question that reliably retrieves it, then save the answer, retrieved passages, document identifiers, and citations as your baseline.
Next, unpublish the article through the normal editorial workflow. Do not clean the vector database manually unless manual deletion is genuinely the production process. Run or wait for the same scheduled sync used in day-to-day operations.
Verify the outcome at every layer:
- Confirm that the CMS or source export reports the item as inactive or removed.
- Inspect the ingestion log to prove the connector noticed that lifecycle event.
- Query the search index or vector store by stable ID, not just by article title.
- Repeat the original customer question in a new conversation and inspect the retrieved sources.
- Test likely paraphrases, because one query returning nothing does not prove every retrieval route is clean.
Set a realistic completion window. Pinecone documents eventual consistency, meaning a short delay can occur before changed records are reflected in queries. Your test should define how long deletion may take, retry after that interval, and fail visibly if the old source remains retrievable beyond it. “Eventually” is a technical characteristic; it is not an operational deadline.
Define the correct customer-facing fallback
A successful deletion test does not require the bot to become silent. It may answer from an approved replacement, explain that the information is no longer available, or route the customer to a person. The pass condition is that the withdrawn source contributes neither facts nor citations.
Write that expected behaviour into the test. For a retired procedure, the expected answer might direct users to the current workflow. For a withdrawn policy with no replacement, escalation is safer than improvisation. This gives support and content teams control over the experience rather than leaving the model to decide.
Metadata is part of the control system
Deletion becomes manageable when every derived chunk carries a stable source ID and useful lifecycle metadata. At minimum, retain the canonical path, locale, content type, product area, publication status, version or modification time, and responsible owner.
That information lets an integration remove every chunk created from one article, identify duplicate or translated copies, and distinguish archived material from approved support content. It also makes investigations faster: an operations lead can trace a questionable answer back to its source without reverse-engineering the entire pipeline.
Do not confuse this serving-path test with a complete legal erasure process. If the source contains personal or regulated information, logs, backups, conversation histories, analytics systems, and provider retention policies may also be in scope. That requires a broader data-retention review.
Keep deletion in release testing and monitoring
Add withdrawn policies, replaced instructions, renamed products, and archived files to a small regression dataset. LangSmith’s current evaluation guidance separates offline evaluation before deployment from online evaluation of production traces; deletion benefits from both. The offline test protects releases, while production monitoring can surface an unexpected stale citation that should become the next regression case.
Someone must also own the result. Content decides when a source is no longer valid. Engineering owns propagation. Support defines the acceptable fallback. Operations tracks whether the agreed deletion window is being met. Without those responsibilities, stale knowledge becomes everybody’s concern and nobody’s task.
A useful support bot project is therefore more than model and prompt selection. It includes the content lifecycle, connector behaviour, metadata rules, deletion evidence, fallback responses, and repeatable monitoring. If your team needs help turning those pieces into a workable delivery plan, Greg can scope and manage the deletion test with you.
Related on GrN.dk
- Drupal Wiki: Build a Knowledge Base People Can Actually Use
- AI crawler policy now has verbs: separate search, RAG, and training
- AI search is eating the click: measure the queries before rewriting pages
Need help with this kind of work?
Plan a support bot deletion test Get in touch with Greg.