Before Your Support Bot Learns the Help Center, Make Sure It Can Forget

By Greg Nowak. Updated 24 October 2026.

Your support bot can answer questions from the help center. Before signing off the project, ask the delivery team to demonstrate what happens when one of those articles becomes wrong.

A refund policy changes. A product is retired. Someone withdraws a workaround that customers should stop using. Removing the page is straightforward; making sure the bot stops recommending its advice takes more coordination.

For business owners, operations leads and agencies commissioning support automation, this deserves a place in the acceptance criteria. Run one article through the normal withdrawal process, measure how long removal takes, and check what customers hear afterwards.

What does “forgetting” mean for a support bot?

Many help center bots use retrieval-augmented generation, or RAG. They search stored copies of your content and pass relevant passages to a language model. Connecting an article usually does not retrain the model itself.

One page can produce extracted text, several searchable chunks and records in both keyword and vector indexes. Answers may also be cached. A page disappearing from the website proves only that the website has changed.

Define the goal precisely: the withdrawn source must stop reaching the answer process, and the bot must follow the current approved guidance. An answer resembling the old article does not, by itself, prove retrieval failed. Inspect the retrieved passages and conversation context as well.

Agree on the evidence before testing

Ask for a short test report with timestamps and traceable record IDs. A reassuring answer in one chat window is useful, but it cannot establish that every retrieval route has been cleared.

Run one withdrawal through the real workflow

Choose a low-risk article with a distinctive answer. For a live bot, agree the test with its content owner so customers retain access to necessary guidance. Use a controlled test article if withdrawing real content would create confusion.

1. Capture the baseline. Record the stable source ID, canonical URL, locale, version and publication status. Save a question that retrieves it, together with the answer, citations, retrieved passages and chunk IDs.
2. Withdraw it normally. Use the editorial process planned for everyday operation. Record the withdrawal time. Manually cleaning the database would bypass the connector you need to test.
3. Follow the sync. Check the CMS API or export, ingestion logs and deletion job. Confirm that an unpublished article cannot still enter through an alternative feed.
4. Inspect every serving path. Check associated record IDs or exact source metadata across vector search, keyword search, relevant namespaces and answer caches. Include translations or duplicate pages that carry the withdrawn advice.
5. Test customer questions. Repeat the baseline question and several paraphrases in fresh conversations. Inspect retrieval traces as well as the visible answer.
6. Run the next sync. Confirm that a scheduled import or retry does not recreate the removed records.

Measure the deadline from editorial withdrawal to verified removal from customer answers. Starting the clock only when the database receives a delete request hides connector delays.

Give every chunk an identity you can remove

Keep a stable source ID on every derived record, alongside its locale, version, publication status and canonical URL. Maintain a way to find all chunk IDs belonging to that source. Titles and URLs can change; neither should be your only link back to an article.

For sources that disappear completely from an API, preserve a withdrawal record or reconcile the index against an authoritative inventory of published articles. Otherwise, a connector that sees only additions and edits may never notice the missing page.

Platform behaviour needs checking. Azure AI Search’s storage-indexer documentation distinguishes automatic change detection from deletion detection, which needs a configured strategy. It says to establish that strategy from the first indexer run; adding it later does not clean up previously orphaned documents. Its documented soft-delete approaches also exclude certain one-to-many indexing scenarios, which require explicit index deletion requests.

Pinecone supports deletion by ID or metadata filter within a namespace. Use known IDs where practical, or an exact filter tied to the source identity and intended scope. Deleting one chunk is insufficient when the article produced several.

Do not treat a successful delete response as proof that queries have caught up. Pinecone documents eventual consistency and, for supported serverless operations, log sequence numbers to check whether queries reflect a deletion. Aggregate record counts alone cannot prove that a particular article is gone.

Decide what customers should hear afterwards

Write the expected response before running the test. If an approved replacement exists, the bot should use it. If a policy has been withdrawn without replacement, it should acknowledge the gap and offer an appropriate support route.

Test an existing conversation too: earlier messages may already contain the obsolete instructions. Decide how the bot should correct them when the customer asks a follow-up. Fresh-session success does not cover that case.

This test verifies withdrawal from live answers. Full erasure from logs, backups, conversation histories or training data is a separate scope.

Make withdrawal part of the handover

Keep the questions as regression tests, updating expected answers when policy changes. Retain a withdrawn source ID as a negative check rather than leaving obsolete advice labelled as the correct answer. LangSmith’s evaluation guidance distinguishes offline regression testing from online production monitoring; both are useful here.

The handover should name who withdraws content, who fixes propagation failures, who approves fallback answers and who receives deadline alerts. Include a temporary way to block affected answers if removal stalls.

If you are commissioning a support bot or reviewing an agency delivery, Greg can help turn these checks into a practical acceptance plan, with clear ownership, evidence and launch criteria.

Related on GrN.dk

• Agentic AI: How It Works and When It’s Worth Using
• AI Search Is Eating the Click: Measure Before You Rewrite
• Your AI workflow has logs. Can they explain one bad decision?

Need help with this kind of work?

Plan your support bot acceptance tests with Greg Get in touch with Greg.

Sources

• Change and delete detection using indexers for Azure Storage in Azure AI Search
• Delete data — Pinecone Docs
• Check data freshness — Pinecone Docs
• Evaluation types — LangSmith Docs