The Answer Was in the Old Handbook: Keeping an AI Assistant Current

By Greg Nowak. Updated 31 October 2026.

Imagine asking your internal AI assistant how to approve a supplier invoice. It gives a tidy answer, complete with a handbook citation. Unfortunately, that handbook describes last year's approval process.

The citation makes the answer look dependable. It only shows where the information came from. For a business owner or operations lead, the real question is whether staff can act on it today.

Keeping an internal AI assistant current means managing approved documents, searchable copies, permissions and answer checks as one operating process. Here is how I would scope that work.

Why an updated handbook can still produce an old answer

Many document assistants use retrieval augmented generation, or RAG: they search selected material and pass relevant passages to a model to write an answer. That searchable material may be a separate copy of your files, split into smaller passages.

Replacing a PDF in a shared folder does not establish that every old passage has disappeared. Nor does a recent upload prove that a policy is approved or already in force. The assistant needs an explicit way to select the version that applies.

Decide which documents are authoritative

Start with a small source register. For each document, record a stable identifier, its business owner, approved version, effective date, review date, audience and status. Keep the identifier consistent through renames so the technical team can trace all indexed passages back to their source.

Separate drafts, approved material, superseded versions and withdrawn documents. If procedures differ by country, department or client, record that scope too. An agency's client-specific instructions should never become the default for another account.

The document owner decides what is authoritative. The technical owner makes that decision take effect in search. A review date should trigger an owner check; agree whether an overdue review also suspends the document. Otherwise, a date field can create false reassurance.

Make each document change a complete workflow

Use the following as an acceptance checklist for whoever builds or maintains the assistant. Agree how quickly each change must take effect, based on what people might do with the answer.

Keep a simple change record linking the source version, indexed identifiers, processing result and reviewer. Where practical, test a replacement collection before switching live queries to it. If the previous procedure is no longer valid and the replacement fails, route the question to its owner instead of serving obsolete instructions.

Check deletion separately from refresh

Two implementation details matter here. Microsoft's Azure AI Search indexer guidance states that resetting and running an indexer does not remove orphaned search documents whose sources no longer exist. Use the appropriate deletion mechanism and verify the affected document identifiers.

The official OpenAI retrieval documentation says file removal from a vector store is eventually consistent: removed content can briefly remain in search results. File processing is also asynchronous, so check completion before relying on a replacement. Attributes and search filters can help select approved material; store expiration does not establish whether a policy is current.

For urgent withdrawals, design an application-level exclusion that takes effect before passages reach the model, or temporarily disable the affected collection. A search for distinctive old wording is a useful extra check, but one empty result does not prove every passage was removed.

Enforce access before content reaches the model

A current handbook can still be inappropriate for the person asking. Resolve identity and group membership in trusted application code, then enforce document permissions on every retrieval request. Treat indexed copies as protected information.

Microsoft's security-filter guidance makes a useful distinction: an identity string in a filter does not authenticate anyone. The application must supply and enforce the correct filter. Test with a user who should be denied, including after access is revoked.

Also control who can publish searchable documents. OWASP's vector and embedding guidance identifies both access leakage and poisoned sources as risks. Include a controlled test document containing instructions aimed at the assistant, and check whether those instructions improperly change its behaviour.

Retest the questions people actually ask

Keep a small set of realistic questions for each important procedure, with the expected answer, approved source and permitted audience. Rerun them after revisions, withdrawals and permission changes. Inspect retrieved passages as well as the final wording.

Test what happens when sources conflict or no approved answer exists. The assistant should explain the gap and direct staff to the responsible owner. Check ongoing conversations and cached answers too: an index update cannot retract information already supplied.

Start with one collection and a named owner

Choose one useful collection, such as operations procedures, and follow a real document change through to a verified answer. Budget for ownership, connector maintenance, failure alerts and review time alongside the initial build.

I can help audit your sources, define the update and withdrawal workflow, and turn these checks into acceptance criteria for your internal team or agency. Get in touch with Greg to discuss one collection you need staff to trust.

Related on GrN.dk

• Who Can Read That Answer? Permissions for Internal AI Search
• Why Your Website’s Third-Party Stack Needs a Real Owner
• Your AI Reads Supplier PDFs. Can They Rewrite Its Instructions?

Need help with this kind of work?

Discuss your internal knowledge assistant Get in touch with Greg.

Sources

• OpenAI: Retrieval
• Azure AI Search: Run or reset indexers, skills, or documents
• Azure AI Search: Security filters for trimming results
• OWASP LLM08:2025 Vector and Embedding Weaknesses