MyDot Consultancy

Featured image: AI-generated conceptual illustration. Sources checked 26 October 2026.

A knowledge assistant can cite a document and still give the wrong answer. It may retrieve the wrong policy, overlook an exception, use an outdated version or combine two passages that apply to different employees. Retrieval-augmented generation helps connect an answer to external information, but the connection needs to be evaluated.

RAG is a useful label for a pipeline that retrieves relevant material and uses it when generating a response. It is not a promise that the material is authoritative or that the response accurately represents it. Business teams should examine the retrieval step and the answer step separately.

Follow a policy question through the pipeline

Consider a fictional employee asking whether the company will reimburse a home-office monitor. The knowledge base contains a general equipment policy, a country-specific allowance, an old onboarding guide and a temporary exception for a particular team. Several documents mention monitors. Only some apply to this employee now.

The retrieval system searches for useful passages. The OpenAI retrieval guide describes semantic search, which can return conceptually related material even when wording differs. That is helpful when an employee asks about a “screen” and the policy says “display equipment.” It does not decide which policy has authority.

The generation step receives selected material and produces an answer. If the right passage was not retrieved, the model may lack the evidence needed to answer correctly. If the right passage was retrieved alongside conflicting material, the model still needs rules and context to interpret the conflict. Both failures can sound equally confident.

Diagnose the right layer

When an answer is wrong, first inspect the material supplied to the model. Was the applicable policy present? Was the effective date available? Did the query include enough information about country, employment status or expense category? A prompt change cannot reliably compensate for an index that never returns the relevant document.

If the evidence was present, inspect how the answer used it. Did the assistant overlook a condition, substitute a different employee category or state an inference as if it were a quoted rule? Those are answer-construction problems. Keeping the layers separate makes remediation more specific than simply asking for a “better model.”

Also consider the user’s question. “Can I expense this?” may require a product category, purchase date or location. A responsible assistant should ask for missing facts when they change the answer. It should not fill them in from a stereotype or from a prior conversation that may no longer apply.

Make source authority explicit

Assign metadata that helps the system distinguish current policy from historical discussion. Useful fields can include owner, effective date, region, document status and superseding version. The retrieval guide documents attribute filtering; how those filters map to your policy hierarchy is an application design decision.

Do not use the assistant to quietly resolve an organizational dispute. If two current policies contradict each other and no precedence rule exists, surface the conflict to the owner. The model can identify the disagreement and prepare a concise explanation, but it should not invent the rule the business has failed to establish.

A good answer might say that the general policy permits the item but the applicable regional document requires approval, with links to both passages. A poor answer collapses those conditions into a universal yes. The difference comes from evidence handling and decision rules as much as language quality.

Treat citations as a verification aid

Citations should help a reader inspect the exact support for a claim. A link to a long handbook is less useful than a link or reference that identifies the relevant section. Where the platform supports it, retain document identifiers and locations through the retrieval and generation steps.

Check that each important claim is actually supported. A genuine citation can be attached to an unsupported sentence. Reviewers should be able to distinguish policy text, the assistant’s interpretation and advice about next steps. If the assistant cannot establish the answer, it can still identify the documents reviewed and explain what remains unresolved.

Keep retrieval inside the access boundary

A shared index can contain material that different users are allowed to see. Access filtering needs to happen before unauthorized content reaches the answer context, not merely before a citation is displayed. The same concern applies to snippets, summaries and cached responses.

OWASP’s vector and embedding weaknesses guidance identifies access-control and cross-context leakage risks. For the equipment-policy example, that means a general employee should not receive a confidential executive exception simply because it is semantically similar to the question.

Test answerable and unanswerable cases

• A straightforward question with one current authoritative answer.
• A question whose wording differs from the policy terminology.
• A case requiring an exception or regional condition.
• A question for which the knowledge base has no answer.
• A conflict between current-looking documents.
• A request whose relevant document is outside the user’s access.

For each case, record expected retrieval and expected response separately. Include the correct outcome “insufficient evidence” when appropriate. The Anthropic evaluation guide supports the broader practice of defining tasks and grading logic rather than relying on a handful of attractive examples.

Improve the knowledge service, not just the answer box

A reliable RAG application needs document ownership, indexing operations, permission checks and a route for correcting errors. Give users a way to flag an outdated source or unsupported answer, and route that report to someone who can repair the underlying issue.

Retrieval can make an assistant substantially more useful by supplying relevant evidence. The business value comes from a maintained chain between the question, the authorized source and the answer. Design and test that chain explicitly, and the assistant can be helpful without pretending that finding a related paragraph is the same as establishing the truth.

Sources and further reading

OpenAI: Retrieval. Semantic retrieval, vector stores and attribute filtering; checked October 2026.

OWASP: Vector and Embedding Weaknesses. 2025 guidance on retrieval access and cross-context risks; checked October 2026.

Anthropic: Demystifying evals for AI agents. Task and grading methodology; published January 2026.