Retrieval quality is a knowledge-management problem before it is a model problem: sources need ownership, permissions, freshness controls, and a way to show users where an answer came from.
Chat is the last layer, not the foundation
Retrieval-augmented generation can make approved material easier to use, but connecting a folder to a chat box does not establish a reliable knowledge service. An answer can sound confident while relying on a superseded policy, a partial document, or material the user should not see. The first design decision is therefore not chunk size or model selection. It is the boundary of the knowledge product.
Map the knowledge supply chain
For each intended answer, identify the authoritative source, its owner, update frequency, audience, sensitivity, and approval state. Then decide what the system will do when the source is missing, stale, contradictory, or outside the user’s entitlement. A useful assistant cites its source, flags uncertainty, and declines to invent an answer when evidence is insufficient.
| Control | Implementation question | Evidence for release |
|---|---|---|
| Source authority | Which version is the approved answer? | Owner and review date recorded. |
| Access | Who may retrieve it? | Permissions tested with representative roles. |
| Freshness | How does an update reach the index? | Documented publish-to-index path. |
| Citation | Can a user inspect the answer basis? | Answer returns source links or identifiers. |
| Gaps | What happens when evidence is absent? | Test cases confirm abstention and escalation. |
Design retrieval around the user’s decision
A support analyst, implementation consultant, and executive often require different depth from the same body of knowledge. Design retrieval with the task in mind: a response may need a concise policy answer, a cited procedure, or a structured case summary. The output should state its limits rather than blend sources into a generic narrative.
Evaluation needs a gold set
Build a small reviewed set of real questions: common requests, policy edge cases, questions with conflicting sources, and requests that should be refused. Score source relevance, citation accuracy, completeness, and appropriate abstention separately. This produces a diagnostic: poor retrieval requires knowledge work; poor grounded synthesis may require prompt or model work; poor access behavior requires a security fix.
Architecture pattern
Authoritative source → approval and metadata → permission-aware index → retrieved evidence → constrained response → user feedback → content owner.
This keeps systems of record and subject-matter ownership in control while making knowledge usable in context.
Sources and further reading
- NIST AI Risk Management Framework — a practical risk-management vocabulary for AI systems.
- OWASP Top 10 for LLM Applications — implementation risks to test and control.
- NIST AI RMF Playbook — actions for governing, mapping, measuring, and managing AI risk.
Turn the framework into a working plan
SoloSoft can help define the workflow, system boundaries, success measures, and delivery sequence before implementation begins.