Method · 9 min
A reliable document assistant:
the checks that make the difference.
A practical protocol for building, testing and operating an assistant that cites sources and respects permissions.
Key takeaways.
- 01
A good model cannot compensate for outdated, poorly segmented sources or access granted to the wrong people.
- 02
Answers must show their sources, know when to abstain and retain a usable audit trail.
- 03
Quality is measured with known-answer questions, unanswerable questions and permission tests.
A reliable answer is a chain of checks.
In a document assistant, the model should not improvise solely from its own memory. The question is associated with the user, permissions are checked, relevant passages are retrieved and the answer is then written from that context.
Each step can fail independently. Quality therefore needs to be diagnosed across the whole chain, not just in the final text.
Seven checks to make visible.
These checks turn a convincing demonstration into an operational professional tool.
- Authorised sources: clear inventory, identified owner and update date.
- Permissions before search: a prohibited passage must never enter the model's context.
- Controlled index: managed segmentation, metadata and versions.
- Citations: a document, passage or link for every important claim.
- Abstention: an explicit answer when sources are insufficient.
- Evaluation: a versioned question set rerun after every change.
- Operations: logs, alerts, user feedback and an owner responsible for fixes.
A minimum protocol before opening access.
Build a test set with business teams. Keep it small enough to rerun often, but varied enough to expose failures.
- 01
20 questions with known exact answers and sources.
✓ - 02
10 questions the documents cannot answer.
✓ - 03
5 attempts to access content restricted to another role.
✓ - 04
5 cases involving old, ambiguous or contradictory documents.
✓ - 05
A defined acceptance threshold for citations, abstention and permission leaks.
✓ - 06
Human review of errors, with their causes and corrective actions.
✓
Measure what the user can verify.
Track the proportion of answers supported by a valid source, abstention quality, permission incidents, response times and requested corrections. A single overall score often hides the actual problem.
Helpful / not helpful feedback is useful, but does not replace factual checking. Fluent text can still be wrong; a brief answer with a good source can be much more useful.
DECISION POINTObjective: an answer that can be traced and challenged. The assistant supports decisions; it does not remove human responsibility.
Check and explore further.
We prioritise official texts and reference frameworks. This guide's recommendations are our practical interpretation of those sources, to adapt to your context.
- 01NIST — AI Risk Management Framework ↗
Structured approach to governance, measurement and risk management.
- 02ANSSI — Security of generative AI systems ↗
Recommendations for architecture, data and operations.
- 03CNIL — Personal data security ↗
Organisational and technical measures to protect data.