Grounded Q&A over approved employment-law sources, with abstention
A retrieval assistant that answers only from an approved corpus, cites the passage inline, and says "not supported by the approved sources" instead of guessing.
The problem
Knowledge retrieval is the most requested and most misused kind of AI assistant. The failure that matters isn’t a wrong answer that looks wrong — it’s a fluent, confident answer with no basis in the firm’s own guidance. This prototype tests whether an assistant can be built to refuse exactly those answers, and how to measure that it does.
Constraints
- Public-domain corpus only for the prototype: Department of Labor wage-and-hour fact sheets, EEOC guidance, and one state’s leave statute — roughly 150–300 pages with section and date metadata.
- Every answer carries inline citations to passages the reader can open.
- Questions outside the corpus get an abstention, not an improvisation.
- Approved sources are managed on an access-controlled admin page, not in a config file.
Planned architecture
- Ingest splits documents by section, keeps the source, section heading, effective date and page, and indexes them for both keyword (BM25) and embedding search.
- Hybrid retrieval merges the two rankings; the top passages are handed to the model with the instruction to answer only from them and to cite each claim.
- Abstention check rejects any answer whose claims aren’t supported by the retrieved passages.
- Chat interface with a sources panel and click-to-open passages.
- Admin page behind a login where approved sources are added and removed, with every change logged.
Evaluation plan
Around 40 questions in three groups: 25 answerable questions with gold passages, 10 that are unanswerable or out of scope, and 5 adversarial (false-premise questions and an instruction planted inside a document). Two retrieval configurations run against the same set.
| Measure | How it is scored |
|---|---|
| Answer correctness | Rubric graded, with an LLM judge checked against a hand-graded sample |
| Citation precision | Share of citations that actually support the claim they’re attached to |
| Abstention accuracy | Correct refusals on unanswerable questions, false refusals on answerable ones |
| Retrieval recall@k | Whether the gold passage is among the top k retrieved |
| Latency and cost | p50 and p95 per question; cost per question |
Human review points
Answers are advisory and every claim is one click from its source. The admin page is the control point: only an approver can change what the assistant is allowed to read.
Known limitations going in
- Public guidance is not the firm’s guidance. The obvious next corpus is internal, and the ingest step is built to accept it without changes.
- Effective dates matter in employment law; the first version surfaces them but doesn’t yet reason about them.
Changelog
- v0.0 — Oct 13, 2026. Plan published. Build starts.