EEOC charge intake: LLM extraction with source spans and human review

The problem

An EEOC charge of discrimination is the first document in a large share of employment-defense matters. Someone reads the charge and the notice letter, keys a dozen fields into a matter system, calculates the position-statement deadline and writes a short summary of the allegations. It is routine, it is repetitive, and a wrong date is expensive.

This prototype tests whether a language model can do the first pass reliably enough to be worth a pilot — and, more importantly, whether we can tell when it gets something wrong.

Constraints

• No real charges. The test set is synthetic: generated charge forms and notice letters with known ground truth, including deliberately ugly cases (missing charge number, rotated scans, multi-page particulars, two respondents).
• The model never guesses. A field it cannot support with a passage in the document comes back as null with a reason.
• Nothing leaves the tool without a human accepting it field by field.

Planned architecture

1. Synthetic data generator produces ~30 charge-plus-notice PDF pairs and a ground-truth JSON for each.
2. Extraction service (FastAPI) renders each PDF to text with page offsets, prompts the model for a fixed JSON schema, and attaches to every field a source span (page number and quoted text) plus a confidence.
3. Span verification checks that each quoted span actually appears in the document text. A field whose span can’t be found is downgraded to null before a reviewer ever sees it.
4. Review screen (React) shows the PDF on the left and the fields on the right, with accept, edit and reject per field. Export is enabled only after every field has a decision.
5. Model adapter swaps providers by configuration so the same evaluation runs against two approved models.

Fields

Charge number, charging party, respondent, EEOC office, bases, issues, earliest and latest dates of harm, the continuing-action flag, the position-statement due date from the notice, and a three-sentence summary of the particulars.

Evaluation plan

The results table at the top of this page fills in when the suite runs. It will report, for each of two models:

Measure	How it is scored
Field-level accuracy	Exact match against ground truth, per field and overall
Unsupported-output rate	Share of non-null fields whose cited span is not in the document
Null correctness	Precision and recall of null where the ground truth is absent
Latency	p50 and p95 per document
Cost	Per document, at list prices on the day of the run

Each change between versions is recorded in the changelog below with its effect on these numbers.

Human review points

Every field. The reviewer sees the model’s quoted evidence next to the value, and the export button is inert until each field is accepted, edited or rejected.

Security and logging

Secrets come from environment variables; the repository ships an .env.example and nothing else. Every run writes a structured log with a request id, model, token counts, cost, and per-field outcomes. The synthetic documents contain no real names.

Known limitations going in

• Synthetic charges are cleaner than scanned reality; OCR quality on real intake will be the first thing a pilot has to measure.
• Deadline calculation from the notice is a rule, not a model output, and it depends on correctly extracting the notice date.

Changelog

• v0.0 — Oct 13, 2026. Plan published. Build starts.