What gets lost when a government letter is rewritten in plain language?
18 real federal notices (IRS, Medicare, VA), rewritten 8 ways by Claude, with every deadline, amount, right, condition and contact the reader needs checked one by one. Results below are for the 15 test notices.
At grade 6, Claude's rewrites kept 96% of reader-critical facts. Pushing to grade 4 cost 6 points more (95% range 2 to 10), and 93% of grade-4 rewrites lost at least one critical fact.
Every other difference (bigger models, a "keep everything" prompt, the rewrite tool) was smaller than the gap between two runs of the same setup (4.6 points per letter on average).
Critical facts kept, by setup
| Critical facts kept | 95% range | Lost per letter | Letters with a loss | Reading grade | Words | |
|---|---|---|---|---|---|---|
| The original letter | 99.6% | 98.9%–100.0% | 0.1 | 7% | 8.7 | 376 |
| Haiku, grade 6 | 94.5% | 91.7%–97.1% | 1.5 | 60% | 6.0 | 390 |
| Sonnet, grade 6 | 96.2% | 93.5%–98.2% | 0.8 | 60% | 4.9 | 389 |
| Opus, grade 6 | 94.9% | 92.0%–97.2% | 1.2 | 73% | 4.4 | 436 |
| Sonnet, grade 4 | 90.2% | 86.6%–93.5% | 2.4 | 93% | 4.2 | 365 |
| Sonnet, grade 8 | 96.0% | 93.9%–97.9% | 1.1 | 60% | 5.4 | 382 |
| Sonnet, grade 6, 'keep everything' prompt | 97.8% | 95.8%–99.5% | 0.7 | 33% | 5.5 | 437 |
| Rewrite tool (Sonnet, grade 6) | 97.2% | 94.7%–99.2% | 0.8 | 40% | 5.7 | 393 |
"Letters with a loss" needs care: the judge wrongly flags 3.4% of facts that were kept, so with about 30 critical facts per letter a perfect rewrite would still show a loss 56% of the time. Reading grade is Flesch-Kincaid, measured on the rewrite; a grade says how long the words and sentences are, not whether anyone understands.
How far the judge can be trusted
Errors were planted in real rewrites that had been checked by hand, with the right answer known by construction. The judge (Claude Sonnet, checked by code for exact quotes and values) caught:
| Planted error | Caught (test) |
|---|---|
| Delete | 9/9 |
| Value | 5/5 |
| Condition | 5/6 |
| Alternative | 3/4 |
| Modal | 6/9 |
| Vague | 2/2 |
It's weakest on a "must" that becomes a "may" (the one type below the bar set in advance), and on unedited rewrites it agreed with only 1 of 5 subtle real changes found by hand, such as a 60-day deadline that dropped "in writing". So the numbers above are likely an upper bound on what's kept.
The notices
- Advance Beneficiary Notice of Non-coverage (ABN) (Centers for Medicare & Medicaid Services)
- Detailed Explanation of Non-coverage (DENC) (Centers for Medicare & Medicaid Services)
- Home Health Change of Care Notice (HHCCN) (Centers for Medicare & Medicaid Services)
- Important Message from Medicare about your rights (hospital discharge) (Centers for Medicare & Medicaid Services)
- Medicare Outpatient Observation Notice (MOON) (Centers for Medicare & Medicaid Services)
- Notice of Medicare Non-Coverage (NOMNC) (Centers for Medicare & Medicaid Services, dev)
- CP05, refund held while the return is reviewed (Internal Revenue Service)
- CP12, a miscalculation corrected and an adjusted refund (Internal Revenue Service)
- CP14, balance due (Internal Revenue Service)
- CP21B, changes to your tax return and a refund due (Internal Revenue Service, dev)
- CP503, second reminder of unpaid taxes (Internal Revenue Service)
- CP59, no tax return on file (Internal Revenue Service)
- CP75, audit of credits and part of a refund held (Internal Revenue Service)
- CP88, refund held because an earlier return wasn't filed (Internal Revenue Service)
- LT11, notice of intent to levy and your right to a hearing (Internal Revenue Service)
- Failure to provide claim information (before) (Department of Veterans Affairs, dev)
- Change in enrollment status (Department of Veterans Affairs)
- Pension claim: medical evidence needed (Department of Veterans Affairs)
How it was done
- Facts were drafted by Claude Opus, checked by code (every quote must be in the letter, every sentence accounted for) and reviewed by hand, then frozen before any rewrite was generated. "Critical" was decided by a written rule, not by the judge.
- The judge prompt was tuned on three dev notices only, then frozen; the planted-error gate ran once on test notices.
- Everything is in the repository: code, data, every rewrite and every verdict. Total cost about $26.