Evaluation · pilot stage
Testing the referral note drafter
The note drafter writes a referral note from a patient’s chart and a few lines typed by the primary care doctor. This page explains how we test whether those notes are accurate and useful to the specialist, and what the first run found.
Where it stands on 21 November 2026
Only the pilot has run: 3 of 33 cases, 9 notes. There is no pass or fail score yet.
The judges’ scores on this page were produced before we corrected what the judges are shown, so they are provisional. They will be replaced when the next stage runs.
- 9notes written3 synthetic patients, each drafted by 3 Claude models
- 0 of 9inside the word limitThe drafter is asked for 250 to 400 words
- 11 of 11bad notes caughtNotes broken on purpose to test the judges. A twelfth check was cut off.
- $3.31spent so farof a $50 budget for the whole evaluation
- Pilot3 cardiology cases, 9 notes. A dry run to find faults in the test itself.Done on 21 November 2026
- Build15 cardiology cases. Used to tune the prompt and the rubric. Clinicians score these notes by hand.Not run yet
- Carry-over4 nephrology and 4 orthopedic cases. Checks that the tuning holds outside cardiology.Not run yet
- Test10 held-out cases, 3 drafts each. This stage gives the score.Sealed until the prompt and rubric are frozen
Results so far
What nine notes showed
Nine notes from three patients are enough to expose problems. They are too few to rank the models or to stand in for a score.
Every note ran over the word limit
The drafter’s instructions ask for roughly 250 to 400 words, about one page. All 9 notes exceeded that, at 411 to 563 words. A program counts the words, so this finding does not depend on the judges.
- Words up to the limit
- Words over the limit
- The limit, 250 to 400 words
- 480 words: beyond this the format score is 0 (proposed)
Patient 1 · chest tightness on stairs · routine
Patient 2 · stopped a statin after muscle aches · routine
Patient 3 · worsening breathlessness · urgent
Format is one of eight scored criteria. Under the cut-offs we propose, a note of 401 to 480 words scores 1 of 2 for format and a longer one scores 0. 4 notes scored 1 and 5 scored 0. No clinician has confirmed these cut-offs yet.
The same program checks that the urgency was not changed. It found the selected urgency stated in 8 notes. In the ninth it missed it, because the note gives the urgency as a one-word sentence (“Routine.”). Read by hand, that one matches too.
One menu label made the notes query a diagnosis the patient does not have
The tool files every cardiology referral under a single menu label, “Ischemic heart disease / chronic heart failure”. Two of the three pilot patients have ischemic heart disease and no heart failure anywhere in their chart extract. All 6 notes written for those patients added a line under “To confirm before sending” saying that the referral names heart failure and the chart does not. One of them reads:
The referral condition lists chronic heart failure, but there is no heart failure diagnosis, echocardiogram or ejection fraction in the chart. Please confirm or remove it, and add the EF if known.
The notes handled the mismatch correctly, but the doctor has to clear that line on every such referral. The cause is in the tool. It should send the patient’s actual diagnosis, entered or confirmed by the doctor, in place of a combined label.
One error that two judges caught independently
A note written by Haiku 5.5 for patient 3 called one NT-proBNP value the peak while quoting a higher value on the same line.
The Opus and Sonnet judges each marked this as a wrong key result, which is critical failure F3. The Haiku judge’s answer on this note was cut off. Two of three judges agreeing is the rule for counting a critical failure, so this note fails.
Three more flags, each from a single judge
A flag from one judge of three is a query for a clinician to settle. It does not count as a failure.
| Note | Quoted from the note | Why it was flagged | Judge |
|---|---|---|---|
| Opus 5.5, patient 1 | LDL 113.7 mg/dL on 2025-12-18, down from 133.1 on 2021-12-07. | The comparison starts from a high point in 2021, so LDL reads as falling. From the first recorded value, 108.5 in 2017, to the latest, 113.7, it rose, and it has risen at each reading since mid-2024.F3: Wrong value, date or direction of trend for a key result | Opus |
| Haiku 5.5, patient 2 | Ejection fraction and cardiac imaging: none in the record. | The chart records a coronary angiogram on 2023-09-05, so “no cardiac imaging” is wrong. No ejection fraction is recorded, which the note got right.F3: Wrong value, date or direction of trend for a key result | Haiku |
| Sonnet 5.5, patient 3 | CKD (stage 2 on problem list), microalbuminuria. ... eGFR 36.2, creatinine 1.4 on 2026-11-09 (eGFR 62.5 on 2026-10-19, 30.5 on 2026-06-29... | The problem list says CKD stage 2 while the latest eGFR, 36.2, is in the stage 3b range. The note states both and does not list the conflict under “To confirm before sending”.F7: Hides a gap or a conflict in the chart | Sonnet |
The pilot also found two faults in our own test
- About the menu label the judges were not shownA fault in the test, now fixed. These flags are set aside.8
- The wrong NT-proBNP peakOne note, flagged by two judges. It counts as a critical failure.2
- Raised by one judge onlyThree notes. Queries for a clinician.3
Because of the first fault, all 21 pilot judgments are set aside. The same 9 notes will be judged again in the build stage.
Can the judges catch a bad note?
Before trusting a judge, we hand it notes we know are bad and see whether it objects. Each of the three judges was given four such notes, all presented as the note for patient 1.
| Known-bad note | Opus judge | Sonnet judge | Haiku judge |
|---|---|---|---|
| An empty note | Caughtmarked F4, F6, F7 | Caughtmarked F6, F7 | Caughtmarked F6 |
| A note that says only “I don’t know.” | Caughtmarked F4, F6, F7 | Caughtmarked F6, F7 | Caughtmarked F6, F7 |
| A real pilot note with an invented symptom and examination finding added | Caughtmarked F1, F3 | Caughtmarked F1, F4 | Cut offthe answer hit the ceiling |
| Another patient’s note | Caughtmarked F1, F2, F3, F4, F5, F6, F7 | Caughtmarked F1, F2, F3, F4, F5, F6, F7 | Caughtmarked F1, F2, F3, F4, F5, F6, F7 |
All 11 completed checks caught the problem. The twelfth, the Haiku judge on the invented findings, was cut off by the ceiling described above and will be repeated. Rejecting an obviously bad note is a minimum. It does not show that the judges find subtle errors.
Provisional scores
Read these as a preview of the scorecard. They come from the judgments that were set aside, with three notes per model. They do not show which model writes better notes.
| Writer | Notes | Failed by 2+ judges | …only because of the menu label | Mean score of 16 | Lowest note |
|---|---|---|---|---|---|
| Opus 5.5 | 3 | 0 | 0 | 13.1 | 12.5 |
| Sonnet 5.5 | 3 | 2 | 2 | 12.7 | 12.0 |
| Haiku 5.5 | 3 | 1 | 0 | 12.7 | 12.5 |
| Mean per criterion, 0 to 2 | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 |
|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | 2.00 | 2.00 | 1.50 | 1.50 | 1.67 | 1.83 | 1.89 | 0.67 |
| Sonnet 5.5 | 2.00 | 2.00 | 1.22 | 2.00 | 2.00 | 1.44 | 2.00 | 0.00 |
| Haiku 5.5 | 2.00 | 1.89 | 1.00 | 2.00 | 2.00 | 1.67 | 1.44 | 0.67 |
C1 to C7 are the judges’ averages. C8, format, comes from the program. The criteria are listed further down.
| Mean C1 to C7 total, of 14 | Opus 5.5 notes | Sonnet 5.5 notes | Haiku 5.5 notes |
|---|---|---|---|
| Opus judge | 12.03 of 3 notes | 12.33 of 3 notes | 12.33 of 3 notes |
| Sonnet judge | 12.73 of 3 notes | 12.73 of 3 notes | 11.73 of 3 notes |
| Haiku judge | 14.01 of 3 notes | 14.01 of 3 notes | 12.01 of 3 notes |
Shaded cells are a judge scoring notes from its own model. The Haiku judge completed only 3 of its 9 note judgments.
On whether a note had any critical failure, the Opus and Sonnet judges agreed on 5 of 9 notes (56%). Agreement beyond chance, Cohen’s kappa, was 0.14, which is poor. Some of the disagreement came from the missing label. We will measure it again after the fix.
How the evaluation runs
Three models write, three models judge, and a program checks what can be counted
The framework is the team’s Referral Note Eval rubric. It is still a draft with open questions for clinicians. One case is one patient under one referral condition.
- Chart extractThe drafter’s own code pulls the part of the synthetic chart that this specialist needs.Program
- Doctor’s inputA scripted urgency, an optional clinical question, and the reason for referring now.Written by Claude Fable 5.2. Clinician review pending.
- DraftOpus 5.5, Sonnet 5.5 and Haiku 5.5 each write the note with the drafter’s real prompt and settings.3 writer models
- Format checkA program checks the eight headings, the word count, and that the urgency was not changed.Program
- JudgingEach judge lists the note’s statements beside their source lines, answers seven critical-failure questions with quotes, and scores seven criteria.3 judge models
- ScoreTwo of three judges agreeing makes a critical failure. The score is the judges’ average plus the format points, out of 16.Program
- Clinician reviewFlagged notes, and anything a judge was unsure about, go to a clinician to settle.Clinician
The cases
| Set | Cardiology | Nephrology | Orthopedics | Drafts per case | What it is for |
|---|---|---|---|---|---|
| Build | 15 | – | – | 1 | Tune the prompt and the rubric. Clinicians score these notes by hand to check the judges. |
| Carry-over | – | 4 | 4 | 1 | See whether what was tuned on cardiology holds for the other specialties. |
| Test, held out | 5 | 2 | 3 | 3 | The score. Opened once, after the prompt and rubric are frozen. |
| All cases | 20 | 6 | 7 | 33 cases in total |
All 33 patients are synthetic, generated by Synthea. 6 cases are marked urgent. 14 leave the clinical question blank, so the drafter has to propose one and mark it as a suggestion.
What a judge looks for
Any one critical failure fails the note, whatever its score. PCP means the primary care provider, the doctor who is referring.
| Critical failure | Example | |
|---|---|---|
| F1 | States a clinical fact that is in neither the extract nor the PCP's input | Describes breathlessness or an exam finding when the PCP typed none |
| F2 | Wrong medication, dose or allergy | Lists a stopped drug as current; says "no known allergies" when two are recorded |
| F3 | Wrong value, date or direction of trend for a key result | Calls eGFR stable when the first and latest values show a fall |
| F4 | Changes what the PCP entered | Urgency differs from the one selected; a suggested question is not marked "[suggested - confirm]" |
| F5 | Wrong patient details, joint or condition | Hip written where the chart says knee |
| F6 | Leaves out something in the chart that would change the specialist's management | Antiplatelet drug not mentioned in a referral for joint surgery |
| F7 | Hides a gap or a conflict in the chart | Problem list says CKD stage 3, latest eGFR is in the stage 4 range, and the note picks one without flagging it |
Each criterion scores 0, 1 or 2, so the most a note can score is 16.
| Criterion | What a 2 looks like | Checked by | |
|---|---|---|---|
| C1 | Reason and question | The top of the note says why the patient is being sent now and what is being asked | Judges |
| C2 | What this specialist needs | Covers every item on the condition's list below that the chart contains | Judges |
| C3 | Results and trends | Values carry dates; a trend is given as direction with first and latest values | Judges |
| C4 | Medications and allergies | Matches the extract and points out the drugs, and the obvious gaps, that matter to this specialty | Judges |
| C5 | Workup and treatment to date | States what has been tried or investigated, and for how long | Judges |
| C6 | Gaps made visible | "To confirm before sending" lists what this specialist would expect but the chart lacks, and any unanswered follow-up question | Judges |
| C7 | Selective | Leaves out chart content this specialist has no use for | Judges |
| C8 | Format | The eight headings in order, 250 to 400 words, plain text with no markdown, and no greeting or sign-off | Program |
The list that C2 refers to depends on the specialty.
| Referral to | What that specialist needs, where the chart has it |
|---|---|
| Cardiology | Cardiac diagnoses with dates; prior events and interventions (MI, PCI, CABG, heart failure admissions); latest ejection fraction and cardiac imaging; cardiac medications and obvious gaps (antiplatelet, statin, beta blocker, ACE inhibitor or ARB, diuretic); blood pressure and heart rate; lipid and glycemic control; renal function and potassium; recent emergency or inpatient visits; smoking status |
| Nephrology | CKD stage and how fast eGFR is falling, with first and latest values and dates; degree of albuminuria; blood pressure control; glycemic control; whether on an ACE inhibitor, ARB or SGLT2 inhibitor; nephrotoxic or renally cleared drugs (metformin, NSAIDs); potassium, bicarbonate, hemoglobin and bone-mineral labs; retinopathy and neuropathy |
| Orthopedic surgery | Which joint and how long since diagnosis; conservative treatment tried and for how long (analgesics, NSAIDs, injections, physical therapy, weight management); pain scores over time; imaging of the joint; BMI; fitness for surgery: diabetes control, cardiac disease, kidney function, anticoagulant or antiplatelet use, anemia, smoking status |
The pass bar
The score comes from the held-out test set: 10 cases with 3 drafts each, which is 30 notes for each writer model. To pass, a model needs all three:
- no critical failure in any of the 30 notes
- a mean score of at least 13 of 16
- no single note below 10
Results are reported separately for cardiology and for the other two specialties, because the prompt is tuned on cardiology cases.
Checks on the judges
- Known-bad notes. Each judge must reject notes we broke on purpose.
- Agreement. We report how often each pair of judges agrees, and their agreement beyond chance.
- Own-model check. Every judge scores every note, including its own model’s. We compare what a judge gives its own model with what it gives the others.
- Clinician hand-scoring. Clinicians score the 15 build notes by hand. The model judges are trusted only if they match.
Settings
| Writers | claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5. Each uses the drafter’s own system prompt and effort setting (medium), with up to 16,000 output tokens. The drafter itself runs on claude-opus-5-5 by default. |
|---|---|
| Judges | The same three models at high effort, with up to 32,000 output tokens. Each answer must fit a fixed form: the note’s statements with their sources, seven failure verdicts with quotes, and seven scores with reasons. |
| Versions | Drafter prompt e6a0a28e3c, rubric 03795be61a. Every note and judgment records the version it was made under, and the scorecard warns when versions are mixed. |
| Held-out set | The runner refuses to write test-set notes until it is told the prompt and rubric are frozen. |
| Budget | $50. The runner records the cost of every call and refuses to start a step that would pass the limit. |
| Batches | Full stages go through the Message Batches API at half price. The pilot ran live. |
| Model identity | No fallback model is configured. The runner rejects any answer served by a different model from the one it asked for. |
The commands
The runner, the rubric and the cases are in the eval folder of the code. Each stage is written, collected, judged, collected and scored.
python eval/run_eval.py selftest # offline checks of the runner, free python eval/run_eval.py plan # what is left to run, and the projected cost python eval/run_eval.py pilot # 3 cases, written and judged live python eval/run_eval.py write --stage build # then: collect --wait python eval/run_eval.py judge --stage build # then: collect --wait python eval/run_eval.py score --stage build # the scorecard python eval/build_report.py # rebuilds this page from the records
Budget
$3.31 spent, about $26.34 projected for the whole evaluation
- Spent on the pilotNine notes written and judged, plus the known-bad notes. Run live, at list price.$3.31
- Projected for the three stages still to runBuild $6.41, carry-over $3.50, test $13.12.about $23.03
- Left over if the projection holdsEnough to repeat those stages once if the prompt or rubric changes.about $23.66
| Pilot spending | Calls | Cost |
|---|---|---|
| Writing the 9 pilot notes | 9 | $0.29 |
| Judging those notes (later set aside) | 21 | $2.29 |
| Judging the known-bad notes | 11 | $0.67 |
| Haiku judge calls that were cut off | 7 | $0.06 |
| Total | 48 | $3.31 |
| Still to run | Notes | Judgments | Projected cost |
|---|---|---|---|
| Build, 15 cases | 36 | 135 | about $6.41 |
| Carry-over, 8 cases | 24 | 72 | about $3.50 |
| Test, 10 cases with 3 drafts each | 90 | 270 | about $13.12 |
| Total still to run | 150 | 477 | about $23.03 |
| Measured in the pilot, per call | Calls | Tokens in | Tokens out | Cost at list price |
|---|---|---|---|---|
| Writing a note, Opus 5.5 | 3 | 4,759 | 2,482 | $0.0687 |
| Writing a note, Sonnet 5.5 | 3 | 4,759 | 1,516 | $0.0247 |
| Writing a note, Haiku 5.5 | 3 | 4,759 | 5,438 | $0.0032 |
| Judging a note, Opus 5.5 | 13 | 9,314 | 5,940 | $0.1561 |
| Judging a note, Sonnet 5.5 | 13 | 9,314 | 5,042 | $0.0690 |
| Judging a note, Haiku 5.5 | 6 | 8,813 | 8,755 | $0.0053 |
Tokens are the units of text the models are billed for. Tokens out include the model’s own reasoning before it answers, not only the note or the verdict, which is why they run far above the length of a note.
$2.29 of the pilot’s spending bought judgments that were later set aside. Finding that fault in the pilot cost less than finding it in the full run.
List prices per million tokens, input and output, checked on 21 November 2026: Opus 5.5 $4 and $20, Sonnet 5.5 $2 and $10, Haiku 5.5 $0.10 and $0.50. Full stages go through the batch service at half price. These are our own counts from the token use the API reports, not an invoice.
Before quoting a number
What this does not show yet
- Nine notes, three patients, one specialty. Nothing here supports a ranking of the models.
- The judges are AI models, the same three that write the notes. No clinician has scored a note by hand yet.
- The doctor’s input for each case was written by an AI model and is waiting for clinician review.
- The patients are synthetic. Their charts have no free-text notes and are tidier than real records.
- Only the first draft is tested. The drafter’s follow-up questions and its revised note are not.
- The word-count cut-offs for the format score are proposed, not agreed.
- The pilot’s judgments were made with a faulty judge input and are set aside.
What comes next
Five steps to a score
- Clinician review of the scripted doctor’s input for all 33 cases.
- Build stage. 15 cardiology cases, with the pilot notes judged again. Projected cost about $6.41.
- Hand-scoring. Clinicians score the build notes, and the prompt and rubric are adjusted.
- Carry-over stage. 8 nephrology and orthopedic cases. Projected cost about $3.50.
- Freeze, then test. The prompt and rubric are frozen, the held-out set is opened once, and the score is reported. Projected cost about $13.12.
Prototype built at the Claude Impact Lab for Digital Health in New York. All patients are synthetic (Synthea); no real patient data is used. The evaluation calls Claude models through the Claude API. Not for clinical use. Case codes for the pilot: patient 1 is cardiac-7208c12f, patient 2 is cardiac-ab51950f, patient 3 is cardiac-d50b22fc. Page built on 22 November 2026 from the run’s own records.