Referral Note Eval Report
← Referral Tracker

Evaluation · pilot stage

Testing the referral note drafter

The note drafter writes a referral note from a patient’s chart and a few lines typed by the primary care doctor. This page explains how we test whether those notes are accurate and useful to the specialist, and what the first run found.

Where it stands on 21 November 2026

Only the pilot has run: 3 of 33 cases, 9 notes. There is no pass or fail score yet.

The judges’ scores on this page were produced before we corrected what the judges are shown, so they are provisional. They will be replaced when the next stage runs.

  1. Pilot3 cardiology cases, 9 notes. A dry run to find faults in the test itself.Done on 21 November 2026
  2. Build15 cardiology cases. Used to tune the prompt and the rubric. Clinicians score these notes by hand.Not run yet
  3. Carry-over4 nephrology and 4 orthopedic cases. Checks that the tuning holds outside cardiology.Not run yet
  4. Test10 held-out cases, 3 drafts each. This stage gives the score.Sealed until the prompt and rubric are frozen

Results so far

What nine notes showed

Nine notes from three patients are enough to expose problems. They are too few to rank the models or to stand in for a score.

Every note ran over the word limit

The drafter’s instructions ask for roughly 250 to 400 words, about one page. All 9 notes exceeded that, at 411 to 563 words. A program counts the words, so this finding does not depend on the judges.

Length of each pilot noteWords per note, all on one scale. The pilot patients are three of the cardiology build cases.
  • Words up to the limit
  • Words over the limit
  • The limit, 250 to 400 words
  • 480 words: beyond this the format score is 0 (proposed)

Patient 1 · chest tightness on stairs · routine

Opus 5.5449 words49 over · format 1 of 2
Sonnet 5.5518 words118 over · format 0 of 2
Haiku 5.5468 words68 over · format 1 of 2

Patient 2 · stopped a statin after muscle aches · routine

Opus 5.5463 words63 over · format 1 of 2
Sonnet 5.5483 words83 over · format 0 of 2
Haiku 5.5411 words11 over · format 1 of 2

Patient 3 · worsening breathlessness · urgent

Opus 5.5563 words163 over · format 0 of 2
Sonnet 5.5484 words84 over · format 0 of 2
Haiku 5.5554 words154 over · format 0 of 2

Format is one of eight scored criteria. Under the cut-offs we propose, a note of 401 to 480 words scores 1 of 2 for format and a longer one scores 0. 4 notes scored 1 and 5 scored 0. No clinician has confirmed these cut-offs yet.

The same program checks that the urgency was not changed. It found the selected urgency stated in 8 notes. In the ninth it missed it, because the note gives the urgency as a one-word sentence (“Routine.”). Read by hand, that one matches too.

One menu label made the notes query a diagnosis the patient does not have

The tool files every cardiology referral under a single menu label, “Ischemic heart disease / chronic heart failure”. Two of the three pilot patients have ischemic heart disease and no heart failure anywhere in their chart extract. All 6 notes written for those patients added a line under “To confirm before sending” saying that the referral names heart failure and the chart does not. One of them reads:

The referral condition lists chronic heart failure, but there is no heart failure diagnosis, echocardiogram or ejection fraction in the chart. Please confirm or remove it, and add the EF if known.

The notes handled the mismatch correctly, but the doctor has to clear that line on every such referral. The cause is in the tool. It should send the patient’s actual diagnosis, entered or confirmed by the doctor, in place of a combined label.

One error that two judges caught independently

A note written by Haiku 5.5 for patient 3 called one NT-proBNP value the peak while quoting a higher value on the same line.

What the note saysNT-proBNP 1963.1 pg/mL (2025-04-27), 1584.4 (2025-12-06). Peak 1869.2 (2025-12-02), nadir 307 (2025-12-04).
What the chart extract saysNT-proBNP (pg/mL): 1584.4 on 2025-12-06; earlier: 1236.4 (2025-12-01), 1869.2 (2025-12-02), 1721.2 (2025-12-03), 307 (2025-12-04), 691.8 (2025-12-05); first recorded: 1963.1 (2025-04-27); 16 readings in total

The Opus and Sonnet judges each marked this as a wrong key result, which is critical failure F3. The Haiku judge’s answer on this note was cut off. Two of three judges agreeing is the rule for counting a critical failure, so this note fails.

Three more flags, each from a single judge

A flag from one judge of three is a query for a clinician to settle. It does not count as a failure.

NoteQuoted from the noteWhy it was flaggedJudge
Opus 5.5, patient 1LDL 113.7 mg/dL on 2025-12-18, down from 133.1 on 2021-12-07.The comparison starts from a high point in 2021, so LDL reads as falling. From the first recorded value, 108.5 in 2017, to the latest, 113.7, it rose, and it has risen at each reading since mid-2024.F3: Wrong value, date or direction of trend for a key resultOpus
Haiku 5.5, patient 2Ejection fraction and cardiac imaging: none in the record.The chart records a coronary angiogram on 2023-09-05, so “no cardiac imaging” is wrong. No ejection fraction is recorded, which the note got right.F3: Wrong value, date or direction of trend for a key resultHaiku
Sonnet 5.5, patient 3CKD (stage 2 on problem list), microalbuminuria. ... eGFR 36.2, creatinine 1.4 on 2026-11-09 (eGFR 62.5 on 2026-10-19, 30.5 on 2026-06-29...The problem list says CKD stage 2 while the latest eGFR, 36.2, is in the stage 3b range. The note states both and does not list the conflict under “To confirm before sending”.F7: Hides a gap or a conflict in the chartSonnet

The pilot also found two faults in our own test

The judges were missing one inputThe drafter is told the menu label. The judges were not. When a note mentioned the label, the judges called it an invented fact. 8 of the 13 failure flags in the pilot came from this. The judges now see everything the drafter sees.
One judge ran out of roomThe Haiku judge’s answers were cut off at the 16,000-token ceiling in 7 of its 13 calls. The ceiling is now 32,000.
Where the judges’ 13 failure flags came fromEach flag is one judge answering yes to one critical failure on one note. 21 judgments were completed.
  • About the menu label the judges were not shownA fault in the test, now fixed. These flags are set aside.8
  • The wrong NT-proBNP peakOne note, flagged by two judges. It counts as a critical failure.2
  • Raised by one judge onlyThree notes. Queries for a clinician.3

Because of the first fault, all 21 pilot judgments are set aside. The same 9 notes will be judged again in the build stage.

Can the judges catch a bad note?

Before trusting a judge, we hand it notes we know are bad and see whether it objects. Each of the three judges was given four such notes, all presented as the note for patient 1.

Known-bad noteOpus judgeSonnet judgeHaiku judge
An empty noteCaughtmarked F4, F6, F7Caughtmarked F6, F7Caughtmarked F6
A note that says only “I don’t know.”Caughtmarked F4, F6, F7Caughtmarked F6, F7Caughtmarked F6, F7
A real pilot note with an invented symptom and examination finding addedCaughtmarked F1, F3Caughtmarked F1, F4Cut offthe answer hit the ceiling
Another patient’s noteCaughtmarked F1, F2, F3, F4, F5, F6, F7Caughtmarked F1, F2, F3, F4, F5, F6, F7Caughtmarked F1, F2, F3, F4, F5, F6, F7

All 11 completed checks caught the problem. The twelfth, the Haiku judge on the invented findings, was cut off by the ceiling described above and will be repeated. Rejecting an obviously bad note is a minimum. It does not show that the judges find subtle errors.

Provisional scores

Read these as a preview of the scorecard. They come from the judgments that were set aside, with three notes per model. They do not show which model writes better notes.

WriterNotesFailed by 2+ judges…only because of the menu labelMean score of 16Lowest note
Opus 5.530013.112.5
Sonnet 5.532212.712.0
Haiku 5.531012.712.5
Mean per criterion, 0 to 2C1C2C3C4C5C6C7C8
Opus 5.52.002.001.501.501.671.831.890.67
Sonnet 5.52.002.001.222.002.001.442.000.00
Haiku 5.52.001.891.002.002.001.671.440.67

C1 to C7 are the judges’ averages. C8, format, comes from the program. The criteria are listed further down.

Mean C1 to C7 total, of 14Opus 5.5 notesSonnet 5.5 notesHaiku 5.5 notes
Opus judge12.03 of 3 notes12.33 of 3 notes12.33 of 3 notes
Sonnet judge12.73 of 3 notes12.73 of 3 notes11.73 of 3 notes
Haiku judge14.01 of 3 notes14.01 of 3 notes12.01 of 3 notes

Shaded cells are a judge scoring notes from its own model. The Haiku judge completed only 3 of its 9 note judgments.

On whether a note had any critical failure, the Opus and Sonnet judges agreed on 5 of 9 notes (56%). Agreement beyond chance, Cohen’s kappa, was 0.14, which is poor. Some of the disagreement came from the missing label. We will measure it again after the fix.

How the evaluation runs

Three models write, three models judge, and a program checks what can be counted

The framework is the team’s Referral Note Eval rubric. It is still a draft with open questions for clinicians. One case is one patient under one referral condition.

  1. Chart extractThe drafter’s own code pulls the part of the synthetic chart that this specialist needs.Program
  2. Doctor’s inputA scripted urgency, an optional clinical question, and the reason for referring now.Written by Claude Fable 5.2. Clinician review pending.
  3. DraftOpus 5.5, Sonnet 5.5 and Haiku 5.5 each write the note with the drafter’s real prompt and settings.3 writer models
  4. Format checkA program checks the eight headings, the word count, and that the urgency was not changed.Program
  5. JudgingEach judge lists the note’s statements beside their source lines, answers seven critical-failure questions with quotes, and scores seven criteria.3 judge models
  6. ScoreTwo of three judges agreeing makes a critical failure. The score is the judges’ average plus the format points, out of 16.Program
  7. Clinician reviewFlagged notes, and anything a judge was unsure about, go to a clinician to settle.Clinician

The cases

SetCardiologyNephrologyOrthopedicsDrafts per caseWhat it is for
Build15––1Tune the prompt and the rubric. Clinicians score these notes by hand to check the judges.
Carry-over–441See whether what was tuned on cardiology holds for the other specialties.
Test, held out5233The score. Opened once, after the prompt and rubric are frozen.
All cases206733 cases in total

All 33 patients are synthetic, generated by Synthea. 6 cases are marked urgent. 14 leave the clinical question blank, so the drafter has to propose one and mark it as a suggestion.

What a judge looks for

Any one critical failure fails the note, whatever its score. PCP means the primary care provider, the doctor who is referring.

Critical failureExample
F1States a clinical fact that is in neither the extract nor the PCP's inputDescribes breathlessness or an exam finding when the PCP typed none
F2Wrong medication, dose or allergyLists a stopped drug as current; says "no known allergies" when two are recorded
F3Wrong value, date or direction of trend for a key resultCalls eGFR stable when the first and latest values show a fall
F4Changes what the PCP enteredUrgency differs from the one selected; a suggested question is not marked "[suggested - confirm]"
F5Wrong patient details, joint or conditionHip written where the chart says knee
F6Leaves out something in the chart that would change the specialist's managementAntiplatelet drug not mentioned in a referral for joint surgery
F7Hides a gap or a conflict in the chartProblem list says CKD stage 3, latest eGFR is in the stage 4 range, and the note picks one without flagging it

Each criterion scores 0, 1 or 2, so the most a note can score is 16.

CriterionWhat a 2 looks likeChecked by
C1Reason and questionThe top of the note says why the patient is being sent now and what is being askedJudges
C2What this specialist needsCovers every item on the condition's list below that the chart containsJudges
C3Results and trendsValues carry dates; a trend is given as direction with first and latest valuesJudges
C4Medications and allergiesMatches the extract and points out the drugs, and the obvious gaps, that matter to this specialtyJudges
C5Workup and treatment to dateStates what has been tried or investigated, and for how longJudges
C6Gaps made visible"To confirm before sending" lists what this specialist would expect but the chart lacks, and any unanswered follow-up questionJudges
C7SelectiveLeaves out chart content this specialist has no use forJudges
C8FormatThe eight headings in order, 250 to 400 words, plain text with no markdown, and no greeting or sign-offProgram

The list that C2 refers to depends on the specialty.

Referral toWhat that specialist needs, where the chart has it
CardiologyCardiac diagnoses with dates; prior events and interventions (MI, PCI, CABG, heart failure admissions); latest ejection fraction and cardiac imaging; cardiac medications and obvious gaps (antiplatelet, statin, beta blocker, ACE inhibitor or ARB, diuretic); blood pressure and heart rate; lipid and glycemic control; renal function and potassium; recent emergency or inpatient visits; smoking status
NephrologyCKD stage and how fast eGFR is falling, with first and latest values and dates; degree of albuminuria; blood pressure control; glycemic control; whether on an ACE inhibitor, ARB or SGLT2 inhibitor; nephrotoxic or renally cleared drugs (metformin, NSAIDs); potassium, bicarbonate, hemoglobin and bone-mineral labs; retinopathy and neuropathy
Orthopedic surgeryWhich joint and how long since diagnosis; conservative treatment tried and for how long (analgesics, NSAIDs, injections, physical therapy, weight management); pain scores over time; imaging of the joint; BMI; fitness for surgery: diabetes control, cardiac disease, kidney function, anticoagulant or antiplatelet use, anemia, smoking status

The pass bar

The score comes from the held-out test set: 10 cases with 3 drafts each, which is 30 notes for each writer model. To pass, a model needs all three:

  • no critical failure in any of the 30 notes
  • a mean score of at least 13 of 16
  • no single note below 10

Results are reported separately for cardiology and for the other two specialties, because the prompt is tuned on cardiology cases.

Checks on the judges

  • Known-bad notes. Each judge must reject notes we broke on purpose.
  • Agreement. We report how often each pair of judges agrees, and their agreement beyond chance.
  • Own-model check. Every judge scores every note, including its own model’s. We compare what a judge gives its own model with what it gives the others.
  • Clinician hand-scoring. Clinicians score the 15 build notes by hand. The model judges are trusted only if they match.

Settings

Writersclaude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5. Each uses the drafter’s own system prompt and effort setting (medium), with up to 16,000 output tokens. The drafter itself runs on claude-opus-5-5 by default.
JudgesThe same three models at high effort, with up to 32,000 output tokens. Each answer must fit a fixed form: the note’s statements with their sources, seven failure verdicts with quotes, and seven scores with reasons.
VersionsDrafter prompt e6a0a28e3c, rubric 03795be61a. Every note and judgment records the version it was made under, and the scorecard warns when versions are mixed.
Held-out setThe runner refuses to write test-set notes until it is told the prompt and rubric are frozen.
Budget$50. The runner records the cost of every call and refuses to start a step that would pass the limit.
BatchesFull stages go through the Message Batches API at half price. The pilot ran live.
Model identityNo fallback model is configured. The runner rejects any answer served by a different model from the one it asked for.

The commands

The runner, the rubric and the cases are in the eval folder of the code. Each stage is written, collected, judged, collected and scored.

python eval/run_eval.py selftest               # offline checks of the runner, free
python eval/run_eval.py plan                   # what is left to run, and the projected cost
python eval/run_eval.py pilot                  # 3 cases, written and judged live
python eval/run_eval.py write --stage build    # then: collect --wait
python eval/run_eval.py judge --stage build    # then: collect --wait
python eval/run_eval.py score --stage build    # the scorecard
python eval/build_report.py                    # rebuilds this page from the records

Budget

$3.31 spent, about $26.34 projected for the whole evaluation

The $50 budgetProjections use the token counts measured in the pilot, at batch prices.
  • Spent on the pilotNine notes written and judged, plus the known-bad notes. Run live, at list price.$3.31
  • Projected for the three stages still to runBuild $6.41, carry-over $3.50, test $13.12.about $23.03
  • Left over if the projection holdsEnough to repeat those stages once if the prompt or rubric changes.about $23.66
Pilot spendingCallsCost
Writing the 9 pilot notes9$0.29
Judging those notes (later set aside)21$2.29
Judging the known-bad notes11$0.67
Haiku judge calls that were cut off7$0.06
Total48$3.31
Still to runNotesJudgmentsProjected cost
Build, 15 cases36135about $6.41
Carry-over, 8 cases2472about $3.50
Test, 10 cases with 3 drafts each90270about $13.12
Total still to run150477about $23.03
Measured in the pilot, per callCallsTokens inTokens outCost at list price
Writing a note, Opus 5.534,7592,482$0.0687
Writing a note, Sonnet 5.534,7591,516$0.0247
Writing a note, Haiku 5.534,7595,438$0.0032
Judging a note, Opus 5.5139,3145,940$0.1561
Judging a note, Sonnet 5.5139,3145,042$0.0690
Judging a note, Haiku 5.568,8138,755$0.0053

Tokens are the units of text the models are billed for. Tokens out include the model’s own reasoning before it answers, not only the note or the verdict, which is why they run far above the length of a note.

$2.29 of the pilot’s spending bought judgments that were later set aside. Finding that fault in the pilot cost less than finding it in the full run.

List prices per million tokens, input and output, checked on 21 November 2026: Opus 5.5 $4 and $20, Sonnet 5.5 $2 and $10, Haiku 5.5 $0.10 and $0.50. Full stages go through the batch service at half price. These are our own counts from the token use the API reports, not an invoice.

Before quoting a number

What this does not show yet

What comes next

Five steps to a score

  1. Clinician review of the scripted doctor’s input for all 33 cases.
  2. Build stage. 15 cardiology cases, with the pilot notes judged again. Projected cost about $6.41.
  3. Hand-scoring. Clinicians score the build notes, and the prompt and rubric are adjusted.
  4. Carry-over stage. 8 nephrology and orthopedic cases. Projected cost about $3.50.
  5. Freeze, then test. The prompt and rubric are frozen, the held-out set is opened once, and the score is reported. Projected cost about $13.12.

Prototype built at the Claude Impact Lab for Digital Health in New York. All patients are synthetic (Synthea); no real patient data is used. The evaluation calls Claude models through the Claude API. Not for clinical use. Case codes for the pilot: patient 1 is cardiac-7208c12f, patient 2 is cardiac-ab51950f, patient 3 is cardiac-d50b22fc. Page built on 22 November 2026 from the run’s own records.