At least 52 wrong training labels in MedCalc-Bench Verified that pass a formula re-check

Re-running the formula on 3,794 Verified training rows finds 0 mismatches, against 27 on the same rows of v1.2. Reading the flagged rows against the notes they came from finds at least 52 of 10,538 training labels wrong given their notes: 22 misread from the note, 30 copying a typo the note itself prints. Row 2807's note says pulse 40-50 bpm, and its label scores a heart rate of 4,050. 31 of the 52 sit in calculators the recompute covers, and it passes every one. Only the 194 flagged training rows were read, so 52 is a floor, not an error rate. Separately, a known unit-conversion bug in the HOMA-IR calculator still makes 32 of its 77 training labels 36 times too large.

Red is a stored label. Blue is a value the note supports. Yellow is quoted note text.

MAP is SBP / 3 + 2 × DBP / 3. The stored inputs give 150160 / 3 + 2 × 8090 / 3 = 55,446.67, which is the stored label, so a formula re-check passes it. The note gives two ranges, the one during the seizure and the patient's baseline, and both come before the blood-pressure treatment. Over every pair of their endpoints, MAP runs from 80.00 to 113.33 mmHg, or 76.00 to 119.00 after the dataset's 5% tolerance. A model that reads the note and answers anywhere in that interval gets zero reward, because the label only rewards 52,674 to 58,219. No model has been run on this row yet, so that is arithmetic on the stored limits, not an observed answer.

The Verified pass rewrote some of these labels

The Verified release changed the stored label on 9 of the confirmed rows, with the extracted inputs unchanged. Every new label is still impossible. 8 were re-rounded to more or fewer digits. Row 3211 (MDRD) moved from 12,129.91 to 11,632.93 because of rounding one step earlier: the v1.2 explanation rounded the converted input to 0.012, and Verified kept 0.012443. The note prints creatinine was 1.1 μmol/L, and the old label sits inside the new one's accepted range (11,051 to 12,215). So all 9 are precision changes. None of them touched the impossible input.

A conversion bug in the HOMA-IR calculator

32 of 77 HOMA-IR training rows give insulin in pmol/L. The explanation converts it to µIU/mL by multiplying by 6, as in row 5880: 756.0 * 6 µIU/mL/pmol/L = 4536.0 µIU/mL. About 6 pmol/L make 1 µIU/mL, so the conversion should divide, and every one of these labels is 36 times too large. They run from 52.85 to 233.81, and 1,089.76 for row 5880, while the HOMA-IR rows in other units top out at 19.9. Row 9797 has insulin 55 pmol/L and a stored label of 76.33; divided instead of multiplied, the same inputs give 2.12.

arXiv 2603.02222 lists this bug, and PR #5 (artuskg) proposed the one-line fix in February 2026. It was closed without merging, so 32 of 77 training labels in v1.0.8 still carry it.

This is a separate count from the 52: 31 of the 32 rows are outside it. The gate does not catch these rows, because the formula is applied consistently to inputs that match the note, and a recompute that runs the same calculator code agrees with the label. The fix belongs in one line of the calculator, not in 32 separate relabels. Every row is listed in results/homa_ir.csv, and results/fix_list.csv marks them "fix the calculator".

Headline numbers

Counts sit next to every share. Intervals are Wilson 95%. Gate precision leaves out rows marked unclear.

Two things are not measured yet, because both need model calls and none has completed: how many defects the checks miss among unflagged rows, and how three models of different sizes answer the defect rows (339 calls planned, 1 attempted).

What finds them

Cheap screens find almost all of these rows. A screen that flags any label or input more than 100 times away from the median of its field (G4 below) catches 47 of the 52 train defects, and the label alone is that far out in 25. G3, the rule that diastolic pressure cannot exceed systolic, adds row 1545 for 48. Two plain range checks, heart rate from 15 to 300 bpm and weight up to 400 kg, add 2807 (heart rate 4,050 bpm), 5924 (weight 3,035 lb) and 6596 (heart rate 7,590 bpm) for 51. Those range checks flag 6 train rows in all; the others are 4737, 4865 and 6583. Their bounds were picked after the defects were known, so this shows how cheap the rows are to find, and it is not a held-out test of a screen. The last row, 5880, is caught by G2 because its stored insulin is not in the note. G4 at 30x instead of 100x would also catch it, and on its own would find 51 of the 52, but that threshold was also chosen after seeing the defects.

What reading the note adds is the cause and the answer the note supports. Row 2807 asks for a Wells score. Its note says pulse 40-50 bpm, the stored heart rate is 4050, and the stored label of 1.5 is exactly the 1.5 points the heart-rate criterion gives for a rate over 100. Read as 40-50, that criterion scores 0, not 1.5, so a model that reads the note correctly does not match the key. That reading is what decides what to do with a flagged row: relabel it when the note gives an answer (2807, or 1337 with an interval label), drop it when the note itself is wrong (the copied typos), and fix the calculator when one bug produced many rows (HOMA-IR above).

Where the defects are

Most of the copied typos sit in kidney and electrolyte formulas, where the note prints a unit such as mmol/dl for sodium or creatinine. The extraction defects are spread wider: platelet counts printed per 10⁹/L stored a thousand times too small, heart-rate and weight ranges joined into one number, resected masses read as the patient's weight, and cerebrospinal fluid cell counts read as blood white cell counts. The largest groups are CKD-EPI (7), FIB-4 (6), Calcium correction (5) and SIRS (5).

The four checks

Each check flags rows to read. A flag becomes a defect only after the note span that contradicts the input is quoted. Together the checks flagged 194 of 10,538 train rows and 12 of 1,100 test rows. Across all four, 52 of 185 decided flagged train rows are wrong labels (28.1%), which is the false-flag load a reviewer would face; test is 1 of 12.

G1 is coded from the formula each row's own Ground Truth Explanation states, for 15 renal, electrolyte and blood-pressure formulas. It agrees with every stored label on the 3,794 rows it covers (upper 95% bound 0.10%). On the same rows of v1.2, 27 differ, all in CKD-EPI 2021. That matches PR #7 (open) on the Verified repo, which re-ran the calculators on both splits and reported 0 train mismatches. The maintainer's v1.0.8 release notes (2026-09-08) report 0 audit mismatches across all 11,693 rows, and commits 3df69cf and 192a585 fixed notes that contradicted their own entities: 8 sodium rows and the fentanyl units. So note-against-entity checks are already part of upkeep. G1 is here to show the formulas are right, so the remaining errors sit in the inputs.

G2 accepts a number that matches a note number times a power of ten from 10^-3 to 10³, to allow unit rescaling, and reports an input whose digits equal two note numbers joined across a hyphen as a merged range. Its precision is poor: 11 of 132 (8.33%). Of its 121 train false flags, 112 are number formats the reader does not parse, such as the decimal comma in 3,79, the space in 12 900 and the lost superscript in 16.4×104.

A format-aware reader, written after reading only a seeded random half of the G2-flagged train rows, handles decimal commas, spaces as thousands separators, a space after the decimal point, and a power of ten written without its superscript. On the other half, precision goes from 5 of 68 to 5 of 10 (50.0%, 95% CI 23.7% to 76.3%) and no defect is lost. On test it goes from 0 of 11 to 0 of 7. On the half it was written on, it goes from 6 of 64 to 2 of 9, and it drops 4 real defects: 4309, 4318, 4323 and 4347, platelet counts stored a thousand times too small. The old reader only caught those because it could not read "×109"; once it can, the 10^-3 rescaling allowance accepts the wrong value. G4 still flags 4 of the 4. The format list was already known from the first pass over all rows, so the held-out half is held out for the exact rules, not for which formats exist. Every other number on this page uses the original reader.

G3 follows from definitions and needs no clinical bounds. It flagged 2 train rows and is the only check that fires on row 1545 below. G4 is a fixed 100x screen against the median of the same field in the same file, set before any flagged row was read. It finds the copied typos and needs a file large enough for the median to mean something.

9 train rows were marked unclear and are left out of every count: 4550, 6414, 6599, 6639, 6653, 6769, 6787, 6789 and 6805. Most give no unit or never state the input. 8 train rows have a wrong input that does not change the label; they count as not a defect.

On a second check, 4 of the unclear rows (6414, 6639, 6653 and 6769) look like the same patterns as counted rows, so 52 is a floor.

Worked examples

Each example shows the note sentence, the extracted input, the stored label, the recompute from the extracted input, and what the note supports, so the arithmetic can be redone by hand. Quotes are literal substrings of the row's Patient Note, and tests/test_handcheck.py checks that.

Row 1545: systolic and diastolic from different readings

The systolic input of 70 comes from a sentence about a drop in pressure, and the diastolic of 74 from a later reading. The stored MAP of 72.67 sits below its own diastolic input, which no MAP can do, and G3 caught it. The only pair the note states together was taken after ECMO cannulation on epinephrine and norepinephrine, and the question asks for values from admission before any treatment. So the note supports no answer: the task cannot be computed, and the label is a number for a question with none.

Row 2685: a calcium range read as one value

The note gives calcium as a range, and the extracted input is a single value with the digits run together, 952.25 g/L. The unit the note prints, g/L, would give a corrected calcium near 200 mg/dL, which no patient has. The note's own reference range is the usual mmol/L range, so the blue interval assumes mmol/L. That is an assumption about a second typo in the source, and it is labeled as one.

Rows 6171, 8473 and 8678: one note typo, three wrong labels

One note prints sodium in mmol/dl. The pipeline copied the unit as written, and the three calculators built on that note got labels of 1,230, 1,218 and -609. The note is wrong here, so this page gives no corrected answer and states only what each label rewards.

Row 267: an impossible creatinine unit copied from the note

The note prints creatinine in mmol/dl, the extraction copied it, and the stored GFR is 0.0009675 mL/min/1.73 m². Both the input and the label sit more than 100 times away from their medians in the file, which is what G4 measures. No interval, for the same reason as the sodium rows.

All confirmed rows

The full list, so any row can be checked against the dataset. It matches results/handcheck.csv and the draft issue in upstream/ISSUE.md.

Method

The four CSVs (Verified and v1.2, train and test) are downloaded from pinned Hugging Face commits and checked against sha256 before anything runs. Joined on Calculator ID plus Note ID, 331 of 10,538 train labels differ between v1.2 and Verified by more than 1e-6. Of those, 57 differ by less than 0.01%, which is re-rounding; 219 differ by more than 0.1%, and 147 by more than the dataset's 5% tolerance. On test, 35 of 1,099 differ, 23 of them by more than 5%.

Every flagged row has a line in results/handcheck.csv with the checks that fired, the quoted note span, the stored input and label, a decision, a defect type from a fixed list, and the note inputs where they can be computed. A row is a defect when the input contradicts the note, or the note prints an impossible unit that was copied, and correcting the input changes the label or leaves the task with no answer. A copied typo is called only where the note prints the unit literally and G4 fired. Each flagged row has a first-pass call and reason. Dev then read every defect and unclear row (62 in all) against its note on 2026-10-18 and agreed with each call. The 144 rows called not a defect keep their first-pass call. Most calls are a quote plus arithmetic. Some are contextual clinical reads: whether a cell count came from cerebrospinal fluid or blood (6665, 6732 and 6805), whether a pressure on pressors counts as pre-treatment (1545), and whether a weight is the patient's or a resected mass's (4711, 4943, 4981 and 5088). None of them was made by a clinician, and a clinician may overrule any of them.

For a range in the note, the label is computed over every combination of endpoints and widened by the 5% tolerance. For mixed readings, it uses pairs stated together in one reading. For a unit tag that does not match, it uses the unit the note prints, unless that unit is itself impossible; for a lost power of ten, it restores it; for an input taken after treatment, it uses the pre-treatment value the note gives. Copied typos get no interval. 11 extraction rows have one, and a test checks that no stored label falls inside its own interval.

The model step is written and not run. It sends the zero_shot system and user strings from MedCalc-Bench's evaluation/run.py, verbatim, through the command in scripts/run_models.py with tools off, and caches every raw response before parsing: 100 re-extractions of unflagged rows by the mid-size model to estimate misses, 53 defect rows on each of three models of different sizes, and 40 clean control rows on the two smaller ones. Per 1,000 training rows, 2.09 carry an extraction defect (22 of 10,538).

Prior work, cited for what it covers. arXiv 2512.19691 audited the v1.0 test labels, estimating at least 27% are likely erroneous or incomputable, and recomputed labels from the note alone for a random half of the v1.0 training split. Its RL experiment trained on 4,593 of those relabeled training rows and beat training on the original labels by 13.5 points. It does not list which training rows were wrong or why, and it predates Verified. It names feature-extraction errors as a cause of wrong labels; this gate finds them on the training split without a model, by checking each Relevant Entity against a quoted span of its note, and lists each row. arXiv 2603.02222 fixes over 20 calculator implementation errors, including the HOMA-IR conversion above. arXiv 2509.16584 scores formula, extraction and arithmetic steps separately in model outputs. PR #7 re-ran the calculators and reported 0 train mismatches.

All 52 rows are inherited from v1.0. 27 fell in 2512.19691's training sample, and its released labels already disagree with the stored label on 18 of them, without naming the rows or the cause. The remaining 34 of the 52 have no public corrected label that a search on October 18, 2026 found, and none of the 52 is fixed in Verified v1.0.8.

Limitations

• This checks labels against the note they came from. It cannot tell whether the note itself is correct, and 30 of the 52 train defects are typos in the original case report that the pipeline copied faithfully.
• It does not judge whether a task makes clinical sense. The gate flagged rows 4656, 4850, 4860 and 6805, but they are not label defects under this method: Holliday-Segar maintenance fluids on a fetus and on very small neonates (rows 4656, 4850 and 4860, at 150 g, 420 g and 268 g), and adult SIRS criteria on a 4-year-old (6805). Two counted defects are also adult formulas applied to a child: 6732, adult SIRS criteria on an 8-year-old, and 458, adult CKD-EPI on an 11-month-old.
• The grounding check looks for numbers. It can miss an input read from the wrong sentence that still appears somewhere in the note, and there is no estimate yet of how many it misses.
• Dev read every defect and unclear row against its note; the 144 rows called not a defect keep their first-pass call. No clinician has read the rows. The quotes are checked by script, so any reader can redo a call.
• No RL training run was done, so this does not show that the label noise measurably hurts training. The only RL evidence cited is 2512.19691, on labels with a much higher estimated error rate.
• The defects look like what any pipeline that extracts fields from clinical text can produce: a range read as one number, two readings mixed, a unit tag that does not match the value. That transfer is a hypothesis. The checks have run on MedCalc-Bench and a three-line fixture, and not on FHIR resources or any other format.
• MedCalc-Bench Verified is maintained and changing. Every number here is pinned to commit 58fc3de (release v1.0.8), and a later release may fix some of these rows. A draft issue listing the confirmed rows is in upstream/ISSUE.md and has not been filed.

Rerun

From the labelgate folder. The gate needs no model calls. The library uses only the Python standard library; the tests need pytest.

make data     # download the four pinned CSVs, check sha256
make gate     # run G1 to G4, draw samples, write results/
make models   # three models, cached in raw/
make report   # rebuild this page

Sources

• Verified train, pinned: huggingface.co/datasets/nsk7153/MedCalc-Bench-Verified
• v1.2 train, pinned: huggingface.co/datasets/ncbi/MedCalc-Bench-v1.2
• Verified repo and calculator code: github.com/nikhilk7153/MedCalc-Bench-Verified
• Recompute audit, PR #7: github.com/nikhilk7153/MedCalc-Bench-Verified/pull/7
• Original zero-shot prompt: github.com/ncbi-nlp/MedCalc-Bench, evaluation/run.py
• CKD-EPI 2021, Inker et al., NEJM: doi.org/10.1056/NEJMoa2102953
• MDRD, Levey et al. 2006: doi.org/10.7326/0003-4819-145-4-200608150-00004
• Cockcroft-Gault 1976: doi.org/10.1159/000180580

Both datasets are CC-BY-SA 4.0. Note excerpts on this page are quoted from them with attribution.