Testing our parser on famous workouts hid its worst bugs

PRzilla turns freeform workout text into structured data. You paste what was on the whiteboard, and we work out the rounds, the rep scheme, the loads, the time domain, and which movement each line refers to.

To know whether that works, we built an eval harness that scored the parser against our catalog of named benchmarks: Fran, Grace, Helen, Annie, Angie, Cindy, Mary, Chelsea, CrossFit Total, Bear Complex. Ten workouts covering every benchmark type, with verified ground truth and per-level deviation scoring. The numbers looked good.

The numbers were measuring the wrong thing.

Famous workouts are in the training data

Every one of those benchmarks appears thousands of times across the public internet. Any modern language model has read all of them many times over. When you hand a model 21-15-9 thrusters and pull-ups, you are not testing whether it can parse a descending rep scheme. You are testing whether it remembers Fran.

That is the difference between a recognition ceiling and a capability floor. A model can score well on all ten benchmarks while being unable to read the workout your coach wrote on Tuesday.

What un-named workouts surfaced

So we ran the real parsePrescription against hand-authored workouts that do not exist anywhere. No names, no catalog entry, nothing to recall. Three failures showed up immediately, none of which the benchmark tables had ever produced.

A rotating EMOM collapsed. Given emom 12 / min 1: 15 wallballs 20/14 / min 2: 12 t2b / min 3: 9 power snatch 75/55, the correct reading is that each movement comes up every third minute, four times each. Haiku 4.5 emitted a single EMOM block with rounds: 12 and all three movements inside it, which means: do all three, every minute, for twelve minutes. Roughly three times the actual work.

Rep schemes got double-encoded. On a 22-16-10 couplet, the same model attached reps: 22 to the movements alongside the correct rep scheme. Rounds two and three would have rendered 22 reps instead of 16 and 10.

Details appeared that were never written. Gemini 3.5 Flash produced correct structure but invented scaled loads that were nowhere in the source text. On another input it kept an every-minute accessory as prose in a notes field, so the cadence survived as English and vanished as structure.

Worth being fair about the result: all three models genuinely parse unfamiliar text. They handled kb halo, sa devil press ea, 35s as a dumbbell pair, and 12 cal echo without a catalog to lean on. The problem was never that novel input is unparseable. The problem was that our eval could not see the difference between a model that reads structure and a model that recognizes a title.

What we changed

The scaled version runs 32 hand-authored workouts chosen for structural variety rather than fame: rotating EMOMs, buy-in/buy-out formats, per-side work, prescribed rest, pood and kilogram loads, partner formats, odd objects, two-part sessions. Each is a shape the parser has to get right, and none carries a name worth memorizing.

Because there is no ground truth to score against, a judge model rates each parse for structural fidelity to the original text. It sees the candidate parses under rotated anonymous labels, so it cannot favor a model by position or reputation. Every candidate runs once per workout and every workout is judged once, with results cached per cell so a re-run costs nothing.

The part that generalizes

If your evaluation set is famous, you are scoring recall.

This is not specific to workouts. Any benchmark old enough to be well known is old enough to have been memorized, and the more standard it is, the more thoroughly it has leaked. A good score tells you the model has seen the answer. It tells you very little about the input your users will actually send.

The fix is not clever. Write your own test cases, keep them off the public internet, and choose them for structural coverage rather than recognizability. Three hand-written inputs were enough to surface a bug that would have tripled the volume of somebody's workout.

When you paste your gym's whiteboard into PRzilla, it is not Fran. The eval that matters is the one built from workouts nobody has memorized.