Why did a 12:21 JT score grade as Rookie?

A 12:21 Rx score on JT came back as Level 1, Rookie.

PRzilla had graded it correctly. The stored benchmark ladder told the app that anything slower than 12:00 belonged at Level 1. It also put the median Rx cutoff at 8:00 and the top cutoff at 4:00.

Those numbers were the bug.

How one score exposed the ladder

PRzilla stores nine cutoffs for each benchmark workout. They map to Levels 2 through 10. For a workout scored by time, lower is better, so each faster cutoff earns the next level. A score slower than the Level 2 cutoff receives Level 1.

Here is what that meant for JT:

Cutoff	Old ladder	Corrected ladder
Level 2	12:00	21:00
Level 5	8:00	12:30
Level 10	4:00	5:00

Twelve minutes and 21 seconds is 741 seconds. Under the old ladder, 741 was slower than the first 720-second cutoff, so the production grading function returned Level 1. Under the corrected ladder, it clears the 750-second Level 5 cutoff.

The fix for JT was small. The uncomfortable part was where the old numbers came from: most catalog ladders had been generated in bulk when the catalog was built, and only a small set of famous workouts had been reviewed as ground truth.

The conclusion I took from JT was simple: stop trusting the rest by default.

Roughly 50 ladders later

The follow-up audit read each workout, counted its work, checked movement and skill constraints, compared the cutoffs with the workout's own duration guidance, and then ran every proposed ladder through the production grader.

Thirty canonical WODs needed changes. The failures were not one repeated arithmetic mistake:

• Bergeron Beep Test had its scoring direction inverted. Lasting longer made the grade worse.
• Bam Bam has a maximum possible score of 225 reps, while five of its old cutoffs ran from 250 to 350. A perfect score could not get past Level 5.
• Seven workouts, including Karen, Manion, and Roy, had women's ladders faster than the men's ladders even though the prescribed loads were relatively heavier or the work was gender-neutral.
• Candy and Blake, two different workouts, carried byte-for-byte identical ladders.
• Several ladders had duplicate cutoffs or large gaps that made levels unreachable or collapsed a whole performance band.

There were also plausible-looking ladders that were wrong in opposite directions. A median effort on Bert graded too low. A 24-minute Michael graded Level 9. Both arrays were monotonic, valid JSON, and easy to accept in review because none of the individual numbers looked absurd on its own.

Tests verified mechanics, not calibration

The grading code could verify structure and behavior. It could prove that a time ladder descended, that every cutoff mapped to the intended level, and that lower times earned better grades. It could not prove that 8:00 was a sensible median time for 135 upper-body pressing reps on JT.

That distinction changed the verification step. Each corrected ladder now had to pass two separate questions:

1. Does the production grader interpret the data correctly?
2. Does the data describe a plausible range of completed work for this workout?

Code tests the first question; workout-specific evidence and judgment settle the second. My rule now is that a generated array can pass every schema and monotonicity test and still be unfit to show an athlete.

Why existing scores fixed themselves

PRzilla stores the completed score and reads the current benchmark ladder when it computes a level. The app does not permanently stamp the old level onto the score.

Once the corrected ladders shipped, existing results used them on the next read. Nobody had to re-log JT or run a score migration. The reported 12:21 result became Level 5, Intermediate.

The remaining audit queue is explicit: a few ground-truth workouts need careful rebaselining, some fixed-duration workouts have scoring-model problems rather than ladder problems, and capped competition workouts need progress-aware thresholds. Those cases stay out of a bulk rewrite until the scoring semantics are handled directly.

If a benchmark grade looks wrong, send the workout, the completed score, whether it was Rx or scaled, and any time cap. That is enough to test both the ladder and the grader against the work you actually completed.