Our QA agents needed QA

Our first blind QA run confirmed 1 of 7 known problems.

We gave agents an older version of the app with problems we already knew about, kept the answer key away from them, and had a separate agent grade the report afterward. The requirement was at least 5 of the 7, including the keyboard problem (five other findings still wouldn't pass). We missed it.

This was part of adding QA to our shipping workflow for PRzilla, an AI-powered training companion for CrossFit and hybrid athletes. The workflow was already quite involved: brainstorming, planning, a design pass when needed, code review, automated tests, screenshots, CI, and another review before merging. The new step was to have agents use the running app with a person's goal, then describe where they got stuck.

We start by working through the problem: who needs this, what they are trying to do, and which approach fits the app. Brainstorming feeds the plan, with the scope and tradeoffs spelled out. A small fix can keep this fairly short (it doesn't need a novel before someone can change a label).

When the change affects how a surface looks, the design pass happens during planning. Claude Design gets the actual app references and design system, then produces frames in light and dark. The mocks are reviewed, revised where needed, and brought back for founder approval. The plan is also checked against the repository before implementation (a plausible approach can still depend on something the code doesn't do). Those approved frames become the reference for checking the built screen later — the design decision carries through to verification.

Unfortunately, some observations never became useful findings. A selection moved to another row. A wait lasted longer than ten seconds without enough explanation. The keyboard resisted dismissal (with some uncertainty about the input driver). These remained unconfirmed, unresolved, or insufficiently investigated. There was also a valid input-loss finding outside the answer key — the 1/7 measures known problems confirmed, not the total number of bugs found.

A screenshot containing a problem wasn't enough. The agent had to recognize it, explain its effect, and reproduce it. Reading every returned frame could still leave an obvious problem unreported, as the later timer comparison showed.

The shipping pipeline now includes exploratory QA before screenshot proof and the draft PR. It runs when a change affects a user flow: a screen, navigation, input, waiting, or access. Copy-only and server-only changes skip this step (a server change that affects a visible value can still require screenshot proof).

Open the full-size pipeline diagram. Design and QA steps apply when the change calls for them.

Each participant gets a persona and a goal. For example: log a workout, correct a mistake, and find the saved result again. They start in a fresh conversation and use the app and its screenshots. The implementation, answer key, and route through the controls stay out of the brief. This "blindness" depends on following instructions — it isn't a tool sandbox.

A change confined to one existing screen uses one participant; a new or restructured flow uses two or three. They run sequentially on an isolated iOS simulator, with local test accounts and a separate database. Each gets 50 actions or 15 minutes (whichever comes first). The coordinating agent then checks things a participant might never try: leaving halfway through, canceling, dismissing the keyboard, reopening, etc.

The distinction between "noticed" and "confirmed" is pretty important here. Confirmation requires the initial observation plus two fresh account-and-install replays — preferably before spending the remaining time on another optional variant. An unresolved observation stays in the report, with the missing evidence explained (running out of replay time doesn't make the problem go away).

One later example was fairly easy to explain with pictures. Entering run 1500 put 1500 under a km heading in the editor. The summary of the same run showed 1.5 km. Reopening the editor brought back the mismatch. That's enough to demonstrate conflicting units. A claim about corrupted stored data would need another check (the screenshots don't tell us what's in the database).

Actual simulator captures from October 14, 2026, using a synthetic test account. The summary has an onboarding overlay, but the distance is readable. These show the historical display mismatch, not proof of the later fix.

After QA, we capture proof of the intended behavior and open a draft PR. The report travels with it: screenshots, open problems, skipped cases, and the code version tested. The default ship command stops there for founder review. An explicit merge request continues through updating from main, full CI, review feedback, merging, and applicable post-merge checks — including whether the original plan was actually delivered.

We also compared two model configurations on a smaller frozen set of timer and distance tasks. Perhaps that deserves its own post. It was a separate experiment, and it doesn't replace the first score or establish reliability across the app.

The revised instructions still haven't passed a new blind calibration. For now, "QA ran" tells us which work was done and what evidence came back. How much would you trust it before testing it on another set of bugs it hasn't seen?