Trained against the same model before training

Where the answers land

Each dot is one test case, drawn at the option the model chose. Teal means that option is the gold label. Red means it chose something else. A model that has not learned the question piles onto one or two options.

Each question, by confidence

One dot is one test question. Teal means the top option matched the gold label. Red means it did not. Dots sit further right as the model assigns more probability to its top option. The line is the gate: at or above it, an application would act on its own.

Accuracy

Share of questions whose top option matches the gold label. These bars are the full test split. The filter above changes the dots and the gate, not these bars.

Does the confidence match the hit rate?

Each point is a band of stated confidence. It should land on the diagonal: questions the model calls at 80% ought to be right about 80% of the time. Points below the line are overconfident.

Time for one case