OAI Labs — classification-model evaluation | oai-labs.com
Assign each evaluated case to TP, FP, FN or TN, and verify the totals before computing any metric.
A confusion matrix is a counting table. For a binary classifier, each evaluated case is compared against its actual label and its predicted label, and the pair determines the cell. Actual positive with predicted positive is a true positive; actual negative with predicted positive is a false positive; actual positive with predicted negative is a false negative; actual negative with predicted negative is a true negative.
Build the table by iterating over the evaluation set once and incrementing the matching cell. Do not compute metrics during the same pass; finish the counts first. Then check that TP + FP + FN + TN equals the number of cases you intended to evaluate. A mismatch almost always means duplicate rows, dropped rows, or a label that was neither positive nor negative and silently skipped.
On narrow screens, swipe or scroll the plate sideways.
The row sums and column sums are useful cross-checks. TP + FN is the number of actual positives, and TP + FP is the number of predicted positives. In the worked example those are 12 and 10 respectively out of 20 cases. Writing these two derived counts next to the matrix makes the precision and recall denominators obvious and prevents the most common arithmetic slip.
Keep the matrix at case level rather than aggregating early. If a case has more than one label in a multilabel setting, the binary matrix does not apply directly; evaluate each label separately or use a multilabel formulation, and say which one you used. For single-label multiclass problems, a per-class one-versus-rest matrix is the usual route to the same four cells.
Finally, store the matrix alongside the model version and the evaluation set identifier. A metric without its cells cannot be audited later, and a comparison between two runs is only meaningful if both matrices were built the same way.
Further reading