OAI Labs — classification-model evaluation | oai-labs.com
Choose and interpret an averaging method for multiclass evaluation by deciding what the summary number should represent.
In multiclass evaluation you first compute a metric per label, then combine the per-label numbers. Macro averaging takes an unweighted mean, so every label contributes equally. A rare label with poor performance lowers the macro score as much as a common label would, which is exactly what you want when every class matters.
Weighted averaging multiplies each label metric by its support, the number of true instances of that label, and divides by the total support. The result tracks overall per-case performance and is dominated by the large classes. It is appropriate when the evaluation set's class mix resembles the mix the model will meet in use.
On narrow screens, swipe or scroll the plate sideways.
Micro averaging pools the counts across labels before computing the metric rather than averaging per-label scores. In a single-label multiclass problem, micro F1 equals accuracy, which makes it a per-case measure. That equality is a useful check: if your micro F1 and accuracy disagree on a single-label problem, one of the two was computed from different data or with a different convention.
The three methods can rank models differently. A model that is strong on common classes and weak on rare ones may lead on weighted and micro scores while trailing on macro. Before comparing runs, state which averaging the decision will use and why, and keep the per-label table available so the summary can be unpacked.
Averaging does not repair a bad evaluation set. If a label has very few examples, its per-label metric is noisy and the macro average inherits that noise. Report supports alongside the averages so the reader can see how much weight each label carries.
Further reading