Why good calibration is not enough when data change

What “80 percent” means

Calibration describes how well the confidence a model states for its predictions matches the observed frequency of correct predictions. Guo and colleagues explain this with a simple example: of 100 predictions, each made with 80 percent confidence, one would expect 80 to be correct under perfect calibration.[1] The example illustrates the concept; it is not a measured result. In any case, with a finite number of predictions, calibration can only be measured approximately.[1]

Calibration is not the same as accuracy. On the image dataset CIFAR-100, Guo and colleagues compared an older, small network with a modern, much deeper network. The older one was wrong more often, but its average confidence closely matched its accuracy. The modern network was right more often, but on average it was considerably more confident than its accuracy justified.[1]

Both quantities describe groups of predictions. They do not show which individual prediction will be wrong. A figure of 80 percent is no guarantee for the individual case; it can only be checked across many comparable cases.

What recalibration changes

Temperature scaling adjusts the probabilities after training. Its single parameter is determined on separate validation data.[1] The predicted class stays the same, so accuracy on the same examples does not change.[1] In the experiments by Guo and colleagues (2017), the method reduced the measured calibration error for almost all of the examined combinations of image or text dataset and model. The exception was a text dataset that was already well calibrated beforehand.[1]

In doing so, the authors explicitly assumed that training, validation and test data come from the same distribution.[1] Their results therefore show what the method achieves under unchanged conditions.

When the data change

New images may, for example, be blurrier than the previous test images. A favourable calibration score from the original environment then does not answer how reliable the probabilities are under the changed conditions.

In 2019, Ovadia and colleagues studied several methods under such dataset shifts. For the image datasets CIFAR-10 and ImageNet, they artificially altered the test images, using 16 types of corruption such as blur, each at five intensities.[2] As the alteration increased, accuracy and calibration quality declined.[2]

Temperature scaling achieved low calibration errors under weak alterations and helped clearly compared with the uncalibrated model. Under stronger alterations, however, its calibration error rose sharply. Recalibration on data from the original distribution therefore did not guarantee calibration under dataset shift.[2] Other methods, above all ensembles of several independently trained models, performed better under stronger alterations but likewise lost quality.[2]

The results apply to the examined models, datasets and artificially generated alterations. They do not provide a fixed error rate for another model or its deployment.

Which questions follow from this

From these findings, this post derives three questions for checking. They are an editorial assessment, not a result of the studies:

• With which data were the probabilities recalibrated?
• Were accuracy and calibration then tested on data used neither for training nor for recalibration?
• Which changes in the input data relevant to the planned use were additionally examined?

A dataset shift does not necessarily make a model unusable. It may, however, require additional testing before existing statements about the reliability of its probabilities are carried over.

Note: This post puts two published studies into context. No original model measurements were carried out.