Uncertainty, Calibration and Abstention Thresholds
A model that predicts a probability should be honest about its uncertainty. A predicted probability of 0.7 should mean that similar cases are actually positive about 70 percent of the time. If a model is miscalibrated, its probabilities are not reliable and you cannot trust the confidence it expresses.
A model that is overconfident predicts 0.9 for cases that are actually positive only 60 percent of the time. This overconfidence can lead to serious errors in high-stakes decisions. For example, in medical diagnosis, a model that predicts a 90 percent chance of cancer may actually be correct only 60 percent of the time.
If clinicians rely on this model without understanding its miscalibration, they may proceed with treatments that are not necessary or delay critical care.
Calibration is often overlooked by data scientists but matters greatly for high-stakes decisions. If a credit model assigns a probability of default of 0.05 to an applicant but applicants with similar profiles actually default 0.20 of the time, the model is severely miscalibrated. You would deny credit to applicants who are actually creditworthy based on miscalibrated probabilities.
The model fails in production despite good test accuracy. In a real-world scenario, a lending institution using such a model may reject loans for individuals who are financially stable, resulting in lost business and customer dissatisfaction. This also increases the risk of regulatory scrutiny and reputational damage.
You can measure calibration using a calibration curve: plot predicted probability on the x-axis and actual positive rate on the y-axis. If the model is perfectly calibrated, the curve is a diagonal line from (0,0) to (1,1). If the curve is above the diagonal, the model is underconfident (says 0.3 but is actually positive 0.5 of the time).
If below, it is overconfident (says 0.7 but is actually positive only 0.5 of the time). A practical example is a fraud detection system that flags transactions at a 0.8 probability. If the actual fraud rate at that level is only 0.4, the model is overconfident and may lead to excessive manual reviews or false positives that waste valuable resources.
Miscalibration often arises from class imbalance during training or when you train and test on different distributions. A model trained on 50 percent defaults will produce different probabilities than a model trained on 5 percent defaults, even if both have the same accuracy. The probabilities shift because the model learns different patterns based on class balance.
For instance, a model trained on a dataset where only 2 percent of customers default may predict a 0.01 probability for a customer who is actually a default risk. This can occur in insurance underwriting, where rare events like claims are underrepresented in training data.
In such cases, the model may appear accurate on average but fails to reflect the true risk for specific segments.
You can post-hoc calibrate a model by fitting a function to correct predicted probabilities. Platt scaling and isotonic regression are common approaches. Post-hoc calibration does not change the model’s decision boundary but makes probabilities more reliable and honest.
An example of this in practice is a model used in legal risk assessment. After initial training, the model is post-hoc calibrated using a validation set. This process ensures that a 0.6 probability of reoffending truly reflects a 60 percent chance, allowing judges to make more informed decisions.
This is especially critical in systems where human oversight is required and model confidence must be trusted.
An abstention threshold is a confidence level below which the model does not make a prediction and instead asks a human to decide. If the model’s confidence is below the threshold, the human makes the call. This allows you to handle cases where the model is uncertain and should not decide alone.
High-stakes decisions should almost always have an abstention threshold to catch edge cases. For example, in autonomous driving systems, a model may predict a 0.5 probability that an obstacle is present. If the abstention threshold is set at 0.6, the system would prompt a human driver to take over.
This is essential in scenarios where the model’s uncertainty is too high to make a safe decision autonomously. Similarly, in healthcare, a model that predicts a 0.4 probability of a critical condition may prompt a physician to conduct further tests rather than proceed with treatment based on uncertain data.