A prediction model can place people in the right order while giving them misleading probabilities. It can also match the overall event rate while failing to distinguish anyone's risk. Reading only one performance statistic leaves these possibilities unresolved.
The learning task is to separate discrimination, calibration and validation, then explain what each piece of evidence supports. The examples concern a future binary outcome measured at a fixed time. They do not establish a clinical decision threshold or recommend action for an individual.
Define the prediction before inspecting performance
In an original fictional rehabilitation study, a model predicts the probability that a participant has not resumed a predefined daily task at eight weeks. Predictors are recorded at entry; the task and outcome definition are fixed in advance. Every fictional participant has a known eight-week outcome.
That specification matters. “Not resumed by eight weeks” is different from eventual recovery, time to recovery or a person's priorities. A model developed for one endpoint cannot quietly become a prediction of another. Before reading its AUC, write down the population, prediction time, outcome and follow-up horizon.
For binary outcomes, the AUC summarizes ranking: how often a participant with the event receives a higher predicted probability than one without it, with ties receiving half credit. Calibration concerns agreement between predicted probabilities and observed event frequencies. These answer different questions. Steyerberg and colleagues
Fictional worked example: inspect grouped calibration
The unchanged model is applied to 200 new participants. For teaching, they are grouped by predicted probability. Each row reports the mean prediction within its group, rather than asserting that every person received the same prediction.
| Group | Participants | Mean predicted probability | Predicted event count | Observed events | Observed proportion |
|---|---|---|---|---|---|
| 1 | 50 | 10% | 5 | 10 | 20% |
| 2 | 50 | 30% | 15 | 20 | 40% |
| 3 | 50 | 50% | 25 | 30 | 60% |
| 4 | 50 | 70% | 35 | 40 | 80% |
| Total | 200 | 40% overall | 80 | 100 | 50% overall |
Predicted counts are sums of probabilities: 50 × 0.10 = 5 in group 1. They are expected counts under the model, not assertions that exactly five named people will experience the event.
The model predicts 80 events altogether, while 100 are observed. Its mean prediction is 80 ÷ 200 = 40%; the observed event rate is 100 ÷ 200 = 50%. In every displayed group, the observed proportion exceeds the mean prediction by 10 percentage points. This is a descriptive pattern of underprediction in these data.
Do not call that “a 10% error” without specifying the denominator. For group 1, ten percentage points separate 10% and 20%; the observed proportion is twice the prediction. For group 4, the same absolute gap separates 70% and 80%, but the relative comparison differs.
The grouped table makes the mismatch visible. It cannot reveal every pattern within a group, and its observed proportions contain sampling uncertainty. A calibration curve with uncertainty and information about where predictions are concentrated provides a more detailed assessment. Agreement of the overall mean alone is weaker evidence than agreement across the prediction range. Van Calster and colleagues
Nor can this table identify the exact individual-level AUC. Group means and counts omit the ranking of event and non-event participants within each group. Avoid manufacturing precision from a summary that does not contain it.
A ranking counterexample you can calculate
Consider four different fictional participants with known outcomes:
| Participant | Event occurred? | Model P | Model Q |
|---|---|---|---|
| A | No | 10% | 60% |
| B | No | 20% | 70% |
| C | Yes | 30% | 80% |
| D | Yes | 40% | 90% |
There are two event participants and two non-event participants: 2 × 2 = 4 event/non-event pairs. In both models, C outranks A and B, and D outranks A and B. Both therefore have an empirical AUC of 4 ÷ 4 = 1 in this tiny table.
Their probabilities differ greatly. P predicts a total of 0.10 + 0.20 + 0.30 + 0.40 = 1 expected event. Q predicts 3 expected events. Two events occurred. Moving all four predictions upward preserved their order, so the AUC did not change. A rank statistic is insensitive to this kind of systematic probability shift. Steyerberg and colleagues
Four observations do not establish either model's population calibration. They are a counterexample to the claim that an AUC of 1 determines the probability values. The perfect ranking is an arithmetic property of this constructed sample, not credible evidence of perfect future performance.
The opposite boundary is also useful. Give every person in the 200-participant example a 50% prediction. The mean then matches the observed 50% event rate exactly, but every event/non-event comparison is tied. Its empirical AUC is 0.5. Matching the average has created no risk separation.
Ask what kind of validation was performed
Performance measured on the data used to develop a model can be optimistic. Internal validation, such as bootstrapping or cross-validation, examines performance within the development-data framework. External validation applies the original model to new participant data, for example from a later period or another setting. The 2015 TRIPOD statement distinguishes these activities and emphasizes evaluation of the original model before updating. Collins and colleagues
Imagine the 200 participants came from another Canadian rehabilitation service. Ask whether the outcome was measured the same way, whether entry predictors were available at the intended prediction time, and whether anyone was excluded because data were missing. A polished performance table cannot answer those questions.
If investigators change the model after seeing the 100 events, separate two results: how the original model performed and how the updated model was assessed. Reusing those same outcomes to adjust predictions and advertise the adjusted fit does not provide independent evidence about future performance.
“Externally validated” also needs a location and a population. It does not mean permanently validated everywhere. Differences in populations, event frequency and predictor measurement can affect calibration. Van Calster and colleagues
A reporting checklist helps locate information; completion is not a quality certificate. TRIPOD explicitly distinguishes reporting guidance from a tool that rates methodological quality. Collins and colleagues
Transfer exercise: the overall average agrees
A third original fictional evaluation contains three groups of 40 participants:
| Mean predicted probability | Observed events |
|---|---|
| 20% | 12 |
| 50% | 20 |
| 80% | 28 |
Which conclusion is best supported?
A. The model is well calibrated throughout because total predicted and observed events match.
B. The overall mean agrees, but the grouped pattern suggests predictions are too extreme.
C. Its exact AUC is 0.80.
D. All individual probabilities should immediately be replaced by 50%.
Feedback: B. Predicted counts are 8, 20 and 32, totaling 60. Observed counts also total 60. Both overall proportions equal 60 ÷ 120 = 50%.
Within groups, however, observed proportions are 30%, 50% and 70%. The lower predictions are ten percentage points below their observed proportion; the higher predictions are ten points above it. Overall cancellation conceals the mismatch.
This pattern motivates closer calibration assessment. It does not identify a unique cause, prove a particular calibration slope or establish a ready-to-use correction. A overlooks the groups; C invents an AUC from insufficient ranking information; D turns a descriptive finding into an unsupported model replacement.
Turn the distinction into a useful reading note
Record the exact endpoint and horizon, the discrimination estimate with uncertainty, the calibration evidence and the population used for validation. Then state one remaining question that affects applicability.
Even favourable results on those checks would not establish that using the model improves care. Decisions also involve the consequences of acting, available alternatives and the person's circumstances. No threshold is selected here. The immediate skill is more modest and more useful: identify precisely what the reported numbers show before attaching a decision to them.
Sources and further reading
- Steyerberg et al. (2010): Assessing the performance of prediction models—a framework for some traditional and novel measures.
- Van Calster et al. (2019): Calibration—the Achilles heel of predictive analytics.
- Collins et al. (2015): The TRIPOD Statement. Used for its explanation of validation and reporting scope, not presented as the latest reporting checklist.