A diagnostic accuracy table has two classifications: the test being evaluated and the reference standard used to determine the target condition. Before calculating sensitivity or specificity, ask whether everyone had both classifications established in a comparable way.
If receiving the reference standard depends on the index-test result, the verified group may not represent everyone tested. A neat two-by-two table can then describe a selected subset accurately while misrepresenting accuracy in the intended population.
Your task is to trace that selection, calculate its consequences and identify when a sampling correction is justified.
Find the people outside the table
Partial verification means that only some participants have their condition status established by the reference standard. When selection for verification is related to the index-test result, a naive accuracy calculation among verified participants can be biased. Simply increasing the size of a similarly selected sample does not remove that mechanism. FDA diagnostic-test statistical guidance
The issue is not that every diagnostic study must expose everyone to every possible procedure. A reference procedure may be costly, burdensome or inappropriate for some participants. The research design and analysis must account for how condition status becomes known; the ethical choice cannot be settled by a desire for a complete table.
Worked example: the complete fictional population
Original fictional teaching study. Two hundred people receive an index test at a fixed threshold. For the first part of the exercise, imagine an accurate reference standard establishes everyone's condition status independently of the index result. The condition does not change between assessments, and the index result does not influence the reference classification.
| Index-test result | Condition present | Condition absent | Total |
|---|---|---|---|
| Positive | 48 | 28 | 76 |
| Negative | 12 | 112 | 124 |
| Total | 60 | 140 | 200 |
Sensitivity is the proportion testing positive among those with the condition:
48 ÷ (48 + 12) = 80%.
Specificity is the proportion testing negative among those without it:
112 ÷ (28 + 112) = 80%.
These numbers describe this constructed population at this threshold. They are not estimates for a real diagnostic test or instructions for an individual assessment.
Now change who gets verified
Suppose investigators verify all 76 index-positive participants, but randomly select only one quarter of the 124 index-negative participants for verification. Of the 31 selected negatives, our constructed sample contains three with the condition and 28 without it.
The verified-only table is now:
| Index-test result | Verified condition present | Verified condition absent | Verified total |
|---|---|---|---|
| Positive | 48 | 28 | 76 |
| Negative | 3 | 28 | 31 |
| Total | 51 | 56 | 107 |
A naive calculation gives:
Sensitivity = 48 ÷ 51 ≈ 94.12%.
Specificity = 28 ÷ 56 = 50%.
Neither the test nor the full population changed. The table changed because index negatives were much less likely to have their condition status included. False negatives fell from 12 to three, while true positives stayed at 48. True negatives fell from 112 to 28, while false positives stayed at 28.
This design inflates verified-only sensitivity and reduces verified-only specificity in our example. “Verification bias makes every accuracy statistic look better” would therefore be an incorrect generalization.
The 93 unverified participants cannot be labelled condition-absent merely because their index test was negative. Doing that would use the test under evaluation to supply its own missing reference classification.
A correction that matches a designed sample
A planned design may verify all index positives and a random sample of index negatives, then account for the sampling fractions. That is different from simply ignoring unverified people. Kohn, abstract
Here, an index-positive participant has inclusion probability one. An index-negative participant has inclusion probability one quarter. The corresponding inverse-probability weights are 1 and 4.
Weight the verified counts by their index-result sampling stratum:
| Cell | Observed verified count | Weight | Estimated population count |
|---|---|---|---|
| True positive | 48 | 1 | 48 |
| False positive | 28 | 1 | 28 |
| False negative | 3 | 4 | 12 |
| True negative | 28 | 4 | 112 |
The adjusted sensitivity is 48/(48 + 12) = 80% and adjusted specificity is 112/(28 + 112) = 80%.
They recover the full-population values exactly because we constructed a sample with exactly one quarter of each negative cell. In an actual random sample, the selected condition mix would vary. Weighting estimates the population cells; it does not reveal the true status of each unverified person or create additional observed outcomes.
Uncertainty calculations must also respect the sampling design. Treating the weighted 200 as 200 independently verified observations would misrepresent the information available. FDA guidance, section 4.1
Boundaries: when the simple weights fail
The correction rests on known inclusion probabilities and genuinely random selection within the stated index-result stratum, along with a suitable reference standard. “We verified roughly a quarter of negatives” is not enough.
Imagine clinicians preferentially verify negative results when other findings raise concern. The verified negatives may then have a different condition distribution from the unverified negatives. Assigning everyone a weight of four would not, by itself, correct that additional selection. A more detailed design or analysis would need to address the determinants of verification.
At a stronger boundary, suppose no index-negative participant is ever verified. This study then supplies no observed reference classifications for that stratum. Sensitivity cannot be rescued merely by entering zero in the false-negative cell. Additional information or assumptions would be needed, and the absence of data should remain visible.
Partial verification is also separate from a flawed reference standard. Perfectly recording who received an inaccurate or inconsistently applied reference procedure would not solve its classification errors. Both the selection process and the reference procedure need scrutiny.
Transfer exercise: do not reuse the verified-only denominator
Original fictional exercise. A second population has 200 people: 40 index positives and 160 negatives. Investigators verify all positives and a simple random quarter of negatives.
The verified positives include 20 with the condition and 20 without it. The 40 verified negatives include five with the condition and 35 without it. Assume the same reference and sampling conditions as above.
What are the inverse-sampling-weighted sensitivity and specificity point estimates?
A. 80% sensitivity and approximately 63.64% specificity.
B. 50% sensitivity and 87.5% specificity.
C. 100% sensitivity and 100% specificity because most negatives are unverified.
Feedback: B is correct. Weighting negative cells by four estimates 5 × 4 = 20 false negatives and 35 × 4 = 140 true negatives. Sensitivity is 20/(20 + 20) = 50%; specificity is 140/(140 + 20) = 87.5%.
A uses only verified counts: 20/25 = 80% and 35/55 ≈ 63.64%. C treats missing reference information as proof of accuracy. Neither follows the designed sampling correction. Even B remains a point estimate requiring an appropriate uncertainty analysis.
A practical paper-reading check
Trace the participant flow separately for positive and negative index results. Record how many received the reference standard, why others did not, the timing, and the analysis of incomplete verification. Then read the accuracy table. That sequence helps you recognize when its denominators describe the target population and when they describe only the people whose condition status happened to be established.
Sources and further reading
- FDA: Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests, 2007. Relevant full-text bias and partial-verification sections consulted; statistical guidance, not a Canadian clinical requirement.
- Kohn: Studies of diagnostic test accuracy—partial verification bias and test result-based sampling. Complete PubMed abstract consulted; full article not accessed.