A rehabilitation trial can measure several useful outcomes without doing anything wrong. The interpretive problem begins when the conclusion is chosen after searching the results: the smallest p-value becomes the headline, while the original question and less favourable findings disappear.
Your task in this lesson is to reconstruct the set of comparisons that could have produced the claim. Then ask whether the study's analysis and reporting support that particular conclusion.
The question defines what success means
A protocol might identify one primary outcome at one follow-up time. Another might specify a family of outcomes, with a rule controlling the chance of at least one false-positive conclusion across that family. These are different plans.
Multiplicity can arise from outcomes, time points, subgroups or other analysis choices. The FDA's statistical guidance explains why the intended family and testing strategy should be considered prospectively. It is US regulatory guidance for drug and biological-product trials; we use its statistical reasoning here, not as a Canadian physiotherapy regulatory requirement. FDA multiple-endpoint guidance
“Prespecified” should mean that the relevant choice was documented before the results could guide it. A clear plan also identifies the population, outcome definition, time point, statistical model and handling of deviations. A heading saying “primary analysis” in the final paper does not by itself establish that chronology.
Worked example: six opportunities for a headline
Original fictional teaching trial. Researchers compare two rehabilitation education programmes. They measure three constructs at two follow-up times. Suppose the reported individual p-values come from appropriate two-sided tests of no between-group difference for each listed comparison. The original plan tests its single primary comparison at a two-sided significance level of 0.05.
| Construct | Time | Individual p-value | Role in dated original plan |
|---|---|---|---|
| Functional task score | Eight weeks | 0.180 | Single primary comparison |
| Functional task score | Four weeks | 0.041 | Secondary |
| Task confidence | Four weeks | 0.027 | Secondary |
| Task confidence | Eight weeks | 0.210 | Secondary |
| Task completion performance | Four weeks | 0.018 | Secondary |
| Task completion performance | Eight weeks | 0.330 | Secondary |
After seeing the results, an author proposes: “The programme was successful because task completion performance improved, p = 0.018.”
The first problem is a change of question. The planned primary comparison was the functional score at eight weeks, with p = 0.180. Selecting a different construct and time because its p-value is smaller does not make it the originally planned primary test.
The second problem is incomplete reporting. Readers need the effect estimates, uncertainty intervals and remaining outcomes, including any unfavourable findings. A single p-value does not show the size or importance of an effect, nor the probability that the null hypothesis is true. American Statistical Association's official six-principle summary
A defensible account could state that the prespecified primary comparison did not meet its planned significance criterion, then describe the secondary pattern with its estimates and limitations. It should not turn the primary result into proof of no effect, or the selected secondary result into a confirmed broad programme benefit.
A probability calculation with explicit conditions
For an illustration, imagine six independent, valid tests, each performed at a 0.05 level. All six null hypotheses are true, and each test has an actual rejection probability of exactly 0.05 under this model; this additional assumption excludes conservative tests with a smaller probability. Define a false positive as rejection in either direction for each two-sided test.
For one test, the probability of no false positive is 0.95. Independence allows multiplication:
Probability of no false positives = 0.95⁶ = 0.7350919.
Therefore:
Probability of at least one false positive = 1 − 0.95⁶ = 0.2649081, or about 26.49%.
This is a repeated-study property under the stated all-null model. It is not a 26.49% probability that the selected claim in our fictional trial is false. It also does not describe only falsely favourable results: the calculation counts rejection in either direction.
Our actual six outcomes are plausibly correlated, so independence is a teaching assumption, not an established feature of the table. At the opposite boundary, repeating exactly the same test six times gives the same rejection event six times, leaving its probability at 5%. Counting comparisons remains useful, but the numerical familywise probability depends on their dependence and decision rules.
What a correction can—and cannot—do
Suppose a different trial had prospectively defined all six comparisons as one family, using a Bonferroni rule with familywise significance level 0.05. Each individual comparison would need:
p ≤ 0.05 ÷ 6 = 0.008333…
None of the six listed values meets that threshold. The smallest, 0.018, has a Bonferroni-adjusted value of min(1, 6 × 0.018) = 0.108.
Bonferroni controls the probability of any false rejection in the defined family when its individual tests are valid; independence is not required. It can be conservative, and other prespecified approaches may be more suitable. FDA guidance, Bonferroni and related procedures
Notice that this is a hypothetical alternative plan. We should not silently rewrite the original single-primary plan into an equal-status six-outcome family. Nor does applying a correction afterward to six reported tests repair a search that actually included dozens of unreported thresholds, transformations and subgroups. The visible table might not contain the whole selection process.
Keep useful exploration visible
Exploratory findings can suggest a more focused future study. The four-week task-performance result may be worth investigating, especially if its direction, magnitude and measurement properties make sense. Its value does not require relabelling it as confirmation.
A transparent interpretation identifies the finding as exploratory, reports its effect and uncertainty, explains the selection process and resists a conclusion broader than the measure supports. A later study can test the new question prospectively.
Equally, multiplicity is not a reason to hide outcomes. Harms, burdens and unexpected patterns can matter even when no single p-value crosses a threshold. Reporting and confirmatory testing serve different purposes. A table can be complete without declaring every entry a separate established effect.
Plans can change for legitimate reasons, such as discovering that an instrument was unavailable. Record what changed, when, why and whether anyone had seen comparative results. A deviation is information to examine, not automatic evidence of misconduct.
Transfer exercise: follow the stipulated family rule
Original fictional exercise. A separate trial prespecifies four primary comparisons as one family. Its analysis plan uses Bonferroni control at 0.05. The four valid individual p-values are 0.009, 0.021, 0.200 and 0.700.
Which conclusions meet this stated testing rule?
A. The first two, because both are below 0.05.
B. Only the first, because the individual threshold is 0.0125.
C. None, because any study with multiple outcomes is uninterpretable.
Feedback: B is best. 0.05 ÷ 4 = 0.0125. The first value qualifies; the second does not. Their adjusted p-values are 0.036, 0.084, 0.800 and 1.000, with the final value capped at one.
A applies the individual 0.05 rule while ignoring the stipulated family rule. C mistakes a manageable design issue for a reason to discard all evidence. Nothing in these four p-values alone tells us which effect is largest, whether it is clinically important or how precisely it was estimated.
Your next appraisal step
Find the earliest accessible protocol or analysis plan and compare it with the final report. Write down the planned primary question, the full set of reported comparisons and any changes. Then ask whether the conclusion follows the plan or follows the smallest number. That check gives the statistical result a context that a significance label cannot supply.
Sources and further reading
- FDA: Multiple Endpoints in Clinical Trials—Guidance for Industry, 2022. Relevant full-text sections on multiplicity, prospective strategies and Bonferroni procedures consulted.
- American Statistical Association: official 2016 release summarizing six principles on p-values. Full three-page official summary consulted; this is not presented as the full journal commentary.