A rehabilitation paper reports a statistically significant benefit in one subgroup and a nonsignificant result in another. The discussion says the programme “only works” for the first group.
That conclusion has skipped a comparison. A test within each subgroup asks whether its effect differs from a reference value. An interaction analysis asks whether the effects differ from each other. The answers need not agree.
The studies, outcomes and numbers in this article are original fictional teaching examples. They do not identify a subgroup that should receive or avoid a real treatment.
Begin with the contrast that matches the claim
Suppose a fictional trial measures a 0–100 ability score at twelve weeks. Higher values indicate better ability. Effects are new programme minus comparator, so positive differences favour the new programme.
Investigators prespecified two nonoverlapping subgroups using symptom duration recorded before randomization. To keep the arithmetic transparent, the supplied effect estimates are independent and their sampling distributions are treated as approximately normal.
| Baseline subgroup | Estimated effect | Standard error | Approximate 95% interval |
|---|---|---|---|
| Shorter duration | +4 points | 1.5 points | +1.06 to +6.94 |
| Longer duration | +3 points | 2 points | −0.92 to +6.92 |
The first interval excludes zero. The second does not. Yet the estimated effects are only one point apart, and their uncertainty is substantial.
Altman and Bland describe the direct comparison of independent estimates and the corresponding interaction calculation. The general lesson is to test the difference between effects rather than compare separate significance decisions. Interaction revisited: the difference between two estimates.
Work through the interaction
For this specific simplified example:
difference between effects = 4 − 3 = 1 point
SE of that difference = √(1.5² + 2²) = 2.5 points
approximate 95% interval = 1 ± 1.96 × 2.5 = −3.9 to +5.9 points
The interval includes zero and differences in either direction. The data do not provide convincing evidence that the programme effect differs between these duration groups under this analysis. They also do not establish that the effects are identical: differences of several points remain compatible with the estimates.
“Only works in the shorter-duration group” is therefore not justified. The longer-duration estimate is positive, and its interval includes effects similar to the shorter-duration estimate. Its failure to cross a significance threshold can reflect imprecision rather than an absence of benefit.
For this exercise, a useful sentence is: “Separate subgroup intervals differ in whether they exclude zero, but the direct interaction estimate is one point with an interval from −3.9 to +5.9.” It states the actual comparison without turning uncertainty into certainty.
Notice the assumptions behind the shortcut
The square-root formula above assumes independent estimates. If the subgroups overlap, the same participants contribute repeatedly, or the estimates share a comparator in a way that creates dependence, their covariance matters. An appropriate model or analysis must account for that structure.
The effect scale matters too. These are differences on one continuous scale. Comparing risk ratios usually requires an interaction calculation on the log-ratio scale; subtracting two risk ratios and copying this standard-error formula would not reproduce the analysis.
A published interaction estimate from a suitable prespecified model is generally more useful than reconstructing one from rounded figures. In a real appraisal, inspect the model, subgroup definition and uncertainty instead of treating the arithmetic here as a universal calculator.
A stronger interaction still needs a credibility argument
Statistical evidence of interaction is one part of a subgroup claim. Sun and colleagues describe additional considerations, including baseline rather than postrandomization subgroup definition, prespecification, the number of hypotheses examined, consistency and a plausible rationale. These are credibility considerations, not a mechanical guarantee. Is a subgroup effect believable?.
Imagine our fictional investigators first inspected twenty possible definitions, including several duration cutoffs, and reported only the most favourable one. The chosen contrast would carry a different evidential status from a single hypothesis specified before results were available. A neat explanation written afterwards would not turn the exploration into confirmation.
Now imagine a subgroup defined by completing at least eight sessions. That characteristic occurs after randomization and may be influenced by the programme and by prognosis. Comparing treatment effects among selected completers would require a more careful causal analysis; the original randomization does not automatically protect that selected comparison.
Conversely, a well-planned interaction can remain imprecise. “Prespecified” improves transparency; it does not manufacture information. Read the uncertainty and the design together.
Different effects can appear on different scales
Consider another original numerical example with an undesirable event. Suppose the new programme halves risk in each of two baseline-risk groups:
| Group | Comparator risk | New-programme risk | Risk ratio | Absolute risk reduction |
|---|---|---|---|---|
| Higher baseline risk | 20% | 10% | 0.5 | 10 percentage points |
| Lower baseline risk | 4% | 2% | 0.5 | 2 percentage points |
There is no difference between the supplied risk ratios, but the absolute reductions differ by eight percentage points. Saying “there is no subgroup difference” without naming the scale would hide useful information. The example follows directly from the supplied risks; it is not a claim that a constant risk ratio applies in every study.
For a decision, an absolute benefit can matter even when the relative effect is similar. For an interaction hypothesis, the planned scale determines what equality means. Do not change the scale after examining results merely to obtain a more attractive subgroup story.
Transfer exercise: calculate, then qualify
A second fictional trial supplies independent subgroup estimates of +6 points with SE 1, and 0 points with SE 1. Use the same normal approximation. What does the direct interaction calculation show? What changes if the subgroup definition was selected after examining twenty candidate analyses?
Reasoned solution: The difference is six points. Its SE is √(1² + 1²), approximately 1.414 points. The approximate 95% interval is 6 ± 1.96 × 1.414, or +3.23 to +8.77 points. This provides statistical evidence of different effects on the specified scale under the stated assumptions.
If the definition was selected after twenty analyses, the calculation itself does not change, but the unadjusted interval does not account for that selection process. The claim becomes exploratory and needs cautious interpretation, appropriate consideration of multiplicity, and further evidence. It would be misleading to present it as a uniquely prespecified confirmation.
The opposite error would be dismissing every subgroup analysis because some are unreliable. A direct, precise, well-supported comparison can be informative. The task is to describe the strength and limits of this particular claim.
Takeaway for your next journal discussion
Before repeating a subgroup headline, identify the effect scale, the interaction estimate and interval, when the subgroup was defined, and how many alternatives were examined. Replace “significant here, nonsignificant there” with the actual estimated difference between effects. That one change makes the reasoning easier to evaluate.
Sources and further reading
- Altman and Bland: Interaction revisited: the difference between two estimates, 2003 — comparison of independent effects and standard-error calculation; indexed excerpts reviewed.
- Sun et al.: Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses, 2010 — subgroup credibility considerations; indexed publisher excerpts reviewed.