“Randomized trial” describes an important feature of a study. It does not finish the appraisal. You still need to know what happened after allocation, which outcomes were measured and which result the paper is asking you to trust.
Cochrane’s risk-of-bias framework assesses a specific result through the trial’s design, conduct and reporting. That focus is useful for students: a paper is not simply “good” or “bad” in every respect. Different outcomes can have different vulnerabilities. Cochrane Handbook, Chapter 8.
The walkthrough below is an original learning exercise, not a replacement for the full RoB 2 tool. The trial and numbers are fictional.
Specify the result before judging it
Suppose a study assigns 120 adults to either a new education programme or an existing education programme. The main result is a self-reported ability score at 12 weeks. Sixty people enter each group.
Write one sentence defining the comparison: “The study estimates the effect of assignment to the new rather than existing education programme on the reported score at 12 weeks.” That is clearer than “the study tests rehabilitation.” It identifies the programmes, outcome, time and effect of interest.
Next ask whether the report’s analysis matches that question. Restricting analysis to people who attended every session no longer estimates the effect of assignment in the full randomized group. It also does not automatically estimate the causal effect of completing the programme: attendance may depend on prognosis, barriers or early response. Estimating that effect requires an appropriate method and additional assumptions, not simply deleting non-attenders.
Follow the participant through the study
Start with enrolment. How was the allocation sequence generated, and could anyone predict the next assignment while deciding who entered? Concealing future assignments and keeping outcome assessors unaware of treatment are different safeguards at different stages. Do not treat the word “blinded” as an explanation of both.
Then examine what each group actually received. Did one receive additional contact, encouragement or access to other services? Depending on the question, those differences may be part of the programme or a complication in attributing the result to one ingredient. Write the ambiguity rather than silently choosing an interpretation.
Finally, locate the outcome. Was it collected in the same way for both groups? If participants knew their allocation and reported their own score, think about how that knowledge could affect this particular measurement. That concern differs from the concern for an outcome recorded automatically by a device.
Make missing data visible
In the fictional report, 54 of 60 participants in the new-programme group provide the 12-week score. Only 36 of 60 in the comparison group do so. The article reports higher average scores among those measured in the new-programme group.
The missing proportions are 10% and 40%. Those numbers raise a question, but they do not reveal the direction or size of bias on their own. Ask why scores are missing and whether the reasons relate to the unobserved outcomes.
For instance, comparison participants doing especially well may have skipped the follow-up because they no longer felt a need to attend. Alternatively, people struggling may have withdrawn. These stories point in different directions. Neither becomes a fact because it is plausible.
Calling an analysis “intention-to-treat” does not resolve missing outcomes. Read who was included, how missing values were handled, and whether sensitivity analyses examine plausible departures from those assumptions.
A useful appraisal note says: “Follow-up differs substantially between groups. The report needs to explain reasons for missing outcomes and show how conclusions depend on assumptions about them.” A weak note says only “small sample” and moves on. It misses the mechanism that could distort this result.
Check what was planned
Now imagine the trial registration names a 12-week activity outcome, while the paper’s headline emphasizes a favourable four-week confidence score. That does not automatically prove selective reporting. It does make the timing, outcome list and explanation for changes important to inspect.
Record the exact discrepancy. Search for the protocol, amendments and other reported results. If you cannot find them, say so. “Not reported” and “not done” are different claims.
Avoid inventing an overall quality score by counting favourable features. Five reassuring details do not necessarily cancel one serious threat to the result you care about. Use the full appraisal method required by your course and justify its judgements.
Separate bias from applicability
Suppose the trial’s conduct is convincing, but it enrolled only people who could attend three daytime appointments each week. Your fictional placement serves many people working shifts. The estimate may be credible for the study population while the delivery model remains difficult to use locally.
That is an applicability question. It does not mean the study is fraudulent or useless. It means you should identify the access requirement before assuming the result transfers.
Create two headings in your notes: “Can I trust this estimate?” and “How closely does it fit this question?” The first concerns the result’s credibility. The second concerns the population, setting, programme and outcome you need to understand.
Practise a balanced conclusion
For the invented trial, a defensible conclusion is:
The reported result favours the new programme among participants with recorded outcomes. Unequal follow-up and limited information about missing scores reduce confidence in that estimate. Attendance requirements also need comparison with the intended setting before considering practical relevance.
This conclusion is neither an endorsement nor a dismissal. It explains the uncertainty that matters.
For your next paper, choose one main result and trace it from allocation through analysis. Bring one documented strength, one mechanism that could bias the result and one applicability question to your discussion. Each should point to a specific part of the report.
Sources and further reading
- Cochrane Handbook Chapter 8 — Risk of bias in a randomized trial — Result-specific appraisal of randomized trials and sources of bias.