A familiar questionnaire can be easy to score and still answer the wrong question. Before asking which outcome measure is “best,” decide what the score must represent, for whom and for what decision. Those choices shape what evidence you need.
COSMIN distinguishes the construct being measured from the instrument used to measure it. Its selection guidance starts by defining the outcome precisely enough to judge whether an instrument covers it. COSMIN: deciding what to measure.
This article concerns selecting a measure. The separate QuietStride article on measurement error and important change concerns interpreting a change after an appropriate measure has been chosen. The programme, instruments and evidence summaries below are entirely fictional.
Write a measurement brief before comparing scores
Imagine a Canadian PT teaching team planning a fictional evaluation of an upper-limb rehabilitation programme. Their stated question is: “How does participants’ reported difficulty carrying out ordinary household tasks change over twelve weeks?”
A useful measurement brief specifies:
- Construct: self-reported difficulty with household tasks, rather than confidence, satisfaction or isolated strength.
- Population: adults with the relevant upper-limb limitations and the language and reading needs represented in the planned study.
- Purpose: measuring change over twelve weeks, with group comparisons as the main analysis.
- Administration: the same supported mode and recall period at each assessment.
The brief does not make the study good by itself. It prevents the team from selecting a tool because its name sounds relevant and discovering the mismatch after data collection.
“Recovery” would be too broad for this task. Different scores might legitimately represent different aspects of recovery. They would not thereby become interchangeable.
Compare three plausible candidates
The teaching team receives these invented instrument summaries:
| Candidate | What it measures | Supplied evidence and practical features |
|---|---|---|
| Grip Output Test | Force under a specified test procedure | Very repeatable in the target population; equipment already available |
| Daily Task Confidence Scale | Confidence in handling household tasks | Two minutes to complete; good evidence for its confidence construct in a relevant population |
| Household Task Difficulty Questionnaire | Reported difficulty with a range of household activities | Relevant content and change-score evidence in a similar population; six minutes to complete; suitable version available |
The grip test’s repeatability does not make it a direct measure of reported task difficulty. The confidence scale is closer in everyday wording, but confidence and experienced difficulty can change differently. Someone could feel more confident while still reporting the same difficulty, or find a task easier while remaining cautious.
Given the supplied facts, the difficulty questionnaire is the best-supported primary candidate for this particular brief. Its case rests on alignment and relevant evidence, not the size of its name, the convenience of its format or a claim that questionnaires are always preferable to performance tests.
The other measures might answer worthwhile secondary questions. Adding them should have a reason and an analysis plan; it should not become a way to avoid defining the primary outcome.
Ask what the validation evidence actually examined
A label such as “validated” is incomplete. Which version, population, language, purpose and measurement property were investigated? A favourable result in one setting is evidence to consider, not permanent approval for every use.
For a patient-reported measure, inspect whether the items are relevant, cover the intended content and can be understood as intended. COSMIN also treats responsiveness as evidence about detecting change in the specified construct; a significant before-and-after difference alone does not settle that property. COSMIN manual, version 2.0.
In the fictional case, imagine the difficulty questionnaire’s strongest study enrolled people with a much narrower range of tasks than the new programme. The team should inspect what is missing. If the questionnaire omits important household responsibilities in the intended group, excellent reliability cannot restore the absent content.
Now change another fact: the only available version uses wording many intended participants cannot understand. Reading it aloud may help some people, but changing wording or switching administration mode without supporting evidence could change what the scores mean. The next step is to investigate a suitable version or permitted administration method, not to silently improvise a new instrument and retain the old validation claim.
Match the evidence to the purpose
Suppose the difficulty questionnaire has strong evidence for distinguishing groups at one time but little evidence about change. That does not establish that it is poor at measuring change. It leaves a relevant uncertainty for this twelve-week evaluation.
Conversely, evidence that group averages can detect change does not automatically establish sufficient precision for a consequential decision about one person. A group average can be estimated more precisely than an individual score. The measurement brief must identify which inference the study intends to make.
Consider a ceiling effect in an invented 0–40 scale where 40 represents the least difficulty. If many participants begin at 40, the scale cannot record further numerical improvement for those people. A stable score could then reflect limited room on the scale rather than no change in their experience. Before selecting it, inspect score distributions in a population with similar starting status.
These checks connect measurement properties to a concrete use. They are more informative than collecting a large number of favourable reliability coefficients without asking what decision they support.
Make feasibility explicit without letting it decide everything
The six-minute questionnaire takes longer than the confidence scale. That burden matters, especially when participants already complete several assessments. However, saving four minutes does not make a different construct equivalent.
The team can investigate an established shorter version, reduce redundant secondary measures or change the collection schedule. Each option has a tradeoff. If no feasible measure adequately covers the intended outcome, the honest conclusion is a measurement gap requiring further work.
Costs, licensing, scoring access, equipment, staff training and accessible administration also belong in the choice. Record practical limitations beside the evidence rather than presenting a statistical property as the entire decision.
A useful selection note would say: “We provisionally choose the difficulty questionnaire because its content and change-score evidence fit our stated outcome. We will confirm the intended language version, administration procedure and coverage of participants’ tasks. We are not using grip strength as a substitute for the primary construct.”
Transfer exercise: change the question, not just the tool
A second fictional project asks whether participants become more confident about handling household activities after an education session. It does not ask whether their task difficulty changes. The same three candidates are available. The confidence scale now has relevant evidence for change in this population and is feasible to administer.
Which choice is most defensible from the information provided?
The confidence scale fits the revised question. Selecting the difficulty questionnaire merely because it won the first comparison would carry over a decision after its purpose changed. Selecting grip force because it is “objective” would also answer a different question.
The correct choice remains conditional on the supplied evidence. If the project instead needed both confidence and difficulty, two clearly justified measures could be appropriate. One instrument need not represent every outcome that matters.
For your next study proposal, write the measurement brief first and make each candidate earn its place against it. Finish with one unresolved limitation and the information that would resolve it. That is a defensible selection process, even when the eventual answer is that more measurement work is needed.
Sources and further reading
- COSMIN — Deciding what to measure — Defining the construct before choosing an instrument.
- COSMIN manual, version 2.0 — Population and purpose, content validity and responsiveness; relevant sections consulted, not the entire manual systematically reviewed.