Choosing Cases for Comparative Policy Research
Choose policy research cases around a comparative question and explain which contextual differences limit the conclusions and transfer of findings.
Choose comparative policy cases by making the comparison carry an explicit analytical burden. The question should state what contrast the cases are meant to supply, why each case is included rather than merely accessible, which contextual differences could alter the mechanism or outcome, what the set deliberately excludes, and what transfer claim the evidence can support. Case selection is an argument about scope and comparison, not a search for illustrative examples.
The comparison is the design
A case is a bounded unit: a jurisdiction, a programme, a delivery organisation or an episode of reform. Before naming cases, define the claim the comparison is meant to examine. If a case would not change the answer, clarify the claim or reconsider the case. If the case list could be reordered without altering the argument, the comparison is probably descriptive rather than analytical.
Two failures are common. The first is selecting cases because they are easy to access and then presenting the set as representative. The second is selecting cases that differ on so many dimensions that the comparison cannot identify which difference matters. Neither failure is repaired by adding more cases. A clearer claim and a narrower scope usually do more.
Convenience is a constraint, not a rationale
Access, language, data availability, prior relationships and timing shape what is feasible. Those constraints belong in the case-selection note. They become a problem when they are presented as substantive selection criteria. A convenient case can be used if the claim is narrowed accordingly and the limitation is stated. Convenience should be visible in the method, not hidden in a footnote.
The relevant distinction is between a case that is available and a case that is informative for the stated comparison. Availability affects feasibility. Informativeness affects what the comparison can support. Both matter, but they answer different questions.
Comparison logics and what they can support
The following table is a prompt for reasoning, not a taxonomy to apply mechanically. A comparison may combine logics, but the combination should be stated.
| Comparison logic | Selection question | What it can support | Limit to state |
|---|---|---|---|
| Most similar | Which cases are alike on background conditions but differ on the outcome or policy of interest? | Examination of a suspected difference, supported by within-case evidence | Cannot establish that the differing feature caused the outcome without further evidence |
| Most different | Which cases differ on background conditions but share the outcome or policy? | Exploration of whether a shared outcome survives divergent contexts; theory development | Cannot show that the shared feature is sufficient |
| Typical | Which case sits near the centre of a defined set on specified dimensions? | Description of a configuration within that defined set | No population representativeness unless the population and selection rule are defined |
| Deviant or extreme | Which case departs from an expected pattern? | Examination of an unexpected outcome and the conditions associated with it | Cannot support prevalence claims |
| Nested | Which cases sit within a wider comparison and allow closer examination of a mechanism? | Linkage between a wider comparison and a within-case mechanism | The wider set may not be representative, and the nested case may be atypical |
The logic should be written before the cases are described. A reader should be able to see what each case contributes and what would count as a reason to replace it.
Material differences: a test
Not every contextual difference matters. A difference is material if it could change the meaning of the intervention, the measurement of the outcome, the operation of the proposed mechanism or the delivery chain. Legal competence, budget rules, data definitions, administrative capacity, eligibility criteria, political salience and timing can all be material. They are not a checklist to score. They are candidate explanations to examine.
For each material difference, record the evidence for it and its expected implication. If the implication cannot be stated, treat the difference as description rather than a selection criterion. This discipline keeps the comparison focused. It also makes later analysis easier, because the relevant contextual features have already been identified.
Comparability of evidence belongs to the same test. Two cases may be substantively comparable but have incompatible administrative data or different reporting periods. The comparison can still proceed with a narrower outcome or a different evidence type, but the note should say so.
Exclusions, representation and causal caution
List the cases considered and rejected, with reasons: no comparable data, a policy change during the period, access refused, ethical risk or a scope boundary. Exclusions are part of the argument. They show what the set is not.
A set defined by access cannot support a claim about a wider population. A pair of cases cannot isolate causation by selection alone; causal inference requires within-case evidence and a stated counterfactual. Case selection can improve the plausibility of a comparison, but it cannot supply population representativeness or causal identification on its own. State that boundary plainly rather than leaving it for the reader to infer.
Transfer claims and the external framework
The United Kingdom's Magenta Book, HM Treasury's central government guidance on evaluation, includes a Quality in Qualitative Evaluation framework with explicit attention to generalisation. See Quality in Qualitative Evaluation and the Magenta Book central government guidance.
A practical recommendation follows: write the transfer claim as three sentences. First, name the mechanism that is thought to produce the outcome. Second, name the scope conditions on which the mechanism depends. Third, name the features of the receiving context that would need to be present for the mechanism to operate there. If the mechanism depends on a feature absent in the destination, transfer is a hypothesis, not a finding. This is a recommendation for organising the argument, not a requirement of the framework itself.
A case-selection note: suggested working format
Use this as a one-page note before fieldwork. It is a suggested format, not a validated instrument or an institutional requirement. Adapt the fields to the decision at hand.
- Question and claim. What comparison is being made, and what would each case need to show?
- Unit and boundary. What is the case a case of, at what level and over what period?
- Comparison logic. Most similar, most different, typical, deviant or nested; state what the contrast can and cannot support.
- Inclusion basis. Give the substantive reason for each case, not only its availability.
- Material differences. Record contextual differences that could alter the mechanism, outcome or delivery chain, with the evidence for each.
- Exclusions. List cases considered and rejected, with reasons and the scope they define.
- Transfer claim. State the mechanism, its scope conditions and the receiving context.
- Access and ethics. Record permissions, data protection issues and what a change in access would do to the comparison.
- Evidence plan. Identify comparable documents, administrative data and interviews for each case.
- Revision trigger. State what finding would force a case to be added, replaced or reclassified.
Fictional worked example: two local service delivery settings
The following example is fictional. It does not describe a real jurisdiction, programme or finding, and it does not support a causal claim from a pair of cases.
A research team is asked whether a referral pathway from adult social care to employment support improves take-up of a work-related benefit. The team has access to two fictional local authority areas, Riverton and Lakemoor. The case-selection note records the following.
- Question and claim. The comparison examines whether the referral mechanism depends on co-location of social care and employment advisers.
- Comparison logic. Most different delivery settings within one national policy framework: Riverton has a co-located team, while Lakemoor has a distributed team.
- Inclusion basis. Both areas use the same national referral form and report the same benefit take-up measure. Both have comparable administrative records and have agreed to interviews.
- Material differences. Co-location, shared case records, local discretion over referral thresholds and the timing of a local reorganisation. Each difference is recorded with its expected implication for the referral mechanism.
- Exclusions. A third area with no comparable take-up data; a fourth that changed its referral form mid-period; rural areas outside the defined scope.
- Transfer claim. If the mechanism depends on a named referral owner and shared records, transfer to a distributed model requires both features. Without them, the claim is limited to the co-located setting. The pair cannot establish the size of any effect or rule out other explanations.
The note would be revised if interviews showed that the referral form was interpreted differently in the two areas, or if the administrative measure excluded a client group central to the question. The example shows the intended discipline: the comparison is stated before the cases are described, and the limits are part of the design.
Draft the note before access is negotiated, keep it with the research protocol and revise it when the evidence changes the comparison.