Process Evaluation and Impact Evaluation: Choosing the Question
Choose evaluation questions by separating delivery from attributable change, and identify which evidence is needed to answer each one.
The choice between a process evaluation and an impact evaluation is settled by the claim the reader needs to be able to defend, not by a preference for methods. A process evaluation supports statements about what was delivered, to whom, in what form and at what cost. An impact evaluation supports statements about what changed relative to a stated comparison and how much of that change can reasonably be attributed to the intervention. Methods follow from that decision. Where delivery cannot yet be described reliably, an impact study tends to measure noise; where no credible comparison exists, a process study cannot fill the gap.
The claim determines the design
Both designs can use the same instruments (interviews, administrative records, surveys), so the labels describe the question rather than the toolkit. A programme may need both, and often needs them in sequence. What distinguishes the two is the inference each is entitled to support, and the discipline of writing that inference down before fieldwork begins.
A theory of change is the common structure. It sets out the expected pathway from inputs through delivery to outcomes, with the assumptions that must hold at each step. The Magenta Book, HM Treasury's central guidance on evaluation for UK government, distinguishes process evaluation (implementation), impact evaluation (attributable change) and value-for-money evaluation (benefits and costs), and treats a theory of change as the statement of expected causal pathways and assumptions (HM Treasury, Magenta Book). The practical use of that framework is not to label a study but to locate which link in the pathway the decision depends on.
Process evaluation: answering what was delivered
A process evaluation asks whether the programme was implemented as intended, who received it, how delivery varied, and what delivery cost. Its questions are concrete: reach against the intended population, targeting and eligibility decisions, dosage and duration, fidelity to the intended model, adaptations made locally, the experience of participants and staff, and the quality of the records that would support any later analysis.
Evidence usually comes from delivery and management records, observation of practice, interviews with staff and participants, document review, and cost data. Possible approaches include descriptive analysis against stated standards, fidelity assessment, and qualitative process tracing to identify where the pathway held or broke.
The permitted inference is descriptive and specific: this programme was delivered in these respects, to these people, at this cost, with these deviations. The prohibited inference is causal. A service that reached its target number of participants and was well regarded by them has shown that it was delivered, not that it changed anything. A common failure is measuring the plan rather than the practice: recording what staff were trained to do instead of what they did.
Impact evaluation: answering what changed, relative to what
An impact evaluation requires an explicit comparison. Change for participants alone (before and after, or higher than an external benchmark) describes a trajectory but does not isolate the effect of the intervention from maturation, selection, concurrent policy or measurement effects. The comparison may be a randomly assigned control group, a matched comparison group, a pre-intervention trend in a comparison area, or a threshold that determines access. Which is feasible depends on ethics, access to data, the number of eligible participants and the timing of the decision.
The permitted inference is an estimate of change relative to a stated comparison, under stated assumptions, for a defined population and period. The prohibited inference is explanatory. An impact estimate can show that change occurred without showing why; mechanism claims require process evidence or further testing. Generalising beyond the population, setting and period studied is a further step that the design may not support.
Principal threats differ by design but rarely disappear: selection into the programme, confounding by concurrent events, spillovers to comparison areas, attrition, and outcome measures that are available but not meaningful.
Combining the two
Process findings discipline impact interpretation. A null result from an impact study of a programme that was barely implemented is evidence about implementation, not about the idea. Conversely, an impact estimate is hard to act on without knowing which components were delivered, to whom, and at what cost. The combination is most efficient when both are designed together: a shared theory of change, a shared set of indicators, one data collection calendar, and an agreed rule for how process findings will be used when interpreting the impact estimate.
Value-for-money analysis sits on top of both. It requires the costs of what was actually delivered and an estimate of the benefits that can defensibly be attributed, which is why it usually follows rather than substitutes for the other two.
A worksheet for matching question to permitted inference
The following format is a suggested working aid for scoping a study, not a validated instrument or an institutional requirement. It is best completed with the decision-maker present, because the last two rows are where disagreements surface.
| Element | Process evaluation | Impact evaluation |
|---|---|---|
| Primary question | What was delivered, to whom, how, and at what cost? | What changed relative to what, and how much is attributable? |
| Object of measurement | Implementation, reach, fidelity, adaptation, participant and staff experience, delivery cost | Outcomes for participants and for a stated comparison, at defined points in time |
| Evidence sources | Delivery and management records, observation, interviews, documents, cost data | Outcome measures collected consistently for both groups; baseline and follow-up data; administrative records |
| Possible approaches | Descriptive analysis against stated standards; fidelity assessment; qualitative process tracing | Random assignment, matched comparison, difference-in-differences, interrupted time series, regression discontinuity—subject to ethics, access and numbers |
| Permitted inference | The programme was delivered in these respects, to these people, with these deviations and this cost | A change of this size relative to this comparison, for this population and period, under stated assumptions |
| Prohibited inference | That outcomes changed because of the programme | Why the change occurred, or that it would recur in another setting, without further evidence |
| Principal threat | Recording the plan rather than the practice; observer effects; recall error | Selection, confounding, spillover, attrition, weak outcome measures |
| Decision informed | Delivery management, targeting, cost control, readiness for impact study | Continuation, scaling, redesign or disinvestment |
A workable sequence is to state the decision, state the sentence the decision-maker wants to be able to say, and then check whether the permitted inference of the proposed design matches that sentence. If it does not, either the design or the sentence has to change. Writing both the permitted and the prohibited inference before data collection also protects the eventual report from overreach under pressure.
Worked example (fictional): an advice service without a comparison
The following example is invented for illustration and does not describe a real service or a PEN engagement.
A fictional telephone advice line for households disputing benefit decisions was funded for eighteen months to serve 600 households, staffed by four advisers and a volunteer roster. Management data show 540 households reached, most within ten working days of referral, but outcome fields are completed for fewer than half of closed cases. The funder wants a statement that the service improved outcomes.
A process evaluation can answer a great deal: whether triage matched the intended model, which households were not reached and why, whether advice followed the planned sequence, why outcome recording failed, and what delivery cost per household. It can also report participants' own assessments of helpfulness—useful evidence about experience, but not about change. The permitted inference is that the service reached a defined population and operated in a particular way; the prohibited inference is that it improved decision outcomes.
An impact claim requires a comparison and an outcome measure that exists for both groups. Plausible options include linking advice records to administrative decision data for advised and unadvised applicants, a phased rollout in which later cohorts act as a temporary comparison, or a matched comparison drawn from referral records. Each depends on a data-sharing agreement, ethical review and a sufficient volume of cases; none can be created retrospectively for the first cohort if the relevant records were never captured.
The defensible course in this situation is to commission the process evaluation now, use it to fix the recording and referral pathway, and specify the impact question, the comparison and the outcome measure before the next cohort begins. If no comparison can be constructed, the funder's claim should be rewritten to match the evidence: what was delivered, to whom, and what participants reported. That is a weaker claim, but it is one the evaluation can support.
When the ideal design is unavailable
Constraints sometimes make the better design impossible. If implementation is unstable or records are incomplete, an impact study is premature and the process question comes first. If a comparison cannot be constructed within ethical and legal limits, the impact question may still be addressed in a more limited form (association with plausible adjustment, or contribution analysis combined with process evidence), provided the limits are stated in the report rather than left to the reader.
Where the work requires specialist method expertise (sampling, administrative data linkage, or the analysis of delivery records), the scope should be agreed in writing before the work starts. The guidance on what an invitation to an expert project should specify sets out the elements that make a brief answerable, and organisations commissioning research can review the support offered for organisations before approaching specialists.
A practical first step is to write two sentences: the claim the evaluation must support, and the claim it must not be used to support. If the second sentence is the one the funding decision actually rests on, the design is not yet sufficient, and the question or the design has to change before fieldwork starts.