Choosing Evaluation Criteria That Fit the Decision

Select evaluation criteria around the intended decision and evidence, using the OECD DAC framework without turning it into a mechanical scorecard.

Evaluation criteria should be selected from the decision that the evidence must support, not from a standard list applied by default. The practical test is whether a candidate criterion can be expressed as a question that available evidence and methods can answer within the project's time, access and ethical limits. Criteria that cannot alter a decision, or cannot be investigated with defensible evidence, may still be worth recording, but they should not be allowed to shape the work as though they were central.

Begin with the decision, not the catalogue

A criterion is a lens for judgment. It is not a data collection instrument, and it does not prescribe a method. Before comparing criteria, state the intended use of the evaluation in one or two sentences: whether a programme should continue, be redesigned, be scaled, be ended, or be accounted for to a funder or governing body. Different uses place different demands on evidence.

For a continuation decision, the central question may concern whether the intervention produced enough benefit to justify further funding. For a redesign decision, the question may concern which components worked for which participants and under what conditions. For an accountability decision, the question may concern whether agreed outputs and outcomes were delivered. Each use implies a different mix of criteria, evidence and inference.

A criterion that cannot change the decision is a candidate for exclusion, or for treatment as contextual information. A criterion that can change the decision but cannot be answered with available data, permissions and methods is a candidate for reframing. Both decisions should be recorded.

The six common names and their provenance

The OECD's Development Assistance Committee (DAC) sets out six evaluation criteria for development co-operation: relevance, coherence, effectiveness, efficiency, impact and sustainability. The OECD DAC evaluation criteria page is the primary reference. Their development provenance matters. They were designed for assessing public interventions in development co-operation, and they are not a universal method, a certification or proof that an intervention succeeded.

In brief, and in original wording:

  • Relevance asks whether the intervention responded to the problem it was meant to address and to the circumstances of intended beneficiaries.
  • Coherence asks whether the intervention fitted with other interventions, policies or institutions operating in the same space.
  • Effectiveness asks whether the intervention achieved its intended results.
  • Efficiency asks whether resources were converted into results in a reasonable way, including consideration of cost and timeliness.
  • Impact asks whether the intervention produced significant or lasting change, including unintended effects.
  • Sustainability asks whether benefits are likely to continue after external support ends.

For evaluations of United Kingdom central government interventions, HM Treasury's Magenta Book provides guidance on evaluation, including how to frame evaluation questions and choose methods proportionate to the decision. That guidance is jurisdiction-specific. It does not settle which criteria fit a particular project, and it should not be treated as a substitute for reasoning about the decision at hand.

A selection note as a suggested working format

A useful practice is to maintain a selection note for every criterion considered, whether it is ultimately included or excluded. The note is a working format, not a validated instrument or an institutional requirement. It has six fields:

  1. Intended use. The decision, audience and timing the evidence must serve.
  2. Criterion. The judgment lens under consideration.
  3. Question. A specific question stating the population, outcome, comparison or threshold, and period where relevant.
  4. Evidence. The data source, its owner, coverage, quality, definitions, missingness, linkage assumptions and permissions.
  5. Method. The analysis or design that would allow the question to be answered, and the inference it would permit.
  6. Reason for inclusion or exclusion. Why the criterion belongs in the evaluation, or why it is being set aside.

The note forces an explicit link between a criterion and a feasible inquiry. It also makes exclusions visible. An omitted criterion is not necessarily unimportant; it may be excluded because the decision does not turn on it, because the evidence cannot support a credible judgment, or because the ethical or data-access conditions cannot be met.

Worked example: a fictional continuation decision

The following example is fictional. It is included to show the format, not to describe a real service, a standard practice or a typical funding arrangement.

A fictional regional advice service, the Northbridge Employment Advice Service, has completed an 18-month pilot across three local offices. It provided one-to-one employment advice, including support with benefit claims and training applications. A funding body must decide whether to continue funding for a further period and whether to retain the same delivery model.

A selection note for this decision might look as follows.

Criterion Question Evidence and method Selection reason
Relevance Did the service reach residents in the eligible areas and circumstances for which it was funded? Eligibility and referral records; descriptive analysis and document review. Continuation depends on whether the service addressed an identified gap. Existing records can answer the question at low additional burden.
Effectiveness Among recipients who completed at least three advice sessions, what proportion reported progress on agreed advice goals at follow-up? Case closure codes and a follow-up survey; descriptive analysis, with no comparison group. Central to whether the service produced intended benefits. The design can describe reported progress but cannot establish attribution.
Sustainability Could advice functions continue through partner organisations or other funding if the pilot ended? Partner statements, funding documents and staff interviews; document review and interviews. The continuation decision includes questions of handover and abrupt closure.
Efficiency What was the cost per completed appointment, and how did it vary across offices? Budget records and appointment counts; unit-cost calculation. Included only as contextual information. The decision is not a choice between delivery models, and unit cost alone does not show value.
Impact Did the service change longer-term employment outcomes relative to what would have happened otherwise? No credible counterfactual is available from the pilot records or survey. Excluded because the period and data access cannot support a causal comparison. A claim of impact would exceed the evidence.
Coherence Did the service align with other employment support in the area? Partner interviews and local strategy documents; document review. Considered but given low priority because no linked intervention is being changed at the same time. It may matter for future design.

The note does not imply that relevance, effectiveness and sustainability are the only criteria that could matter. It records why they carry the decision, why efficiency is contextual, and why impact and coherence are not central in this instance.

From criterion to answerable question

A criterion becomes useful when it is converted into a question with a specified population, outcome, comparison and period. "Was the service effective?" is not answerable as stated. A more workable question is: "Among residents who completed at least three advice sessions between January and December, what proportion reported a resolved benefit issue at follow-up, and how does that compare with the situation at intake?"

The question should also state what kind of inference is intended. A descriptive question can establish what was reported or recorded. A process question can establish what was delivered and to whom. An impact question requires a credible comparison, whether through a control group, a natural comparison, a before-and-after design with suitable caveats, or another defensible counterfactual. Where no credible comparison is available, the evaluation should not present descriptive change as causal impact.

Method follows from the question and the evidence. Administrative records may support questions about reach, eligibility and recorded outcomes. Interviews may support questions about mechanisms, implementation and perceived change. Surveys may support questions about participant experience. Each method carries its own limitations: records measure what was recorded, interviews capture accounts rather than observed behaviour, and surveys depend on response and recall.

Access and ethics constrain the choice. Where permission to access individual records is absent, the question must be reframed around aggregate or anonymised data, or the criterion dropped. Consent, data minimisation and data protection obligations should be settled before collection, not after a question has been agreed.

Trade-offs that shape the choice

Selecting criteria involves trade-offs, not only technical decisions.

Breadth against depth. Adding criteria adds data collection, analysis and ethical burden. A focused evaluation that answers two or three questions well may be more useful than a broad assessment that answers many questions weakly.

Timeliness against attribution. Impact questions often require longer periods and stronger comparison designs than a decision cycle allows. Where the decision cannot wait, a process or effectiveness question with explicit limits may be more honest than a delayed impact claim.

Accountability against learning. A funder may need assurance that outputs were delivered. A delivery team may need to understand why some participants benefited and others did not. The selection note should make the primary use clear rather than blending incompatible purposes.

Competing criteria. A relevant service may be inefficient. An effective service may not be sustainable. A coherent intervention may have modest impact. Criteria can point in different directions, and the evaluation should not conceal that tension. Weighting is a judgment, and it should be recorded.

Limitations to state plainly

A criterion does not guarantee evidence quality. The six names carry development-evaluation assumptions about public intervention, beneficiaries and outcomes. They can be adapted, but the adaptation should be explained. Their use is not a certification of methodological rigour, and a reference framework does not establish that a particular evaluation design is adequate.

Evidence limitations should be stated in the selection note and carried into the final reporting. Records may change definitions over time. Survey response may be uneven. A comparison group may not be available. Permissions may restrict access. These are not administrative details; they determine what the evaluation can legitimately conclude.

The note should also record what was not assessed. A decision-maker reading the evaluation should be able to see which criteria were excluded and why, rather than inferring that an omitted criterion was found to be satisfactory.

A short documentation standard

Before commissioning or accepting an evaluation role, write the decision in one sentence, then complete the selection note for each criterion under consideration. Record the intended use, the question, the evidence, the method, the reason for inclusion or exclusion, the assumptions and the limitations. Review the note with the person or group who will use the findings, and revise it if the decision or access conditions change.

If the note cannot be completed, the evaluation question is not yet specified. That is a useful finding in itself. Policy Expert Network's guide Before You Accept an Expert Project covers a related check on framing a project engagement before committing to it.

No advertising or analytics cookies are used. Your preference and application receipt may be stored in this browser; CV files are not.

Read the storage explanation