Recording Evidence Gaps in Policy Research
Record evidence gaps by the decision they affect, then choose a proportionate next step rather than treating missing information as a finding.
An evidence gap register records, claim by claim, what evidence a policy research project holds, what it lacks, what that absence prevents a reader from concluding, and what will be done about it before delivery. It is a working document rather than an annex produced at the end, and it exists to stop a project from making a recommendation its evidence cannot carry. A gap is defined against a claim, not against a topic: "we need more evidence on implementation" is not a register entry, while "we cannot say whether the training changed practice, because no observation of practice before the training exists" is.
The fields that make an entry usable
A register works when each row can be read by someone who was not involved in the fieldwork. Seven fields are usually enough.
Claim. Write the sentence as it would appear in the report, briefing or recommendation. If the claim is still vague at design stage, that is itself a finding worth recording.
Evidence available. State what exists, how it was obtained, its date and scope, and its known weaknesses. Separate primary collection from secondary sources and from material supplied by the client.
Missing element. Name the specific thing absent: a population, a time period, a counterfactual, a delivery mechanism, a cost component, a sub-national breakdown, or a comparison case.
Consequence for the decision. Describe what the decision-maker cannot do, or must do differently, if the gap remains. This is the field that ranks the register. A gap with no decision consequence belongs in a methods note, not in the top half of the table.
Next check and owner. Specify a feasible action, a named person and a date, or record an explicit decision to accept the gap. A named individual is more useful than a team, because it makes silence visible.
Reporting limit. Write the sentence that must accompany any downstream use of the claim, for example: "Descriptive of participants only; no causal attribution." That sentence then travels into the report, the slide deck and any later reuse.
The register is distinct from a risk register. A risk register tracks threats to delivering the work on time and within budget; the gap register tracks limits on what the finished work can assert. The two meet where an access problem threatens a claim, but they answer different questions and should not be merged into one table.
Missing evidence, weak evidence and evidence of absence
Three states are routinely conflated, and the distinction changes what a project may say.
Weak or indirect evidence exists but is dated, drawn from another jurisdiction, based on a different population, or dependent on an assumption that has not been tested. It can support a qualified claim.
Missing evidence has not been collected, or cannot be accessed within the project's constraints. It supports no claim at all, and the entry should say why: no data held, no permission, no time, no counterpart.
Evidence of absence is a negative finding produced by a defined search over a defined population and period, where the search would have surfaced the item had it existed. Only then does "we found none" become "there is none within this scope".
The practical test is whether a search was actually run and can be described. Absence of recorded incidents in a monitoring system is not evidence that incidents did not occur: if reporting is open only to registered providers, incidents among unregistered providers are invisible by design. The register entry should record the system, its coverage and who can use it, and then state the limit on the inference. A negative finding from a documented search is a result; an unexamined area is a gap. Confusing the two is the most common way a careful project overstates its conclusions.
Narrowing gaps before delivery
Gaps are easier to close at design than at drafting, and some cannot be closed at all. Four moves are worth considering, each with conditions.
Narrow the claim. If the evidence supports a statement about participants only, the claim becomes a statement about participants. This costs nothing except the ambition of the sentence.
Add a targeted check. A small number of interviews, a document request, an administrative extract or a short site visit can close a specific gap. Whether this is feasible depends on access, ethical approval timelines and the client's willingness to extend scope; a check that cannot be completed before the delivery date is not a next step but a note for the next project.
Substitute a comparable source with an explicit caveat. An earlier period, an adjacent region or a similar programme may provide indirect evidence. Label it as indirect in the register and in the report, and state the assumption that makes the substitution reasonable.
Reframe the deliverable. An interim finding, a conditional recommendation, or a decision deferred pending a named check is often more honest than a conclusion assembled from insufficient material. This requires the client's agreement, and it should be agreed early.
Rank gaps by consequence rather than by ease. Where a wrong answer is cheap to correct later, accepting the gap and saying so is a reasonable professional choice. Where a wrong answer would be costly or difficult to reverse, the gap should be escalated to whoever owns the decision, with a clear statement of what the research can and cannot support.
Fictional worked example
This example is fictional. A family-learning programme has attendance records and participant feedback, but no information on eligible families who did not join.
| Proposed claim | Available evidence | Gap | Decision consequence | Next check | Limit |
|---|---|---|---|---|---|
| Participation improved home learning | Participant feedback before and after attendance | No comparison; incomplete follow-up | Change cannot be attributed to participation | Examine feasibility and permissions for a suitable comparison | Respondents only; no causal claim |
| The programme reached families most in need | Enrolment records | Unknown coverage of eligible non-participants | Reach and equity remain unresolved | Compare coverage with an authorised eligibility source | No estimate of unmet demand |
| Delivery can transfer elsewhere | Delivery notes | Unknown staffing and access conditions elsewhere | Transfer remains a question | Compare the conditions and missing evidence | No scale-up recommendation |
Feedback can describe respondents while failing to establish reach or impact. Keep those limits attached to the claim they constrain.
Assurance, review and the limits of scoring
The register is more useful inside an assurance conversation than as a compliance artefact. HM Treasury's Aqua Book, which sets expectations for quality analysis across UK government, asks that analysis be clear about its purpose, assumptions and limitations, and that assurance be proportionate to the decision it informs (The Aqua Book). A register operationalises part of that expectation by making each limitation attach to a specific claim, which gives a reviewer something concrete to challenge. Applying the same discipline outside UK central government is a professional choice, not a requirement.
Review the register at three points: when the design is fixed, before fieldwork closes, and before sign-off. Record changes with dates so the document shows whether each gap was closed, persisted or accepted. That history is often more informative than the final version.
Resist attaching numeric confidence scores unless the project has a calibrated basis for them. A score of "0.7 confidence" without a defined meaning invites false precision and travels badly into decision papers. Qualitative descriptors with a stated reason, such as "direct, recent and in scope" or "indirect and dated", communicate more and mislead less.
Frequently asked
How large should the register be? As many rows as there are claims carrying a decision consequence. A register that lists every minor unknown becomes unreadable and stops being maintained; a register with three rows that are genuinely load-bearing will be used.
Should the client see it? Generally yes, as a live document with named owners on both sides. Some entries need restricted circulation where they touch on commercially sensitive information or on individuals who could be identified. Agree that boundary at the start rather than discovering it at delivery.