Using Administrative Data in Programme Evaluation
Assess how administrative records were produced and whom they cover before treating them as evidence of programme outcomes or impact.
Administrative data is a by-product of an operational process, not a measurement instrument designed for evaluation, and that single fact governs how it should be used. Before asking what a dataset shows about a programme, an evaluator has to reconstruct why each record exists: who created it, what decision it supported, which population it covered at that moment, and where a person could enter, leave or appear twice. In most evaluations the gap between what the data appears to say and what the programme actually did is explained at this stage rather than at the analysis stage.
What these records can and cannot support
Service registers, payment and claims systems, case-management logs, enrolment databases and workforce records share a set of strengths. They usually cover an administrative population rather than a sample, they run continuously rather than at survey intervals, and they are tied to consequential decisions such as a payment, an eligibility determination or a referral. The intervention is often documented by the same system that delivered it, which is useful for both implementation and impact work.
The weaknesses mirror those strengths. Fields are defined by operational rules, refreshed on operational cycles and validated only to the standard the decision required. A flag labelled "completed" may mean a form was signed, a payment was released or a case was closed. Those are three different constructs that behave differently under analysis, and treating them as one outcome measure is a common source of error.
| Record type | Designed to capture | Characteristic blind spot |
|---|---|---|
| Service or case register | Contact with the service, status changes, caseworker actions | People who never made contact; informal support not logged |
| Payment or claims system | Verified transactions and eligibility at the point of payment | Activity that generated no claim; timing tied to claim cycles, not delivery |
| Enrolment or training database | Registration and course participation | Drop-out before first attendance; repeat enrolment counted per course |
| Workforce or human-resources record | Contracted roles, pay, headcount | Unpaid or contracted-in staff; turnover recorded only at payroll dates |
Reconstruct the purpose of the record first
A short record biography, agreed with the data owner before any extract is requested, will usually save more time than it costs. It should establish five things: who creates the record in the ordinary course of work; which decision the record triggers; which business rule governs a change of status; which version of the system generated the data being supplied; and what happens to the record when a person leaves the programme, changes provider or dies.
Questions to put to the data owner follow from those five points. Is a record created at first contact or at first attendance? Can a person hold more than one active record at a time, and if so under what conditions? Which fields are mandatory and which are populated only when a caseworker chooses? Has the definition of any key field changed since the programme began, and on what date? Is the extract a snapshot or a history of changes? The answers are usually documented somewhere, but rarely in the form the evaluation needs.
Coverage: missing, duplicated and misaligned
Coverage has to be judged against the evaluation's target population, not against the programme's registered population. Three failure modes account for most problems.
People absent from the system. Groups may never appear because they were eligible but never enrolled, dropped out before the first recorded transaction, were served by a partner organisation with its own system, or declined the consent that a research use of the data requires. Their absence is rarely random, which is what makes it consequential: the people who leave a service early are often those about whom the evaluation most needs to say something.
People counted more than once. Duplication tends to come from identifier practice rather than from error. A learner enrolled on three courses may be three enrolment records and one person. A household-level record and a person-level record may be mixed in the same table. People who transfer between providers may be re-registered on arrival. The unit of analysis has to be settled explicitly, and the duplication rule written down.
People whose records cannot be linked. Where evaluation depends on joining two datasets, linking depends on identifiers that may be missing, inconsistent or unavailable for legal reasons. The match rate becomes part of the evidence, and unmatched cases need to be described rather than dropped silently.
A provenance and coverage worksheet
The worksheet below is a suggested working format, not a validated instrument or an institutional requirement. It is meant to be completed with the data owner present and signed off before the analysis plan is fixed.
| Column | What to record |
|---|---|
| Field name and definition | The operational wording, including any code list in use |
| System and version | Source system, extract date and whether the extract is a snapshot or a change history |
| Operational purpose | The decision this field supports in ordinary work |
| Population in scope at creation | Who could appear, by rule |
| Known exclusions | Groups absent by design or by practice |
| Duplication rule | How a person is identified, and how repeat records are resolved |
| Definition changes | Dates on which wording, codes or thresholds changed |
| Fitness for the evaluation question | Direct measure, proxy, or unusable |
| Action required | Clarification, supplementary source, sensitivity check or exclusion |
The final column matters most. A field can be unfit for one purpose and adequate for another: a completion flag may be unusable as an outcome measure but perfectly serviceable as a covariate describing delivery intensity.
Fictional worked example
The following example is fictional and is used only to show how the worksheet operates. No findings from any real programme are reported here.
A two-year adult skills programme operates through contracted providers in two regions. The evaluation team receives an extract of 4,300 enrolment records and 2,950 completion flags, alongside provider payment claims. Completing the worksheet produces three findings before any analysis begins.
First, records are created at the point of enrolment, which follows an initial assessment session. People who enquire but never attend an assessment are absent from the extract, and providers report that this group is larger in the region with poorer transport links. The evaluation's target population includes them, so the extract cannot measure take-up on its own.
Second, the completion flag is set when a tutor signs off the learner's portfolio, which happens before external assessment results are known. The flag therefore measures delivery completion, not attainment, and the evaluation's outcome question requires the awarding body's results file instead.
Third, learners who transfer between providers are re-registered, and 190 records share a learner identifier with another record. The unit of analysis is agreed as the learner, with the earliest enrolment date retained and transfers noted as a delivery variable. The fictional 190 figure is illustrative only; the duplication rate in any real dataset has to be counted, not assumed.
Linking records to evaluation questions
Where a programme operates across several jurisdictions, the same outcome can be recorded under different administrative definitions: an employment outcome may appear as a benefit exit in one system, a payroll notification in another and a self-reported status in a third. Pooling these without a crosswalk produces comparisons that reflect differences in administrative practice rather than differences in results.
The practical response is a common indicator set with documented country-specific additions. For each indicator, record which administrative definition was used, which population it covers and what the known divergence is. Where two definitions cannot be reconciled, report them separately rather than forcing a single series.
Governance and constraints to settle early
Purpose limitation, legal basis, retention and disclosure control are not administrative afterthoughts; they shape what can be analysed and where. Establish who controls each dataset, whether analysis must take place inside a secure environment, whether record-level data may cross a border, what the minimum necessary field list is, and how small counts will be suppressed in published tables. Where research use depends on consent, the consent wording and the timing of its collection determine whether the data can be used at all. These are matters to verify in the relevant jurisdiction and against the specific data-sharing agreement, not matters to assume.
On the quality of evidence, HM Treasury's Magenta Book sets out UK guidance on evaluation, and its supplementary material includes guidance on quality in qualitative evaluation, which treats quality as a property of the inference an evaluation is trying to support. Applying that discipline to administrative data means judging a field against the specific claim it will be asked to bear, and documenting that judgement, rather than treating a dataset as good or poor in the abstract.
What to write into the evaluation plan
The plan should name each data source, its owner, the extract date and the field-level definitions in use, together with the known exclusions and the duplication rule. It should state which indicators are direct measures and which are proxies, and describe the sensitivity checks that will test whether the conclusions survive plausible coverage error. It should also record what will happen if a source proves unusable after the plan is agreed. Identifying data sources at inception, rather than after the fieldwork, is the difference between an evaluation that can answer its questions and one that can only describe the data it was given.