What to Check When Piloting a Research Questionnaire
Pilot questionnaire wording, interpretation and response options before relying on answers, using an issue log to guide changes and retesting.
A questionnaire pilot is a diagnostic exercise, not a rehearsal. Its purpose is to establish, before main fieldwork, whether the instrument can produce the evidence the study needs. The checks that repay attention most consistently concern interpretation, item structure and option coverage: whether respondents read the wording as intended, whether an item asks for one judgement rather than two, and whether the answer options allow a truthful answer. A pilot can surface these problems. It cannot establish that the eventual survey is representative, that the questions measure what they claim to measure, or that people who did not take part resemble those who did.
What a pilot can and cannot establish
The UK Government Analysis Function's questionnaire design guidance treats question design as a matter of information needs, wording and respondent interpretation, and notes that testing helps identify comprehension and response problems. It is also explicit that a pilot is not evidence of population representativeness. That distinction should remain visible throughout the work: the exercise checks an instrument, and the guidance does not present it as a guarantee that the resulting survey will be valid.
A pilot is therefore best designed around the decisions it must inform. Those decisions usually fall into four groups: whether respondents understand the items and the accompanying instructions; whether the options cover the answers people actually hold; whether routing and administration work in the intended mode; and whether the coding and analysis plan can handle what comes back. One small exercise rarely serves all four equally well. Deciding in advance which matter most stops the pilot from becoming an unstructured conversation that yields impressions but no record.
Cognitive interviewing and piloting overlap without being identical. Cognitive probes examine how a respondent arrived at an answer. A pilot also examines the instrument as a working system, including item order, skip logic and the practicalities of fielding. Combining the two is reasonable when resources are limited, provided the interpretation probes are not sacrificed to logistics.
Interpretation: what the wording means to the respondent
Most serious questionnaire faults are visible in the moment a respondent explains, in their own words, what a question was asking. Common sources of misreading include policy vocabulary with more than one referent, unstated time periods, vague frequency terms, assumed knowledge, and negation.
Referents deserve particular scrutiny. "Your organisation" may mean a team, a site, a division or a parent body. "Local area" may mean a neighbourhood, a council boundary or a travel-to-work area. "Access" may mean availability, affordability, physical entry or administrative eligibility. Where the intended meaning sits in an introduction or a definition rather than in the item itself, test the introduction as well; respondents often answer from the definition they remember rather than the one printed above the question.
The order of probing matters. Ask for a paraphrase before asking for an opinion. If the opinion comes first, the answer anchors the subsequent explanation, and the respondent may rationalise a reading they would not otherwise have taken. Useful probes are specific and short: what did that phrase mean to you; what were you thinking about when you chose that answer; what period were you counting. Record the respondent's words rather than a summary judgement such as "confused".
Item structure: one question, one judgement
An item that asks for two judgements produces an answer that cannot be interpreted. The pattern is familiar in attitude statements (clear and timely, affordable and accessible, satisfied with quality and cost), and it also appears in factual items, such as asking how often someone uses two different services in one frequency scale.
The test is behavioural rather than editorial. After a response, ask which part of the question the respondent was answering. If different participants answer different halves, the item is not measuring a shared construct. The remedy is usually to split it. Splitting has costs: a longer instrument, correlated items and more analysis decisions. Where the combined judgement is genuinely the object of interest, state the combination explicitly ("taking clarity and reliability together, how satisfied are you…") and check that respondents treat the combination as a single thing to judge.
Also check conditional items and negatives. A question that only applies to people who have used a service needs a route for everyone else. A negative statement inside a negative response scale invites misreading that no amount of interviewer training fully removes.
Response options: room for a truthful answer
Option problems are easier to miss than wording problems because the item can read well while offering no accurate place to stand. Four checks cover most cases.
Coverage: is there an option for each state the respondent could legitimately be in, including no experience, not applicable, refusal and genuine uncertainty? These are not interchangeable. Collapsing them into a single "don't know" loses the distinction between someone who has not used a service and someone who has used it but cannot judge it, and that distinction usually matters downstream.
Mutual exclusivity: does any pair of options overlap, so that a respondent could reasonably choose either? Frequency bands and income bands are frequent offenders.
Balance: does the scale offer symmetrical positions around a defined midpoint, and is the midpoint labelled in a way respondents understand? A scale that runs from "very satisfied" to "very dissatisfied" without a stated middle invites improvisation.
Order and presentation: option order can shape responses, particularly in self-completion where the first plausible option is visually salient. Test the presentation as it will be fielded, on paper or on screen, rather than reading options aloud in an interview if the main mode is different.
A simple option-sorting task helps: give respondents the options on separate cards and ask which they would choose, then ask what they would have selected if their true answer had not been listed. The second question is often the most productive part of the session.
A worksheet for recording decisions
The following is a suggested working format, not a validated instrument or an institutional requirement. Its value lies in forcing a written link between what the pilot showed and what was changed.
For each item, record the wording as fielded, the judgement the item is meant to capture, what the pilot showed about interpretation, whether the options allowed a truthful answer, and the decision with its reason. Keep the original wording alongside the revision; a log that shows only the final version cannot be audited later.
The worked example below is fictional. It illustrates how the worksheet records decisions; it does not report a real study or a measured improvement in data quality.
A fictional district council is preparing a consultation on bus information. Six pilot participants are recruited to cover a range of travel habits and familiarity with council terminology. Three items produce usable findings.
| Item | Wording as fielded | Judgement sought | What the pilot showed | Option coverage | Decision |
|---|---|---|---|---|---|
| 1 | How satisfied are you with the clarity and reliability of bus service information in your area? | Satisfaction with the quality of information | Three participants answered about timetable legibility, two about the accuracy of real-time displays, one about the reliability of buses themselves | Very satisfied to very dissatisfied, plus don't know | Split into two items — clarity of presentation and accuracy of information — and add a screening item on whether the respondent has used bus information in the past 12 months |
| 2 | Do you agree the council provides good information and responds quickly to enquiries? | Assessment of information quality and responsiveness | Participants who had never contacted the council chose the midpoint; one said they were answering only the first half | Agree–disagree scale with a midpoint | Split into two items; add "have not contacted the council" as a separate response route before the scale |
| 3 | How often do you use bus services? | Frequency of use | A seasonal user found no suitable option; two participants used "rarely" for very different frequencies | Daily, weekly, monthly, rarely, never | Define the reference period, replace "rarely" with a band below monthly, and test the revised item with new participants |
The example shows a decision trail, not an improvement in data quality. No comparison group was used and no measurement of error was taken.
Running the sessions
Recruit for range rather than representation. Include people at the edges of the eligible population: those who have and have not used the service, those whose circumstances the wording might exclude, and those unfamiliar with policy vocabulary. The aim is to encounter interpretations the drafting team has stopped noticing, not to estimate how common those interpretations are.
Use the intended mode where feasible. A self-completion questionnaire piloted as a face-to-face interview will not reveal layout, navigation or instruction problems that appear when someone completes it alone. If the mode must differ, treat the pilot as an interpretation check and test the presentation separately. Note also that an interviewer's presence can either suppress confusion or prompt it; state which condition applied when reporting.
Consent, recording and storage are project-specific obligations rather than matters settled by the pilot method. Where sessions are recorded or observed, participants need accessible information about purpose, use, sharing and withdrawal before they take part.
Revise, then re-test the changed items with new participants. Returning to the same people tests their memory of the first version rather than their comprehension of the second. Keep the revision log current, with dated entries and the pilot evidence behind each change.
There is no fixed number of sessions at which a pilot becomes sufficient. The practical stopping point is a judgement: when new participants stop producing new interpretation problems on the items that carry the most analytical weight, and when option gaps have been resolved or explicitly flagged. That is a recommendation about how to decide, not an observed threshold. Items that remain uncertain should be carried forward as flagged uncertainties, not presented as settled.
Carrying the results into fieldwork
The pilot's output is a fielding pack: final wording, option lists, routing and screening rules, show cards, coding instructions, and the revision log with its reasons. Add a short note for each item stating the judgement it is meant to capture, so that interviewers, programmers and analysts work from the same understanding. Where the study combines questionnaire data with other evidence, the analysis plan should say which claims rest on the questionnaire and which do not.
Before fieldwork begins, ask someone who did not draft the instrument to work through the worksheet against the final wording. A fresh reader is more likely to notice an ambiguous referent than the team that has been reading the same sentence for weeks. If the pilot raises a doubt about whether a questionnaire is the right instrument for part of the evidence, the comparison of written research and expert calls sets out the trade-offs to weigh before committing to fieldwork.