Reducing Bias in Expert Elicitation
Make expert judgments easier to examine by recording assumptions, testing contrary explanations and separating independent views from group discussion.
Bias in expert elicitation is produced mainly by the design of the exercise rather than by the experts themselves: the wording of the question, the order in which participants see each other's answers, the boundary between what a specialist actually knows and what the decision needs, and the absence of a record of why a judgment changed. Reducing it means settling those four points in writing before the first estimate is collected, then keeping the record intact through discussion and revision.
Where bias enters, and what controls it
Most of the distortion in a structured elicitation is introduced before anyone speaks. A question that names a figure invites anchoring on that figure. A question that asks for a single number conceals whether the expert holds a range or a point. A discussion that opens with the most senior participant's view converts conformity into apparent consensus. None of these requires a badly intentioned facilitator; they follow from ordinary meeting habits.
| Design choice | Failure mode it invites | Low-cost control |
|---|---|---|
| Question wording | Anchor, scope ambiguity, or a unit mismatch drives the number | Draft two equivalent framings; require units, baseline and time horizon |
| Expertise boundary | An estimate outside a specialist's competence is weighted like one inside it | Ask each expert to mark where their knowledge ends for each question |
| Order of sharing | Conformity recorded as agreement | Collect sealed first judgments; log what was shared, with whom, and when |
| Aggregation | A mean hides a split distribution | Report the spread and the stated reasons, not only the central value |
| Revision | Silent drift between first and final answer | Set revision triggers in advance and code each change by reason |
The table is a design checklist, not an evidence claim that one format produces more accurate estimates than another. Its value lies in the audit trail it leaves: after the exercise you can say what each expert saw, when they saw it, and why any judgment moved.
Framing: how wording shifts the answer
Three features of a question do most of the work.
The quantity being requested. "What proportion of eligible households will apply in the first year?" and "How many households will apply in the first year?" draw on different mental arithmetic, and the second depends on a denominator the expert may not share with you. Ask for the quantity in the form you will report, and ask for the denominator separately.
The reference point. A question that opens with last year's figure invites adjustment from that figure, and adjustment is typically insufficient. If a baseline is unavoidable, present it identically to every participant and record that you did so. Where the baseline is disputed, ask for the estimate under two stated baselines rather than letting each expert choose silently.
The evaluative verb. "Acceptable", "tolerable", "feasible" and "likely" carry different thresholds across institutions and disciplines. Replace them with an operational test, such as the level at which a specified body would expect to intervene, or the probability you would need before committing a stated budget.
A practical check is to write a reversed or reframed version of each substantive question and ask a small number of experts both versions, separated in time. Divergence between the two answers is diagnostic of the question, not of the expert, and it is worth recording even when you cannot resolve it.
Independence: whether views are genuinely separate
Independence is a property of the process, not of the participants. Two experts who read the same briefing note, see the same draft report and speak in the same meeting are not independent sources, however distinct their careers. Before claiming that agreement across experts is meaningful, establish:
- whether each participant produced a judgment before seeing anyone else's;
- what shared material was circulated, in what version, and to whom;
- whether any participant knew the identity or affiliation of the others;
- whether the final position was reached by explicit argument or by accommodation.
Where a client wants a deliberative discussion, that is a legitimate design choice, but the initial sealed judgments should still be retained and reported. The difference between the sealed and deliberated positions is often the most informative output of the exercise.
A working worksheet
The following format is a suggested structure for a single elicitation, not a validated instrument and not an institutional requirement. Adapt the fields to the decision at hand; drop the ones that produce nothing.
- Decision this informs. The specific choice the estimates will feed, and the date by which it must be made.
- Question as put to participants. Full wording, units, population, baseline and time horizon.
- Reframed version. The equivalent question used to test wording sensitivity.
- Expertise boundary. For each question, the part of the judgment the expert is competent to make and the part they are not.
- Assumption register. Conditions the estimate depends on, stated by the expert rather than inferred by the analyst.
- Sealed initial judgment. Point or range, with a one-line rationale, collected before any sharing.
- Sharing log. What was circulated or said, in what order, and to whom.
- Revision record. The revised judgment, the reason code, and whether the change followed new information or social pressure.
- Attribution status. Whether the contribution may be quoted, paraphrased anonymously, or used only in aggregate.
Fictional worked example
The following is invented for illustration; the organisation, participants and figures are not real.
A fictional regional authority asks a fictional research institute to estimate first-year take-up of a home insulation subsidy, to inform a budget decision. Four specialists take part: a housing economist, a retrofit engineer, a local authority benefits adviser and a consumer behaviour researcher.
The first question reads: "What proportion of eligible households will apply in year one?" The reframed version asks for the number of applications and the eligible household count separately. The benefits adviser draws a boundary: she can judge application behaviour among households she advises, not regional take-up. The economist draws the reverse boundary. Both boundaries are recorded rather than averaged away.
Sealed initial estimates range from 8 per cent to 22 per cent, a spread the analyst would have concealed by reporting only a mean. After the sharing log records that the economist's 22 per cent was presented first, the retrofit engineer revises from 11 to 15 per cent, giving the reason "new information on boiler grant eligibility". The behaviour researcher revises from 9 to 14 per cent with the reason "the first figure sounded optimistic", which is coded as social influence rather than new evidence and reported separately. The engineer later identifies that one estimate had mixed dwellings with households, a definition mismatch corrected before aggregation. Final figures are reported as a range with the reasons for movement attached.
What should cause a revision
Revision triggers are better agreed before the exercise than negotiated during it, because the pressure to move towards a dominant view is strongest while the group is together. Reasonable triggers include:
- a factual correction to the baseline, population or units;
- an arithmetic or definitional error identified in a stated rationale;
- a mechanism raised in discussion that the expert had not considered and accepts as relevant;
- a boundary problem, where discussion reveals the question asked for a judgment outside the expert's competence;
- new evidence supplied after the sealed round.
"I was persuaded by the room" is not a trigger; it is an observation about the process and should be logged as such. Recording it is not an accusation. It indicates where the aggregate reflects the order of speaking rather than the state of knowledge, which matters to whoever uses the estimate.
Consent, attribution and what may be quoted
Consent to take part in an elicitation is not the same as consent to be named, and neither is the same as consent to have a professional judgment attributed to an employer. UK Research and Innovation's ESRC consent guidance treats consent as a substantive element of ethics review rather than a formality, and the UKRIO Code of Practice for Research sets expectations for how research is conducted and reported in the UK. Both are worth reading before designing the consent step, but neither supplies a template you can adopt unaltered.
A useful approach is to separate the permissions in the record: participation, anonymous paraphrase, and attributed quotation. A fictional participant might permit the interview and the anonymous use of their estimate while declining named quotation; that choice should be visible in the record, not reconstructed afterwards. It also helps to state plainly what anonymity you can and cannot offer, since a small expert pool in a narrow field may make a judgment identifiable even without a name. The guidance on confidentiality in expert consulting sets out how those limits can be described to participants before they agree.
Limits of this approach
Documenting divergence diagnoses the process; it does not tell you how to weight competing judgments, and it will not remove prior commitments that participants bring to the room. Small expert pools make some of these signals weak, because a single dissenting view and a single error look similar in a sample of four. The worksheet also imposes a recording burden on the facilitator, which may exceed what a short exercise can justify. Where the decision is low-stakes or the expertise is uncontested, two fields, the question as put and the sealed judgment, are likely to be enough.
A reasonable next step is to run one forthcoming elicitation with sealed initial judgments and the reason-code log, then compare the sealed and final records. If that comparison shows nothing the discussion would have revealed anyway, keep the process minimal; if it shows movement that the aggregate would have hidden, you have a concrete reason to extend the format.