Research Confidence Is Not Scenario Probability
Separate confidence in a research claim from the likelihood of a future event, and use numerical estimates only where their basis is defensible.
Research confidence and scenario probability answer different questions, and conflating them produces claims that look measured but are not. Research confidence describes how well the available evidence and stated assumptions support a finding within a defined scope. Scenario probability describes the estimated chance that a specified event will occur in a specified period. A study can hold high confidence in an exposure estimate while assigning no probability to the scenario in which that exposure becomes consequential. The first is a judgment about evidence; the second requires a defined event and time window, a reference class or model, and a basis for the estimate.
What confidence describes
Confidence is a statement about support, not about occurrence. It combines judgements about the relevance of the evidence to the question, the quality and coverage of the data, the consistency of findings across sources, and the sensitivity of the conclusion to its assumptions. It is conditional: high confidence in an exposure estimate for a defined population and period does not transfer automatically to another population, period or definition.
A confidence label should always have an object. "High confidence" alone is incomplete; the reader needs to know whether the confidence concerns a measurement, a causal claim, the internal coherence of a scenario, or the sufficiency of the evidence base. It should also carry its conditions: data years, geography, definitional choices and known gaps. Without those, the label is a tone rather than an assessment.
Uncertainty about measurement, coverage or future change informs the confidence judgement; it is not itself a probability. A finding can be uncertain in its measurement and still support a confident statement about a bounded quantity, or be precisely measured and still say nothing about whether a future event will occur.
Is there a probability model?
Usually there is not. A probability claim requires at least a defined event, a time window, a reference class or model, and a basis for the estimate: historical base rates, a statistical model, or a structured and calibrated elicitation. Many policy research products have none of these. They have evidence about exposure, mechanisms and institutional constraints, which can support a scenario's plausibility but not its frequency.
Where a model exists, the probability is model-conditional. It should be reported with the assumptions that generate it and the conditions under which it would be revised. Where no model exists, a numerical probability is not a more precise version of a qualitative judgement; it is a different kind of claim that the evidence does not support. The UK Government's Futures Toolkit, published by the Government Office for Science, presents scenario development as a way to explore plausible futures and test policy under uncertainty rather than as a forecast of which future will occur (Futures Toolkit). That framing is useful because it separates the construction of a scenario from a claim about its occurrence.
Four judgments that should stay separate
The table below is a suggested working vocabulary, not a validated instrument or an institutional requirement. Its purpose is to keep four judgments distinct in drafting, review and communication with decision-makers.
| Judgment | What it describes | What it does not describe | Suggested wording | Wording that misleads |
|---|---|---|---|---|
| Evidence support (research confidence) | The strength, relevance and consistency of evidence for a stated finding, conditional on scope and assumptions | The chance that a future event will occur; a calibrated probability | "High confidence in the exposure estimate for the defined period, with coverage gaps in two districts noted" | "High confidence this will happen"; a percentage attached to confidence without a calibration basis |
| Assumption sensitivity | How much the conclusion changes if a named assumption changes | The probability that the assumption is correct | "The estimate is sensitive to the assumption that the route remains open; closure would substantially increase the affected volume" | "The assumption is likely to hold" when no basis for likelihood is given |
| Scenario plausibility | Whether a constructed scenario is internally coherent and consistent with known drivers and constraints | The scenario's relative likelihood or frequency | "Plausible under the stated drivers; not ranked against alternative scenarios" | "The most likely scenario"; a decimal probability attached to a narrative |
| Event probability | The estimated chance that a defined event occurs within a defined period, from a model, base rate or calibrated judgement | The strength of evidence for a scenario's logic; research confidence | "Model-based annual estimate, conditional on assumptions A and B" (only where such a model exists) | "High confidence, therefore high probability" |
The fourth row is the one most often borrowed without its conditions. A probability estimate is only as good as its event definition, its reference class and its calibration. If those are absent, the honest output is a conditional scenario with explicit assumptions, not a number.
A fictional example: exposure evidence without occurrence probability
The following example is fictional and is constructed only to show how the labels work. It is not a forecast, and it does not describe a real region or event.
Suppose a research team is examining access to a supply route under contrasting administrative conditions. The team has freight-volume data and route-dependency estimates for a defined period. The data have known gaps, but the exposure estimate is reasonably sound: the team can say with high confidence that a substantial share of freight in the fictional region moves through a single corridor.
The team also constructs a scenario in which administrative changes restrict access to that corridor. The causal narrative is coherent and consistent with the evidence on how access decisions are made. But the team has no base rate for such restrictions in comparable settings, no calibrated model and no agreed time window. What can it say?
A defensible formulation would be:
Exposure is high and well evidenced for the defined period. The access-restriction scenario is plausible under the stated drivers and institutional conditions. The occurrence probability of the scenario is not estimated, because the evidence base does not support a calibrated probability.
An earlier draft might have said: "There is a high probability of disruption." That sentence conflates confidence in the exposure evidence with the chance of the event. A revised version separates them:
Confidence in the exposure estimate is high. Confidence in the causal narrative is moderate. The scenario is plausible, not ranked against alternatives, and carries no calibrated probability.
If the team later needs an event probability, it would need to define the event and period, identify a reference class or build a model, and state the assumptions that generate the estimate. Until then, the conditional scenario is the more accurate product. The same discipline applies when a decision-maker asks for a single likelihood: the researcher can explain what would be required to produce one and what the current evidence can support instead.
Labeling rules that avoid implying percentages
A few drafting rules reduce the risk of a qualitative label being read as a number.
Name the object and the evidence base. Write "high confidence in the exposure estimate, based on the named data source and period, with the named gaps" rather than "high confidence". The reader should be able to see what the label attaches to.
Define the scale in the document. If a project uses high, moderate and low, state what each means for that project and what would move a finding between them. A scale defined once and applied consistently is easier to interpret than adjectives chosen sentence by sentence.
Keep occurrence language out of confidence sentences. Words such as "likely", "probable" and "unlikely" carry probability connotations. Use them only where a defined probability scale and a calibration basis exist. Otherwise use "plausible under the stated conditions", "consistent with the evidence" or "not estimated".
Record revision triggers. A confidence judgement should state what new evidence, data revision or assumption change would alter it. This converts the label from a static verdict into a testable statement and gives the reader a basis for updating.
Do not translate confidence levels into probability bands. There is no general mapping from "high confidence" to a percentage, and inventing one produces false precision. Different questions, evidence types and decision contexts require different treatments.
What this means for project-based research
For researchers considering project-based policy, regional or social research, the distinction is a scoping issue as much as a writing issue. A project brief may ask for a likelihood, a risk rating or a "most likely scenario". The researcher's contribution can be to clarify which of the four judgments the decision actually needs, what evidence exists for each, and what would be required to move from a conditional scenario to a probability estimate. That conversation is easier before the deliverable is drafted.
It also affects how findings are defended. A confidence statement tied to evidence and assumptions can be explained and revised. A probability claim without a model cannot. Where the evidence supports exposure and mechanism but not frequency, the defensible product is a scenario record: assumptions, causal narrative, exposure, evidence to monitor, response options and revision reasons. The record can be useful to a decision-maker without implying that the scenario is probable, exhaustive or validated as a prediction.
Researchers reviewing the scope of a potential project can use the questions in Before you accept an expert project to test whether the requested output matches the evidence that can be supplied. The relevant test is not whether a confidence label sounds strong, but whether the document distinguishes evidence support from occurrence probability and states what would change each.
A practical next step is to mark every sentence in a current draft that implies occurrence, then either attach the model and assumptions that support it or rewrite it as a conditional statement.