(Behaviour Science) #21 Planning Fallacy

 

Principle · Cognitive bias category

Planning fallacy

The systematic tendency to underestimate the time, cost, and risk required to complete a future task — while simultaneously overestimating the benefits — even when the planner has direct experience of similar tasks running over time and over budget in the past. It is a specific, predictable manifestation of optimism bias applied to personal and organizational planning.

90%

of large infrastructure projects go over budget (Flyvbjerg, 258 projects)

44%

average cost overrun in real terms on major public projects

×2–3

typical ratio of actual completion time to originally estimated time

1979

Kahneman & Tversky — the coining paper

1. What it is and the science behind it

Kahneman and Tversky coined the term "planning fallacy" in 1979 to describe a specific bias they had observed in project estimation: people consistently produce optimistic forecasts for their own future projects even when they know perfectly well that similar projects in the past ran long and over budget. The bias is not explained by ignorance — people know the base rates — but by a systematic failure to apply that knowledge to their own situation.

The deepest explanation lies in the distinction between two fundamentally different ways of approaching a forecast: the inside view and the outside view. These are not just different data sources — they reflect different cognitive orientations toward the problem, and most planners default to the inside view without realizing it.

Inside view vs. outside view — the core distinction

Inside view

Project-specific reasoning

Focus on the specific details of this project — the plan, the team, the milestones, the unique features that make it different. Build the forecast by imagining the sequence of steps and estimating each one. Feels rigorous and specific.

Result: optimistic forecast that ignores base rates of similar projects. The "this time is different" cognitive trap.

Outside view

Reference class reasoning

Begin with the question: what happened to similar projects in the past? Find the relevant reference class, establish the base rate distribution of outcomes, and anchor the forecast on that distribution. Adjust only for genuinely unique factors.

Result: calibrated forecast grounded in empirical history. The "what actually happens to things like this" approach.

Kahneman describes the inside view as the natural, almost irresistible default. When you are planning your project, it feels wrong — even disrespectful to your team and your work — to say "I'll just treat this like all the other projects that failed." But the outside view is not pessimism. It is calibration. The question is not "will this project fail?" but "what does the historical distribution of outcomes look like, and where should my estimate sit within that distribution?"

Why it happens — four mechanisms

Scenario thinkingWhen planning, people construct a narrative of how the project will unfold — imagining steps, milestones, and a smooth execution path. This scenario focuses attention on the plan working, not on the distribution of things that can go wrong. The scenario feels complete, so estimates are anchored on it.

Motivational optimismPeople want their projects to succeed — and wanting a forecast to be optimistic subtly distorts the estimate. This is not deliberate dishonesty; it is the same desirability-contaminates-probability effect documented across optimism bias research. Personal investment in the project amplifies the effect.

Unpacking failure to generalizePlanners who have experienced overruns on past projects still exhibit the fallacy on new ones — because each new project feels different enough to justify a fresh start. The brain does not automatically aggregate past experience into a calibrated base rate; it treats each project as a novel case.

Singular focus on best caseInside-view planning naturally produces best-case estimates because it focuses on how the project succeeds, not on the probability distribution of outcomes. Risks are noted but treated as exceptional rather than as the expected texture of complex work.

Reference class forecasting — the evidence-based counter

Flyvbjerg's reference class forecasting method — four steps

1

Identify the reference class

What type of project is this? Find a large, representative sample of completed projects of the same type — same domain, similar scope, comparable complexity. Resist the urge to define the class so narrowly that your project appears unique.

2

Establish the base rate distribution

What is the distribution of outcomes for that reference class? What percentage went over budget and by how much? What was the median completion time vs. estimated time? What was the distribution of cost overruns — is it skewed? This is the empirical anchor for your forecast.

3

Position your project within the distribution

Given what you know about your specific project, where does it sit within the reference class distribution? Are there documented factors that justify a better-than-median forecast — or factors that suggest higher risk? Adjustments should be modest and evidence-based.

4

Apply the optimism bias uplift

Add a mandatory contingency buffer based on the historical overrun distribution for this reference class. The UK Treasury Green Book mandates specific uplift percentages by project type (e.g., 44% for standard buildings, 66% for equipment). This is not pessimism — it is calibration to what actually happens.

Key studies

Student thesis completion — the classic lab study

Buehler, Griffin & Ross, 1994

Students were asked to predict when they would complete their senior thesis. The average predicted completion time was 33.9 days. Actual average completion: 55.5 days — 64% longer. Crucially, when students were separately asked how long similar projects had taken in the past, they accurately recalled that past projects ran significantly over their original estimates. But they did not apply this knowledge to their own current forecast — because they believed their current project was somehow better planned or more under control than past ones. Even being explicitly asked about past experience did not eliminate the fallacy; it was applied to others, not to self.

Actual completion 64% longer than predicted — even with accurate recall of past overruns

Global infrastructure megaprojects

Flyvbjerg, Holm & Buhl, 2002 — Journal of the American Planning Association

Analysis of 258 large infrastructure projects across 20 nations and 70 years found that 90% went over budget, with an average real-terms cost overrun of 44%. Rail projects averaged 45% over budget; fixed links (bridges, tunnels) 34%; roads 20%. The pattern was statistically consistent across time periods and geographies — ruling out random error and pointing to a systematic bias in how large projects are estimated. Flyvbjerg argued the overruns are not primarily technical surprises but the predictable result of inside-view planning by teams who ignore the base rate of comparable projects — and, in some cases, deliberate strategic misrepresentation to secure project approval.

90% over budget; average 44% overrun — consistent across 70 years and 20 nations

Outside view intervention — reducing the fallacy

Lovallo & Kahneman, 2003; Buehler et al., 2010

Experimental studies tested whether prompting planners to take the outside view — explicitly asking "what happened to similar projects?" before producing their own estimate — reduced the planning fallacy. The intervention consistently improved forecast accuracy, with estimates significantly closer to actual outcomes when planners were forced to anchor on base rates before building their own scenario. The effect was strongest when the reference class was specific and the base rate data was concrete. Generic prompts ("think about similar projects") were weaker than structured reference class methods with actual data.

Outside view prompt significantly improved forecast accuracy vs. unaided inside view

IT project overruns — the Chaos Report

Standish Group, replicated annually 1994–present

The Standish Group's annual surveys of IT project outcomes consistently show that fewer than 30% of software projects are delivered on time, on budget, and with the originally specified features. Average cost overrun in large IT projects exceeds 50%; a significant proportion are cancelled mid-delivery after substantial investment. The pattern has not meaningfully improved over three decades of the report — suggesting that awareness of the planning fallacy at an industry level has not translated into calibrated estimation practice. Structural interventions (agile methods, shorter sprints, reference class data) show more promise than awareness training alone.

Fewer than 30% of IT projects on time and on budget — consistent for 30 years

2. Real application examples

Business

Product development sprints

Engineering teams that anchor sprint estimates on historical velocity data — actual story points completed per sprint over the last 10 sprints — produce significantly more accurate delivery forecasts than teams that estimate from scratch each time. The velocity data is the reference class; the current sprint plan is the inside view. Agile's empirical approach is essentially a structural counter to the planning fallacy.

Business

New product launch timelines

Product managers who track time-to-launch for previous product releases — and use that historical distribution as the anchor for new launch estimates — consistently outperform those who estimate from the project plan alone. The reference class data captures the distribution of unforeseen delays that the inside view systematically ignores.

Business

Pre-mortem analysis in project planning

The pre-mortem technique — asking teams to imagine the project has failed and write down why — is a structured inside-view intervention that forces planners to generate failure scenarios they would otherwise suppress. Combined with reference class data, it produces estimates that account for both the statistical base rate and the specific risks of the current project.

Public policy

UK Treasury Green Book uplifts

Following Flyvbjerg's research, the UK Treasury formally embedded reference class forecasting into its project appraisal guidance. Teams must apply mandatory "optimism bias uplifts" — percentage increases to cost and time estimates derived from historical overrun data for each project type. The policy institutionalizes the outside view as a requirement, not an option.

Public policy

Infrastructure procurement reform

Denmark, Norway, and the Netherlands adopted reference class forecasting for major public investment decisions following Flyvbjerg's advocacy. Projects appraised using RCF showed measurably smaller cost overruns than comparable projects using traditional estimation — the first policy-scale evidence that the fallacy can be structurally reduced.

Public policy

Strategic misrepresentation in bid processes

Flyvbjerg documented that some infrastructure cost overruns are not purely cognitive — they reflect deliberate underestimation to win project approval, with the expectation that costs will be revised upward once committed. Procurement processes that require reference class forecasting and independent review reduce the incentive for strategic misrepresentation by making the base rate publicly available as a benchmark.

Personal habit

Personal project timelines

The most reliable personal counter to the planning fallacy is tracking actual completion times for recurring tasks and using that data to estimate future ones. "How long did the last three tax returns actually take?" is a better predictor than "how long should this one take?" Most people do not keep this data — but those who do systematically make better time estimates.

Personal habit

Home renovation and DIY projects

Home renovation is one of the most reliably planning-fallacy-afflicted domains in everyday life — budgets and timelines routinely double. The standard folk counter ("double your estimate") is a rough outside view correction. More accurate: ask people who have recently done the same type of renovation what it actually cost, not what they expected.

Personal habit

Writing and creative projects

Writers, designers, and creative professionals consistently underestimate project completion times — partly because creative work is non-linear and partly because the inside view imagines a smooth creative flow. Writers who track word counts per session and use historical data to estimate project completion dates produce significantly more accurate timelines than those who estimate from the manuscript plan.

3. Design guidance — when and how to use it

The central design insight

The planning fallacy is not cured by asking people to "be more realistic" or "add some buffer." These prompts invoke the inside view with a slight pessimism adjustment — which is still the inside view. The only robust counter is a structural switch to the outside view: what does the historical distribution of outcomes look like for projects of this type? Start there. Adjust from the base rate rather than building up from the plan. This is a fundamentally different cognitive operation, not just a recalibration of the same one.

When counter-design is needed

Use when

Any project or task involves significant time, cost, or resource commitments — especially where overruns have irreversible consequences on budget, schedule, or downstream dependencies.

Use when

The team or individual has personal investment in the project's success — which amplifies motivational optimism and strengthens the inside-view pull.

Use when

Historical data on similar projects exists and can be accessed — reference class forecasting is only as good as the reference class. Build and maintain internal project completion databases.

Use when

Projects involve novel technology, complex dependencies, or multiple stakeholders — all factors that increase variance and make the inside view's scenario-based estimates especially unreliable.

Be careful when

Applying reference class thinking too rigidly to genuinely novel situations where no reference class exists. In truly unprecedented projects, honest uncertainty ranges are more honest than a false precision derived from a poorly matched reference class.

Be careful when

Over-correcting to the point of demotivating teams. The goal is calibration, not pessimism. Presenting realistic ranges as "this will probably fail" undermines team morale without improving forecasts.

Step-by-step counter-design process

1. Before building any project plan, find the reference class first — identify comparable completed projects of the same type, scale, and domain. Resist the urge to define the reference class so narrowly that your project appears unique. Narrow reference classes are a cognitive escape hatch from the outside view.
2. Gather actual outcome data from the reference class — time to completion, budget vs. actuals, scope changes, cancellation rates. The distribution matters more than the average: if 20% of similar projects are cancelled and 70% run more than 50% over budget, that is the landscape your project is entering.
3. Set the baseline estimate at the reference class median, not the best case — your project's initial schedule and budget anchor should be the median outcome for similar projects, not the plan-as-designed. The plan is then an argument for why you expect to do better than median, not the starting point.
4. Apply an explicit contingency buffer derived from the reference class overrun distribution — not a round number ("let's add 20%") but a figure grounded in the historical data. UK Treasury tables provide specific uplifts by project type; building your own internal database over time is even more valuable.
5. Conduct a pre-mortem before finalizing estimates — ask the team to imagine it is 12 months from now and the project is 60% over budget and 8 months late. What happened? The scenarios generated should be used to stress-test the estimate and add specific contingency lines, not just absorbed into a general buffer.
6. Separate estimation from advocacy — the person or team most motivated to deliver the project should not be the sole estimator of its probability of on-time, on-budget delivery. Independent estimation or structured red-team review introduces an outside perspective that is not contaminated by personal investment in the plan.
7. Track and publish actuals after each project — the long-term fix for the planning fallacy at an organizational level is a culture of honest retrospective data. Every project's actual vs. estimated outcomes should be documented and accessible for future reference class use. Organizations that do this accumulate their own reference classes and improve their forecasting over time.

Before and after — estimation framing

Software feature delivery estimate

Inside view (weak)

"We've scoped the feature into 12 tasks. Based on the plan, we estimate 6 weeks. We've flagged some risks but are confident in the timeline."

Outside view first (strong)

"The last 8 features of similar complexity took between 7 and 14 weeks, median 10. Our plan-based estimate is 6 weeks — here's specifically what we believe makes this faster: [X, Y]. Our committed estimate is 10 weeks with a range of 8–14."

Home renovation budget

Inside view (weak)

"The builder quoted £45,000 and 12 weeks. We've budgeted £50,000 to be safe."

Outside view first (strong)

"Kitchen renovations of this scope typically run 30–50% over quoted price. We're budgeting £65,000 and planning for 18 weeks, treating £50,000 and 12 weeks as the optimistic scenario rather than the expected one."

New product launch planning

Inside view (weak)

"Our launch roadmap has us shipping in Q3. The team is aligned and we have no major blockers currently."

Outside view first (strong)

"Our last three product launches took 4, 6, and 5 months longer than the original roadmap. We're communicating Q3 internally and Q4 externally — and treating Q1 next year as the realistic worst case we're planning infrastructure for."

The strategic misrepresentation problem — not all overruns are cognitive

Flyvbjerg's infrastructure research revealed an uncomfortable finding: a significant proportion of large-project cost overruns are not purely the result of cognitive bias. They reflect deliberate strategic underestimation — teams and sponsors who know the real expected cost but present optimistic figures to secure project approval, intending to revise estimates once commitment is locked in. This is sometimes called "survival of the unfittest": projects with honest, realistic estimates lose funding to projects with optimistic, unrealistic ones. Counter-designs that only address cognitive bias — better estimation methods, reference class training — will not fix strategic misrepresentation. That requires institutional incentives: procurement processes that penalize large post-approval cost revisions, independent cost review that is genuinely independent, and a political culture that rewards honest uncertainty over false confidence. Both problems are real; they require different solutions.