REFERENCE CLASS FORECASTING

Reference Class Forecasting (RFC) is a forecasting method developed by taking into account the outcomes of similar actions taken in the past and their outcomes.

Reference Class Forecasting assumes that the behavioral pattern of an entity to external changes in environment does not change with time. That is, if some changes occur in the environment, then the object will behave in the same way it would have done, if the same changes had occurred in the past. Taking the outcomes of certain decisions taken by the same or a different organization in the past, it is possible to predict the outcome of a decision taken in the present.

Need for Reference Class Forecasting

Whenever the outcome of a certain action or decision is to be forecast, there are two stances adopted by people:
a) The Inside View
b) The Outside View

The Inside View is one in which the current situation is thoroughly analyzed without taking the past experiences into account. Forecasts by made by focusing tightly on the case at hand, i.e. by considering its objective, the resources available and required, and the obstacles.

The Outside View is the same as Reference Class Forecasting, which predicts the outcome of an action by analyzing past outcomes under different environmental conditions.

Traditional forecasting methods suffer from the biasness and human factor, and adoption of the Inside View. There are 2 main reasons for the errors in forecasting due to Inside Views.

a) Optimism Bias (Physiological Explanation):
Most managers, especially those who are intuitive, are over-optimistic about the future of a project, or an action. They tend to underestimate the costs, completion times, and risks of planned actions while overestimating the benefits of those same actions.
In his article “Delusions of Success: How Optimism Undermines Executives' Decisions”, author Daniel Kahneman described a situation where an expert, adopting an inside view, estimated that a certain project would be completed in 18-30 months. But similar projects in the past had taken 7-10 years for completion. The expert ignored this outside view and went ahead with the project, which ultimately took 8 years to complete.

b) Strategic Misrepresentation (Political Explanation):
When forecasting the outcomes of projects, managers deliberately overestimate the benefits and underestimate the costs in order to increase the likelihood that it is their projects gain approval & funding and not that of their competitors'. For example, in a company, the IT approval committee chooses various IT projects to implement out of a portfolio on the basis of their risk, returns and costs. A project sponsor may deliberately show increased benefits of his projects to get approval.

These types of errors of judgment are often systematic and predictable rather than random and occur as a result of bias, rather than confusion. It is because of these two reasons that there was a need to include all the past distributional information while making forecasts. Reference Class Forecasting, in this sense suggests that forecasting be done in a manner in which judgment is controlled by a more critical evaluation of the evidence and normal faith in one's impressions must be suspended.

Forecasting Procedure

The following 3 steps are to be followed for Reference Class Forecasting.

1. Select a Reference Class:
While choosing a Reference class, first the variables need to be identified on which to compare the current situation with the reference scenario. After that, those scenarios are chosen as reference in which the variables values are similar. The article “Delusions of Success: How Optimism Undermines Executives' Decisions” provides the example of a studio executive trying to forecast sales of a new film, where the reference class is based on the variables: Genre, Actors, Budgets, etc. The reference projects are the recent movies of the similar genre that have similar budgets and the same actors. The Reference Classes should be broad enough to be statistically meaningful but narrow enough to be truly comparable to the project at hand.

2. Assess the distribution of outcomes:


After the reference class is chosen, the similar recent projects are identified and their outcomes are arranged in a distribution ranging from the least to the most favorable. Theses outcomes are the results of different environmental conditions in each of the cases. If all the outcomes between the extremes have not taken place, the distribution can be interpolated using the mean and standard deviation.

3. Compare current project with the Reference Class:
Based on the environmental conditions of the project at hand and that for the other projects, it is needed to predict where the current project would fall along the distribution, and as a result, find the most likely outcome. The judgment of the environment can sometimes become biased, which is why 2 additional steps may need to be followed for a more accurate forecast.

4. Assess the reliability of your prediction:
Different persons can predict different events with different accuracies. From past experiences, one needs to find the correlation between his predictions and actual outcomes. If one feels that he has, in the past, reliably predicted similar events (correlation nearly equal to 1), then there is a fair chance that the environment comparison is correct and current prediction will hold good. On the other hand, if the correlation is low (near to zero), the current prediction needs to be modified somewhat. This method requires a thorough statistical analysis of past predictions and outcomes.

5. Correct the Intuitive Prediction:
Based on the predictive correlations found in step 4, the initial estimate is refined and the final forecast is found. The final forecast is given as per the formula,

Final Forecast =
Initial Forecast + Predictive Correlation * (Mean Outcome-Initial Forecast)

When the predictions are uncertain, the variation is quite large.

Advantages

1. Reduces Optimism Bias:
The primary advantage of using Reference Class Forecasting comes from the fact that the optimism bias of forecasters is negated by statistical study of past actions, which leads to much greater accuracy than traditional forecasting models.

2. Takes Environment into account:
By analyzing outcomes of several actions belonging to a similar Reference class, the environmental effect on the outcome of the system can be known. This is a major leap from other Forecast based planning methods, which do not take the environment into consideration at all.

Disadvantages

Although Reference Class Forecasting has some advantages over traditional forecasting methods, but it still suffers from some of the limitations of the later. They are described below:

1. Finding the appropriate Reference Class:
Finding the appropriate Reference Scenario with which to compare the current situation and state of the company is a difficult task. First the variables which describe the states of the companies need to be identified in order to compare the states, which can become a lengthy process.
Secondly, even though two states may seem similar, they might actually be quite different. For example, while setting up a business ‘A’ in country ‘X’, it might seem convenient to compare it with a situation where another company set up the same business ‘A’ in another country ‘Y’. But if the government rules and policies of the countries are quite different, then it is more beneficial to compare the state with a case of a company setting up a different business ‘B’ in the same country ‘X’.

2. Non Existence of Reference Classes:
If a company is going to take up a totally new initiative, which has never been done before, then there won’t be any reference states to the current state. For example, when introducing a new technology or an entirely different kind of product the company can only compare its state with cases where other new technologies or products were introduced; but those products might be having completely different attributes to what is needed. In these cases, RFC can be very inaccurate.

3. Limited Environmental Changes Considered:
The different reference classes used provide different environmental conditions under which outcomes are different. But since the past is not a complete reflection of the future, there can be different and extreme environmental conditions also which have not been taken into account in the reference classes. Reference Class Forecasting fails in this scenario. Long run forecasts are generally inaccurate with this model.

4. Failure in Strategic Misrepresentation:
RFC works out fine when the problem with forecasting is optimism bias and inaccuracy is because of honest mistakes. But when the cause is Strategic Misrepresentation, and the forecaster deliberately wants to manipulate the forecast data, this model fails.

Conclusion

After studying the model in some detail, we can conclude that Reference Class Forecasting can reap great benefits, but only if it is used in the right context and for the right organization. Ultimately, organizations will have to employ advanced planning techniques such as Interactive and Strategic Planning in order to thrive, not just survive.

References

http://en.wikipedia.org/wiki/Reference_class_forecasting

Lovallo, D., & Kahneman, D.(2003, July). Delusions of success: How optimism undermines executives' decision. Retrieved August 25th 2008 from http://hbswk.hbs.edu/archive/3630.html

Flyvbjerg, B. (2006). From Nobel Prize to Project Management: Getting Risks Right. Retrieved August 25th 2008 from http://flyvbjerg.plan.aau.dk/Publications2006/Nobel-PMJ2006.pdf

Flyvbjerg, B. (2004, June). The British Department for Transport: Procedures for dsealing with Optimism Bias in Transport Planning - Guidance Document. Retrieved August 25th 2008 from http://flyvbjerg.plan.aau.dk/0406DfT-UK%20OptBiasASPUBL.pdf