Daniel Kahneman's favorite paper

I heard this story from a Credit Suisse seminar held by Michael Mauboussin. He is excellent investment thinker who has written a number of provocative books on decision-making. Make time for Mike.

The great behavioral psychologist and Noble Prize Winner in Economics stated that of all the papers that he wrote the one that was his favorite was On the Psychology of Prediction written in 1973. In this work, Kahneman with his co-author Amos Tversky discuss the difference between intuition and statistical prediction. They discuss the problem of representative bias and how there are three types of information for any prediction. There is the base case or outside view, there is the specifics about the particular case or inside view and finally the relative weights you assign to each. 

This view is related to any good Bayesian view of the world. You have to look at a base case and compare against the specifics. This is also similar to the shrinkage estimators which look at the long-term average plus the difference between a sample and the long-term average scaled by a shrinkage estimate. Put differently, you are never as good as you think you are on the best of days nor as bad as you think you are on the worst days.

Any forecast has to be tempered by long-term averages, You cannot extrapolate from a small set of facts. Some call this mistake the law of small numbers. Temperance in any forecast is a good thing. Mauboussin would say there is a difference between luck and skill and you should weight events within this framework. When there is more luck in an event, there will be more mean reversion. If there is more skill, there will less revision to them mean because the actual event is more likely.

This paper is 40 years old but still provides strong insights.