The replication problem in finance - You don't get what you thought

 

There is a problem with replication of finance studies on risk premia and trading strategies. Additionally, there is a problem of out-of-sample results not matching in-sample returns. You don't get what you think with most financial studies. There are a couple of reasons for this disconnect. One, the construction of the test was poor. Two, the articles published are only extremes from data mining. Three, risk premia, once observed are arbitraged away. Four, market behavior and structures change. Overall, the buyer needs to beware.

Investors are aware of research and exploit it so that future returns are lower. There is a life expectancy for strategies and once the cat is out of the bag and the general trading public knows the strategy, excess returns are quickly gone. The out-of-sample results also show that there are strong performance declines.

So, what is the slippage that you should expect from models? Studies have found that portfolio returns for a strategy may fall about 25% for out-of-sample work. The drop can be over 50% during the five years after publication. 

Quantpedia did a study of out-of-sample results and found that the return decline is universal across strategies. Testing a large set of strategies, they found an average drop of 33% out-of-sample and with the median drop over 40%. 

Nevertheless, there is a significant performance difference between in and out-of-sample results. There is a lot of variation. However, a careful analysis suggests that since there is factor momentum, there can be ways to reduce the out-of-sample problem. Hold the factor premium or strategy that is trending higher.