P-hacking is an important concept for understanding the value or failure of financial research. Whether called data mining or data snooping, the process of searching for significance can lead to results that are spurious and will not follow in out of sample analysis and trading. What was significant for a back-tested sample will not be replicated out of sample or in real life. There is now a zoo of factors that have showed significance in financial research which begs the question of whether there really are so many unique factors in an efficient market. There is also the question of what are the drivers for these significant factors.
P-hackling is real, but the question is whether it applies to all factor tests. A recently published paper "The Limits of p-hacking; some thought experiments" takes a step back and focuses on the likelihood of a p-hacking problem for all of the research results in the factor zoo. He focuses on the likelihood of p-hacking with factors that have t-stats close to 6.
This p-hacking problem may be especially present on the margins for t-stats around 2. While there are hundreds of studies on unique risk premium, the field of studies that show high levels of significant are limited. The author of this study states that the likelihood of p-hacking for high t-stat factors is low. There would have to be millions of tests to find these levels of significant.
P-hacking is real and should be feared, but data snooping cannot be the explanation for all the results on pricing anomalies. There is a continuum of market efficiency. The bar for finding unique factors, risk premium, and anomalies is high, and the null should be that markets are likely to be significant, but that does not mean that markets will always be efficient. Mispricing, pricing anomalies, structural opportunities, and limits to arbitrage do exist. Be a skeptic, but sometimes the results from testing are real.
No comments:
Post a Comment