The history of evidence-based investing - data and techniques using computing and storage to beat competition

 

The above chart is from Marcos Lopez de Prado powerpoint presentation "Escaping The Sisyphean Trap: How Quants Can Achieve Their Full Potential". The slide does a good job of dividing the world into four major time periods in the move to evidence-based investing. From technical analysis, fundamental analysis, and market microstructure, we are now in the era of machine learning. I may not agree with the framework, but it provides a good starting place for discussion. 

The ascent of evidence-based investing matches the growing power of computing and data storage. The two cannot be separated. With computing and storage came the development of sophisticated databases that allowed for testing of hypotheses. While academics and a few large firms had access to strong computing power, the ascent of the quants only began with the personnel computer that allowed desktop testing of evidence. 

The explosion of early quants came with using the best data available, prices from exchanges. Technical quant trading grew because the databases and computing power was easy to access. The fundamental data explosion came with the availability of the CRSP database from the University of Chicago which started 60 years ago but exploded in use as computing power improved.

Once data and computing became readily available, the demand for quants and programmers grew in an arms race that has not abetted. The first part of the arms race was focused on more data which again started with prices. If everyone had open, high, low, and close, the next race was for hourly and minute data. The only limitation was storage. The same applied with adding more firms with fundamental data.  

In this arms race, once the existing data usage were exhausted, the focus had to be on techniques to reconfigure data. There was a limit on the number of linear regressions that could be done on any database. The move to data science and machine learning is the continuation of trying to better competition that has the same data and computing power. The second line of work in this competitive battle is the use of alternative databases. Once all the existing data was mined, the search for new data began in earnest. 

The quest to deeper evidence-based investing is not about the quest for knowledge albeit this is a side result, but a drive for profits in one of the most competitive industries in the world.