Fight model overconfidence by having a threshold for failure

 

"Overconfidence Could Be Investors’ Biggest Mistake" - Richard Thaler, Behavioral Economist, University of Chicago Professor Barron's December 2020

There can be overconfidence with models. Markets change and sticking with one model in the face of changing conditions can lead to underperformance. A model that has worked in the past may not always work in the future. No method for adaptation is a recipe for model overconfidence. There is no acceptance for failure.

All models will fail or have periods of poorer performance. There is a frequency of success and while it may be stable and high, it can still be subject to periods of underperformance. Hence, there needs to be consideration on when the researcher should throw-in the towel and make changes. The modeler and thus their model can be overconfident. To not consider failure and the need for change is no different than the discretionary trader who has subjective beliefs that do not change or match reality. 

The question is when model change should be considered. Certainly an ad hoc approach for model change based on a run of failures is the most frequent approach but it does not take advantage of advances in data science which focus on prediction and success based on a defined fitness function. Fitness can have a threshold which if breached represents failure. Action can be triggered by predetermined criteria. This can range from the simple, a maximum drawdown threshold to more complex criteria which are regime based. The important issue is having a plan for when you lose confidence in a model.