Ensemble modeling - A solution to ambiguity and improved forecasting

Models will fail. Models will miss variables. Models will change with the sample side used. We are imperfect modelers. It is hard to find the right models because it is hard to differentiate between theories. Data relationships change so the importance of some variables change through time. Market see structural changes and regime changes. Hence, a good model today may not work tomorrow. These fundamental issues are not new and discussed even in introductory econometric classes, yet the real world of market forecasting has to find meaningful solutions.

Of course, the need for good theory is paramount to drive an econometric model, but the advancements in machine learning deemphasize the formation of modeling and allow data to speak for itself. An atheoretical approach to modeling will focus on measured success without theory. This is problematic for theorists, but ultimately there needs to be a focus on success. A number of ensemble modeling approaches have been developed to help with prediction. Two major schools can be applied to forecasting problems. The first is the simple forecasting approach of model averaging which was introduced by Clive Granger fifty years ago as a simple combination of forecasts. The second set is more formal process for developing alternative models used in machine learning.

This first ensemble approach:

Average or weighting models - 

• Take a set of model forecasts and average or form some type of weighting scheme for multiple models. This has been shown to be successful especially when the model used are uncorrelated. However, this approach has not been formalized into a process of finding alternative models. 
• The average can be done through some weighting or voting scheme. A better model is given greater weight. This combination of forecasts has developed a large literature on ways to reduce the estimation error of forecasts. For example, a simple application to trend-following would be to use the signal from different trend lengths.

The second set of ensemble approaches which have come out of statistics and computer science include: 

Bagging models - 

• Look for simple variations on the same theme of model averaging. Bagging stands for bootstrap aggregating. A very high-level view of bagging is using a single dataset tested over different subsamples of data sets or bags of data to generate different model choices. The bags are created with resampling to test different models. The modeler who applies the same basic approach across different markets can be thought of as using a bagging method. The forecast results are bundled or averaged to create the bagged model results. 

Boosting models - 

• A boosting approach uses past predictions to help with new models. The idea is to find what has not worked with one model and build a new model that will reduce or offset the failure of the prior model. It can be described as using a modeling method to make a set of weak learners into a stronger learner. A model can be developed, but it will have some errors. A new model can be formed that focuses on the errors of the prior model. A model that is adjusted to account for errors could be thought of as being boosted. Using the gradient boosting method, a model is formed which will have residuals. A new model will be formed to explain the residuals (if they are not random) which is then added to the original model.  

These ensemble approaches are important methods to increase prediction as opposed to drawing some conclusion of model failure. While we have just provided a very simple overview, ensemble techniques have been used for decades to improve quant results. 

Adding uncorrelated models to get better forecasts, training or testing on different data sets to find common parameters or different models, building or adding models that address failure and reduce estimation errors have all been used to help forecasts; however, there are now more structured approaches to generate more efficient ensembles.