Machine learning and portfolio optimization

Machine learning can be used to enhance classic portfolio optimization through dividing the portfolio selection problem into two parts. By making this simple division, investors can use the best of both worlds to improve stock portfolio construction. Use machine learning data science techniques to improve predictions of expected return and optimization techniques to find the right diversified mix of stocks meet an objective like maximizing the Sharpe ratio.

The core problem of mean-variance optimization is not with employing an optimizer but choosing the expected return for the stocks to be selected. The optimizer will do a good job of selecting stocks given the expected returns, correlation, and volatility provided. However, if you get the expected return wrong, the optimizer has limited value. Using past returns does not work. Forming some expected value using linear regression may provide some improvement, but the variation explained through most of these models is low. These approaches may have theoretical validity, but their predictive power renders poor results. 

Instead of throwing out optimization completely, machine learning can be employed to improve the expected return predictions. Different machine learning (ML) techniques have been compared with linear regression predictions and it has been found that these ML tools will often beat the traditional finance techniques. By ranking expected return using ML tools, the optimizer can then form portfolios that have a better chance for success.

ML focuses on prediction and optimization techniques focus on building diversification with constraints. The division of tasks allow for specialized skills and allows for a clear breakdown of value-added between tasks. The success of the ML can be measured by accuracy predictions. The success of the optimizer can be measured through portfolio efficiency. The two can be used to build better portfolios.