Confusion matrix and forecasting

The core for performance forecasting is getting the forecasts right. It not about the model albeit the model should have a strong economic foundation. However, getting it right is not always simple, nor can it be assessed through a single number. The sense of being right should also account for the false positives and false negatives, type 1 and type 2 errors. It should account for such numbers as the likelihood of success as well as accuracy. 

The best approach is to use the confusion or error matrix, a 2x2 contingency table that compares forecasts with outcomes. Analysis of errors is critical to good decision making, and if forecasts can be measured and success and failure counted, a number of measures can be used to determine the quality of forecasts. Below is the set of statistics that can be generated from the 2x2 matrix.

A simple tool like the confusion matrix is often overlooked yet for any quant trading firm engaged in forecasting, the ability to get the direction right may be more important than the size of the move. Of course, getting a few trades wildly right is more important than just having the odds of success correct. Knowing how to size bets is critical, but at the forecast stage, the confusion matrix is critical.