Data mining - more than collecting data ore

There are fads and fashions in finance and business. Some statistical techniques come and go in and out of favor based on expectations of success and the realization that some tools just cannot solve certain problems. Data mining in now a hot topic in many areas of business, yet there are not clear definitions of what data mining means, what tools it represents, and how it can solve problems. Clearly, the low cost of computing and storage has made the collection of data much easier, but the problem is not with the data. The issue is how you extract the data "ore", how it is processed, and what you do with it once it is modeled.

I don't have all of the answers and am still learning, but the graph below was helpful at presenting the rich and diverse set of tools that can be applied to large data sets. Data mining can be broken into two parts: one, explaining the past and two, modeling or predicting the future. The explaining of the past can be viewed as a problem of data reduction. How can you take a large data set and categorize it into smaller more meaningful chunks of information. The second part is taking the data and making inferences that can be applied to future behavior. You cannot do the second without extensive work on the first.

The problem with data mining is that with so many tools for explaining the past and predicting the future it is not clear what is the right tool for the right problem. For example, is regression always the best tool or would techniques of classification and clustering better solve the problem. Mining is complex and a first order problem is finding the right tools to match the data.

You can have favorable odds and lose money - show me the sizing

You should expect that if a group of college-aged students in economics and finance and young financial professionals were given a fixed amount of money and told they can flip and bet on a biased coin with a 60% likelihood of heads they should be able to make money. Don't bet on it. "Rational decision-making under uncertainty: Observed betting patterns on a biased coin" by Victor Haghani and Richard Dewey  is one of those simple studies that really make you question the decision-making skills of even financial professionals. Just under 30% of the players in this game went bust and another 5% lost money. Just under 25% got it right in terms of using an optimal betting strategy that maxed wealth.

The game was simple with the subjects given $25 dollars and told they can bet and flip for 30 minutes in a controlled environment to maximize their wealth. Now, given the framework, the optimal betting rule to maximize wealth would be the Kelly criterion which would be a fixed portion of wealth used for each bet, (2*p-1) where p is the probability of success. Even assuming that the subjects did not know the Kelly criterion, a limited proportional scheme might make sense. Well, the researchers found subjects who bet it all, used fixed betting sizes, followed a process of doubling-down, and reversing the bet on heads based on the expectation of a reversal. You name it, and there were some educated 'traders" who tried something that was not optimal. 

The researchers noted that none of the top business schools and most of the subjects have never been told or knew of the optimal betting strategy. That type of decision-making is not taught in finance.

The results really make you think. On the one hand, it makes you almost want to cry knowing that there are so few who may know how to size bets. Alternatively, you want to clap for joy for those who do know how to play the game, for those traders, there is opportunity. You have to say hats off for to all of the systematic traders who have embraced optimal betting strategies to ensure survival and maximize wealth. 

Due diligence of trading skill - Can it be improved?

Being on both side of the table concerning the due diligence of managers, I can argue that there has been a significant improvement with the skill at conducting operational due diligence. Operational risks can be effectively identified and measured. There will be fund failures, but investors can do a good reasonable job of handicapping firm-specific risk. There are checklist and processes that can support the choice of managers. The operational due diligence has been effectively institutionalized.

While there have also been improvements with investment due diligence, there still needs to be better ways of assessing skill than asking the usual set of canned questions about performance, risk management, and portfolio construction.

We have advanced with our ability to decompose performance numbers versus factors, but it is less clear whether we have improved our ability to assess investment skill. Alpha generation can move in cycles and skill may require adaptation. Firms do a good job of tearing apart the numbers verse benchmarks, but our ability to understand how managers cope with change in order to generate future returns is less clear. It is not clear they are as effective at doing a deep dive into the behavior of the manager. Recurring behavior drives skill.

The hedge fund investment due diligence can be divided into two parts: one, determining whether there has been measurable skill and two, determining whether the skill is repeatable. Investors have made strong advancements with the measuring skill through past performance. Investors have factor analysis, alternative measures of different risk premiums or betas, as well as peer group benchmarks to measure skill ex post. We can slice a portfolio any number of ways to determine the risk of the manager and find their ability to generate alpha from their past performance.

In fact, we can see performance breakdown in four dimensions. There is peer analysis through a number of hedge fund peer indices. This provides good relative value performance measures. There is asset class factor analysis where we can determine what are a fund's risks based on exposures to asset classes. Factor analysis can be used to analyze a broader set of risks. For example, a classic factor set would be the Fama-French three or four factor set. From these performance measure sets, we can isolate the alpha for a manager which will be the constant term from a regression equations. We have moved well beyond a measure of alpha relative only to the market portfolio, yet the amount of variation explained by these models is still relatively low. So much of skill cannot be measured through systematic means.

The due diligence problem is extrapolating this performance breakdown of risk into the future. Our ability to predict future returns of managers given their risk breakdown is at best, less precise. In a world where hedge funds can dynamically adjust their risks, past data may not tell us about the risks in the future.

Yet, there is help with measuring trading skill through focusing on three key issues. A primary focus should be on case-based reasoning and story-telling. Can managers explain how they made money in certain situations? What are the stories for how gains were made? Similarly, cases can also describe failure. This will include what a model may have been telling the manager and why it did not work. It may include how and why position were exited. We separate the description of case situations with the overall story-telling of the manager’s edge and how he views the market and the process for generating returns. The narrative is important for explaining how a manager makes money and why he should make money in specific cases.

As important is a focus on adaptability. How have managers learned and acted to improve their performance? The ability to admit mistakes and adjust is the most important skill for managers who have long track records that show periods of underperformance. There will be loses. The question is how managers deal with these underperformance periods and learn to change their thinking in both the short and long-run. 

Due diligence of skill should be systematic so a framework for manager skill discussions is as repeatable and comparable as operational reviews.