How do we detect skill? An ongoing problem in finance

 

The age-old question for any investor - how do you detect skill in a trader or strategy? It is not easy, but you can go back to simple principles of gain and loss. Instead of focusing on the Sharpe ratio, the emphasis may be better placed on the actuarial concepts of frequency and magnitude. A good test of skill is answering two questions: 1. how often does the investment or strategy make money? and 2. how much does it gain (loss) when it is correct (wrong), that is, what is the probability of gain and what is the conditional expected gain or loss? See "Gains and Losses Revisited: Skill Detection and Similarity Assessment" from Sid Browne for a review and extension of these key concepts. 

The proportion of times a gain occurs can be viewed as the hit rate or the batting average for a trader. Of course, an investor wants to have trader working for him who has a high batting average. However, the second component can be considered the slugging percentage which is the ratio of the average gains to loss. There are now two skills that can be measured: 1. the timing skill which is your ability to get bets right and the sizing skill which is your ability to make more on your winning trades over your losing trades. This view of looking at the probability of gain and the gain versus loss is the foundation for key measures like the Kelly criterion. 

A key statistic for assessing trader is the Gain/Loss Ratio or GLR which is a product of two ratio: 1. the conditional expected gain versus conditional expected loss, and 2. the probability of a gain versus the probability of a loss. One ratio is the magnitude of gains versus loses while the other is the likelihood of gains versus loses.  To get a finer edge on the skill of traders, you can look more deeply than just the GLR but at the two components of the gain to loss ratio. 

Different strategies will have different hit rate and slugging percentages. For example, a carry strategy will likely have a higher hit rate but may have a lower profit to loss ratio because there may be small gains but outsized or concentrated losses. A trend strategy may have a lower hit rate but greater gains versus losses as profitable positions are held, and losses are cut.

The nice part of breaking trading skill into these two parts is that traders can be compared between backtested and life periods. Similarly, one trader can be compared to another to measure timing and magnitude skill. There are well formed test statistics for comparison. This approach will provide deeper insight on the type of skill any trader may possess.