Give me some metrics! Beware of those metrics

I have been one of those "if you cannot measure it, you cannot manage it" types. Most quants are like this, but there are limits to measurement or trying to fit qualitative characteristics into a measure. Difficulty does not mean that attempts should not be tried. Most will admit that measurement is very useful in some areas of study but have to be more tempered in others. Jerry Muller, in his book The Tyranny of Metrics, explains the potential problems of a culture that focuses too deeply on measurement. This is a good counter to the always measured to manage crowd.

Muller provides a breakdown of the recurring flaws of metrics. These flaws fall into two areas, the distortion of information collected by the analyst and the gaming of information by the persons being measured:

• The distortion of information problem -
• Measuring the most easily measurable 
• Measuring the simple when the desired outcome is complex
• Measuring inputs rather than outcomes 
• Degrading information quality through standardization 
• The gaming of information problem -
• Gaming through creaming
• Improving numbers by lowering standards 
• Improving numbers through omission or distortion of data 
• Cheating 

The author describes measurement and gaming problems with colleges, schools, medicine, policing, the military, business, finance, philanthropy, and foreign aid. It is everywhere, so every analyst should appreciate the problem of measurement. You will be gamed. Measurement changes incentives. You will measure the wrong things. Use your numbers, but beware of your numbers.