An operational due diligence conference panel - What would I ask them?

I got an email inviting me to an operational and governance due diligence panel of experts up in Toronto. I will not be able to attend and hear their comments, but I would have liked to ask some questions that have been perplexing me about how to measure the success of due diligence.

Due diligence can be digitized as an either/or question. There are a set of rules or a standard and a firm either meets those minimums or not. The due diligence committee can veto any manager do does not meet a set criteria. More likely, there is a continuum of due diligence quality. Under this environment, the due diligence question is more complex. This complexity can apply even if there is a minimum standard. Some firms are just better than others with respect to operational due diligence and governance.

1. Why do hedge funds close?

a. What percentage of closures is based on operational issues versus performance?

Everyone talks about survivorship bias, but there is not much known about the failures. This is  what due diligence is supposed to eliminate, the failures. If an emerging manager is undercapitalized, it needs performance to improve operational issues. If there is no performance, the operation issues are never solved and the firm never passes due diligence. Similarly, better performing managers can gain capital to improve the business and gather more assets. If this story is correct, then the key issue is performance and the capitalization of the hedge fund. Many operational issues can be solved with good performance.

2. How do you know whether due diligence is doing a good job?

a. How do you measure type I and type II errors for due diligence?

A firm can focus only on large and established firms and never have a due diligence failure. Would that focus on established firms be a successful due diligence process? How do firms measure false positives and false negatives with respect to due diligence? My guess is that most firms do not conduct deep reviews based on some statistical measure.

3. Is the impact of poor operational due diligence measurable on performance?

Clearly, if the firm fails, it will have a measurable impact on performance, but most firm failures occur over time and can be seen. What is more likely is that a firm potentially has inadequate staffing or systems and this leads to lower performance. Can this be measured? For example, what is the value of a third party risk management system on performance? 

4. What has been your greatest due diligence failure?

Should a firm take pride in never having a due diligence failure or does that just mean that the firm is too cautious? The type of failure by a due diligence team is important. Was the failure caused by a change in behavior or was it from something that was missed in the due diligence process? Was it a known risk that was taken or was it a complete surprise? 

5. Does size cure all problems?

Quantitative credit analysis focuses on size. Big firms do not fail as often. Their failure will get more headlines but the likelihood is less. Can a firm save time and effort by just looking at big firms and avoid all new managers.

6. Is transparency the only relevant factor? If a firm will not share information, is it ever a good credit risk?

A first pass for due diligence can be made easy. If they don't answer questions or share all information, pass. This is always an issue with large successful managers who do not want to give full transparency? Secrecy is important but if the manager does not trust the client to keep confidential information secret, then it is the beginning of a bad relationship. Similarly, is there such a thing as too much transparency?

7. How much added return do you need to overlook a governance and operational issue?

There may not exist a perfect firm with respect to operational due diligence. Many firms could improve, so there has to be some added value from firms that are rated lower on some due diligence scale. How much return do you need to move down the due diligence scale?  To put this more concretely, large firms will likely have a better due diligence record. Smaller firms will be weaker. What extra performance is necessary to switch from the larger firm? "Depends" is an answer, but not a good one.

My guess is that the panel will not be able to answer these questions to the satisfaction of the audience. There is no clear solutions to these complex issues. Firms have tried to make due diligence very structured and that has been helpful for the hedge fund industry, but as we have gotten better at doing this analysis we have uncovered bigger questions on the meaning of risk and how it can be measured.