The danger in in the covariance matrix

 

While most investors focus on volatility, the covariance matrix can significantly affect performance and is hard to measure. A large covariance matrix with N assets will have N(N+1)/2 free parameters and T degrees of freedom based on the number of observations. A large portfolio will be hard to calculate and difficult to use out of sample. 

Work has also been done on shrinkage estimates, which suggests that the raw sample covariance should contribute only about 20% to the new covariance matrix. Linear shrinkage does better than nonlinear out-of-sample estimates. Overall, shrinking toward zero correlation will help long-short portfolios because they rely less on extreme values. 

The problem is that, in optimization, weights depend on risk aversion, the inverse of the covariance matrix (the precision matrix), and expected returns. Estimation errors are amplified during inversion. A 10% mistake in the covariance matrix will cause the precision matrix to take much larger values, which can be catastrophic. 

Additionally, illiquidity can distort covariance in hidden ways. Illiquidity creates positive autocorrelation in the return series, so measured volatility is lower than true volatility and measured covariance is lower than true covariance. 

These issues are one reason funds should focus on covariance across the environment and try to adjust for relative volatility and risk contribution. It is not just the overall volatility that is an issue but the link across markets which defines diversification and the risk hidden within portfolios. 

What can LLMs do and more importantly, not do?

 

A very interesting position paper worth a read for any follower of AI: “LLMs Can't Jump: Why the Abductive Leap is the Final Frontier of AI Discovery.” The thought is simple. AI may be good at some forms of inference, like deduction and induction, but it is not able to do the third type: abduction. “While AI can compress data (Induction) and prove theorems (Deduction), it cannot yet recreate the intuitive leap Albert Einstein took to formulate the axioms of General Relativity—a process rooted in embodied physical simulation rather than symbolic manipulation.”

The Three Pillars of Inference

The paper adopts the framework of Charles Sanders Peirce to categorize AI’s missing link:

• Deduction (Rule + Case → Result): Applying a known law to a specific situation. (AI status: SOTA - Achieved)
• Induction (Case + Result → Rule): Spotting patterns in data to find a general rule. (AI status: SOTA - Achieved)
• Abduction (Rule + Result → Case/New Rule): Inventing a hypothesis to explain a surprising or singular phenomenon. (AI status: The Missing Jump)

There is still room for importing thinking by analysts. The market has to understand which skills are important for analysts and what can be done by quants.  

Momentum and network learning

The paper “Network Momentum across Asset Classes” introduces a multi-asset quantitative trading strategy based on network momentum—a trading signal derived from momentum spillover across different asset classes. Momentum spillover occurs when past performance in one asset predicts future returns in linked assets. While traditionally studied in pairwise connections (e.g., stock-bond pairs or supply chains), this paper expands the concept to a systemic network across four distinct asset classes.

The study analyzes daily price data from 1990 to 2022 across 64 continuous futures contracts covering Commodities, Equities, Fixed Income (FI), and Foreign Exchange (FX).

Methodology

Graph Learning - Uses a linear and interpretable graph learning model to infer dynamic asset networks strictly from historical price data, overcoming the lack of explicit fundamental links between disparate asset classes.

Momentum Features - Leverages 8 individual momentum features per asset, including volatility-scaled returns over multiple horizons (1-day to 1-year) and normalized Moving Average Convergence Divergence (MACD) indicators.

Portfolio Construction - Combines learned graph adjacency matrices with a linear regression model to forecast 1-day volatility-scaled returns across all assets.

Out-of-sample backtesting from 2000 to 2022 yields an annualized return of 22% and a Sharpe ratio of 1.51 after volatility scaling. The strategy exhibits low downside risk and low correlation with traditional individual momentum strategies. Inter-class (cross-asset) connections significantly enhance return predictability and diversification.

Causal inference and critical statistical thinking

Causal inference is one of the most important topics in finance today. There is a difference between what correlates with or is associated with X and Y and saying that X causes Y. We can thank the work of Judea Pearl for truly focusing our attention on causality rather than correlation. 

You should not ask what tends to happen to Y when X is high. Of course, you can ask, but that only refers to the association. The real question for causality is, "What will happen to Y if we set X to a specific vlaue and all other factros are held constant?". To answer that question, we have to consider the relationship between X and Y, and also ask what other factors may influence Y, such as variable Z. Does Z cause X, which then affects Y? Does Z affect Y directly? This type of thinking is not about fitting a set of past data into a relational model, but about asking the primary question of whether there is a reasonable link between these variables. 

Before you run a statistical test, think about causal relationships and how they may be linked together. What type of relationship are you trying to find? 

Multi-agent LLM systems for profit

 

LLM can be used to develop autonomous trading systems. The key is to have agents mimic the workflow within a multi-agent trading framework that divides a trading system’s tasks. When you look at a deep decomposition of the tasks associated with a trading system, you will get better risk-adjusted returns. In the paper, “Toward Expert Investment Teams: A Multi-agent LLM System with Fine-Grained Tradign Tasks”, the researchers develop what they call a fine-grained system that looks at detailed tasks and instructions and compares it with a coarse-grained LLM system that does not provide detailed instructions to the agents. 

Across different stock portfolios, the fine-grained approach performs much better. It is notable that we can examine the effects of dropping specific agent tasks and assess their impact on the Sharpe ratio.

The results are clear - be detailed with the LLM tasks given to agents. The more specific you get, the better the results.

Do LLMs make market more efficient? Yes

 

We have seen an explosion of new papers on how to use LLM in finance. Clearly, computing power can enable analysts to use vast amounts of data more effectively across companies. Many papers have shown that excess returns can be generated by using LLMs, but a more fundamental question is whether markets have become more efficient since the introduction of LLMs. This is the focus of the new paper “Do LLMs make markets more efficient?". 

The answer to the question of efficiency is yes. LLM availability eliminates 40 - 60% of post-new drift into the next trading day. LLM helps with short-horizon price discovery. The researcher used a novel way of determining this value-added by examining the effect of plausible exogenous outages of major LLM providers. When LLM providers are unavailable, there is more drift. 

Get on the LLM bandwagon, even though markets have already become more efficient with the use of this tool.  If you are not usng this tool, you are falling behind competitors.

Is AI a stochastic parrot?

There’s a well-known phrase that AI is not true intelligence, but just a “stochastic parrot”. But human beings do quite a bit of parroting in conversation, as it turns out. So maybe the comparison is a little more complicated.

Six Questions on the Value of Humanities Research

A Podcast Interview with Chris Yeomans, Justin S. Morrill Dean of Liberal Arts, Purdue University

I found this phrase thought-provoking. We all parrot what we hear from others. Perhaps most of our conversations are parroting what we may have learned from someone else. AI is just a sophisticated parrot because it can search through more work than we can. AI, like us, does this with noise. It is a stochastic process. 

So what makes us or AI more than a parrot? The difference is what we can call creativity. It is rephrasing or making extensions or narratives that are not just a repeat of what we have heard before. The reordering of fact. The prioritizing of data is a part of the creative process. This is done by AI, so it is more than just parroting with noise. However, we should be aware that AI is creating something new through this reordering. If we ask the questions differently or use a different algorithm, we will create something different.

More on the Sharpe ratio - Look for its stability

 

A recent paper introduces a new concept to help investors assess managers and trading strategies: the Sharpe Stability Ratio (SSR). See “The Sharpe Stability Ratio:Temporal Consistency of Risk-Adjusted Performance”. This performance metric accounts for the temporal consistency of risk-adjusted returns. An investor, if given two Sharpe ratios with the same value, should choose the one that has more stable characteristics. You should like the persistent Sharpe ratio. This paper treats the Sharpe ratio as a rolling performance measure. It defines stability as the ratio of the mean rolling performance to the heteroskedasticity- and autocorrelation-consistent (HAC) standard deviation. 

Using the time-series approach can help analyze point-in-time SR or the probabilistic Sharpe ratio (PSR). Given strong serial correlation in the Sharpe ratio, arising from the rolling average and return consistency, the HAC correction provides a better measure than simply scaling by the standard deviation.

The important issue for investors is to look at persistence and consistency with strategies. This may be the true hallmark of skill.

Think of global equity markets as a network

 We have been spending more time thinking about markets as networks or clusters. Don’t think about asset returns in isolation, but through the connections across markets and regions. Some of the latest work in this is presented in the paper, “Clustred Network Connectedness: A New Measurment Framework with Applicaitons to Global Eqiuty Markets” by Buchwalter, Diebold, and Tilmaz. 

These authors have been working on the network process for asset returns through variance decomposition of VAR models. From these models, the authors have been able to distinguish causality from and to markets across a wide set of markets. Their latest work on global equity markets seeks to address econometric issues arising from the decomposition method. This process of decomposition will provide a different narrative but will also answer questions about whether there is contagion or just co-movement across the network. The graph above shows the traditional method for forming the clustered identification. The graph below looks at the same data, accounting for groupings within the network after accounting for generalized identification.

Note that in the clustered identification, the US equity market serve as the center of netwrok behavior while the generalized idienificaiton which accoutns for correlation within groupinsg of the 16 equity markets studied, shows the high connection that is the focus of the EU cluster. Both provide interesting interpretations for how equity markets are connected. 

 

What drives performance of machine learning

 

A recent paper, "What drives the performance of machine learning factor strategies?" seeks to disentangle two key ingredients in modeling: expanding the dataset and allowing for flexible functional forms. Now, as expected, as you move closer to a realistic setting, you find that the value of both deteriorates. However, this research finds that the value of an expanded dataset is more persistent than the functional form employed. While many may think that machine learning is a form of holy grail for investing, the reality is that real-world constraints and transaction costs are key drivers of performance. Reality indicates that adding nonlinear complexity does not add value, whereas being non-sparse is beneficial. 

Deep learning and asset management - it is here but requires significant work

The use of deep learning techniques has exploded in finance, but few papers summarize what has been achieved. That has changed with a new paper,  see Deep Learning in Asset Management: Architectures, Applications, and Challenges. This work provides a good survey of how to apply deep learning within asset management. While the authors suggest that the use of deep learning is promising, they also note significant challenges, including low signal-to-noise ratios, non-stationarity in financial time series, and the market's adaptive nature, which can face regime changes and adapt to patterns. Along with outlining the problems with CNNs, RNNs, and transformers, there are many practical challenges in using deep learning models, from data usage to the usual overfitting problems. While deep learning has focused on quality predictions, there may be useful ways to integrate deep learning into holistic approaches to asset management 

Chaos and machine learning

 

"Chaos theory -the qualitative study of unstable aperiodic behavior in deterministic nonlinear dynamical systems" - Stephen Kellert 

That definition of chaos theory is all-inclusive, yet in a world before ML, it would be hard to model using simple linear regression techniques. ML is helpful because it can address the characteristics of chaotic systems. It also helps define the type of ML necessary to employ when faced with a chaotic system. Foremost, ML learning can address nonlinear relationships. All neural network ML can address nonlinear relationships. ML can also work with dynamic systems that have strong cross-asset relationships. ML can also address aperiodic behavior by examining time-series relationships using techniques such as long short-term memory (LSTM) recurrent neural networks (RNNs). What takes more work is dealing with unstable systems that change over time. This requires ML models that are compact and can be retrained regularly. 

The AI model used differs by task

 

I found this simple graph that matches how I use AI models. There is not one single provider that works best. The choice of AI model will differ by the task. Given these differences, I will sometimes use more than one model for he same task by asking the same questions or using the same prompts across several models and then comparing the results. Claude is easy to work with, but ChatGPT will sometimes provide more useful answers. Gemini and Co-Pilot are easy to use because of their integration with workflow, although their answers for more complex questions are not as clear.

While good at answering quick questions, their usage is harder to intergrate with normal quant work.

CME Outage, Quant Models, and Prices

 

Prices are the lifeblood of any quant model. If you have the wrong input, you will get incorrect predictions. In the case of an outlier, a single incorrect price may generate misleading signals about future opportunities, so any model should be thoroughly reviewed to assess its sensitivity to wayward signals. 

If there are incorrect inputs, the model output should be adjusted to reflect the change in data when a replacement is made. Users have to be alerted to any changes. These are easy cases to deal with. There are also more difficult issues, such as data oddities or anomalies. 

For example, the CME outage during the Thanksgiving period is a market anomaly that has to be addressed, especially given that it occurred at month-end. To provide context, there was an 11-hour system outage ending at 1335 GMT. It was during the Thanksgiving break, which is associated with low trading volume in the US, but it impacted all global markets on a Friday, a month-end. An outage will lead to a change in trading and a surge upon reopening. Hence, the inputs for open, high, low, and close will be distorted from what they would be in the absence of an outage. This will lead to slightly different signals generated from any model. 

So what should a modeler do about this? One response is that the price is the price and to do nothing. It is reasonable and defensible, yet it may seem odd not to account for some distortion; however, there is no way to determine the impact of any outage. What would be the right price? Another option is to drop the price to the last close. This can be defended, but replacing data seems somewhat arbitrary. 

The best response is to focus on the output and look at the marginal trade signals generated. Does it matter? An output sensitivity analysis can be conducted to see what happens with the new prices, rather than looking at what would have happened if no change had been made. If there are small marginal changes, then keep the latest prices. If there is a large set of new signals, investigate further on why and flag the changes. The prices can be kept, but the flagged trades can be ignored. However, this creates another set of problems if the trade is closing an existing position. When do you exit the old position?

These real-life problems tell the user that there is no such thing as a fully automated system.

Beware of the Sharpe ratio - Use the Sharpe ratio

 

The new paper "How to use the Sharpe Ratio" is an important read for any analyst looking to compare two managers or strategies using the Sharpe ratio. Many of the implications of the Sharpe ratio have been discussed in previous papers; however, this paper presents all the limitations and possible solutions in a single reading. The authors present several adjustments to the Sharpe ratio to address issues such as non-normality, and they also list and comment on all prior research associated with the Sharpe ratio. The overall conclusion is that using quick calculations should be done with peril. Oftentimes, minor adjustments will help on the margin. This is important because often Sharpe ratios for many strategies are close to each other, so rankings will flip once you account for some of these adjustments

The cause and effect link is not always obvious

This comic addresses many economic problems. There is no reason to think about unintended consequences or second-order effects, but you don't get the cause and effect right. 

Currently, there is an increased focus on cause-and-effect thinking within finance, as a result of the proliferation of factors used to explain returns. Some have stated that there are hundreds of factors that may drive returns, yet they often do not work outside of the training period, or they seem to periodically move in and out of favor. The problem is that there is insufficient consideration of the relationship between cause and effect. Factors are found and stories are developed to explain why a factor may be relevant, but that is not the same as positing a theory and then testing it. Correlation with a story is not causal.

Choose your modeling technique wisely

 

One of the key challenges for any quantitative modeler is selecting the appropriate modeling technique. This involves not selecting the features or predictors to be included in the model, but rather the statistical techniques that will generate the best predictions. Not all modeling approaches are the same, so knowing when to use the best techniques is an important skill. The paper "From OLS to neural nets: Comparing regression models in systematic trading" is not the final word on this question. Still, it does a good job of making comparisons across techniques. It does not address all the machine learning techniques, yet it does compare many of the simple techniques that most researchers may employ. 

The tables from the paper are nice to have handy for anyone starting out when thinking about modeling.

Causal discovery and trading

 

Causal discovery techniques can help any quantitative hedge fund, but may be especially helpful for enhancements to trend-following through finding causal links with other markets. The basic structure for a trend-following model is to use past values of a variable to extrapolate ot the future. Look for the trend, yet it would add significant value if you could learn whether other markets may have some causal impact on another variable. 

The standard approach to time series causality is to use Granger causality tests, which simply determine whether some time series Y causes or has an impact on the prediction of X. However, a growing number of alternative techniques are available to aid in causal discovery, thereby improving trading, such as time series data causal inference, vector autoregressive linear non-Gaussian acyclic models, and time-varying interactions models for nonlinear observations. The code for these algorithms is already written, so it is relatively easy to implement for a set of assets.

We are not planning to explore all of these techniques, but there are ways to support better causal discovery that can be used to improve the inputs in investment strategy. See "Trading with Time Series Causal Discovery: An Empirical Study" for a simple application of causal discovery for long-short equity portfolios. Now, these algorithms are not easy to implement due to the time required for computation; however, this seems to be a fruitful area for further research, especially given the growing interest in causal reasoning in finance.

Choose your correlation carefully - Kendall's Tau

Portfolio construction is fundamentally based on the correlation between assets. The lack of correlation creates diversification, yet limited work has been conducted in testing alternative forms of correlation. Most construction work is based on Pearson correlation, which looks at linear dependence across assets. There are limitations with Pearson correlation, so sometimes an alternative is used, Spearman's rho correlation, which is based on rank ordering. Spearman's correlation can adjust for non-normality and outliers, but there is a problem with the assumption of monotonicity. The third alternative, Kendall's Tau is based on measuring the concordance between asset pairs through counting the sign of movements across assets. 

A relatively simple paper looks at the portfolio construction of daily foreign exchange pairs using all of the same parameters except for the correlation matrix. See "Beyond Correlation: Enhancing currency portfolio construction through Kendall's Tau and Correspondence Analysis". I was surprised by the results. Yes, volatility is higher, but the overall portfolio performs better. There are clear benefits from using Kendall's Tau. The numbers are compelling enough to ask a simple question: "Why not try this alternative?"

 

The stock market - It is not your normal distribution

The stock market is not your normal distribution, yet we often focus on normal return stats. For risk management, assume normality at your own peril. Market extremes usually come in clusters when there is a regime switch in volatility, which does not change the fact that outliers exist. You can windsorize to take out extremes; however, that does not change the risk. Yes, April has been an unusual month.