Semantic Similarity Covariance Matrix Shrinkage (2023)

Abstract: An accurate estimation of the covariance matrix is a critical component of many applications in finance, including portfolio optimization. The sample covariance suffers from the curse of dimensionality when the number of observations is in the same order or lower than the number of variables. This tends to be the case in portfolio optimization, where a portfolio manager can choose between thousands of stocks using historical daily returns to guide their investment decisions. To address this issue, past works proposed linear covariance shrinkage to regularize the estimated matrix. While effective, the proposed methods relied solely on historical price data and thus ignored company fundamental data. In this work, we propose to utilise semantic similarity derived from textual descriptions or knowledge graphs to improve the covariance estimation. Rather than using the semantic similarity directly as a biased estimator to the covariance, we employ it as a shrinkage target. The resulting covariance estimators leverage both semantic similarity and recent price history, and can be readily adapted to a broad range of financial securities. The effectiveness of the approach is demonstrated for a period including diverse market conditions and compared with the covariance shrinkage prior art.

## introduction a wide range of domains rely on the analysis of high-dimensional time-series data. for example, this includes medical systems with mri denoising @xcite , radar sensors @xcite post-processing, physics and chemistry, engineering, neuroscience, speech recognition, and quantitative finance @xcite . these applications often require a reliable estimate of the covariance matrix. in the financial domain, the price of a company's stock is influenced, among other factors, by the fundamental characteristics of the company, news articles, and figure 1 : instead of using the semantic similarity as a direct approximation to the covariance matrix, we use it to build a target for linear shrinkage. this estimator can then be used for convex portfolio optimization. the volatility and the expected return of an asset are both critical ingredients when forming an investment strategy. risk-return trade-off is a fundamental trading principle describing the typical inverse relationship between a given investment risk and investment return. in the common case of a multi-asset portfolio, an investor with a given risk tolerance seeks to maximise their return within the allowed risk level. in this case, the degree of covariance between asset returns can be utilised to achieve an aggregate portfolio risk that is lower than its components. modern portfolio theory formalises this trade-off @xcite . the mean variance portfolio aims at maximizing expected returns while minimizing expected variance: existing approaches leveraging semantic stock representations have focused on improving the estimate of the expected returns µ, using a rough approximation of the covariance matrix σ by directly using the semantic similarity matrix as an estimator for portfolio optimization @xcite . however, these similarity matrix estimators may not be morphologically similar to a covariance matrix and are prone to bias. indeed, while semantic models are effective at ranking similarities, their absolute similarity value is typically uncalibrated and depend on the fraction of the hypersphere occupied by the embeddings. furthermore, their expected cosine similarity value depends on the dimension of the embeddings @xcite . since the true covariance matrix is obviously independent of the chosen embedding dimension, this would imply that a single embedding dimension is valid for building a semantic estimator, which is unacceptable. finally, they give no guarantee of being positive semi-definite and ignore recent stock price observations. an alternative estimator for the covariance matrix is the sample covariance matrix. while unbiased, the sample covariance matrix is singular when the number of observations n is lower than the number of random variables p (preventing the use of convex solvers for equation 1) or ill-defined when n ̸ ≫ p, making the portfolio optimization task inaccurate @xcite . covariance shrinkage refers to a family of regularization methods aiming at combining the maximum likelihood sample covariance matrix with a structured, lower-dimensionality, regularizing model (shrinkage target). the regularizing targets are built on a set of assumptions (e.g., constant variance and 0-covariance between variables) whose degree of validity varies between asset classes and market regimes. furthermore, the shrinkage estimators can be interpreted as empirical bayes estimators where the prior is data-dependent, and computed from the same data as the sample covariance matrix @xcite . in this work, we highlight the limitations of using semantic similarity as a direct estimation of the covariance matrix. we address these limitations by extending the established framework of linear shrinkage to semantic similarity matrices as seen in figure 1 . the semantic representations can be derived from both structured data stored in a curated knowledge base or unstructured data in the form of natural language text. we share a python implementation foot_0 of our method, allowing its integration as a simple post-processing step when building covariance matrix estimators from semantic similarity matrices. we aim at validating the following: ## related work portfolio optimization using semantic representations: du and tanaka-ishii (2020) investigate the potential of using news data to train a classifier to make buy and sell decisions over time. they take the cosine similarity matrix of intermediate stock representations as a direct estimator for the covariance matrix to perform portfolio optimization. similarly, @xcite use normalized stock embeddings trained on fundamental and price data to build a similarity matrix and use it as an estimate of the covariance matrix for mean-variance portfolio optimization. both approaches suffer from the assumption that the similarity matrix is an unbiased estimator of the covariance. our work instead uses semantic similarity as a shrinkage target to the sample covariance matrix resulting in estimators that are asymptotically unbiased. @xcite leverage a knowledge graph (kg) to model the correlation between stocks through temporal hyperbolic graph learning on riemannian manifolds. they train a model to directly output a ranked list of trade candidates, and therefore do not rely on the modern portfolio theory mean-variance optimization problem. our approach instead leverages established portfolio optimization routines allowing better interpretability and modularity (decoupling the semantic representations generation from the investment strategy). covariance shrinkage: linear shrinkage methods, taking a convex linear combination between the sample covariance matrix and a regularization target were originally proposed by @xcite for the estimation of high dimensional multivariate mean, choosing the 0-vector as a shrinkage target. @xcite built upon this approach by choosing a 1-parameter structured target built as the identity vector multiplied by the sample mean. this work was extended to a robust estimation of the covariance matrix in a high dimensional space by @xcite , proposing a singleparameter shrinkage target built as the scalar product of the identity matrix and the average sample variance. their choice of shrinkage target made the strong assumption that the stock price returns had a constant, common variance and no covariance. @xcite instead assumed a constant and positive correlation between stock price returns, resulting in a 2-parameters shrinkage target. the single-factor model @xcite ) assumes the asset prices can be approximated using the capital asset pricing model (capm) and linearly depend on the average market returns with sensitivity β i and an offset α i capturing the stock idiosyncratic returns. the resulting shrinkage target contains p parameters to estimate, higher than the previous two models, but typically significantly lower than the p(p -1)/2 parameters of the full (symmetric) covariance matrix. these regularization targets are estimated using the same data used to compute the sample covariance matrix and are therefore prone to estimation error if the number of samples is low relative to their number of parameters. our approach instead utilises similarity-based targets that are not derived from recent observations but rather trained on large scale semantic datasets. these therefore do not suffer from this parametrization trade-off and are less sensitive to market regime changes. ## preliminaries in this section, we formally describe the framework of linear shrinkage and derive the optimum shrink-age (regularization) factor coefficient. let x n be a de-meaned price returns matrix of n observations for p assets (we omit the p for notation simplicity). the sample covariance matrix is a maximum likelihood estimator (mle) of the covariance and is defined as: the sample covariance matrix is symmetric and has p(p-1) 2 parameters to estimate. when the number of observations n is lower than the number of features p, this matrix is non-invertible. furthermore, unless p ≪ n , it is numerically ill-conditioned and the instabilities in the estimate lead to the underestimation of the smallest eigenvalues and overestimation of the largest ones @xcite ). an alternative is to replace this mle by a lower capacity model making a series of assumptions on the random variables observed (for example constant variance, no correlation or a factor model as introduced previously). this option is also unsatisfactory as it would suffer from a high bias because of the strong modeling assumptions implied. the approach of linear shrinkage is to build a new estimator from a convex linear combination of (1) the high variance, low bias of the sample covariance matrix; and (2) the low variance, high bias of the shrinkage target (lower capacity model) t : the objective is to minimize the mean square error between this estimator and the population covariance σ * . the loss function is therefore given by the population covariance is unknown and not observable but @xcite show that the optimum λ * coefficient is asymptotically con-stant over n and has the following form: ## semantic similarity shrinkage our proposed approach is based on the availability of embeddings (vector representation) for the random variables considered. in the context of financial markets, this translates into the availability of a (p, k) embeddings matrix e built from k-dimensional semantic vectors for the p financial assets considered for the portfolio optimization. these embeddings may be generated using text description of the assets (e.g., description of the company that issued a stock) and a sentence embeddings model such as @xcite , kg embeddings, or intermediate representation generated from a downstream task such as a buy/sell classifier @xcite . we first propose a semantic shrinkage target built using these embeddings and then derive the optimum shrinkage factor. ## discussion in this work, we present a family of covariance shrinkage estimators leveraging semantic similarity between the random variables. we show that semantic similarity cannot be used as a direct estimator of the covariance, but instead as regularization for the sample covariance using linear shrinkage. the shrunk estimators include priceindependent semantics and are superior in periods of high volatility, where an accurate estimation of the covariance is typically more critical. this demonstrates the potential of semantic models for covariance matrix estimation and extends beyond traditional applications to predicting returns. the proposed method is agnostic to the embeddings generation process and can be applied to most existing semantic representation algorithms. this includes semantic representations that capture static (company description) and dynamic (company news, recent price data) components from text, knowledge graphs or multi-modal datasets. by decoupling the semantic model from the portfolio optimization, the proposed method provides a robust and modular mechanism for integrating semantic representations into established portfolio management and investment decision systems. ## limitations while showing encouraging results, the proposed similarity-based targets are built from static semantic information. a potential future work direction could be the extension of these static embeddings with dynamic components, including for example news data or dynamic knowledge graphs. text embeddings from news data (capturing company recent events) can be combined with the static embeddings (capturing company fundamentals) to compute either a single similarity matrix target for shrinkage, or use techniques such as multiple targets shrinkage. building semantic representations from news data will involve challenges related to sampling for companies with high news coverage, the lack of news for some companies and the choice of importance given to the static versus dynamic components. the input to the semantic models (text and kg triples) was gathered as of 2021 and 2022, potentially leading to data leakage for the 2007-2023 period backtest. future work may use historical wikipedia snapshots to measure the impact of this assumption. however, we estimate this risk to be low given the focus on company fundamentals. the price data frequency is typically much higher than frequency of change of the fundamental data (description or kg triples): the company characteristics (e.g., industry sector) of the large capitalization investigated did not change significantly over the period considered. furthermore, the analysis does not show a drop in performance before and after the datasets snapshot date. future work including dynamic semantic components (e.g., news) should however ensure that only news released prior to an experiment time window are used, since news do have a significant impact on short term price data (and correlation) as illustrated by recent work referenced in our work. similarly, the rolling window evaluation excludes companies that stopped trading during the period considered: this inclusion/exclusion process is a-priori unknown. the approach requires both rich semantic representations and an effective way to match the stock price returns to their embeddings. the matching process may be more challenging for international companies or smaller capitalization stocks than for the s&p 500, nasdaq-100 or nikkei 225 components. finally, while the approach could be extended to any financial asset class including commodities or derivatives, obtaining a valid text description may prove more challenging. note that the knowledge-graph embeddings do not suffer from this limitation and can generalize to any asset class where structured data is available, and future work may investigate this method potential for multi-asset portfolio (i.e., equities, fixed income and commodities) covariance estimation.