Evaluating Topic Model on Asymmetric and Multi-Domain Financial Corpus (2024)

Abstract: Multiple recent research works in Finance try to quantify the exposure of market assets to various risks from text and how assets react if the risk materialize itself. We consider risk sections from french Financial Corporate Annual Reports, which are regulated documents with a mandatory section containing important risks the company is facing, to extract an accurate risk profile and exposure of companies. We identify multiple pitfalls of topic models when applied to corporate filing financial domain data for unsupervised risk distribution extraction which has not yet been studied on this domain. We propose two new metrics to evaluate the behavior of different types of topic models with respect to pitfalls previously mentioned about document risk distribution extraction. Our evaluation will focus on three aspects: regularizations, down-sampling and data augmentation. In our experiments, we found that classic Topic Models require down-sampling to obtain unbiased risks, while Topic Models using metadata and in-domain pre-trained word-embeddings partially correct the coherence imbalance per subdomain and remove sector’s specific language from the detected themes. We then demonstrate the relevance and usefulness of the extracted information with visualizations that help to understand the content of such corpus and its evolution along the years.

## esma guidelines on risk factors disclosure to extract risk distributions, we identified three main approaches. campbell et al. rely on a handmade dictionary and iterate on its corpus to obtain hundreds of terms related to 4 different risks. @xcite proposed a supervised approach with a set of 25 different risks and obtained good performances with a multi-label categorical k-nearest neighbor. exposures defined in these documents rely on complex ontologies with interrelations between multiple types of risks, classes we could imagine are therefore overlapping depending on the granularity level at which we try to extract risk descriptions. time is also an important factor, some risks being ignored before an exogenous shock as we've seen with covid-19 and the pandemic risk @xcite . because of those characteristics, creating a classification oriented annotated dataset or a dictionary based on pre-defined labels would have a limited interest overtime and would be restricted to the original representation granularity. in order to correct such pitfall, bao and datta and israelsen proposed to use unsupervised topic modeling such as latent dirichlet allocation (lda) @xcite to discover risk distributions from unstructured texts. this type of modeling is still widely used in nlp applied to risk analysis in finance @xcite @xcite . @xcite are the first to incorporate metadata in their model with the structural topic model (stm) @xcite to correct idiosyncratic terminology within industry, unfortunately they didn't evaluate to which extent this added covariate results in more reliable distributions of risks. also, their stm is an entirely probabilistic model and does not incorporate prior knowledge such as the one found in embeddings, which are known to be helpful to obtain more coherent and diverse topics. at the best of our knowledge, there exist no opensource annotated dataset nor model that can be used for the risk analysis in the french language, increasing the need for unsupervised approaches. bayesian graphical probabilistic models and matrix reduction techniques were the cornerstone with the well known latent dirichlet allocation (lda) @xcite and non-negative matrix factorization (nmf) @xcite . variants have been proposed, such as supervised topic model @xcite , labeledlda @xcite , labeled medlda @xcite and partially labeled topic models @xcite and have been applied to supervised tasks in specific domains @xcite ). other variants are meta-data based topic models which include features either to parametrize the topic distribution @xcite or to add supplementary feature specific topics @xcite . the first one was introduced by adding feature information to condition the topic distribution depending on the observed features @xcite and recently upgraded to allow unstructured features such as images as conditions @xcite . the second was proposed by @xcite to separate topics from sentiments in webblogs and upgraded by @xcite . similar systems are called facets topic models. since 2017, variational inference has been widely used in topic modeling since it releases the constraint to build inference function for each model and let the neural network approximate the marginal distribution of latent variables. this architecture is far more flexible than probabilistic graphical topic models such as lda and originate from the autoencoding variational bayes (aevb) @xcite and upgraded into prodlda by @xcite . in 2018 @xcite published the scholar model, which incorporates a facet topic model variant, interactions between covariates and topics, or supervision. since skip-gram @xcite and even more since bert @xcite , nlp has been in a shift towards vector-based representations, which is quite distant from the bag-of-words usual representation for topic models. since then, authors have been incorporating embeddings into topic models @xcite or into clustering models with relative success @xcite . neural topic models are highly flexible, we advise the reader to look at the following surveys : @xcite as large language models took the lead on the last few years, we expect such models to be able to perform topic modeling. at the best of our knowledge llms have barely been experimented in such a way @xcite , therefore we do not include them in our study. although imbalance is a well known impairment for supervised and unsupervised learning, it has not been extensively studied in the topic modeling framework, especially in a multi-domain corpus. such work on topic imbalance was initialized by @xcite who introduced an asymmetric prior on topic probabilities and demonstrated an overall increase in lda performances. in a recent work @xcite found out that topic capacities (tc), defined as the overall presence of each discovered topic in a corpus, are often close to each others in matrix factorization based probabilistic topic models. they proposed to increase the imbalance degree using a specific regularizer in the additive regularized topic model (artm) framework while @xcite proposed a causal inference approach to increase the identification of rare risks.