D oc F in: Multimodal Financial Prediction and Bias Mitigation using Semi-structured Documents (2022)

Abstract: Financial prediction is complex due to the stochastic nature of the stock market. Semi-structured financial documents present comprehensive financial data in tabular formats, such as earnings, profit-loss statements, and balance sheets, and can often contain rich technical analysis along with a textual discussion of corporate history, and management analysis, compliance, and risks. Existing research focuses on the textual and audio modalities of financial disclosures from company conference calls to forecast stock volatility and price movement, but ignores the rich tabular data available in financial reports. Moreover, the economic realm is still plagued with a severe under-representation of various communities spanning diverse demographics, gender, and native speakers. In this work, we show that combining tabular data from financial semi-structured documents with text transcripts and audio recordings not only improves stock volatility and price movement prediction by 5-12% but also reduces gender bias caused due to audio-based neural networks by over 30%.

## introduction financial risk modeling is of great interest to capital market participants for making sound investment decisions. earnings calls are quarterly audio conference calls wherein company executives discuss their companies' performance and future prospects with outside analysts and investors @xcite . mergers and acquisitions (m&as) conference calls are held preceding financial transactions involving two or more entities such that either one of the participant companies takes over the other(s) ("acquisition") or combines with another to become a joint entity ("merger") @xcite . both kinds of events consist of a prepared speech delivery by company executives on analysis and future expectations followed by a spontaneous analyst-driven question-answer session to seek additional information @xcite . several past works have utilized the text transcripts and audio recordings from these calls to improve the stock forecasting @xcite @xcite @xcite @xcite . however, most prior works exclusively focus on vocal verbal information, often ignoring information from official financial documents. financial semi-structured documents such as 10-k and 10-q reports are publicly available, recurrent mandatory filings made by public companies to disclose their financial performance. these semistructured financial documents present comprehensive financial data in tabular format, such as earnings, profit and loss statements and balance sheets, and can often contain more than 100's of tables worth technical analysis. information contained in such financial documents also includes a textual discussion of corporate history, management analysis, compliance, risks, and future plans about new projects relevant for investment decision-making @xcite . recent studies such as sawhney et al. (2021a) have highlighted the downside of utilizing audiobased multimodal approaches for financial risk prediction due to the inherent gender bias induced in learning models due to the imbalance of speaker demographics in call recordings . audio features such as speakers' pitch and intensity can vary greatly across genders. under-representation of female executives in conference calls is amplified by deep learning models, leading to high error disparity between stock predictions across sensitive attributes. we combine tabular from financial semistructured documents input with existing vocalverbal information from audio call recordings to improve stock price movement and volatility prediction. we demonstrate that supplementing existing conference calls transcripts with the tabular figure 1 : we combine the text transcripts and audio call recording input to a neural model with tabular text from semi-structured financial documents. here we illustrate how m3a @xcite for volatility and stock price prediction on m&a calls uses dot product attention to extract condensed table representations relevant to text from transcripts, weight averages each modality through softmax layer to obtain a fused embedding, combines the fused embeddings with speaker and positional embedding, and finally passes through the transformer model for stock price movement and volatility prediction tasks. measuring stock volatility : following @xcite , we formulate volatility as a regression task. for a given stock with a close price of p k on the trading day k, we calculate the average log volatility as the natural log of the standard deviation of return prices r in a window of τ days as. price movement prediction : following @xcite , we define price movement y d-τ,d over a period of τ days as a binary classification task. for a given stock, we employ its close price, which can either rise or fall on a day d compared to a previous day dτ , to formulate the classification task as: given a conference call χ, we experiment with several baseline and state-of-the-art multimodal financial prediction models (example m3a(sawhney ## results and discussion effect of tabular datat on financial predictions: ## conclusion and future work in this work, we show that combining tabular data from financial semi-structured documents with text transcripts and audio recordings improves stock volatility and price movement prediction by 5-12% along with reduction in gender bias learned by audio-based neural networks by over 30%. we empirically show that our approach is generic and extensible to recurrent, attention-based and transformer models. future work can utilize advances in document-nlp to extract temporal information extraction @xcite , temporal dependency parsing @xcite , and nli @xcite for better contextual understanding of financial reports. predicting the correct layout can also helps align audio with transcripts @xcite .