An Empirical Investigation of Bias in the Multimodal Analysis of Financial Earnings Calls (2021)

Abstract: Volatility prediction is complex due to the stock market’s stochastic nature. Existing research focuses on the textual elements of financial disclosures like earnings calls transcripts to forecast stock volatility and risk, but ignores the rich acoustic features in the company executives’ speech. Recently, new multimodal approaches that leverage the verbal and vocal cues of speakers in financial disclosures significantly outperform previous state-of-the-art approaches demonstrating the benefits of multimodality and speech. However, the financial realm is still plagued with a severe underrepresentation of various communities spanning diverse demographics, gender, and native speech. While multimodal models are better risk forecasters, it is imperative to also investigate the potential bias that these models may learn from the speech signals of company executives. In this work, we present the first study to discover the gender bias in multimodal volatility prediction due to gender-sensitive audio features and fewer female executives in earnings calls of one of the world’s biggest stock indexes, the S&P 500 index. We quantitatively analyze bias as error disparity and investigate the sources of this bias. Our results suggest that multimodal neural financial models accentuate gender-based stereotypes.

## introduction earnings calls are publicly available, quarterly conference calls where ceos discuss their company's performance and future prospects with outside analysts and investors @xcite . they consist of two sections: a prepared delivery of performance statistics, analysis and future expectations, and a spontaneous question-answer session to seek additional information not disclosed before @xcite . researchers have studied the post earnings announcement drift (pead) to observe that statements made by upper management affect the way information is digested and acted upon impacting short-term price movements @xcite @xcite . audio features contextualize text and connotate speaker's emotional and psychological state @xcite @xcite . hence, when used with textual features, audio features significantly determine the effect of earning calls on the stock market @xcite . past research has shown that audio features such as speakers' pitch, intensity, etc. vary greatly across genders @xcite @xcite . moreover, female executives are highly underrepresented in these earnings calls @xcite . the variation in audio features is amplified by deep learning models due to a dearth of female training examples and is manifested as a gender bias. the system learns unneeded correlations between stock volatility and sensitive attributes like gender, accent, etc. it further perpetuates gender-based stereotypes and generalizations like female executives are less confident than male executives @xcite , men are assessed as more charismatic than female executives under identical conditions @xcite , and nurses are female and doctors are male @xcite . biased models further perpetuate stereotypes that can harm underrepresented communities, specifically in the financial and corporate world. @xcite even show that female speakers have to deliver better acoustic-melodic performance to seem as charismatic as men. taking a step towards fair risk forecasting models, we analyze gender bias by studying the error disparity in the state-of-the-art for multimodal volatility prediction, mdrm @xcite . bias in finance public financial data is impacting virtually every aspect of investment decision making @xcite . prior research shows that nlp methods leveraging social media @xcite , news @xcite , and earning calls (wang and hua, 2014) can accurately forecast financial risk. companies and investors use statistical and neural models on multimodal financial data to forecast volatility @xcite ) and minimize risk. these models although effective, may be tainted by bias due to individual and societal differences, often unintended @xcite . for example, models trained on the audio features extracted from ceo's speech in earnings calls @xcite , may be prone to bias given the underrepresentation of several demographics across race, gender, native language, etc. in the financial realm. bias in ai bias is prevalent in ai based neural models owing to the lack of diversity in training data @xcite . the design and utilization of ai models trained on gender imbalanced data, pose potential deprivation of opportunities to underrepresented groups such as females @xcite . with over 75% of ai professionals being men, male experiences also dominate algorithmic creation @xcite . in terms of natural language representation, embeddings such as word2vec and glove, trained on news articles may inherit gender stereotypes @xcite @xcite . recent studies also show the presence of bias in speech emotion recognition @xcite . ## formulation and experiments stock volatility following @xcite ; @xcite , for a given stock, with a close price of p i on trading day i, we calculate the average log volatility over n days following the day of the earnings call as: ## results and analysis bias in multimodal volatility prediction for evaluating gender bias in mdrm, we analyze the error disparity quantified by ∆g for the individual text and audio modalities and their combination for τ = 3,7,15,30 days. we tabulate the error disparity in terms of ∆g across modalities in over amplification refers to bias that occurs in a system during model fitting. the model learns imperfect generalizations between the attributes and the final labels and amplifies them while predicting on the test set. in our case, since female examples are very less in comparison to the male counterparts, the model discriminates between male and female examples by inferring insufficient information beyond its source base rate as shown in to study this effect we train the model for different training sample ratios as per gender to observe performance variation in figure 2 . we note that as the male:female training ratio increases, the test loss is amplified the most in audio modality followed by audio+text and text. test m se male decreases in comparison to increase in m se f emale . m se f emale increases as the percentage of female examples in the train set decreases as the generalised notions of this underrepresented community are learnt and the incorrect inferences do not harm the overall performance much. since the difference in test loss for male and female is significantly less when the number of samples across genders is equal. through this observation, we note that performance for female examples can be improved by augmentation techniques or cross domain adaptation, which we leave for future work. semantic bias occurs in embeddings and representations of audio and textual data which learn unwanted stereotypes. for our case semantic bias occurs as the audio features are significantly different for male and female distributions. we analyze each audio feature for both distributions in ## ethical considerations degradation in the performance of speech models could be due to discernible noise and indiscernible sources like demographic bias: age, gender, dialect, culture, etc @xcite @xcite . studies also show that ai can deploy biases against black people in criminal sentencing @xcite . although we only account for the gender bias in our study, we acknowledge that there could exist other kinds of bias due to age, accent, culture, ethnic and regional disparities in audio cues, as the publicly available earnings calls majorly have companies belonging to the us. moreover, only publicly available earnings calls have been used limiting the scope of the data. this also limits the availability of genders in the data to only male and female. in the future, we hope to increase the amount of data to expand our study to more categories and types of sensitive attributes. ## conclusion earnings calls provide company insights from executives proving to be high risk-reward opportunities for investors. recent multimodal approaches that utilize these acoustic and textual features to predict the financial risk achieve state-of-the-art performance, but overlook the gender bias associated with speech. we analyze the gender bias in volatility prediction of earnings calls due to gender sensitive audio features and underrepresentation of women in executive positions. we observe that the while adding speech features improves performance, it also perpetuates gender bias, as the audio modality has the highest error disparity. we further probe into the sources of bias, and analyze audio feature variations across gender, and perform experiments with varying training data distributions. our study presents the first analysis of its kind to analyze gender bias in multimodal financial forecasting to bridge the gap between fairness in ai, neural financial forecasting and multimodality.