Overview of the F in NLP -2022 ERAI Task: Evaluating the Rationales of Amateur Investors (2022)

Abstract: This paper provides an overview of the shared task, Evaluating the Rationales of Amateur Investors (ERAI), in FinNLP-2022 at EMNLP-2022. This shared task aims to sort out investment opinions that would lead to higher profit from social platforms. We obtained 19 registered teams; 9 teams submitted their results for final evaluation, and 8 teams submitted papers to share their methods. The discussed directions are various: prompting, fine-tuning, translation system comparison, and tailor-made neural network architectures. We provide details of the task settings, data statistics, participants’ results, and fine-grained analysis.

## introduction in the financial market, people have different reasons to make trading/investment decisions. thanks to the development of social media platforms, people can share these reasons and discuss them with others rapidly. however, there are hundreds of thousands of posts on social media platforms every day. selecting the posts (opinions) that have the potential to help investors make profitable investment decisions becomes a challenge. inspired by the ideas of persuasive essay scoring @xcite and argument quality assessment @xcite , we proposed a new task: evaluating investment opinions based on the rationales in the post @xcite . there are some steps when reading and evaluating investment opinions. first, as in most sentiment analysis studies @xcite , investors need to identify the sentiment of the opinion (bullish/bearish/neutral). second, investors will read the reasons that are provided to support the sentiment. third, investors will evaluate whether these reasons are rational, and further decide whether to follow the suggestions in the opinion. when we attempt to select useful investment opinions automatically, we think that systems also need to follow the above steps. however, in many cases, it is hard to decide the ground truth for the opinion quality because it is somehow subjective and varies due to the viewpoints. in the debate scenario, we can use the voting records as a proxy for evaluation. in the financial market, we can use historical information as a proxy to assess forecasting skills @xcite . therefore, we propose to use maximum possible profit (mpp) and maximum loss (ml) as evaluation metrics to measure the quality of investment opinions @xcite . in this shared task, we propose two kinds of settings, pairwise comparison and unsupervised ranking. the findings under these settings not only can be used in investment recommendations in the future, but also can be used in evaluating the generated reports and investor education. additionally, we also expect that we can improve models' performances in market information forecasting tasks by sorting out high-quality opinions and filtering out low-quality opinions in the first step when selecting input data. participants explore various directions for solving these challenges. there are several interesting discussions for a better understanding of where we are in the financial opinion scoring. we summarize the details of their methods in section 3. ## participants' methods ## participants' results team top 10% mpp team top 10% ml promptshots_2 24.39% baseline (chen et al., 2021) -2.46% promptshots_3 23.76% yet_3 -3.24% promptshots_1 22.53% lipi_1 -4.11% lipi_2 18.27% aimi_1 -4.17% baseline (chen et al., 2021) 17.61% yet_1 -4.35% lipi_1 17.46% lipi_3 -5.56% uccnlp_3 14.81% yet_2 -5.77% yet_3 14.61% uccnlp_3 -5.85% aimi_1 14.02% uccnlp_1 -6.22% dcu-ml_1 13.97% uccnlp_2 -6.77% uoa_1 12.35% promptshots_1 -7.80% yet_2 12.10% lipi_2 -7.81% lipi_3 11.83% dcu-ml_1 -8.25% uccnlp_2 11.34% uoa_1 -9.39% uccnlp_1 11.10% promptshots_3 -12.33% yet_1 8.52% promptshots_2 -13.04% ml instead of mpp (dcu-ml_1). in the unsupervised setting, sentiment lexicons still play important roles (promptshots_1,2,3). most supervised results with the model trained with pair-wise setting dataset cannot outperform lexicon-based method and the baseline @xcite , which count the expert-like sentences in the post. on the other hand, the ml results in unsupervised setting imply that expert-like sentences matters in sorting out the opinions containing lower risk. ## future directions we want to highlight that before we try to sort opinions, we may need to first filter out those posts that do not contain trading ideas. for example, there are 57 of these kinds of posts in the unsupervised set. these posts follow the same format but may just ask questions. there are two reasons why we need to remove such posts. firstly, in most cases, the models' input length is limited. under this limitation, ideally, we should only use those considered important. secondly, since this kind of posts does not contain opinion, putting them into a model may lead to incorrect claims and increase the noise. following this line of thought, one of the future directions is to filter out both irrelevant and low mpp posts in the preprocessing process. on the other hand, the proposed idea can also use in a recommendation system for investors. instead of only suggesting the relevant opinions as previous work @xcite , we think that recommending high potential suggestions would be more preferred in the investment scenario. ## conclusion this paper introduces the methods explored in the erai shared task, and summarizes the performances of these methods. we think this is a pilot exploration for evaluating the rationales of investors, and plan to dig into this direction more deeply in the future. the first step is exploring the role of argument in these tasks. we will present several datasets for extracting argument features from financial opinions, and we think that it will be useful in scoring investors' opinions. the enlarged dataset for evaluating investors' opinions will also be proposed. please refer to the finarg@ntcir for more details. 5 jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. jacob eisenstein. 2017. unsupervised learning for lexicon-based classification. in proceedings of the aaai conference on artificial intelligence, volume 31. debanjan ghosh, aquila khanam, yubo han, and smaranda muresan. 2016. coarse-grained argumentation features for scoring persuasive essays. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 549-554, berlin, germany. association for computational linguistics. sohom ghosh and kumar sudip naskar. 2022. lipi at the finnlp-2022 erai task: ensembling sentence transformers for assessing maximum possible profit and loss from online financial posts. in proceedings of the fourth workshop on financial technology and natural language processing, abu dhabi, united arab emirates. association for computational linguistics. alolika gon, sihan zha, sai krishna rallabandi, parag pravin dakle, and preethi raghavan. 2022. jetsons at the finnlp-2022 erai task: bert-chinese for mining high mpp posts. in proceedings of the fourth workshop on financial technology and natural language processing, abu dhabi, united arab emirates. association for computational linguistics. md kamrul hasan, james spann, masum hasan, md saiful islam, kurtis haut, rada mihalcea, and ehsan hoque. 2021. hitting your marq: multimodal argument quality assessment in long debate video. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 6387-6397, online and punta cana, dominican republic. association for computational linguistics. yi-ting liou, chung-chi chen, tsun-hsien tang, hen-hsen huang, and hsin-hsi chen. 2021. finsense: an assistant system for financial journalists and investors. in proceedings of the 14th acm international conference on web search and data mining, pages 882-885. tim loughran and bill mcdonald. 2011. when is a liability not a liability? textual analysis, dictionaries, and 10-ks. the journal of finance, 66(1):35-65. lefteris loukas, manos fergadiotis, ilias chalkidis, eirini spyropoulou, prodromos malakasiotis, ion androutsopoulos, and georgios paliouras. 2022. finer: financial numeric entity recognition for