Jetsons at the F in NLP -2022 ERAI Task: BERT - C hinese for mining high MPP posts (2022)

Abstract: In this paper, we discuss the various approaches by the Jetsons team for the “Pairwise Comparison” sub-task of the ERAI shared task to compare financial opinions for profitability and loss. Our BERT-Chinese model considers a pair of opinions and predicts the one with a higher maximum potential profit (MPP) with 62.07% accuracy. We analyze the performance of our approaches on both the MPP and maximal loss (ML) problems and deeply dive into why BERT-Chinese outperforms other models.

## introduction natural language processing (nlp) has the potential to uncover meaningful insights from the vast amounts of unstructured data and impact the financial services industry. the use cases for financial nlp range from quantitative trading, portfolio selection, and risk assessment to speech recognition and customer chatbots on various unstructured sources, including transcripts of quarterly earnings calls, research reports, company filings, and social media chatter. people frequently express opinions about financial products, services, investments, and the stock market on social media. such financial opinions can be effectively mined to provide recommendations and influence user/enterprise perception. the evaluating the rationales of amateur investors (erai) shared task @xcite focuses on opinions that would lead to profitable outcomes by using forecasting skills as a proxy. it is formulated as follows: given two opinions about a company extracted from chinese social forums by amateur investors, predict the opinion that would lead to a higher profitable outcome or higher loss. we approach this problem using several strategies to represent and classify opinions, including: (1) using bert-chinese 1 on the original chinese posts, (2) using roberta @xcite and other bert @xcite variants on the english translated opinions, (3) using ( 2 ) in conjunction with pos tag features and (4) an ensemble of some of these approaches. our approach using bert-chinese topped the test leaderboard for the mpp task. we present the results of all these approaches and analyze how these models perform for the mpp task. we also examine what we may have lost in translation between chinese and english and why the bert-chinese model outperforms the englishlanguage bert models. ## dataset and methods the training dataset contains 200 instances of opinion pairs in chinese, their english translations, along with an mpp (maximum potential profit) label and an ml (maximum loss) label @xcite . in every instance, the pair of opinion posts have associated mpp i and ml i values where @xmath0, 2}, mpp @xmath1.0, 0.16) and ml @xmath2.24, 0.0]. if opinion 1 leads to higher mpp than opinion 2 i.e., mpp 1 ≥ mpp 2 , the mpp label is 1, otherwise 0. similarly, if opinion 1 leads to higher loss than opinion 2 i.e. ml 1 ≤ ml 2 , the ml label is 1, otherwise 0. out of the 200, there are only seven instances where the absolute difference between ml values of opinion 1 and 2 is greater than 0.1. similarly, for mpp values, there are only six instances where the absolute difference between opinions 1 and 2 is greater than 0.1. the dataset distribution of ml and mpp labels as shown in ta-dataset labels ml mpp training 1 105 109 0 95 91 testing 1 63 44 0 24 43 ## experimental results all models are trained using 10-fold crossvalidation on the training set. the model corresponding to the best fold accuracy is used to obtain predictions on the test set. ## analysis this section focuses on analyzing the impact of using the original chinese posts for classification. the analysis is carried out in two ways -understanding the translation errors and probing the bbc model. all analysis presented in this section is for the mpp classification task. ## conclusion this paper discusses the models submitted for the erai pairwise comparison subtask organized at finnlp 2022. of the submitted models, the bert-chinese model trained on the chinese posts ranks first on the mpp label leaderboard. we investigate why using chinese posts over translated english posts results in higher accuracy and attribute the behavior to errors in translation. additionally, we probe the bert-chinese model using different training and testing language combinations to evaluate the impact of two language pre-training. we show that the model did better when trained on chinese posts and tested on english translations. lastly, we show that the accuracy metric is not suited for the task owing to its inability to handle class imbalance.