LIPI at the F in NLP -2022 ERAI Task: Ensembling Sentence Transformers for Assessing Maximum Possible Profit and Loss from Online Financial Posts (2022)

Abstract: Using insights from social media for making investment decisions has become mainstream. However, in the current era of information ex- plosion, it is essential to mine high-quality so- cial media posts. The FinNLP-2022 ERAI task deals with assessing Maximum Possible Profit (MPP) and Maximum Loss (ML) from social me- dia posts relating to finance. In this paper, we present our team LIPI’s approach. We ensem- bled a range of Sentence Transformers to quan- tify these posts. Unlike other teams with vary- ing performances across different metrics, our system performs consistently well. Our code is available here https://github.com/sohomghosh/LIPI_ERAI_ FinNLP_EMNLP- 2022/

## introduction over the last few years, financial opinion mining has emerged to be an interesting area of research @xcite . several research @xcite al. (2012 ## problem statement for task-1, given two posts, the task is to develop a system for evaluating which of them will lead to greater mpp and lower ml. for task-2, given a set of posts, the task is to develop a system for ranking these posts in terms of higher mpp and lower ml values. results of task-1 were evaluated using accuracy. for task-2, average mpp and ml values of top 10% posts were considered for evaluation. ## datasets the organizers initially provided the participants with two datasets. the first dataset (corresponding to task-1) had 200 instances out of which 2 were null. we dropped the null instances from our experiments. each instance consists of two posts (in chinese as well as in english), their mpp and ml values, and labels corresponding to each post. in the dataset, the ml label is set to '1' for an instance (i.e., a pair of posts) when the ml value of the first post is less than that of the second post, otherwise the ml label is set to '0'. on the contrary, the mpp is set to '0' for an instance (i.e., a pair of posts) when the ml value of the first post is less than that of the second post, otherwise the mpp label is set to '1'. the posts in the dataset were collected from social media platforms like ptt foot_0 and mobile01 foot_1 . we refer to this as d1. for task-2, a dataset consisting of 210 unlabelled posts (in chinese as well as in english) were provided. this dataset is referred to as d2. d2 serves as the test set for task-2. subsequently, the organizers released a test set consisting of 87 pairs of unlabelled posts (in chinese and english) for pairwise comparison. we refer to this as d3. ## experiments and results this section states various experiments we performed and their results. we started with sb-1 which is a linear regression model trained over sentence embeddings. we tried financial sentence embeddings available for chinese as well as the english language. subsequently, we replaced the linear regression model with a multi-layer perceptron model. we further experimented by transforming the original training set d1 to d4 and d5. we also tried altering the last layer of the bert @xcite model for the task of regression. for the pairwise classification task, we used the regression models to get the mpp/ml values for each post in a pair. we then assigned a label to the pair by comparing these values as mentioned in §3. the results are presented in tables 1 and 2 . in this paper we focus on the best-performing systems among all our submissions due to page constraints. the other approaches we tried include classification of posts separated by [sep] token using various variants of @xcite . since the d4 dataset consists of single posts, we use the same training and validation set for both the tasks. as the d5 dataset comprises only of pairs of posts, we are unable to provide its performance in the unsupervised ranking task corresponding to the training and validation set. we ensembled models with varying lengths of the training set, therefore we do not report the performance of the model mentioned in §5.1 for the training set. similarly, for the unsupervised ranking task, we do not report the performances of the models describe in §5.1 and §5.2 as these models were suitable for pairwise comparison task only. the performance of the participating teams has been reported here @xcite .we used labelled instances from d4 to assess the performance of the unsupervised ranking models as well. this helped us in choosing the best performing models. as d5 was suitable for pairwise comparison task only, we could not use it to evaluate the models which were developed for the unsupervised ranking task. it is interesting to observe that our ensemble system's performance (sl.# 1.6) is next only to that of team jetsons in the pairwise comparison task using mpp. moreover, in the same task using ml our subsystem sb-4 (sl.# 2.4) performs as good as that of the best performing team dcu-ml (accuracy: 59.77%). however, we did not submit this sub-system separately as it did not perform well on the validation set and submitted the results of the ensemble model (sl.# 2.6) instead. in the unsupervised ranking using mpp task, only team promptshots's system performed better than that of ours (sl.# 1.2). however, in the unsupervised ranking using ml task, the performance of the system developed by team yet and the baseline solution were better than that of our systems (sl.# 2.3 and 2.5). in this case as well we did not submit the result corresponding to sb-3 (sl.# 2.3) where ml of top 10% post is -3.90% on the test set because the underlying system could not be evaluated on the validation set obtained from d5. we submitted results of sb-5 (sl.# 2.5 ) instead. ## conclusion comparing the performance of our models with that of the other participants, we conclude that our models performed consistently well. we also observe that in most cases we achieve better performances using the chinese texts than the translated version in english. this is because we are losing out on the nuances during translation. we further observe that ensembling helps in improving the overall performance. collecting more financial posts in a resourcerich language like english and incorporating prices of the stock whose mpp and ml are being discussed as input to the model are interesting directions for future work.