P rompt S hots at the F in NLP -2022 ERAI Task: Pairwise Comparison and Unsupervised Ranking (2022)

Abstract: This report describes our PromptShots submissions to a shared task on Evaluating the Rationales of Amateur Investors (ERAI). We participated in both pairwise comparison and unsupervised ranking tasks. For pairwise comparison, we employed instruction-based models based on T5-small and OpenAI InstructGPT language models. Surprisingly, we observed OpenAI InstructGPT language model few-shot trained on Chinese data works best in our submissions, ranking 3rd on the maximal loss (ML) pairwise accuracy. This model works better than training on the Google translated English data by a large margin, where the English few-shot trained InstructGPT model even performs worse than an instruction-based T5-small model finetuned on the English data. However, all instruction-based submissions do not perform well on the maximal potential profit (MPP) pairwise accuracy where there are more data and learning signals. The Chinese few-shot trained InstructGPT model still performs best in our setting. For unsupervised ranking, we utilized many language models, including many financial-specific ones, and Bayesian lexicons unsupervised-learned on both Chinese and English words using a method-of-moments estimator. All our submissions rank best in the MPP ranking, from 1st to 3rd. However, they all do not perform well for ML scoring. Therefore, both MPP and ML scores need different treatments since we treated MPP and ML using the same formula. Our only difference is the treatment of market sentiment lexicons.

## introduction evaluating the rationals of amateur investors (erai) @xcite ) is a shared task on evaluating social media opinions on the topic of investments and whether they are going to be useful or not. mining high-quality opinions by inspecting their supporting rationales might utilize the wisdom of the crowd on social media. previous work @xcite proposes stylistic and semantic features to filter out noisy crowd opinions which may not be high-quality and profitable. there are two settings in this erai shared task, pairwise comparison and unsupervised ranking. these settings sort out the opinions based on two metrics, higher maximal potential profit (mpp) and lower maximal loss (ml). in pairwise comparison, two posts are given with a binary label whether the mpp and ml of the first post are more or less than the second post. in unsupervised ranking, the goal is to filter and keep the top 10% posts based on mpp and ml given a set of unranked posts. for pairwise comparison, our best submission ranks 3 rd on the maximal loss (ml) pairwise accuracy on the leaderboard foot_0 . for unsupervised ranking, our best submission ranks 1 st on the maximal potential profit (mpp) ranking. the codes for our systems are open-sourced and available at our github repository foot_1 . ## experimental results in our experiments, most of our submissions (except t5-small) are intuition-based heuristics, and we did not even measure neither their training nor validation performance at all during the competition. we did not use any data augmentation techniques. ## conclusion this report describes our systems for a shared task of evaluating the rationales of amateur investors at finnlp-2022. from the experimental results in pairwise comparison, we conclude that few-shot prompted instruction-based language models can work reasonably well in low resource settings with minimal training efforts but might need quite accurate data from sources since using translated data seems not to perform well. from the experimental results in unsupervised ranking, financial language models perform well and bayesian-fitting the lexicons helps improve the performance. also, the heuristic function design needs to differ between mpp and ml.