F in E ntity: Entity-level Sentiment Classification for Financial Texts (2023)

Abstract: In the financial domain, conducting entity-level sentiment analysis is crucial for accurately assessing the sentiment directed toward a specific financial entity. To our knowledge, no publicly available dataset currently exists for this purpose. In this work, we introduce an entity-level sentiment classification dataset, called FinEntity, that annotates financial entity spans and their sentiment (positive, neutral, and negative) in financial news. We document the dataset construction process in the paper. Additionally, we benchmark several pre-trained models (BERT, FinBERT, etc.) and ChatGPT on entity-level sentiment classification. In a case study, we demonstrate the practical utility of using FinEntity in monitoring cryptocurrency markets. The data and code of FinEntity is available at https://github.com/yixuantt/FinEntity.

## introduction developing a natural language processing (nlp) system for entity-level sentiment classification necessitates the availability of a dataset with entity tagging and sentiment annotation. to our knowledge, no such public dataset currently exists. in this paper, we fill this gap by constructing a new dataset that annotates both the financial entity spans and their associated sentiments within a text sequence. we outline the development of a high-quality entitylevel sentiment classification dataset for the financial domain, called finentity. subsequently, we benchmark several pre-trained language models (plms) and a zero-shot chat-gpt model (gpt-3.5-turbo) on entity-level sentiment classification tasks. the results demonstrate that fine-tuning plms on finentity outperforms the zero-shot gpt model. this finding suggests that manually collecting a high-quality domain-specific dataset and fine-tuning plms is more suitable than relying on the zero-shot gpt model. we further demonstrate the practical utility of finentity in investment and regulatory applications, extending the work of @xcite ; @xcite . collaborating with a regulatory agency, we apply the fine-tuned plms to a unique cryptocurrency news dataset. experimental results indicate that the individual cryptocurrency sentiment, inferred using the finentity finetuned plm, exhibits a stronger correlation with cryptocurrency prices than traditional sequencelevel sentiment classification models. furthermore, the inferred individual cryptocurrency sentiment can better forecast future cryptocurrency pricesleading to enhanced risk monitoring for regulatory agencies and investors. we make the finentity dataset publicly available and hope it will be a valuable resource for financial researchers and practitioners in developing more accurate financial sentiment analysis systems. financial entity tagging and sentiment dataset. existing financial sentiment classification datasets, such as financial phrase bank @xcite , semeval-2017 @xcite , analysttone dataset @xcite , headline news dataset @xcite and trillion dollar words @xcite , are based on entire text sequence (sentence or article). fiqa, an open challenge dataset foot_0 , features aspect-level sentiment; however, it does not include entity annotations. sentfin @xcite is a dataset for financial entity analysis in short news headlines. however, this dataset uses a pre-defined entity list to match entities and does not have entity tagging, so it is still not applicable to recognize financial entities from text. moreover, most of the headlines contain only one entity, which makes it close to a sequence-level sentiment dataset. for financial entity tagging dataset, finer @xcite and fnxl @xcite are created for financial entity recognition and numeral span tagging respectively, but both lacks sentiment annotation. therefore, we aims to bridge this gap by constructing a high-quality, entity-level sentiment classification dataset, which not only label the financial entity spans in sentences, but also annotate their associated sentiments. ## dataset construction initial dataset. we obtain a financial news dataset from refinitiv reuters database. in the prescreening step, we utilize a pre-trained named entity recognition model foot_1 and a sequence-level sentiment classification model foot_2 to infer the number of org entities and sequence sentiment. subsequently, we compile a dataset with a balanced distribution of positive/negative/neutral articles, ensuring that 80% of the sequences contain more than one entity. following prescreening, we obtain a dataset comprising 4,000 financial news sequences foot_3 . label. entity-level sentiment classification is a sequence labeling task. as such, we employ the bilou annotation scheme. specifically, each token in an input sequence is assigned one of the bilou labels, indicating the beginning, inside, last, outside, and unit of an entity span in the sequence. additionally, each annotated bilu entity is tagged with a sentiment label (positive/neutral/negative), while the o entity does not receive a sentiment label. consequently, each token in the input sequence is assigned to one of thirteen labels (bilu-positive/neutral/negative and one o label). annotators. a total of 12 annotators are recruited, all of whom are senior-year undergraduate students majoring in finance or business at an englishspeaking university. three annotators label the same example to ensure data quality and perform cross-check validation. therefore, each annotator is assigned 1,000 examples. we employ the lighttag platform foot_4 for the annotation job. annotators are instructed to tag all named entities in the sequence and the sentiment associated with each entity. focusing on the financial domain, we limit the named entity to companies (such as apple inc.), organizations (such as the fed), and asset classes (such as equity, reit, and crude oil). annotators are advised not to tag persons, locations, events, or other named entities. a screenshot of the annotation interface is shown in appendix a. annotation consistency. a total of 28,631 entities are annotated by the annotators, and we conduct cross-checks in order to ensure data quality. initially, we employ the jaccard similarity coefficient to measure entity-level consistency between pairs of annotators. the overall jaccard similarity of the dataset is 0.754. furthermore, the number of examples with a jaccard similarity equal to 1.0 is 44.35%, indicating that 44.35% of examples ## benchmarking entity-level sentiment classification plms. we benchmark several plms for entitylevel sentiment classification. we consider finetuning bert (bert-base-cased) @xcite and a finance-domain specific finbert @xcite by incorporating a linear layer on top of each token's hidden output. in order to account for token label dependency, we replace the linear layer with a conditional random field (crf) layer, yielding bert-crf and finbert-crf respectively. we implement those plms using the transformers library @xcite . chatgpt. for comparison with state-of-the-art generative llms, we examine the zero-shot and few-shot in-context learning performance of chatgpt by querying openai's gpt-3.5-turbo model with a 0.0 temperature value. following previous literature @xcite , we construct a prompt designed to elicit structured responses. the detailed prompt for zero-shot and few-shot learning is shown in appendix c. evaluation. we randomly partition the dataset into 80% training dataset and 20% testing dataset. we employ seqeval @xcite for evaluation, reporting f1-scores, which include the negative, positive, and neutral classifications, respectively. it is important to note that for entity-level sentiment classification, a testing example is considered correctly classified only if all the named entities in the example are correctly tagged and their associated sentiments are correctly classified. this implies that this task is much more challenging than traditional sequence-level sentiment classification tasks. results. we present the benchmarking results in ## case study: cryptocurrency market in this section, we demonstrate the practical utility of the finentity dataset for investment and regulatory applications. as a case study, we focus on the cryptocurrency market, which is notoriously volatile and has been plagued by instances of manipulation and fraud, as evidenced by the recent ftx crash. as such, it is crucial for regulators and investors to closely monitor the market sentiment toward different cryptocurrencies. in this case study, we collaborate with a regulatory agency responsible for overseeing the monetary and financial systems in both local and global markets. the team at the regulatory agency shares a cryptocurrency news dataset (non-overlapping with our reuters data) that they have internally col-lected. the dataset comprises 15,290 articles spanning from may 20, 2022 to february 1, 2023. each article is associated with a timestamp, enabling time-series analysis. we select four cryptocurrencies (bitcoin, ethereum, dogecoin, ripple) as the target entities for analysis, owing to their substantial market dominance and the attention they receive from investors. ## conclusion in this paper, we present the construction of finentity, a dataset with financial entity tagging and sentiment annotation. we benchmark the performance of plms and chatgpt on entity-level sentiment classification and showcase the practical utility of finentity in a case study related to the cryptocurrency market. we believe that finentity can be a valuable resource for financial sentiment analysis in investment and regulatory applications.