Related Named Entities Classification in the Economic-Financial Context (2021)

Abstract: The present work uses the Bidirectional Encoder Representations from Transformers (BERT) to process a sentence and its entities and indicate whether two named entities present in a sentence are related or not, constituting a binary classification problem. It was developed for the Portuguese language, considering the financial domain and exploring deep linguistic representations to identify a relation between entities without using other lexical-semantic resources. The results of the experiments show an accuracy of 86% of the predictions.

## introduction in the context of the financial market, the news bring information regarding sectors economy, industrial policies, acquisitions and partnerships of companies, among others. the analysis of this data, in the form of financial reports, headlines and corporate announcements, can support personal and corporate economic decision making @xcite . however, thousands of news items are published every day and this number continues to increase, which makes the task of using and interpreting this huge amount of data impossible through manual means. information extraction (ie) can contribute with tools that allow the monitoring of these news items in a faster way and with less effort, through automation of the extraction and structuring of information. ie is the technology based on natural language, that receives text as input and generates results in a predefined format @xcite . among the tasks of the ie area, it is possible to highlight both named entity recognition (ner) and relation extraction (re). for example, it is possible to extract that a given organization (first entity) was purchased (relation) by another organization (second entity) @xcite . a model based on the bert language model @xcite is proposed to classify whether a sentence containing a tuple entity 1 and entity 2 (e1,e2), expresses a relation among them. leveraging the power of bert networks, the semantics of the sentence can be obtained without using enhanced feature selection or other external resources. the contribution of this work is in building an approach for extracting entity relations for the portuguese language on the financial context. the remainder of this work is organized as follows. section 2 presents news processing for the competitive intelligence (ci) area. section 3 presents the related work. section 4 provides a detailed description of the proposed solution. section 5 explains the experimental process in detail, followed by section 6, which shows the relevant experimental results. finally, section 7 presents our conclusions, as well as future work. ## competitive intelligence and news processing some of the largest companies in the financial segment have a competitive intelligence (ci) sector where information from different sources is strategically analyzed, allowing to anticipate market trends, enabling the evolution of the business compared to its competitors. this sector is usually formed by one or more professionals dedicated specifically to monitor the movements of the competition. in a time of competitiveness that is based on knowledge and innovation, ci allows companies to exercise pro-activity. the conclusions obtained through this process allow the company to know if it really remains competitive and if there is sustainability for its business model. ci can provide some advantages to companies that use it, such as: minimizing surprises from competitors, identify-ing opportunities and threats, obtaining relevant knowledge to formulate strategic planning, understanding the repercussions of their actions in the market, among others. the process of capturing information through news still requires a lot of manual effort, as it often depends on a professional responsible for carefully reading numerous news about organizations to highlight possible market movements that also retain this knowledge. it is then estimated that a system, that automatically filters the relations between financial market entities, can reduce the effort and the time spent on these tasks. another benefit is that this same system can feed the business intelligence (bi) systems and, thus, establish a historical database with market events. thus, knowledge about market movements can be stored and organized more efficiently. ## related work er is a task that has been the subject of many studies, especially now when information and communication technologies allow the storage of and processing of massive data. zhang @xcite proposes to incorporate the position of words and entities into an approach employing combinations of n-grams for extracting relations. presenting a different methodology to extract the relations, wu @xcite proposed to use a pre-trained bert language model and the entity types for re on the english language. in order to circumvent the problem of lack of memory for very large sequences in convolutional networks, some authors @xcite @xcite have adopted an approach using memory cells for neural networks, long short-term memory (lstm). in this sense, qingqing's li work @xcite uses a bidirectional long short-term memory (bi-lstm) network, which are an extension of traditional lstms, for its multitasking model, and features a version with attention that considerably improves the results in all tested datasets. also using bi-lstm networks, florez @xcite differs from other authors in that it uses types of entities and the words of the entities being considered for a relation in addition to using information such as number of entities and distances, measured by the number of words and phrases between the pair of entities. the entry of the bi-lstm layer is concatenation of words and relations, with all words between the candidate entities (included), provided by a pre-trained interpolation layer. yi @xcite proposes to join a bert language model and a bidirectional gated recurrent unit (bi-gru) network, which is a version of bi-lstm with a lower computational cost. finally, they train their model based on a pre-trained bert network, instead of training from the beginning, to speed up coverage. some works @xcite @xcite use attention mechanisms to improve the performance of their neural network models. such mechanisms assist in the automatic information filtering step that helps to find the most appropriate sentence section to distinguish named entities. thus, it is possible that even in a very long sentence, and due to its size being considered complex, the model can capture the context information of each token in the sentence, being able to concentrate more in these terms the weights of influence. pengda qin @xcite proposes a method using bi-gru with an attention mechanism that can automatically focus on valuable words, also using the pairs of entities and adding information related to them. tao @xcite also addresses re with an attention method to capture important parts of the sentence and for that, it uses an lstm attention network for entities at the subsequent level. in this way, he focuses more on important contextual information between two entities. zhou (zhou and zhang, 2018) also implement a model based on rnn bi-gru with an attention mechanism to focus on the most important assumptions of the sentences for the financial market. despite having great importance, the financial domain, specifically, has been little explored in the literature. the authors at (zhou and zhang, 2018) created a corpus collecting 3000 sentence records manually from the main news sites, which was used to recognize the entity and extract relations such as learning and training as a whole. most studies present re solutions for english texts, and, in this way, it is also possible to identify a larger number of data sets in this language. there are few data sets available in the portuguese language, such as the golden collection harem, which is widely used in the literature @xcite @xcite . harem is a joint assessment event for the portuguese language, organized by linguateca @xcite . its objective is to evaluate recognizing systems of ne @xcite . the golden collection (gc) is a subset of the harem collection, being used for the task of evaluating the systems that deal with recognition of named entities. the lack of this type of resource forces researchers to develop their own research corpus. in most cases, it is necessary to first create a set with the sentences and write them down when the classification is supervised to proceed with the re task. besides, the lack of public data sets also makes it difficult to fairly compare related work, as well as requires more time and effort from the researcher. it is possible to observe that there are works that discuss the task of extracting relations between ne and that already employ machine learning techniques for this purpose. however, although we found some works for the re task, few of them are suitable for the portuguese language, and none of them are related to the financial context. considering other languages, the work of zhou (zhou and zhang, 2018) was the only one that came closest to our goals. however, there is a gap in the literature for works that address such tasks using deep learning techniques and portuguese as the main language, especially in the financial-economic context as addressed in this work. ## architecture in this section, we present our bert-based model in detail. as shown in figure 1 , it contains three parts: (1) input layer; (2) bert layer; and (3) output layer, which is composed of a sigmoid activation function and two neurons that represent the classes to be predicted. the input layer consists of a bert encoder used for input sentence tokenization and produces a tuple of arrays (token, mask, sequence ids), which were used as input to the second layer that is the portuguese bert language model @xcite foot_0 from huggingface python package foot_1 @xcite . figure 2 illustrates the input layer of the proposed model. the entry consists of (1) the original sentence with the mentioned entities and (2) the entities to be verified concatenated. a special token [cls] and a token [sep] are added at the beginning and end of the input string respectively, as mentioned in the original bert implementation @xcite . the third layer of the model architecture is identified as the output layer. this layer is fully connected with a tangent activation function. the output of this layer is propagated to a new fully connected layer, with a sigmoid activation function, whose characteristic is the mapping of input values to 0 or 1. in this model, these values represent non-relation and relation, respectively. as shown in figure 1 , this layer still has two output neurons, which indicate the respective classes to be predicted by the model. in the end, we added a dropout layer with a 0.1 rate to avoid model overfitting, which happens when the model memorizes the training data and thereby loses the power of generalization. ## experiments the purpose of this section is to verify the proposed model performance thought experiments on the financial domain corpus. the proposed study follows the classic methodology of knowledge discovery in databases (kdd) @xcite , which contains 5 phases that range from data collection to the evaluation of the results. the following subsections aim to indicate how each step of the methodology was applied in the context of our work. subsection 5.1 refers to the selection step and seeks to indicate what data will be used during the experiments for the re task. subsection 5.2 addresses the pre-processing step, indicating procedures for quality checking, cleaning, correction, or removal of inconsistent or missing data. subsection 5.3 reports the transformation phase, where the transformation processes applied to the data set in the context of our work are explored. subsection 5.4 brings the penultimate phase, of mining, where the data mining process is presented. finally, the last phase of the methodology is presented in the subsection 5.5, which consists of evaluating the performance of the model applied on top of the data that were not used in the training or mining phase. ## results after the training stage of the model, it was applied to the test data set. in this evaluation step, the model obtained reasonable results, achieving an overall accuracy and f-measure of 86%. an important observation to make is that results are also good when it comes to the target class, that is, when the label is positive, as can be seen in as indicated in section 3, the vast majority of studies present re solutions for texts in english or a domain other than finance. thus, it is difficult to nevertheless, it is shown that the proposed model was able to recognize patterns and indicate when two entities are semantically related in the same sentence in the financial domain. the process of finding the best parameters for bert is time-consuming as the predictions made by the network. the time might not be a constraint to using the re task model applied to the context of the financial domain considering that this demand does not require the processing time to be real-time. we believe that if the data set is increased with more samples, the model may have a performance gain. also, we can notice that the data set has a small unbalanced distribution rate, with a greater number of negative samples. this imbalance can help explain the difference in precision and f-measure between the positive and negative class indicated in regarding recall, the study indicates that, even with the imbalance of the data, the proposed model achieved a very good performance of approximately 90% when it comes to the positive class (it has a relation). that is, when it really belongs to the positive class, in approximately 90% of the cases, it identifies correctly. it is also possible to carry out tests with adjustments of more hyper-parameters such as loss function, optimizers, among others. in addition to adjustments to the hyper-parameters of the approach, more contextual information of the samples can be added, such as the type of the named entity, whether it is an organization, person, or place, and scope adopted for the task being worked on. in this way, it is possible to delimit the types of relations between 2 entities, excluding, for example, an acquisition relation between two entities of the person type. ## conclusion and future works the present work proposed an approach to extract relations between named entities, in the financialeconomic context, based on the portuguese bert language model, to our best knowledge, different from what is already in the literature. thus, it provides an insight into the use of pre-trained deep language models for extracting relations for the portuguese language financial market. from the related work section, it is possible to verify that there is little research on the technology for extracting the relation between named entities for the financial domain, for the portuguese language. this domain lacks practical solutions, given a large amount of information in the financial field, and manual analysis becomes difficult to meet the needs and make full use of that information. a model of classification of relations between named entities based on bert was proposed, which replaces explicit linguistic resources, required by previous methods. this approach uses the information from the sentence and the concatenated entity pair, which allows more than one entry to be sent since a sentence can have n pairs of named entities. therefore, the adopted approach allows the sentence and the pair of entities to be inferred to be sent separately. the results demonstrate that the approach used can bring satisfactory results, reaching an accuracy of 86%. during the discussion of results, some adjustments were made to try to improve accuracy, such as testing other combinations of hyper-parameters and also the increase in the corpus. however, the development of memory improvements and optimizations are still in need, especially in the training period, due to the complexity of the pre-trained bert model. as a natural continuation of this work, we will proceed with tests with other combinations of hyper-parameters as indicated in section 6. to try to reduce the chance of the model being surprised with some non-standard samples, new data will be annotated and added to the research corpus. thus, the model can be trained with a greater amount of data and a greater diversity of data patterns. as a continuity, a second model will also be developed, with sequential classification, so that it is possible to highlight the parts of the sentences that represent or describe the relation between the named entities verified. to achieve this goal, this second model will be trained only with the tuples that contain the annotated relation. thus, the output of the model proposed in this work will be the input of the sequential classifier model.