How Can a Teacher Make Learning From Sparse Data Softer? Application to Business Relation Extraction (2022)

Abstract: Business Relation Extraction between market entities is a challenging information extraction task that suffers from data imbalance due to the over-representation of negative relations (also known as No-relation or Others) compared to positive relations that corresponds to the taxonomy of relations of interest. This paper proposes a novel solution to tackle this problem, relying on binary soft labels supervision generated by an approach based on knowledge distillation. When evaluated on a business relation extraction dataset, the results suggest that the proposed approach improves the overall performance, beating state-of-the art solutions for data imbalance. In particular, it improves the extraction of under-represented relations as well as the detection of false negatives.

## introduction nowadays, the web is considered as an important source of business and financial information that can be used to analyze business interactions between market entities. these interactions enable financial institutions to take well-informed decisions @xcite , as well as business professionals to sustain and innovate in a rapidly changing business world. however, structuring this information remains a challenging task, given the volume and velocity of the textual data generated online. hence, the availability of systems that automatically extract business interactions between organizations (e.g., startups, companies, non-profit organizations, etc.) from textual content becomes crucial. business relation extraction (bre) is an nlp task that aims at discovering relations involving different companies (e.g. company-customer, company-partner) at the sentence level @xcite . for example, from the sentence in (1), extracted from bizrel dataset @xcite , a relation extraction system can infer that the company airbus is a supplier for the company inmarsat. example 1 the [airbus] e 1 group has signed a contract with @xcite e 2 for the delivery of three reconfigurable geostationary satellites in orbit. recent works for bre rely on supervised approaches, where neural models are trained on annotated datasets for business relations @xcite @xcite . in general, supervised approaches consider relation extraction (re) as a multi-class classification problem where each class corresponds to a predefined relation type @xcite . in addition to the set of positive relations (henceforth pr) which corresponds to the taxonomy of relations of interest (like hypernymy, meronymy, and cause-effect relationships), most popular datasets manually annotated either for generic (e.g., semeval-2010 task 8 @xcite , tacred @xcite ) or domain specific relations (e.g., chemprot @xcite , bizrel @xcite ) include a negative relation (henceforth nr) to account either for the absence of a relation between two target entities (see no-relation in tacred), or any other types of relations not present in the annotation scheme (see others in semeval-2010 and bizrel). nrs share two main characteristics: (c1) they have irregular and unstable linguistic realizations and (c2) are often over-represented making pr hard to predict due to the highly imbalanced nature of the problem (see the ratio of nr in several solutions have been proposed to address nr: discard them during training @xcite , ignore them at the evaluation stage focusing only on the performances of pr as done in most re shared tasks @xcite , or include them during training by treating all relations equally @xcite . these strategies however fail to deal with the sparseness of pr and the characteristics of nr in a real world scenario. to overcome the data imbalance problem, four main solutions have been proposed in the literature: the first three solutions rely on hard labels supervision, where the ground truth labels are represented using one-hot encoded vectors that are not able to represent the semantic information among relations. indeed, nr can have unstable patterns, and can share similar linguistic realizations with pr. for example, the semeval 2010 task 8 dataset @xcite includes the others relations for near misses of pr as nr instances, while in the bizrel dataset @xcite sentences expressing pr and nr at the same time between different pair of entities are one of the main sources of false negatives. while hard label supervision successes to counter the class imbalance problem (i.e. (c2)), it does not however fully capture the dissimilarities between pr and nr, making the optimiza-tion of model's output probabilities hard. recent studies show that soft labels generated via kd by a teacher model are more adequate to efficiently handle the inherent characteristics of nr (c1) @xcite . in this paper, we aim to continue these efforts by proposing a new knowledge distillation approach based on binary soft labels supervision (bsls). the soft outputs generated by the teacher model trained for binary classification (pr vs. nr), are used to supervise the student model to perform multi-class re. our contributions are three folds: our results show that our approach improves the extraction of under-represented relations as well as the detection of false negatives, addressing therefore both (c1) and (c2). this paper is organized as follows: we first present the related work, then describe our kd architecture. we finally detail the carried experiments and give our results. ## a binary soft-labels supervision for multi-class re (bsls) our binary soft label supervision approach for multi-class relation extraction is based on knowledge distillation where binary soft labels generated by a teacher model noted t are used to supervise the training of a student model noted s (cf. figure 1 ). following (zhou and chen, 2021), both s and t have the same architecture based on an improvement of r-bert @xcite , a transformer model specifically designed to handle re tasks. this architecture has two main components: a) a sentence encoder noted encoder i with @xmath1, t } based on the pre-trained bert model @xcite while using entity markers as sentence representation vectors, b) a relation classifier noted classif ier i composed of two linear layers followed by dropout layer then a softmax activation function. an input sentence is first fed into encoder i , to get its contextual representations that are injected into classif ier i to predict the relation type. let p @xmath2, ..., p in ) the prediction probabilities generated by classif ier i , with n being the number of relations to predict. let p sof tt the soft labels, i.e., the prediction probabilities generated by a pre-trained teacher binary classifier classif ier t whose weights are frozen and shared with s. finally, let y b and y m be respectively the binary and multi-class hard labels that encode the ground-truth labels as one hot vectors. these soft and hard labels are used by two different losses in order to optimize the models parameters through back-propagation: l ct (resp. l cs ) , the classification loss that minimizes the errors between p t and y b (resp. p s and y m ). and l d , the distillation loss calculated between a binarised form of p s and p sof tt . the distillation algorithm consists in the following steps: we compare our model against four baseline models used to tackle data imbalance in re: augmentation of the training data (da), multitask architecture (mlt), optimizing using an adapted loss (als), and knowledge distillation (kd) via soft labels. we describe below each of these configurations. ## discussion and analysis we further compare the performances of the best baseline (als f c ) with our best performing model (bsls f c ). figure 2 gives a confusion matrix that shows the number of false/true positives/negatives between pr and nr. we can see that bsls f c was able to reduce the number of false negative where bslsf c achieves the best f1 (68.9%), followed by alsf c (68.7%). in order to gain insights into the main strengths of the current approach when compared to the best baseline, we analyse well classified instances by bsls f c , that als f c fails to classify correctly. we notice that our approach is able to identify the nr others in some cases where many relations are expressed between different target entities, unlike als f c (see example 5). example 5 while there were few mega acquisi-tions/ mergers primarily chinese players acquiring european and us robotics/ automation companies (kuka ag by [midea group] e 1 , dematic by [kion group] e 2 and kraussmaffei automation by chem-china) and few others by us industry giants (affeymetrix by thermofisher and intelligrated by honeywel), most acquisitions were in the sub @xmath0 128,500 million contract with [electronic data systems] e 2 (eds) to handle the corporation's desktop computer procurement, service, and maintenance activities. bsls f c 's correct label : sale-purchase, als f c 's wrong label: cooperation ## conclusion in this paper, we propose a novel solution to tackle pr vs. nr imbalance and nr irregular patterns problems, relying on binary soft-labels supervision generated by knowledge distillation. when evaluated on a business relation dataset, our approach improves the overall performances by enhancing the detection of under-represented relations and reducing false negative misclassification rates. as future work, we plan to evaluate our method to other generic and domain specific re datasets in order to assess its adaptability to other domains.