Active Learning for Interactive Relation Extraction in a F rench Newspaper’s Articles (2021)

Abstract: Relation extraction is a subtask of natural langage processing that has seen many improvements in recent years, with the advent of complex pre-trained architectures. Many of these state-of-the-art approaches are tested against benchmarks with labelled sentences containing tagged entities, and require important pre-training and fine-tuning on task-specific data. However, in a real use-case scenario such as in a newspaper company mostly dedicated to local information, relations are of varied, highly specific type, with virtually no annotated data for such relations, and many entities co-occur in a sentence without being related. We question the use of supervised state-of-the-art models in such a context, where resources such as time, computing power and human annotators are limited. To adapt to these constraints, we experiment with an active-learning based relation extraction pipeline, consisting of a binary LSTM-based lightweight model for detecting the relations that do exist, and a state-of-the-art model for relation classification. We compare several choices for classification models in this scenario, from basic word embedding averaging, to graph neural networks and Bert-based ones, as well as several active learning acquisition strategies, in order to find the most cost-efficient yet accurate approach in our French largest daily newspaper company’s use case.

## motivation relation extraction is a mature field of natural language processing that aims at finding the relations between identified entities in texts. recent research has focused on the use of trainable models to automatically extract and classify relations between entities. the state-of-the-art research on relation extraction focuses on large, complex models, that require either a long time to train or some pre-training with a fine-tuning phase. task-specific labelled data is needed to train the final classification model, with research relying on benchmarks such as @xcite or @xcite , which are often made of single sentences with clearly tagged entities and definitive, non-ambiguous relations. in a real-life scenario, however, relation extraction is used in a language processing pipeline, e.g., in order to confront different reports of a same event, or to grasp a general picture of a situation. in the specific case of a regional newspaper company such as ours, the intent is to create a knowledge graph from the content of the newspaper's own articles, so as to facilitate journalists' investigation with an easy access to information and to possible relations that are otherwise drowned in pages of text. in this realistic scenario, data is of a very different nature from standard corpora, exhibiting certain features that are specific to the regional ecosystem, thus challenging off-the-shelf models or learning methods. the main feature of this data is that, while it is abundant, almost none of it is annotated, as human expert annotation is expensive. we thus turn to active learning (al) as a means to alleviate the cost of labelling datasets. this approach, opposed to simply annotating samples, allows for a selection of the most helpful training samples, and therefore also allows a reduction of the number of annotations that have to be made by a human to reach satisfactory performance. besides, it also allows annotation to be done in several installments of comfortable length for the annotator, reducing their fatigue and potential labelling errors. another issue than the scarcity of labelled data constrains the models that can be used within this active learning paradox, due to the local nature of the articles. entities being often specific to the local context, making use of external data is not an option, also because of ownership and trust issues. furthermore, relation types vary greatly, from one journalist's interests to the other, requiring the ability to quickly rebuild models from very limited data. on top of these general concerns, in articles, a large number of entities mentioned co-occur but are not actually related. to eliminate such fallacious cases is not straightforward, due to the complex language style and numerous entities mentions within articles. we therefore develop an active learning approach to relation extraction in the newspapers' articles to deal with the scarcity and cost of labelled data. with the underlying idea that two light models are more likely to be accurately trained with a limited amount of data in an active learning scenario than a single large end-to-end model, we separate the task of detecting a relation from the classification of said relation. a first lstm-based model specializes in detecting the fallacious candidates by outputting whether two entity mentions within a sentence are related or not, letting a second classification model focus on subtle differences between relations instead of having to clean the data at the same time. we aim to find, in this active learning on newspaper context and with the data particularities outlined above, whether a complex state-of-the-art classification model is relevant, or whether a shallow approach is better suited. our goal is also to find an active learning strategy that reaches satisfactory results the fastest, so as to reduce human annotation. in the following, we describe the architecture of the whole system and detail the different models used. a first experiment compares the performances of three classification models as data becomes available, therefore checking the amount of data at which a more complex model is better suited than a simpler, lighter model. in a second experiment, we compare three active learning scenarios with a fixed pipeline consisting of the lstm-based relation detection model and a c-gcn classification model. we aim at finding a cost-effective active learning within our framework so as to minimize the amount of annotations needed. we therefore contribute to a study of several active learning scenarios to fit our newspaper use-case with specific, unbalanced local data, using machine learning models with different levels of depth. we notably study the relevance of very deep learning models in such a data-scarce scenario. ## related works all the outlined particularities of the data in our regional newspaper articles render most of the stateof-the-art work in the field of relation extraction inapplicable directly to this scenario, although relation extraction has excellent state-of-the-art models. for instance, even if open information extraction @xcite @xcite revolving around the identification then extraction of potential entities (noun groups) and potential relations between them, is a breakthrough in treating large-scale corpora, this method relies on the extraction of nominal groups, which does not link to entities of the newspaper's knowledge graph, and may lead to the identification of some relations that make sense grammatically, but have no meaning semantically. in unsupervised relation extraction @xcite , most works cluster linguistic patterns if two given entities co-occur a sufficient number of times. used directly, these clusters' usability is limited, as they have to be studied and labelled by hand. preemptive information extraction @xcite uses such clusters of candidate relations as high-precision seeds that feed a second, semi-supervised model. those methods require numerous documents with redundant entities pairs and linguistic patterns, otherwise seeds might be corrupted and the semi-supervised model experiences semantic drift. finally, stateof-the-art supervised models range from learning on syntax trees and hand-crafted features based on dependency parsing @xcite @xcite @xcite to deep learning @xcite . most of the latest approaches are building on the computationally heavy and dataintensive transformer model ( devlin et al., 2019), such as yamada et al. (2020); baldini soares et al. (2019) ; @xcite . all the aforementioned approaches thus fall within one of two categories: either they are heuristic, count-based approaches, that do not work on our very local data as journalists tend to avoid redundancy in their writing; or they are learning-based, which requires large amount of labeled data or pre-existing knowledge which we do not have. the active learning paradigm is a way to address the shortage of annotated data as well as the evolution of the needs, such as the addition of new relation types, by selecting most helpful samples to train the model: the main goal is to find the best strategy for the selection of said samples. in active learning, a small number of samples is chosen, labelled by an oracle and used to optimally train the model, the process repeating until a stopping criterion is met.schröder and niekler (2020) point out the conflicting paradigms of deep learning and active learning: deep neural networks excel, but under the strict requirement that abundant data be available, which defeats active learning's frugality objective. uncertainty sampling has been shown to be adaptable to deep classifiers in @xcite for nlp, and we select this approach, as it only relies on the distribution of probabilities of the sample once it has run through the model. therefore, there is little additional cost to creating an uncertainty-based sample to label, and many models can be used in these framework, as long as they output such probability distribution. for deep learning, @xcite or @xcite report no improvement of uncertainty sampling in image classification scenario, while siddhant and lipton (2018) find that both uncertaintybased sampling and bayesian approaches outperform random across 3 nlp tasks. our aim is to find how these results transfer on real-life data, and whether deep learning is truly an improvement over shallower approaches in this context. ## methodology and protocol we present the global architecture of the active learning relation detection system that we study before providing details on the different classifiers. in the following, several working hypotheses have been made to reduce noise on our particular data. we justify them, and present the specifics of the use-case, as well as the structure of the iterative system that we implemented. ## conclusions in this work, we reported on an experiment to develop an automatic relation extraction system adapted to our newspaper data in a context of scarcity of labelling, exploring various strategies towards the best possible setting. we got confronted to the divergence of objectives between deep learning and active learning, showing that shallower approaches are a safer bet in a context where labelled data is acquired via active learning. although such simple models do reach their limits early, and deep learning consistently breaks records in the literature, deep architectures require more data to train than can be supplied through active learning with a real human as an oracle. we will therefore steer our future work in the direction of shallow classifiers with a small amount of weights to train. given the small amount of data available, all linguistic information might need to be used, such as partof-speech tags, semantic roles or dependency tags, not solely relying on contextualized word vector representation that either takes time to train or has to be obtained via an external source. additionally, adopting a hand-made query strategy relying on sampling from each predicted relation to reduce the cost of annotation was found to be a small improvement compared to using only a small training set. still, the increase in performance needs to be checked in the light of a classification model better suited for active learning. besides, in order to be deployed, our system still needs to be able to take into account new types of relations. while we already store these "autre" relations during the annotation phase, we are yet to incorporate them into the classification model, in order to create a system that evolves with the content of the newspaper.