FAMIE : A Fast Active Learning Framework for Multilingual Information Extraction (2022)

Abstract: This paper presents FAMIE, a comprehensive and efficient active learning (AL) toolkit for multilingual information extraction. FAMIE is designed to address a fundamental problem in existing AL frameworks where annotators need to wait for a long time between annotation batches due to the time-consuming nature of model training and data selection at each AL iteration. This hinders the engagement, productivity, and efficiency of annotators. Based on the idea of using a small proxy network for fast data selection, we introduce a novel knowledge distillation mechanism to synchronize the proxy network with the main large model (i.e., BERT-based) to ensure the appropriateness of the selected annotation examples for the main model. Our AL framework can support multiple languages. The experiments demonstrate the advantages of FAMIE in terms of competitive performance and time efficiency for sequence labeling with AL. We publicly release our code ( https://github.com/nlp-uoregon/famie ) and demo website ( http://nlp.uoregon.edu:9000/ ). A demo video for FAMIE is provided at: https://youtu.be/I2i8n_jAyrY

## introduction information extraction (ie) systems provide important tools to extract structured information from text @xcite @xcite @xcite . at the core of ie involves sequence labeling tasks that aim to recognize word spans and semantic types for some objects of interest (e.g., entities and events) in text. for example, two typical sequence labeling tasks in ie feature named entity recognition (ner) to find names of entities of interest, and event detection (ed) to identify triggers of specified event types @xcite . despite extensive research effort for sequence labeling @xcite @xcite , a major bottleneck of existing ie methods involves the requirement for large-scale human-annotated data to build high-quality models. as annotating data is often expensive and time-consuming, large-scale labeled data is not practical for various domains and languages. to address the annotation cost for ie, previous work has resorted to active learning (al) approaches @xcite where only a selective set of examples are annotated to minimize the annotation effort while maximizing the performance. starting with a set of unlabeled data, al methods train and improve a sequence labeling model via multiple human-model collaboration iterations. at each iteration, three major steps are performed in order: (i) training the model on the current labeled data, (ii) using the trained model to select the most informative examples in the current unlabeled set for annotation, and (iii) presenting the selected examples to human annotators to obtain labels. in al, the number of annotated samples or annotation time might be limited by a budget to make it realistic. unfortunately, despite much potentials, existing al methods and frameworks are still not applied widely in practice due to their main focus on devising the most effective example selection algorithm for human annotation, e.g., based on the diversity of the examples @xcite and/or the uncertainty of the models @xcite @xcite . training and selection time in the first and second steps of each al interaction is thus not considered in prior work for sequence labeling. this is a critical issue that limits the application of al: annotators might need to wait for a long period between annotation batches due to the long training and selection time of the models at each al iteration. given the widespread trend of using large-scale pre-trained language models (e.g., bert), this problem of long waiting or training/selection time in al can only become worse. on the one hand, the long idle time of annotators reduces the number of annotated examples given an annotation budget. further, the engagement of annotators in the annotation process can drop significantly due to the long interruptions between annotation rounds, potentially affecting the quality of their produced annotation. in all, current al frameworks are unable to optimize the available time of annotators to maximize the annotation quantity and quality for satisfactory performance. to this end, we demonstrate a novel al framework (called famie) that leverages large-scale pretrained language models for sequence labeling to achieve optimal modeling capacity while significantly reducing the waiting time between annotation rounds to optimize annotator time. instead of training the full/main large-scale model for data selection at each al iteration, our key idea is to train only a small proxy model on the current labeled data to recommend new examples for annotation in the next round. in this way, the training and data selection time can be reduced significantly to enhance annotation engagement and quality. an important issue in this idea is to ensure that the examples selected by the proxy model are also optimal for the main large model. to this end, we introduce a novel knowledge distillation mechanism for al that encourages the synchronization between the proxy and main models, and promotes the fitness of selected examples for the main model. to update the main model with new annotated data for effective distillation, we propose to train the main large model on current labeled data during the annotation time, thus not adding to the waiting time of annotators between annotation rounds. this is in contrast to previous al frameworks that leave the computing resources unused during annotation time. our approach can thus efficiently exploit both human and computer time for al. to evaluate the proposed al framework famie, we conduct experiments for multilingual sequence labeling problems, covering two important ie tasks (i.e., ner and ed) in three languages (i.e., english, spanish, and chinese). the experiments demonstrate the efficiency and effectiveness of famie that can achieve strong performance with significantly less human-computer collaboration time. compared to existing al systems such as ac-tiveanno @xcite and paladin @xcite , our system famie features important advantages. first, famie introduces a novel approach to reduce model training and data selection time for al via a small proxy model and knowledge distillation while still benefiting from the advances in large-scale language models. second, while previous al systems only focus on some specific task in english, famie can support different sequence labeling tasks in multiple languages due to the integration of our prior multilingual toolkit trankit @xcite to perform fundamental nlp tasks in 56 languages. third, in contrast to previous al systems that only implement one data selection algorithm, famie covers a diverse set of al algorithms. finally, famie is the first complete al system that allows users to define their sequence labeling problems, work with the models to annotate data, and eventually obtain a ready-to-use model for deployment. ## system description in al, we are given two initial datasets, a small seed set of labeled examples d 0 = {(w, y)} and an unlabeled example set u 0 = {w} (the seed set d 0 is optional and our system can work directly with only u 0 ). for sequence labeling, models consume a sequence of k words @xmath0, w 2 , . . . , w k ] (i.e., a sentence/example) to output a tag sequence @xmath1, y 2 , . . . , y k ] (y i is the label tag for w i ). the tag sequence is represented in the bio scheme to capture spans and types of objects of interest. a typical al process contains multiple rounds/iterations of model training, data selection, and human annotation in a sequential manner. let d and u be the overall labeled and unlabeled set of examples at the beginning of the current t-th iteration (initialized with d 0 and u 0 ). at the current iteration, a sequence labeling model is first trained on the current labeled set d. a sample selection algorithm then employs the trained model to suggest the most informative subset of examples u t in u (i.e., u t ⊂ u ) for annotation. afterwards, a human annotator will provide labels for the sentences in the selected set u t , leading to the labeled examples d t for u t . the labeled and unlabeled sets can then be updated via: d ← d ∪ d t and u ← u \ u t . ## usage detailed documentation for famie is provided at: https://famie.readthedocs.io/ . the codebase is written in python and javascript, which can be easily installed through pypi at : https: //pypi.org/project/famie/ . initialization. to initialize a project, users first choose a data selection strategy and upload a label set to define a sequence labeling problem. next, the dataset u with unlabeled sentences should be submitted. famie then allows users to interact with the models and annotate data over multiple rounds with a web interface. also, famie can detect languages automatically for further processing. annotating procedure. given one annotation batch in an iteration, annotators label one sentence at a time as illustrated in figure 3 . in particular, the annotators annotate the word spans for each label by first choosing the label and then highlighting the appropriate spans. also, famie designs the size of the annotation batches to allow enough time to finish the training of the main model during the annotation time at each iteration. output. unlike previous al toolkits which focus only on their web interfaces to produce labeled data, famie provides a simple and intuitive code interface for interacting with the resulting labeled dataset and trained main models after the al processes. the code snippet in figure 4 presents a minimal usage of our famie python package to use the trained main model for inference over new data. this allows users to immediately evaluate their models and annotation efforts on data of interest. 4 evaluation datasets and hyper-parameters. to comprehensively evaluate our al framework famie, import famie # access a project via its name @xmath2.get_project('newproject') # access the project's labeled data @xmath3.get_labeled_data() # access the project's trained target model @xmath4.get_trained_model() # make predictions with the trained model @xmath5.''' @xmath6.predict(doc) print(output) # [('nick', 'b-person'), ('is', 'o'), ('happy', 'o'), ('. ', 'o')] 1 2 3 4 5 6 7 8 9 10 11 12 13 we conduct experiments on two ie tasks (i.e., ner and ed) for three languages using four datasets: conll03-english @xcite and conll02-spanish @xcite for ner, and ace-english and ace-chinese for ed (i.e., using the multilingual ace-05 dataset @xcite @xcite ). the conll datasets cover 4 entity types while 33 event types are annotated in ace-05 datasets. we follow the standard data splits for train/dev/test portions for each dataset @xcite @xcite . for the main target model m main , the full-scale xlm-roberta large model is used as the encoder. our framework for al thus inherits the ability of xlm-roberta to support more than 100 languages. also, we employ the compact minilm architecture (distilled from the pre-trained xlm-roberta) for the proxy model m prx . in all experiments, the main model is trained for 40 epochs while the proxy model is trained for 20 epochs at each iteration. we use the adam optimizer with batch size of 16 and learning rate of 1e-5 to train the models. we follow the al settings in previous work to achieve consistent evaluation @xcite @xcite . specifically, the unlabeled pool is created by discarding labels from the original training data of each dataset; 2% of which (∼ 242 sentences) is selected for labeling at each iteration for a total of 25 iterations (examples of the first iteration are randomly sampled to serve as the seed d 0 ). the annotation is simulated by recovering the ground-truth labels of the corresponding instances. the model performance is measured on the test datasets by taking average over 3 runs with different random seeds. ## related work despite the potential of al in reducing annotation cost for a target task, most previous al work focuses on developing data selection strategies to maximize the model performance @xcite @xcite @xcite . as such, previous al methods and frameworks tend to ignore the necessary time to train models and perform data selection at each al iteration that can be significantly long and hinder annotators' productivity and model performance. to make al frameworks practical, few recent works have attempted to minimize the model training and data selection time by leveraging simple and non state-of-the-art architectures as the main model, e.g., activeanno @xcite and @xcite . however, an issue with these approaches is the inability to exploit recent advances in large-scale language models to achieve optimal performance. in addition, some recent works have also explored large-scale language models for al @xcite ; however, to reduce waiting time for annotators, such methods need to exclude the training of the large models in the al iterations or employ small models for data selection, thus suffering from a harmful mismatch between the annotated examples and the main models @xcite . ## conclusion we introduce famie, a comprehensive al framework that supports model creation and data annotation for sequence labeling in multiple languages. famie optimizes the annotators' time by leveraging a small proxy network for data selection and a novel knowledge distillation to synchronize the proxy and main target models for al. as famie is task-agnostic, we plan to extend famie to cover other nlp tasks in future work. minh-quoc nghiem, paul baylis, and sophia ananiadou. 2021. paladin: an annotation tool based on active and proactive learning. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: system demonstrations, pages 238-243, online. association for computational linguistics. minh van nguyen, viet lai, and thien huu nguyen. 2021a. cross-task instance representation interactions and label dependencies for joint information extraction with graph convolutional networks. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 27-38, online. association for computational linguistics. minh van nguyen, viet dac lai, amir pouran ben veyseh, and thien huu nguyen. 2021b. trankit: a light-weight transformer-based toolkit for multilingual natural language processing. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: system demonstrations. thien huu nguyen and ralph grishman. 2015. event detection and domain adaptation with convolutional neural networks. in the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing. thien huu nguyen and ralph grishman. 2018. graph convolutional networks with argument-aware pooling for event detection. in proceedings of the aaai conference on artificial intelligence. trung minh nguyen and thien huu nguyen. 2019. one for all: neural joint modeling of entities and events. in proceedings of the aaai conference on artificial intelligence.