Generating unlabelled data for a tri-training approach in a low resourced NER task (2022)

Abstract: Training a tagger for Named Entity Recognition (NER) requires a substantial amount of labeled data in the task domain. Manual labeling is a tedious and complicated task. Semisupervised learning methods can reduce the quantity of labeled data necessary to train a model. However, these methods require large quantities of unlabeled data, which remains an issue in many cases.

We address this problem by generating unlabeled data. Large language models have proven to be powerful tools for text generation. We use their generative capacity to produce new sentences and variations of the sentences of our available data. This generation method, combined with a semi-supervised method, is evaluated on CoNLL and I2B2. We prepare both of these corpora to simulate a low resource setting. We obtain significant improvements for semisupervised learning with synthetic data against supervised learning on natural data.

## introduction training models to solve ner tasks requires a considerable amount of labeled data. in most nlp tasks, this data needs to be related to the task domain and must be in the targeted language. while english is a well-covered language, corpora are still being built to cover new domains or expand existing ones. for any other languages, corpora cover fewer domains. data in the private sector is rarely shareable due to privacy reasons. it is also the case in domains such as the medical domain. recent approaches tackle the issue of the absence of resources by leveraging knowledge or data from other sources. zero-shot learning is a learning paradigm trying to solve a target task without any labeled data. it uses the knowledge of how to predict labels of an adjacent task and applies it to predict the unseen labels of the target task @xcite . we do not aim to solve the ner problem in a situation with such strict data restrictions. labeling a few examples is almost always possible. few-shot learning provides training methods to generalize from a few labeled examples. these methods use the labeled examples to build representations of the class, which serve as comparison points for inference @xcite . transfer learning leverages the knowledge learned on tasks of the domain to improve the performance on a specific task @xcite . it is quite common to see cross-lingual transfer from higher-resourced languages where the task exists. however, the most prominent use case of transfer learning in nlp is the use of language models for data representation. we use this type of transfer learning to build high-performing taggers from bert models. semisupervised learning is a paradigm where unlabeled data is widely available. the unlabeled data is used to improve the model's performance by giving a better topology of the data space. we propose to use a semi-supervised learning method in a context where data is scarce enough to be fully labeled. we aim to achieve this by using large language models to generate the necessary unlabeled data. we test whether large language models can generate data that make tri-training a viable option in a low-resource context. the performances of our baseline models are compared against the performances of the ensembles of models trained with tri-training on conll @xcite and i2b2 @xcite . significant improvements are observed using our method on the reduced datasets. language modeling has already been used as an augmentation method to generate labeled and unlabeled examples for ner in daga @xcite . however, our taggers overperform the taggers presented on the gold standard by 30 points at size 1000 and 9 points at full size. the semisupervised method used in daga, self-training, is also prone to errors due to reinforcement of early mistakes. in our case, we generate unlabeled sen-tences using pre-trained large language models. we test this method with subsets of data ranging from 50 examples to 1000 examples vs. over 1000 in daga. thus, our main contribution is using out-of-thebox large language models as tools to obtain unlabeled data for semi-supervised learning in ner in a low-resource setting. the code relative to the experiment will be available in a public repository 1 . section 2 presents state of the art related to data augmentation, semi-supervised learning in ner, and language modeling. section 3 presents tritraining @xcite , and how we fit generation into it. section 4 touches on the technical details of the experiments. section 5 and 6 are the discussion and the conclusion of the article. ## related works learning models in a low-resource setting require extracting every possible information from the available data. data augmentation is a common technique that creates synthetic data from available data. in natural language processing, augmentation is used across various tasks to help achieve better performances. in classification, techniques such as back-translation @xcite or easy data augmentation @xcite are used. however, in tagging, paraphrasing using back-translation @xcite is not bringing significant improvements. recent works show that using language models learned on the training data to generate labeled and unlabeled examples can bring improvements @xcite . inductive semi-supervised learning (van engelen and hoos, 2020) aims at improving the performances of models through the addition of unlabeled data. for named entity recognition, pseudolabeling is a method that has been used @xcite . pseudo-labeling is one of the semisupervised learning methods. the unlabeled data receives pseudo-labels from the models trained. this pseudo-labeled data is then used alongside labeled data to train the models. variants of the method exists @xcite @xcite with varying quantities of models trained. the separation of the data between the different models trained and how the models are used to produce pseudo-labels also creates variants to this method. in our case, 1 https://github.com/hugoboulanger/ tritraining-gen we use tri-training @xcite , which uses three models. this method has been used to solve clinical concept extraction in the medical domain @xcite on new data. semi-supervised learning methods still require a significant amount of unlabeled data. however, with current advances in language modeling, this method could be improved. transformer-based models @xcite have been a revolution in the language modeling landscape. from their first iterations like gpt @xcite to their more recent ones like t5 @xcite and gpt-3 @xcite , transformerbased models have become a staple of natural language processing as fine-tuning or transferring knowledge from these models often outperforms learning a model on the task directly. while our taggers are based on bert models @xcite , we otherwise use the generative power of gpt2 @xcite to provide unlabeled data for the semi-supervised training. gpt2 has been finetuned and used to generate unlabeled data for classification in a high resource context @xcite . ## methods this section provides details on the tri-training process for sentence tagging and how we levy language modeling as an unlabeled data provider. ## experiments in this section, we describe the technical details of the experiments and explain the variants tested. ## discussion while our low-resource setting allows us to compare the impact of the training method in an otherwise similar context, it does not fully represent the nature of the problem. building the development and test set is also a low resource problem. reducing the test set to simulate low-resource will only make any comparison meaningless. simulating the development set in the low resource context is an improvement that could be made. it is also to note that while the application domain is low resource, it is necessary to have a sizeable open-domain language model in the target language. trying this method in languages other than english must be tested. multilingual models might be the solution to the generalization of this method. as it stands, availability of large language model is the hardest limitation of this method. ## conclusion leveraging pre-trained models to improve performances on specific tasks is a common approach. with recent improvements to language modeling, recent models are often used directly to solve tasks. direct usage is the method we use to build our taggers. however, we propose a new use for these sizeable models. they can serve as unlabeled data generators for semi-supervised learning. in particular, we have shown that we can use this method to gain significant improvements to the performances of taggers on ner and clinical concept extraction in a low resource context. we gain between 3 and 4 points of f 1 score on subsets of data of size 50. gains are overall positive on the sizes of the subsets we have tested. the higher the gains, the lower the data size is. we have shown that large language models are suitable tools to generate unlabeled examples for semi-supervised learning for ner.