ECG - QALM : Entity-Controlled Synthetic Text Generation using Contextual Q & A for NER (2023)

Abstract: Named Entity Recognition (NER) state-of-the-art methods requires high-quality labeled datasets. Issues such as scarcity of labeled data, under-representation of entities, and privacy concerns with using sensitive data for training, can be significant barriers. Generating synthetic data to train models is a promising solution to mitigate these problems. We propose ECG-QALM, a contextual question and answering approach using pre-trained language models to synthetically generate entity-controlled text. Generated text is then used to augment small labeled datasets for downstream NER tasks. We evaluate our method on two publicly available datasets. We find ECG-QALM is capable of producing full text samples with desired entities appearing in a controllable way, while retaining sentence coherence closest to the real world data. Evaluations on NER tasks show significant improvements (75% - 140%) in low-labeled data regimes.

## introduction nlp tasks typically require large amounts of highquality labeled data to train sufficiently accurate and useful models. however, in many domains, such as finance and healthcare, access to labeled data is often limited. in these domains, annotating data often requires strong domain expertise and therefore, crowdsourcing of labeled data is infeasible. the cost of annotating data by training an expert workforce is often too high for feasibility. a small collection of labeled data also runs the risk of bias creeping in the data and may result in algorithms and models that reflect or even exploit this inherent bias. it also degrades the capability of models to generalize as small datasets are much less likely to have population groups or patterns under-represented @xcite . these issues need solutions that can perform well in lowlabeled data regimes while combating data bias. this work was done during henry's internship at amazon synthetic data generation presents a promising solution to address the issues outlined above @xcite . by synthetically generating data, we can augment small labeled datasets to build a training set. synthetic data generation can also reduce bias in the data by to sufficiently represent all population groups. in particular, the field of controlled synthetic text generation has received increased attention in recent years. controlled text generation provides the ability to control for traits such as tone, sentiment, and topic in the generation of a language model @xcite . this lends controlled synthetic text generation as a useful technique for augmenting small or privacysensitive datasets. however, there has been limited work on the topic of entity-controlled synthetic text generation, i.e., the task of generating coherent text while controlling for the named entities that appear in the generation @xcite . in this paper, we study the problem of entitycontrolled synthetic text generation. we propose, ecg-qalm, a entity controlled text generation with contextual question answering based pretrained language model, that can produce coherent text which contains specific entity tokens, generated in an order provided by the user. we are motivated by the need to synthetically augment datasets to improve performance on downstream ner tasks @xcite . ecg-qalm provides multiple advantages. it is more sample efficient than other methods, as the model is trained on each block of each sample, unlike just seeing a sample in whole for seq2seq models like @xcite ; b) ecg-qalm sees a block of text which is relatively smaller than whole sample, prompted on entity to be inserted and conditioned on previous generation allowing for generation of more coherent text as demonstrated by generation metrics like perplexity versus sota seq2seq baselines; and c) unlike prior seq2seq methods like rnn @xcite or using a vanilla gpt, where length of text generated is limited to 512/1024, ecg-qalm can generate as many blocks of (maximum) length 1024, as the number of entities to be inserted. we make the following contributions: 1) we propose a novel approach using pre-trained language models to generate entity-controlled blocks of text, which can be chained to produce full synthetic text samples; 2) our method is capable of generating texts semantically closest to the training data while being distinct; and, 3) evaluations on publicly available datasets on ner task show a significant improvement in data augmentation performance for low-labeled data regimes, even by just using a purely synthetic data. ## related work controlled text generation these methods control a certain aspect of generated text @xcite @xcite like sentiment @xcite or concepts @xcite . these methods focus a macro level aspect of the generated text while we want to control a fine grained text generation. ## methodology we use a contextual question and answering based training approach to generate blocks of text with desired entity tags. this approach is able to reliably generate augmented text samples while retaining sentence coherence. our method generates blocks of text delimited by entities to be inserted, and chaining these generated blocks to create full text samples. we use a gpt-2 language model in place of a recurrent network used by @xcite to take advantage of using pre-trained large language models. the intuition being that using a pre-trained model helps in increasing diversity of the generated text. ## experiments we evaluate our model on two datasets described in gold data: refers to the real world training data. daca @xcite : substitution method using a lstm based language model to replace entities in the gold samples, by exploiting context. melm @xcite : substitution method using masked entity language model with xlm-roberta with linearization, for a richer context. entinj @xcite ): text generation method based on lstm seq2seq model. closest work to ours, as it performs actual text generation. ecg-lm: this is our own baseline seq2seq method, which generates the entire text given a list of entities, without a block-by-block generation. note: generated text length in daca, melm, entinj, and ecg-lm is limited by number of tokens model can generate (512/1024) at once; ecg-qalm is not, as it chains the generated blocks. ## conclusion synthetic data generation is a promising approach to train large language models in order to deal with scarcity of labeled data. in this work, we study the problem of conditional text generation where the conditions are provided as a list of entities that must appear in the text in a manner desired by the user. we propose ecg-qalm that can generate blocks of text conditioned on the desired entities. we test our generation system on generation quality metrics and ner task. evaluations show that our method outperforms baselines in terms of both generation quality and ner performance. our blockby-block generation provides significant gains over using a fine-tuned vanilla llm for generation.