Entity-Controlled Synthetic Text Generation using Contextual Question and Answering with Pre-trained Language Models
Abstract
Recent advancements in Natural Language Processing (NLP) algorithms have resulted in state-of-the-art performance on Named Entity Recognition (NER) tasks.
These algorithms typically require high-quality labeled datasets for training models.
However, training NLP models effectively can suffer from issues such as scarcity of labeled data, data bias and under-representation, and privacy concerns with using sensitive data for training.
Generating synthetic data to train models is a promising solution to mitigate these problems.
We propose a contextual question and answering approach using pre-trained language models to synthetically generate entity-controlled text.
Entity-controlled text generation is then used to augment small labeled datasets for downstream NER tasks.
We evaluate this proposed method on two publicly available datasets, and measure the quality of generated texts quantitatively.
We find that the model is capable of producing full text samples with the desired entities appearing in a stochastically controllable way, while retaining sentence coherence closest to the real world data.
Evaluations on downstream NER tasks show significant improvements in low-labeled data regime, and in using purely synthetic data for NER to alleviate privacy concerns.
Introduction
Many tasks in NLP require large amounts of high-quality labeled data to train sufficiently accurate and useful models.
However, in many domains, such as finance and healthcare, access to labeled data is often limited.
In these domains, annotating data often requires strong domain expertise and therefore, crowdsourcing of labeled data is infeasible.
The cost of annotating data by training an expert workforce is often too high for feasibility.
Even if it were financially feasible to annotate data, there are concerns with using customer data for training large language models and potentially endangering customer privacy.
Recent studies have raised concerns about leakage of training data (potentially sensitive information) from the trained language models [2, 9, 11].
A small collection of labeled data also runs the risk of bias creeping in the data and may result in algorithms and models that reflect or even exploit this inherent bias.
It also degrades the capability of models to generalize since they may have been trained on a small dataset where certain population groups or patterns were under-represented [19, 28, 27, 7].
These issues demand solutions that can perform well in low-labeled data regimes and can combat privacy concerns and data bias.
Synthetic data generation presents a promising solution to address the issues outlined above [1, 5].
By generating data synthetically, we can augment small labeled datasets to build a training set large enough for efficient learning of large models.
Synthetic data generation also promotes privacy by hiding sensitive customer data.
Synthetic data generation can also reduce bias in the data by conditionally generating it in a way that all population groups are sufficiently represented.
In particular, the field of conditional or controlled synthetic text generation has received increased attention in recent years.
Controlled text generation provides the ability to control for traits such as tone, formality, sentiment, and topic in the generation of a language model [23, 26].
This lends controlled synthetic text generation as a useful technique for augmenting small or privacy-sensitive datasets.
However, there has been limited work on the topic of entity-controlled synthetic text generation, i.e., the task of generating coherent text while controlling for the named entities that appear in the generation [8].
In this paper, we study the problem of entity-controlled synthetic text generation.
We propose a
Contextual Question Answering based pre-trained language model that can produce coherent text which contains specific entity tokens, generated in an order provided by the user.
We are motivated by the need to synthetically augment datasets to improve performance on downstream NER tasks.
Our contributions in this work are as follows.
We propose a contextual question and answering approach using pre-trained language models to generate entity-controllable blocks of text, which can be chained to produce full training text samples, 2) Our method is capable of generating coherent texts that beat the baseline methods in grammaticality and distinctness metrics, and 3) Evaluations on publicly available NER datasets show a significant improvement in performance in low-labeled data regimes, and for scenarios consisting of purely synthetic training data to promote privacy.
Related Work
There has been limited work in the area of entity-controlled text generation.
We can group prior works into: Controlled text generation, data-to-text generation, and entity-controlled text generation.
Controlled text generation These methods have been designed to control certain aspects of generated text [25, 3, 16] like sentiment [23] or concepts [26].
Furthermore, they have been used in summarization [20] tasks as well.
These methods however, mostly focus on changing one aspect of the generated text like a topic or sentiment.
Our goal is to generate a coherent text based on the set of entities we want to appear in the text.
Data-to-text generation This is a well studied class of problems where the idea is to convert a given set of words or structured data from tables into a piece of text.
Most popular problem is table summary generation, also called table-to-text [14, 15, 4] or keyword to text methods [16, 21].
While similar to our problem, the key difference is that they have a pre-defined set of words that just need to appear in every generated text while we can have a variable number of entities that appear in each generated text.
For example, table-to-text methods would be given a set of four words per row belonging to one entity type, while we need to generate any number of entities (even repeated) as they occur in most real world corpus.
Entity-controlled generation To the best of our knowledge, only Dong et al.
[8] have worked on this problem.
They use a two pronged approach to generate text with given entity types and their mentions.
The entity predictor, is used for generating an entity tag, indicating that an entity must be injected at a particular location in the text.
When these tags are generated, the desired entity type is injected and the second stage using the mention predictor generates the specific entity mention (word instantiation) that belongs to the entity tag.
They use a RNN based sequence-to-sequence architecture to achieve this.
We make a comparison with their method and found that their method generated repetitive text, and does not generate as realistic text as ours.
Additionally, they do not make a comparison on a downstream task, while we present an analysis on downstream NER task.
Methodology
Our methodology uses a pre-trained language model and a question and answering training approach to generate blocks of text with desired entity tokens.
This approach is able to reliably generate augmented text samples while retaining sentence coherence.
Our method builds off the work of Dong et al.
by training on blocks of text and chaining such blocks to generate text samples.
In addition, we opt to use a pretrained transformer-based language model in place of a recurrent network to take advantage of the benefits of using finetuned and pre-trained language models.
We expect that using the pre-trained model helps in diversity of the generated text.
3.1 Training
To use the approach of question and answering, we first preprocess our real world training text samples into blocks, whereby each block is composed of non-entity tags and ends with an entity token.
Every text sample is then decomposed into these blocks of text.
In addition to the entity tokens in the training corpus, an end of text token is also added to the end of every text sample.
Therefore, a full text sample generation consists of chaining generated blocks until a block with an <ENDTEXT>
token appears.
After decomposing text samples into such blocks, we arrange blocks into format block the desired generation.
The context section consists of all blocks preceding a particular block belonging to the same text sample.
This was motivated by the need for the model to be aware of the context for a particular generation.
The generation of each block must be a continuation of preceding blocks in order for sentence level coherence to be maintained.
Consequently, training the model on preceding blocks, provides a signal for the text’s context in order for the generation in the answer segment to be a seamless continuation of the preceding blocks.
The question segment prompts the model for the desired entity to appear in the next block, and is therefore the mechanism by which we control for the desired entity token to be generated.
Following the "Question: " tag is a single token representing the desired entity.
The answer segment contains the desired text block to be generated.
The final token in this block will therefore be the same token as in the question segment.
With this three segment format, every block from the corpus represents a training sample for the language model.
Figure 2 illustrates this three segment structure of each training sample.
3.2 Generation during Inference
At inference time, the language model generates text conditioned on the first two segments of context and question.
To generate the first block of a text, the context segment is blank, while the question segment contains the desired token to be generated in the first block.
The model then completes the answer segment with a generated block, which is inserted into the context segment for the next block generation.
A full text sample then is produced by concatenating blocks until an <ENDTEXT> token.
3.3 Metrics
To evaluate the model, we quantitatively measure the quality of generation and then measure the performance on NER task using precision, recall, and F 1 scores.
We use three generation quality metrics that have been used in the prior works [8].
Grammaticality [24] measures the grammaticality probability evaluated with a Roberta CoLa grammaticality model.
Perplexity measures the ‘surprisingness’ of the generated text evaluated on a GPT model [17].
Distinctness [12] measures the uniqueness of trigrams in the corpus.
Rouge-L [13]: One trivial sanity check is regurgitation.
, i.e., if the generation model is simply memorizing the training set.
Rouge-L score measures the similarity of the generated text with the training data by calculating the longest common substrings.
Ideally, if the model is not just spitting out the training examples, Rouge-L score should be low.
[Table]
Experiments
We evaluate our model on two public datasets described in Table 1.
We use the following baselines to compare our method with the following three baselines:
Original Data: Refers to BERT model trained using the real world training data
Random Swap [22]: We randomly swap entity mentions across the corpus following Vakili et al.
[22], e.g., for the entity ‘B-DNA‘, we replace with another randomly chosen mention of ‘B-DNA‘ in our training data.
Since, we are following a similar procedure to instantiate a generated entity post generation in our model, we use this as a simple baseline.
Additionally, it acts as a sanity check during augmentation, as the training corpus hasn’t changed much except reshuffling of entity mentions.
EntInjection [8]: We use the Entity Injection work by Dong et al [8] as that is the only relevant work closest to our work.
4.1 Experimental Settings
We use the training, validation, and testing data splits provided publicly in the datasets on Huggingface 2 .
We use the training dataset (and its mentioned subsets) for training both the text generation models as well as training the downstream NER model.
We use BERT [6] for downstream NER task.
We measure the generation quality metrics on the generated text from the training dataset.
NER results are reported on the complete test set for both the datasets.
For our pre-trained language model, we use an instance of OpenAI’s GPT-2 [18].
The model is trained with the Adam optimizer on a learning rate of 1e-3, one hundred warmup steps, and an epsilon of 1e-8.
The default CrossEntropy loss function is used, and the model is trained for up to 100 epochs.
For the NER task, we train the BERT model for upto 10 epochs with a learning rate of 2e-3.
These parameters were set based on hyper-parameter tuning on the validation set.
During generation, we exactly mimic the entity distribution of training data samples.
Generation quality results are shown in Table 2 measured on Grammaticality, Perplexity, Distinctnesstrigram.
We clearly observe that our method is lower on all three metrics against the original dataset, which is expected as ours is synthetically generated data. However, our method works better than the baseline EntInjection [8] on all three metrics across both the datasets. Particularly, for the BC5CDR
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
dataset, we observed that EntInjection tends to generate repetitive text.
The correct benchmark is the random swap as our method inserts the entitities in the same fashion .
We observe for the random swap baseline, distinctness is highest, as expected as we have swapped commonly occurring trigram entities, while the perplexity and grammaticality are worse than all the methods.
This shows that random swapping affects the lexical meaning of the text.
While we also insert randomly chosen entities in our generated text, these results indicate that our method generates coherent generic text where semantic meaning of the type of the entity is preserved, unlike other baselines.
Our generated data has one of the lowest Rouge-L scores across the two datasets.
Hence, our generated data is not simply memorizing the training data and is quite different than the original training data.
We can see the huge gap between the generated data through random swapped entities and our generated data; while the former is practically same as the training data, ours is distinct.
This is quite important for privacy, as this ensures that an adversary would not be able to extract the original training data trivially from the generated data.
Based on these metrics, we can claim that generated text is semantically closest to the original corpus for all the datasets, while being distinct.
5.2 Named Entity Recognition Task
We took two subsets of the JNLPBA and BC5CDR datasets: 1% and 10% as we found that the performance on datasets was already saturated at their full sizes as number of samples was enough.
We didn’t find any difference in performance between BERT model trained on the original training dataset, synthetically generated dataset, and augmented dataset with our generated examples.
Hence, we present the results on first 1% and 10% examples of training datasets to show the comparisons.
We present two settings: (a) w/o augmentation with original training data; and (b) augmentation with original training data.
Idea for (a) is to test for privacy-preserving training, and (b) tests effectiveness of the generated data for data augmentation purposes.
Generated text is same size as the training set.
Table 3 shows the results for the two subsets of the two datasets.
From the results four things stand out: 1) Augmenting original data with our synthetically generated data always out-performs a model trained with the original data; 2) using only synthetically generated data is comparable in performance to the original data in medium labeled data setting (10%) subsets; 3) our synthetically generated data outperforms original data in low labeled data setting (1%) subsets; and 4) our synthetically generated data gives better performance vs two baseline methods: random swap [22] and EntInjection [8].
Our finding that using synthetically generated data can get us a comparable performance to the model trained on real data has an application in making the models trained for downstream tasks like NER, privacy preserving as they are not trained on the real data.
This makes it difficult for the model to leak sensitive data [11, 2].
Our results show our method of generation can be quite effective as a data augmentation method in a low labeled data regime .
5.3 Ablation: Generating more text in low resource setting
In the previous results, we only showed the results by generating synthetic text of the same size as the training data.
Next, we perform an experiment to see if there is further improvement in the performance as we add more generated text.
We take the JNLPBA (1%) dataset, and generate more text using Random Swap and our method.
We observe that the results keep improving as measured by the Macro Average F 1 score, with augmentation going up to 0.70, and without augmentation going to 0.64 vs baseline at 0.31.
Note, we only use the entity mentions found in the JNLPBA (1%) dataset to fill in the entity tags in the generated text.
This is more remarkable considering that a model trained on 10x real data for JNLPBA (10%) has a Macro Average F 1 score of 0.72.
This evidence shows that our model is able to generate text that is similar to the real data.
Conclusion and Future Work
Synthetic data generation is a promising approach to train large language models in order to deal with scarcity of labeled data, privacy concerns, and data bias.
In this work, we study the problem of conditional text generation where the conditions are provided as a list of entities that must appear in the text in a manner desired by the user.
We propose a contextual question and answering approach using pre-trained language models that can generate blocks of text conditioned on the desired entities.
We test our generation system on various generation quality metrics as well as on NER tasks.
Evaluations show that our proposed method outperforms baselines in terms of both generation quality and NER performance.
We also achieve comparable performance relying solely on synthetic data, showing that our proposed architecture can preserve privacy.
In future, we will extend this work on more datasets, and explore cross-domain data generation strategies to generate of out-of-distribution data.
A Appendix A.1 Examples of Generated Text
In the section below we shows few examples of generated text by our method and EntInjection [8] method.
Our method generates semantically meaningful examples, while EntInjection generates quite repetitive examples.
Text highlighted in Green marks the entities.
A.1.1 Ours
The examples below seem grammatically correct, as was the observation over the entire generated corpus.
However, as we randomly insert entity mentions after we generated the entity tags, most of the generated examples are not factual.
E.g., DTG is not associated with treatment of blood clotting as generated in the first example.
Our goal was not factual correctness but ensuring that the generated data preserves the distribution of the training data, which seems to be the case based on generation metrics and results on NER task.
The efficacy of DTG in the treatment of impaired blood clotting likewise did not appear to be affected by the rate of administration, although no formal statistical comparisons were made .
The prevalence rate for death was the most important reason for preference, cited by 67 .
3 % of patients preferring Picloxydine and 54 .
2 % of patients who preferred a p < or = 0 .
The reduction of acetaminophen at 1 and 4 days after gestation not glomeruli with ataxic movements than control rats .
The aims of this study were to confirm our previous findings in a separate cohort of patients and to determine the time course of the cardiovascular consequences of stopping sertraline in the expectation that this might shed light on the mechanisms by which the mechanisms by
Tamoxifen is being a significant reduction of the activity on the drug causes the sodium associated with cephalothin sodium associated with povidone - iodine is associated withcocaine and inhibition with the use of tuberculosis and area in this effect.
MR imaging with quantitative diffusion mapping of E4031 ( 0 .
g ), p - choloroaniline ) and outcome in organ transplant controls, and / L and the development of blood coagulation by a potential is also more than the development of systolic dysfunction and possibly .
[-A.1.
-]A.1.2 EntInjection [8]
We observed a lot of repetition in the generated text by EntInjection method.
This looping behavior is shown in Example 2 and 3 below.
Note, unlike our method, EntInjection has access to the exact same entity mentions as they appear in the training data, having an inherent advantage with this additional information. telithromycin - induced bromo tetrahydropyranyladriamycin pituitary carsinom agitation one : a longitudinal study of <unk> :
The cardiovascular responses to standing and standing .
4 patients were studied in the drug .
4 days .
The possibilities that these findings might be the result of non - induced <> is a result of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine oxidase or inhibition of monoamine
In the study was undertaken to the combination of painful , headache , bleed , which was only induced by epilepticus drug , and bronchitis
Investigation of anti - inflammatory agents are warranted in the caudate nucleus .
injection of
Allopurinol injection of bacterial collagenase - induced