Exploring Data Augmentation in Neural DRS -to-Text Generation (2024)

Abstract: Neural networks are notoriously data-hungry. This represents an issue in cases where data are scarce such as in low-resource languages. Data augmentation is a technique commonly used in computer vision to provide neural networks with more data and increase their generalization power. When dealing with data augmentation for natural language, however, simple data augmentation techniques similar to the ones used in computer vision such as rotation and cropping cannot be employed because they would generate ungrammatical texts. Thus, data augmentation needs a specific design in the case of neural logic-to-text systems, especially for a structurally rich input format such as the ones used for meaning representation. This is the case of the neural natural language generation for Discourse Representation Structures (DRS-to-Text), where the logical nature of DRS needs a specific design of data augmentation. In this paper, we adopt a novel approach in DRS-to-Text to selectively augment a training set with new data by adding and varying two specific lexical categories, i.e. proper and common nouns. In particular, we propose using WordNet supersenses to produce new training sentences using both in-and-out-of-context nouns. We present a number of experiments for evaluating the role played by augmented lexical information. The experimental results prove the effectiveness of our approach for data augmentation in DRS-to-Text generation.

## introduction data augmentation is a systematic way of increasing data examples by altering the original data with controlled variations @xcite . it is a prevalent technique in computer vision (cv) for increasing dataset size by introducing slightly different and contextually similar examples @xcite . augmentation approaches are also becoming popular in many natural language processing (nlp) applications as well. the most commonly used approaches to augment textual data are based on random swapping, random insertion, random deletions, synonyms replacement, back translation, and using generative models to get new contextaware data @xcite . notice that data augmentation in nlp is a very challenging task due to the constraint of producing a grammatical augmented text @xcite . moreover, given the continuous nature of images, in cv the augmented version of an image rarely is pragmatically incorrect. in contrast, in nlp, preserving the contextual meaning of the sentence is, usually, a hard constraint. indeed, bad model performance can be the consequence of augmented textual data that is grammatically incorrect or out-of-scope @xcite . recently, researchers working on text generation from meaning representations, i.e., graphbased abstract meaning representation (amr) @xcite or discourse representation structure (drs), have put their efforts into generating text from logical representations, and vice-versa, using transformers and encoder-decoder-based neural models @xcite @xcite @xcite . in this paper, we consider the specific problem of augmenting data in the context of neural drs-to-text generation task. drs represents textual information in the form of events, concepts, and entities, i.e., names as discourse referents usually represented as variables in drs, and logical relations between these entities i.e., quantifiers, conjunctions, negations, disjunctions, etc. @xcite @xcite . in figure 1 a graphical representation of drs in box format (on the left), its flattened version i.e., clausal format (on the right), and its corresponding textual representation (on the bottom) is displayed. neural drs-to-text generation is a type of datato-text generation task that takes the logical representation of a sentence as input and generates text as output @xcite . this is an application of text generation from structured input data similar to knowledge graphs @xcite , rdf triplets data @xcite , and tables @xcite . note that, in contrast to tables and graphs, the ability to represent the structured logical nature of the input as a drs generation allows for a more fine-grained investigation of the relation between input and output in drs-to-text. in other words "changing the meaning of a drs in a controlled way, the robustness of systems can be monitored in detail and assessed accordingly" @xcite . however, this robustness property discourages the application of large language models (llms) for augmenting data because llms would generate noise in the augmented data @xcite @xcite -see also section 4. in this paper, we exploit the robustness property of neural drs-to-text generation by designing and evaluating data augmentation for the specific categories of (i) proper nouns and (ii) common nouns. in particular, we have designed and evaluated a procedure for augmenting a drs training dataset by adding context-aware new sentences that are produced by varying the proper and common nouns in the original sentences. we consider different strategies and propose to use supersense tagging (sst) for creating new training sentences using both inand out-of-context nouns. in this way, we want to analyze the role played by lexical information in the performance of a neural drs-to-text system. the research questions and contributions addressed in this paper are: to the best of our knowledge, apart from the preliminary work on augmentation of verbs presented in @xcite , this is the first work on data augmentation in drs-to-text generation analyzing its impact on model performance. notice that our augmentation techniques could generate factually incorrect texts (e.g., starting from "at dawn, the sun rises", "at night, the sun rises" could be generated. however, since humans can generate texts that are not factually correct (consider, for example, a sci-fi story), preventing this situation would actually be not only beneficial but detrimental for the system. the statistical nature of the neural networks does not allow for an easy analysis of the kind of knowledge really learned by the system. when we provide a specific example as brad pitt is an actor, the network is learning that the verb follows the subject (e.g. grammatical competence), and/or that a man can be an actor (semantic and pragmatic knowledge), and/or that a specific man is an actor (world knowledge)? how can we exploit this multilevel nature of neural learning? a side effect of our study on data augmentation is to investigate these theoretical questions as well. the paper is structured as follows: in section 2, we describe the procedure adopted for noun augmentation; in section 3, we give architectural insights on the neural drs-to-text pipeline; in section 4, we describe the experimental results of drsto-text generation that uses (1) automatic metricsbased and pre-trained model-based evaluations on a standard test set, (2) a reduced test set comparing our neural systems with two general llms, and (3) applying both automatic and human evaluation metrics. finally, in section 5, we conclude the paper. ## logical data augmentation with nouns data augmentation is a relatively complex task in the case of neural drs-to-text: each augmented example in the training set consists of a pair of new drs structures together with a new corresponding sentence. while applying systematic transformations on training data, it is essential to keep track of both types of data representations as they are treated as input value pairs in the neural model. so, data transformations should be identical and symmetrical on both elements by considering the order of meaning representations and textual translations. in the drs-to-text generation task, we applied different augmentation techniques to augment proper and common nouns. we have used the gold version of the parallel meaning bank 1 (pmb) dataset, which is organized in the usual train-devtest split. a graphical representation of transformation for proper (highlighted in blue) and common (highlighted in green) nouns in drs is shown in figure 2 : the drs on the left generates the sentence brad pitt is an actor, while the drs on the right gener1 the pmb is developed at the university of groningen as part of the nwo-vici project "lost in translation -found in meaning" (project number 277-89-003), led by johan bos. ## experimental results evaluation with automatic metrics. we have conducted a series of different experiments that focus on analyzing the model performance based on systematic alterations in lexical semantics-based input representations. we have listed cb-bi-lstm, wb-bi-lstm, and ft-byt5 (see if we compare the overall performance of cb-bi-lstm and wb-bi-lstm, we found as expected that cb-bi-lstm always wins in all aspects of input data. this reflects the fact that the char-level model with the ability to handle oov words is performing very well in capturing micro-level aspects and data patterns of input drs. this also shows the effectiveness and morphological accuracy of the char-level model in generating correct output sequences. however, the ft-byt5 model outperforms the bi-lstm-based models in most experiments. in the proper noun augmentation, our experiments are twofold: (1) inside context and (2) outside context as discussed in section 2. exp. 2 -3, 11-12, and 20-21 (see for common noun augmentation, our experiments are fourfold: (1) inside context with ss, (2) inside context without ss, (3) outside context with ss, and (4) outside context without ss: exp. 4-7, 13-16, and 22-25 regard these four different flavors of common noun augmentation. we believe that the important role played by the vocabulary holds for common nouns as well, with the highest scores of cb-bi-lstm decoder for outside context with ss (exp. 6) and best wb-bi-lstm score for inside context with ss (exp. 13). again, the ft-byt5 shows the best cn augmentation results for outside context with ss (exp. 24) and produced the highest scores for all flavors of cn augmentation: when compared to both lstm models. finally, in exp. 8-9, 17-18, and 26-27, we have applied the best augmentation techniques of proper and common nouns (i.e., outside context for cb-bi-lstm, and inside context for wb-bi-lstm and outside context for ft-byt5) as blended and individual data examples. in exp. 8, 17, and 26, the augmentation techniques have been applied simultaneously to each input data example (i.e., as we are applying 2 data transformations on one example, we name it blended, see proper and common noun example in finally, in exp. 28, we preliminarily evaluated the impact of the size of the augmented data. so, we repeated exp. 21, by halving the size of the augmented part of the training set. the results, with scores that are intermediate w.r.t. the baseline and the best model, suggest that there is a linear increase w.r.t. the size of the augmented training set. however, more experiments are necessary to verify this hypothesis. we tested the statistical significance of the results with a wilcoxon signed rank test @xcite . ## conclusion in this paper, we have performed a series of experiments on data augmentation for neural drsto-text generation. using the pmb dataset, we adopted a novel approach to augment lexical information in drs for proper and common nouns along with the ability to sustain contextual similarly through ss approach on different in-and outof-context transformations. experimental results reflect both individual and blended implementation scores for our seq-to-seq models (from a training perspective) and transformer model (from a finetuning perspective). the significantly improved results for the char, word, and transformer level models prove the effectiveness and reliability of our proposed approach.