Attention Is Indeed All You Need: Semantically Attention-Guided Decoding for Data-to-Text NLG (2021)

Abstract: Ever since neural models were adopted in data-to-text language generation, they have invariably been reliant on extrinsic components to improve their semantic accuracy, because the models normally do not exhibit the ability to generate text that reliably mentions all of the information provided in the input. In this paper, we propose a novel decoding method that extracts interpretable information from encoder-decoder models’ cross-attention, and uses it to infer which attributes are mentioned in the generated text, which is subsequently used to rescore beam hypotheses. Using this decoding method with T5 and BART, we show on three datasets its ability to dramatically reduce semantic errors in the generated outputs, while maintaining their state-of-the-art quality.

## introduction task-oriented dialogue systems require high semantic fidelity of the generated responses in order to correctly track what information has been exchanged with the user. therefore, their natural language generation (nlg) components are typically conditioned on structured input data, performing data-to-text generation. to achieve high semantic accuracy, neural models for data-to-text nlg have invariably been reliant on extrinsic components or methods. while large pretrained generative language models (lms), such as gpt-2 or t5, perform better in this respect, even they do not normally generate text that reliably mentions all the information provided in the input. in this work, we study the behavior of attention in large pretrained lms fine-tuned for data-to-text nlg tasks. we show that encoder-decoder models equipped with cross-attention (i.e., an attention mechanism in the decoder looking back at the encoder's outputs) are, in fact, aware of the semantic constraints, yet standard decoding methods do not fully utilize the model's knowledge. the method we propose extracts interpretable information from the model's cross-attention mechanism at each decoding step, and uses it to infer which slots have been correctly realized in the output. coupled with beam search, we use the inferred slot realizations to rescore the beam hypotheses, preferring those with the fewest missing or incorrect slot mentions. to summarize our contributions, the proposed semantic attention-guided decoding method, or sea-guide for short: (1) drastically reduces semantic errors in the generated text (shown on the e2e, viggo, and multiwoz datasets); (2) is domainand model-independent for encoder-decoder architectures with cross-attention, as shown on different sizes of t5 and bart; (3) works out of the box, but is parameterizable, which allows for further optimization; (4) adds only a small performance overhead over beam search decoding; and (5) perhaps most importantly, requires no model modifications, no additional training data or data preprocessing (such as augmentation, segmentation, denoising, or alignment), and no manual annotation. 1 ## related work several different approaches to enhancing semantic accuracy of neural end-to-end models have been proposed for data-to-text nlg over the years. the most common approach to ensuring semantic quality relies on over-generating and then reranking candidate outputs using criteria that the model was not explicitly optimized for in training. reranking in sequence-to-sequence models is typically performed by creating an extensive set of rules, or by training a supplemental classifier, that indicates for each input slot whether it is present in the output utterance @xcite @xcite @xcite . @xcite proposed an extension of the underlying lstm cells of their sequence-tosequence model to explicitly track, at each decoding step, the information mentioned so far. the coverage mechanism @xcite @xcite penalizes the model for attending to the same parts of the input based on the cumulative attention distribution in the decoder. @xcite and @xcite both introduce different sequence-to-sequence model architectures that jointly learn to generate text and reconstruct the input facts. an iterative self-training process using data augmentation @xcite was shown to reduce semantic nlg errors on the e2e dataset @xcite . among the most recent efforts, the jointly-learned segmentation and alignment method of @xcite improves semantic accuracy while simultaneously increasing output diversity. @xcite use segmentation for data augmentation and automatic utterance planning, which leads to a reduction in semantic errors on both the e2e and viggo @xcite datasets. in contrast to the above methods, our approach does not rely on model modifications, data augmentation, or manual annotation. our method is novel in that it utilizes information that is already present in the model itself to perform semantic reranking. finally, related to our work is also controllable neural language generation, in which the constrained decoding strategy is often used, rescoring tokens at each decoding step based on a set of feature discriminators @xcite @xcite . nevertheless, this method is typically used with unconditional generative lms, and hence does not involve input-dependent constraints. ## semantic attention-guided decoding while we will evaluate the sea-guide method on viggo, e2e, and multiwoz, we develop the method by careful analysis of the cross-attention behavior of different pretrained generative lms fine-tuned on the viggo dataset. viggo is a parallel corpus of structured meaning representations (mrs) and corresponding natural-language utterances in the video game domain. the mrs consist of a dialogue act (da) and a list of slot-and-value pairs. the motivation for selecting viggo for developing the method was that it is the smallest dataset, but it provides a variety of da and slot types (as shown in ## evaluation in order to measure the proposed decoding method's performance in semantic error reduction, we first develop an automatic way of identifying erroneous slot mentions in generated utterances. in a human evaluation we establish that its performance is nearly perfect for all three datasets used for testing our models (see section 4.1). we then use it to calculate the slot error rate (ser) automatically for all our model outputs across all datasets and configurations tested, which would be infeasible to have human annotators do. ## results to maximize the performance of the models using sea-guide, the binarization thresholds (and possibly other parameters of the mention-tracking components) can be optimized for each model and dataset on the validation set. in our evaluation, however, we focused on demonstrating the effectiveness of this decoding method out of the box. that being said, even common decoding methods, such as simple beam search or nucleus sampling (holtzman et al., 2019), usually benefit from parameter optimization (e.g., beam size, or the p-value) whenever used with a different model or dataset. ## discussion in the previous section, we showed that sea-guide is highly effective at reducing semantic errors across different models and domains, and that without compromising on the generated utterances' fluency. on datasets other than e2e, it does not quite match the performance of beam search combined with our slot aligner-based reranking, but then again, the slot aligner is a hand-crafted tool with complex rules, requiring a good deal of domain knowledge, and suffering thus significantly in scalability. while these two decoding methods have a lot in common -both being based on beam search and subsequent candidate reranking -their difference lies in the identification of slot mentions; sea-guide identifies them automatically during the decoding, utilizing the model's cross-attention weights at each step, as opposed to relying on string-matching rules post decoding, which need to be extended for any new domains. despite working conveniently out of the box, sea-guide does not come with a computational overhead caveat. performing inference on a gpu, sea-guide is a mere 11-18% slower than beam search with slot aligner-based reranking, while we observed no performance difference on a cpu (see appendix b.4 for a detailed analysis). ## conclusion we presented a novel decoding method, sea-guide, that makes a better use of the crossattention component of the already complex and enormous pretrained generative lms to achieve significantly higher semantic accuracy for data-to-text nlg, while preserving the otherwise high quality of the output text. it is an automatic method, exploiting information already present in the model, but in an interpretable way. sea-guide requires no training, annotation, data augmentation, or model modifications, and can thus be effortlessly used with different models and domains.