Learning with Contrastive Examples for Data-to-Text Generation (2020)

Abstract: Existing models for data-to-text tasks generate fluent but sometimes incorrect sentences e.g., “Nikkei gains” is generated when “Nikkei drops” is expected. We investigate models trained on contrastive examples i.e., incorrect sentences or terms, in addition to correct ones to reduce such errors. We first create rules to produce contrastive examples from correct ones by replacing frequent crucial terms such as “gain” or “drop”. We then use learning methods with several losses that exploit contrastive examples. Experiments on the market comment generation task show that 1) exploiting contrastive examples improves the capability of generating sentences with better lexical choice, without degrading the fluency, 2) the choice of the loss function is an important factor because the performances on different metrics depend on the types of loss functions, and 3) the use of the examples produced by some specific rules further improves performance. Human evaluation also supports the effectiveness of using contrastive examples.

## introduction we address the task of generating market comments from stock prices as illustrated in fig. 1 . this can be seen as a data-to-text generation task. recently, neural data-to-text generation has been studied in a wide range of domains such as biography @xcite , sports recap @xcite @xcite @xcite , and market comments @xcite @xcite . these models generate fluent sentences, but we often observed problematic generated sentences in terms of correctness. as shown in fig. 1 , the word gain is possibly generated, although the word drop or rebound is expected. the terms that express the fluctuation of stock prices are crucial because such errors could reverse the meaning of the sentence in the worst case. similar issues have been seen in other generation tasks, such as machine translation or summarization. the known solutions are, for example, the use of alignments between input and output @xcite or copy mechanisms @xcite . however, they cannot be directly applied to our task because ours treat sequences of numerical values as an input. in this paper, we consider how to alleviate such errors by using contrastive examples, which are identical to the correct examples except for a single word: nikkei gained vs. nikkei dropped. learning with such examples provides models direct signals on the words that are not to be generated in addition to those to be generated. we propose a learning framework to examine how to use such examples from the viewpoint of loss functions and rules to create contrastive examples. recent studies show the effectiveness of learning methods that exploit explicit negative examples in language modeling. @xcite introduced a margin loss to penalize sentences in a beam, assuming that the generated sentences are imperfect. @xcite used synthesized ungrammatical sentences in addition to the originals to improve the syntactic ability of language models. for generation, @xcite proposed a model with an unlikelihood loss to alleviate the repetition problem. rather, we focus on improving the correctness of generated sentences, which is crucial for data-to-text tasks. ## framework we introduce our learning framework with several losses that exploit contrastive examples. the main aim of this study is to investigate whether models can generate crucial terms more correctly if we train them with contrastive examples. ## experiments in this section, we describe the dataset used for our experiments, the ways to finetune hyperparameters and metrics for automatic evaluation in addition to manual evaluation by a human judge. ## results in this section, we discuss the results of automatic and human evaluations. ## related work neural data-to-text generation has been studied in wide range of domains such as biography @xcite , sports recap @xcite @xcite @xcite , and market comments @xcite @xcite . @xcite and @xcite deal with sentence-level market comments generation tasks, whereas @xcite generate document-level market comments that can be controlled by hand-crafted rules. we follow the most basic setup, that is, sentence-level market comments generation. although each domain of data-to-text tasks has its own difficulty, these studies showed that neural endto-end approaches such as encoder-decoders can generate fluent text. however, the sentences generated by existing models are often problematic in terms of correctness. significant developments have been made to capture input data correctly, for example, encoders with content selection @xcite , decoders with entity modeling @xcite . the problem in terms of correctness is also well knwon in other generation tasks such as machine translation tasks @xcite or summarization @xcite . the use of an alignment dictionary @xcite or copy mechanisms @xcite are common strategies to reduce such errors, but these are difficult to adopt for data-to-text tasks. in this study, our models use various loss functions that take into account contrastive samples. this approach relates to recent studies that propose loss functions that use negative samples for language modeling. @xcite introduced a margin loss that estimates the quality of each beam-searched candidate by comparing it with the reference sentence. more recently, @xcite showed negative examples help to improve the syntactic ability of neural language models. they created negative instances from original instances by injecting a grammatical error and used them to calculate a margin loss that will be added to the cross-entropy loss. for generation, @xcite proposed a model with an unlikelihood loss to alleviate the repetition problem. although their study targets neural language models or the different problems in generation other than improving correctness, we focus on improving the generated sentences in data-to-text tasks in terms of correctness. ## conclusion we presented learning methods with several losses that exploited contrastive examples for data-to-text. the results showed our methods improved the performances in terms of correctness. human evaluation also supported the improvements for the crucial parts that our model attempted to reduce errors. because our methods have wide applicability, we will examine their effectiveness against other models and tasks. the applicability will be wider if effective contrastive examples can be generated automatically.