Semantic Matching for Sequence-to-Sequence Learning (2020)

Abstract: In sequence-to-sequence models, classical optimal transport (OT) can be applied to semantically match generated sentences with target sentences. However, in non-parallel settings, target sentences are usually unavailable. To tackle this issue without losing the benefits of classical OT, we present a semantic matching scheme based on the Optimal Partial Transport (OPT). Specifically, our approach partially matches semantically meaningful words between source and partial target sequences. To overcome the difficulty of detecting active regions in OPT (corresponding to the words needed to be matched), we further exploit prior knowledge to perform partial matching. Extensive experiments are conducted to evaluate the proposed approach, showing consistent improvements over sequence-to-sequence tasks.

## introduction sequence-to-sequence (seq2seq) models are widely used in various natural-language-processing tasks, such as machine translation @xcite @xcite , text summarization @xcite and image captioning @xcite . typically, these models are based on an encoder-decoder architecture, with an encoder mapping a source sequence into a latent vector, and a decoder translating the latent vector into a target sequence. the goal of a seq2seq model is to optimize this encoder-decoder network to generate sequences close to the target. therefore, a proper measure of the distance between sequences is crucial for model training. wasserstein distance between two text sequences, i.e., word-mover distance @xcite , can serve as an effective regularizer for semantic matching in seq2seq models @xcite . classical optimal transport models require that each piece of mass in the source distribution is transported to an equal-weight piece of mass in the target distribution. however, this requirement is too restrictive for seq2seq models, making direct applications inappropriate due to the following: (i) texts often have different lengths, and not every element in the source text corresponds an element in the target text. a good example is style transfer, where some words in the source text do not have corresponding words in the target text. (ii) it is reasonable to semantically match important words while neglecting some other words, e.g., conjunction. in typical unsupervised models, text data are usually non-parallel in the sense that pairwise data are typically unavailable @xcite . thus, both pairwise information inference and text generation must be performed in the same model with only non-parallel data. classical ot is not applicable without target text sequences. however, partial target information is available, for example, the detected objects in an image should be described in its caption, and the content words when changing the style should be preserved. ot will fail in these cases but matching can be performed by optimal partial transport (opt). specifically, we exploit the partial target information representation via partially matching it with generated texts. the partial matching is implemented based on lexical information extracted from the texts. we call our method semantic partial matching (sepam). to demonstrate the effectiveness of sepam, we consider applying it on sequence-prediction tasks where semantic partial matching is needed: (i) in unsupervised text-style transfer, sepam can be employed for content preservation via partially matching the input and generated text; (ii) in image captioning, sepam can be applied to partially match the objects detected in images with corresponding captions for more informative generation; (iii) in table-to-text generation, sepam can prevent hallu-cination @xcite via partially matching tabular key words with generated sentences. the main contributions of this paper are summarized as follows: (i) a novel semantic matching scheme based on optimal partial transport is proposed. (ii) our model can be interpreted as incorporating prior knowledge into the optimal transport to exploit the structure of natural language, while making the algorithm tractable for real-world tasks. (iii) in order to demonstrate the versatility of the proposed scheme, we empirically show consistent improvements in style transfer for content preservation, image captioning for informative image descriptions and in table-to-text generation for faithful generation. ## semantic matching via opt in unsupervised seq2seq tasks without pair-wise information, naively matching the generated sequence with the weak-supervision information (e.g., source text in style transfer) will render deficient performance, even though both sentences share similar content. in supervised settings, target and input sequences are of different lengths but have similarity in terms of semantic meaning, such as table-to-text generation. motivated by this, we propose a novel technique for semantic partial matching and consider two scenarios: (i) text-totext matching and (ii) image-to-text matching. rooms were spacious and food was very well cooked . the rooms are so bad and the food is poorly cooked either . lstm lstm lstm cnn snow 1.000 skiing 0.993 man 0.917 slope 0.898 person 0.889 hill 0.808 covered 0.750 riding 0.627 generated caption: a man riding skis down a snow covered slope a @xmath0. text-to-text matching we consider semantic matching between two sequences in seq2seq models, where partial matching is important: i) in the unsupervised setting, such as non-parallel style transfer, partially matching between the source and target texts is helpful for content preservation. ii) in the supervised setting, such as table -to-text generation, partially matching the input and target sequences can effectively avoid hallucination generation, i.e. text mentions extra information than what is present in the source. figure 1 shows an example of partial matching, where part-of-speech (pos) tags for each word are exploited to provide prior knowledge. in these cases, directly applying ot will cause imbalanced transportation issue or poor performance. image-to-text matching objects and their properties (e.g., colors) are both included in the pair-wise images and captions. consider the imageto-text matching in figure 1 . it is clear that each object in the image has corresponding words/phases in the captions. we can consider matching the labels of detected objects in the image to some words in its caption. please note labels are not in oneto-one correspondence with the text, thus directly applying ot is inappropriate, similar to the case of text-to-text matching. different matching schemes hard matching seeks to exactly match from the source and target. typically, hard matching is too simple to be effective without considering semantic similarity, and if we apply classical optimal transport in unsupervised settings, it causes an imbalance matching, since some unnecessary words are included in the source and the exact target is unavailable. to tackle this issue, one can directly apply the optimal partial transport (opt) here to detect which word has its correspondence and match the word with its target. however, the detection process is computationally expensive, which is not scalable as a constrained optimization in (3) for real-world tasks. fortunately, we can exploit the syntax information from text, and incorporate this information as prior knowledge into opt to avoid the detection procedure. ## semantic partial matching for text generation assume there are two sets of objects @xmath1, we consider a seq2seq model, where the input is 1 x, and the output is a sequence of length t with tokens y t , i.e., @xmath2, y 2 . . . , y t ). one typically assigns the following probability to an observation y at location t: p(y|@xmath3, where @xmath4, y 2 . . . , y t ). this specifies a probabilistic model, i.e., to train the model, one typically uses maximum likelihood estimation (mle): we consider an encoder-decoder framework in this paper, where a latent vector z is given by an encoder enc(•), with input x, i.e., z @xmath5. based on z x , a decoder g(•) generates a new sentence ŷ that is expected to be the same as y . the decoder can be implemented by an lstm (hochreiter and schmidhuber, 1997), gru @xcite , or transformer @xcite . a unsupervised seq2seq model considers x and y as non-parallel, i.e., the pair-wise information is unknown. one typically pretrains the generator with the reconstruction loss: differentiable sepam note that the sepam loss is not differentiable due to the multinomial distribution sampling process ŷt ∼ softmax(g t ), where g t is a logit vector given by the final layer of the generator g(•). to enable direct backpropagation from the sepam loss for generator training, we consider the soft-argmax approximation @xcite to avoid the use of @xcite : sepam regularization sepam training objectives discussed above only focus on generating words with specific meanings and do not consider the word-ordering. to train a proper textgeneration model, we propose to combine the sepam loss with the likelihood loss l mle in supervised settings or l ae in unsupervised settings. hence, we have the training objective in unsupervised settings: @xmath6, where λ is the weight of sepam to be tuned. a similar objective applies for supervised settings: @xmath7. in the following, we discuss how to extract and use prior knowledge for partial matching in three downstream tasks: unsupervised seq2seq learning different from the standard seq2seq model @xcite , parallel sentences for different styles are not provided, and must be inferred from the data. unsupervised machine translation @xcite learns to translate text from one language to another with two sets of texts of these languages provided. @xcite explores the transformer model as the generator, instead of classical auto-regressive models. style transfer @xcite aims at transferring the styles of the texts with non-parallel data. compared with these tasks, unsupervised image captioning @xcite is more challenging since images and sentences are in distinct modalities. copying mechanism this is related to the copy network @xcite , which achieves retrievedbased copying. li et al. (2018) further proposed a delete-retrieve-generate framework for the style transfer. chen and bansal (2018) combine the abstraction with extraction in text summarization, and achieves state-of-the-art results via reinforced word selection. in this work, we proposed the semantic partial matching, which can be regarded as a kind the rooms are so bad and the food is poorly cooked either rooms were spacious and food was very well cooked pad pad pad 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 the rooms are so bad and the food is poorly cooked either rooms were spacious and food was very well cooked pad pad pad 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 a man riding skis down a snow covered slope snow skiing man slope hill cover pad pad pad 0.02 0.04 0.06 0.08 a man riding skis down a snow covered slope snow skiing man slope hill cover pad pad pad 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 of soft-copying mechanism. instead of the retrievalbased exact copying used by @xcite , sepam considers semantic similarity, and thus ideally delivers smoother transformation in generation. ## conclusions we incorporate prior knowledge into optimal transport, to encourage partial-sentence matching via formulating it as an optimal partial transport problem. the proposed sepam shows broad applicability and consistent improvements against popular baselines in three downstream tasks: unsupervised style transfer for content preservation, image captioning for informative descriptions and table-totext generation for faithful generation. further, the proposed technique can be regarded as a softcopying mechanism for seq2seq models.