Transformer-based Lexically Constrained Headline Generation (2021)

Abstract: This paper explores a variant of automatic headline generation methods, where a generated headline is required to include a given phrase such as a company or a product name. Previous methods using Transformer-based models generate a headline including a given phrase by providing the encoder with additional information corresponding to the given phrase. However, these methods cannot always include the phrase in the generated headline. Inspired by previous RNN-based methods generating token sequences in backward and forward directions from the given phrase, we propose a simple Transformer-based method that guarantees to include the given phrase in the high-quality generated headline. We also consider a new headline generation strategy that takes advantage of the controllable generation order of Transformer. Our experiments with the Japanese News Corpus demonstrate that our methods, which are guaranteed to include the phrase in the generated headline, achieve ROUGE scores comparable to previous Transformer-based methods. We also show that our generation strategy performs better than previous strategies.

## introduction following the initial work of @xcite , abstractive headline generation using the encoderdecoder model has been studied extensively @xcite @xcite . in the automatic headline generation for advertising articles, there are requests to include a given phrase such as a company or product name in the headline. generating a headline that includes a given phrase has been considered one of the lexically constrained sentence generation tasks. for these tasks, there are two major approaches. one approach is to select a plausible sentence including the given phrase from several candidate sentences generated from left to right @xcite @xcite . although these methods can include multiple phrases in a generated sentence, they are computationally expensive due to the large search space of the decoding process. in addition, since they try to force given phrases into sentences at every step of the generation process, these methods may harm the quality of the generated sentence @xcite . another approach proposed by @xcite is to generate token sequences in backward and forward directions from the given phrase. @xcite proposed sequence to backward and forward sequences (seq2bf), which applies the method of @xcite to the sequence-tosequence (seq2seq) framework. they use an rnnbased model and adopt the best strategies proposed by @xcite , generating the backward sequence from the phrase and then generating the remaining forward sequence. @xcite introduced the generative adversarial network (gan) to the model of @xcite to resolve the exposure bias problem @xcite caused by generating sequences individually, and used the attention mechanism @xcite to improve the consistency between both sequences. however, their model does not support the seq2seq framework. recently, @xcite used a transformerbased model @xcite , which is reported to achieve high performance, to generate a headline containing a given phrase. they proposed providing an encoder with additional information related to the given phrase. however, their method may not always include the given phrases in the generated headline. in this study, we work on generating lexically constrained headlines using transformer-based seq2bf. the rnn-based model used by @xcite executes a strategy of continuous generation in one direction, and thus cannot utilize the information of the forward sequence when generating the backward sequence. however, transformer can execute a variety of generative strategies by devising attention masks, so it can solve the problem of the rnn-based model. we propose a new strategy that generates each token from a given phrase alternately in the backward and forward directions, in addition to adapting and extending the strategies of @xcite to the transformer architecture. our experiments with a japanese summarization corpus show that our proposed method always includes the given phrase in the generated headline and achieves performance comparable to previous transformer-based methods. we also show that our proposed generating strategy performs better than the extended strategy of the previous methods. ## proposed method we propose a transformer-based seq2bf model that applies seq2bf proposed by @xcite to the transformer model to generate headlines including a given phrase. the seq2bf takes w (= w 1 , ..., w l ; w 1:l ) as the given phrase consisting of l tokens and generates the headline y -m :-1 of m tokens backward from w , and the headline y 1:n of n tokens forward from w . the transformerbased seq2bf is the transformer model with two generation components, consisting of a linear and a softmax layer (see figure 1 ). in transformer-based seq2bf unlike transformer generating tokens from left to right, the token position changes relatively depending on already generated tokens. we determine the token position, inputting to the positional encoding layer of the decoder, l+1 2 in w to be 0, and the position in the backward direction to be negative, and the position in the forward direction to be positive. we consider the following four generation strategies. in addition to two strategies (a) and (b), which extend those proposed by @xcite al. (2016 (c) generating each token backward and then forward alternately. (tok-b) (d) generating each token forward and then backward alternately. (tok-f) transformer-based seq2bf is formulated as to train the model on these generative strategies, we have prepared an attention mask for the decoder. transformer can control the generation order of tokens by devising the attention mask used in the decoder's self-attention mechanism. transformer generates tokens from left to right, so it is sufficient to disable the attention to tokens forward from the input tokens. however, the transformerbased seq2bf needs to specify the areas where input tokens disallow the attention in the backward and forward directions, depending on each generation strategy (see figure 2 ). ## experiment we conducted the experiment to verify the performance of our methods in the headline generation task. the objective of our experiment is to compare our method with previous transformer-based methods that generate tokens from left to right. we also compare seq-b/f, the generation orders proposed by @xcite , with tok-b/f, our new generation orders. ## conclusion we proposed transformer-based seq2bf that generates the lexically constrained headline by devising the attention mask for the decoder and generating backward and forward sentences from the phrase. our experiments using the jnc demonstrated that transformer-based seq2bf always includes the given phrase in the generated headline and obtains comparable performance compared to previous transformer-based methods. we also showed that strategies of generating each token alternately between backward and forward directions are more effective than that of generating a sequence in one direction and then a sequence in another direction. in future work, we will investigate whether transformer-based seq2bf can generate natural headlines even when given a variety of phrases, such as phrases not in the reference or the article, and examine if our methods can creatively generate diverse headlines by giving a variety of phrases quantitatively. also, we will explore methods for generating headlines that include multiple phrases.