A Timestep aware Sentence Embedding and Acme Coverage for Brief but Informative Title Generation (2022)

Abstract: The title generation task that summarizes article content in recapitulatory words relies heavily on utilizing the corresponding key context. To generate a title with appropriate information in the content and avoid repetition, we propose a title generation framework with two complementary components in this paper. First, we propose a Timestep aware Sentence Embedding (TSE) mechanism, which updates the sentences’ representations by re-locating the critical words in the corresponding sentence for each decoding step. Then, we present an Acme Coverage (AC) mechanism to solve the repetition problem and preserve the remaining valuable keywords after each decoding step according to the final vocabulary distribution. We conduct comprehensive experiments on various title generation tasks with different backbones, the evaluation scores of ROUGE and METEOR in varying degrees are significantly outperforming most of the existing state-of-the-art approaches. The experimental results demonstrate the effectiveness and generality of our novel generation framework TSE-AC.

## introduction on account of the existence of many articles and news, automated title generation which summarizes the source content into a succinct title with recapitulatory words can significantly reduce the cost of obtaining information and improve the efficiency of information transmission. as the example shown in we are planning a party it has been a while since the group has had a party and my daughter is going to be a year old. so we are planning a party for november 14th and vandhana and i would like to invite everyone in research and their family. as yet we do not have the ... email subject birthday party invitation to overcome the shortness of previous works that freeze the sentences' embeddings without considering the changes of the decoding states. we propose a novel title generation architecture with a tse mechanism, which finely adopts different related words of each sentence to update the sentence's embedding vector in each decoding step. furthermore, the architecture with tse incorporates the commonly used hierarchical encoder @xcite @xcite , part of speech (pos) information @xcite , and graph structure @xcite to take full advantages of inner relations among words within each sentence as well as among sentences through the article content, so as to re-locate the related key words for each target word more accurately. this encoding method gives insight into the correlation of each keyword and then integrates them well to summarize the article. to the best of our knowledge, this is the first work which finely updates the embedding of each sentence via the target words for each decoding step. besides, repetition is a common problem in generation tasks, which especially stands out in the title generation scenerio due to its succinctness. the coverage mechanism is widely used to address this problem and achieves impressive results @xcite . previous coverage mechanisms focus on maintaining a coverage vector which is the sum of attention distributions over all previous decoding timesteps and make the next attention weights as different from the coverage vector as possible. however, in title generation, the penalization with this kind of coverage vector will penalize all attended keywords, disturbing the attention mechanism in the following decoding steps. and the generated titles may lose some keywords because of the inappropriate penalization. moreover, since our tse relies heavily on the attention mechanism, the superimposed negative influence will seriously impair the accuracy of the generated title. in this paper, we leverage an ac mechanism that only prevents repeatedly attending to the word which has been generated actually to avoid producing repetitive text but without missing other valuable keywords. our contributions are summarized as follows: some other works applied hierarchical frameworks to encode document level inputs. @xcite ; @xcite utilized a hierarchical encoder to model the discourse structure of long documents, but the sentences' embeddings are static during all decoding steps in their works, we update the embedding of each sentence for every targeted word dynamically based on the corresponding valuable words and the decoding state. @xcite presented a hierarchical model to generate summarization for multiple documents. the main differences between their work and ours are twofold. first is the granularity of the encoder, we pay more emphasize on tokens instead of paragraphs since every word is crucial for title generation. the second but critical difference is the related information locating mechanism used in decoder. they first extracted a central location for the key information and select several paragraphs around it. our model proposes a tse mechanism which will focus on all related keywords in each sentence to update their embeddings dynamically for every decoding step. besides, repetition is a common problem in generation tasks, there are several efforts made to address this issue. temporal attention is a technique that has been applied to neural machine translation (nmt) @xcite and summarization @xcite , where each attention distribution is divided by the sum of the previous, which effectively avoids repeated attention but distorts the signal from the attention mechanism and then reduces the performance. n-gram blocking @xcite is another technique proposed to discard the n-gram if it appeared already. coverage mechanism is a widely used technique to dampen repetition, which was first applied to nmt by maintaining a coverage vector @xcite . a simple approach that upholds the coverage vector with the sum of the attention distributions was introduced in summarization task and achieved impressive results @xcite . while this approach penalizes the previous attention distribution indiscriminately, which disturbed the generation of later target words and resulted in degraded performance. we leverage an ac mechanism that maintains a coverage vector based on the attention weights with the highest probability of the final vocabulary, and outperforms difference evaluation metrics compared to the vanilla coverage. ## conclusion in this paper, we proposed a novel title generation framework with timestep aware sentence embedding, which re-locates critical words dynamically for each target word from the source content based on the decoding states to update the corresponding sentence's embedding. moreover, we present acme coverage that can accurately penalize the probability of the word which has been generated actually. the experiments demonstrate that our approach achieves the state of the art performance on different kinds of title generation scenarios.