Generating Repetitions with Appropriate Repeated Words (2022)

Abstract: A repetition is a response that repeats words in the previous speaker’s utterance in a dialogue. Repetitions are essential in communication to build trust with others, as investigated in linguistic studies. In this work, we focus on repetition generation. To the best of our knowledge, this is the first neural approach to address repetition generation. We propose Weighted Label Smoothing, a smoothing method for explicitly learning which words to repeat during fine-tuning, and a repetition scoring method that can output more appropriate repetitions during decoding. We conducted automatic and human evaluations involving applying these methods to the pre-trained language model T5 for generating repetitions. The experimental results indicate that our methods outperformed baselines in both evaluations.

## introduction dialogues can build a trusting relationship with others, thus are essential in our daily lives @xcite . there are several types of responses in dialogues, and the one we focus on is repetitions @xcite . a repetition is a response that uses the previous speaker's words or phrases. figure 1 shows an example. the phrases "a bear" and "came out" are repeated. repetitions frequently appear in a conversation with diverse roles, e.g., to indicate attentive listening, confirm the previous utterance, and show agreement or sympathy @xcite . many linguistic studies investigating repetitions have concluded that they are important for building and strengthening relationships between speakers @xcite @xcite . from the above linguistic point of view, we can say that repetitions are indispensable in dialogues. repetitions are similar to paraphrases and reflections, which are component skills of counsel-speaker: when i was driving, a bear suddenly came out . listener: oh. a bear came out !? ing @xcite , in terms of using the previous speaker's utterance. paraphrases and reflections have been generated using a template-based method @xcite . while many studies have tackled general response generation with neural network-based frameworks @xcite , less attention has been paid to repetitions. this might be because they are buried in a huge amount of response data. therefore, we focus on automatically generating repetitions. to the best of our knowledge, this is the first study on generating repetitions with a neural approach. we used the pre-trained language model t5 @xcite for generating repetitions because it has performed well in language generation in past few years (e.g., @xcite @xcite . in generating repetitions, it is important to take into account which words should be repeated from the previous utterance. the repeated words might represent objective facts, names of people and places, and the speaker's experiences and emotions, though they are different depending on the language @xcite . when we use a pre-trained language model, however, the model cannot explicitly learn the repeat likelihood among words during fine-tuning because it is difficult to directly teach which words are likely to be repeated at this step. to solve this problem, we propose weighted label smoothing (wls), which is an improvement upon label smoothing (ls) @xcite . the method enables a language model-based re-sponse generator to learn the words it should use for each input utterance during fine-tuning. we also propose the repetition scoring method (rsm) to expand the scoring method proposed in @xcite for selecting repetitions that contain appropriate repeated words during decoding. we evaluated the proposed methods on a dataset we created in japanese for automatic and human evaluations. our methods outperformed baselines, i.e., fine-tuned pre-trained language models without our methods, in both evaluations. this indicates that our methods can generate repetitions that contain appropriate words to repeat. our contributions are as follows: ## proposed methods repetitions do not necessarily mean we repeat any word. for the utterance "today's dinner was pizza.", the repetition "oh, you ate pizza." is more appropriate than "oh, you ate today." however, a fine-tuned pre-trained language model alone may not be enough to generate repetitions with appropriate repeated words. therefore, to generate a response that repeats more appropriate words, we introduce repeat scores ( §2.1) to calculate how likely a word is repeated and incorporate the scores into wls ( §2.2) for fine-tuning and rsm ( §2.3) for beam search in decoding. ## dataset we manually created pairs of a speaker's utterance and its repetition as our dataset using a crowdsourcing service. 2 since repetitions often occur when a listener replies to a speaker, we used utterances in a corpus of listening dialogues @xcite between an elderly person and caregiver or clinical psychologist as the speaker's utterances in our dataset. 3 in this corpus, the elderly person tends to be a speaker and the others are listeners. we extracted the elderly person's utterances containing content words for creating a repetition. the number of extracted utterances was 5,548. we asked three crowdsourcing workers to create repetitions for each utterance. specifically, a worker was shown two utterances before each target utterance and asked to create a repetition, that supports the creation of context-aware repetitions. when the workers found it difficult to create a repetition for an utterance, they could discard it. the total number of workers was 333. examples from the dataset are given in to understand what types of words overlap, while we can have at most three repetitions for an utterance in our dataset, we used only one randomly selected repetition for an utterance in the training data. we used all repetitions for an utterance for the evaluation on the validation and test data to consider the diversity of responses. ## conclusion we focused on repetition generation. although repetitions play an important role in dialogues, there has been no neural approach for this task to the best of our knowledge. we proposed wls, which is an improvement upon ls, during fine-tuning and rsm, which is an extended scoring method, during decoding for repetition generation. through automatic and human evaluations, we confirmed that our model can generate repetitions that contain more appropriate words to repeat than baseline models. for future work, we will take into account synonyms and multiple gold repetition instances to calculate repeat scores for improving the diversity of responses. we are also planning to incorporate our repetition model into a general response generation framework.