Controlling keywords and their positions in text generation (2023)

Abstract: One of the challenges in text generation is to control text generation as intended by the user. Previous studies proposed specifying the keywords that should be included in the generated text. However, this approach is insufficient to generate text that reflect the user’s intent. For example, placing an important keyword at the beginning of the text would help attract the reader’s attention; however, existing methods do not enable such flexible control. In this paper, we tackle a novel task of controlling not only keywords but also the position of each keyword in the text generation. To this end, we propose a task-independent method that uses special tokens to control the relative position of keywords. Experimental results on summarization and story generation tasks show that the proposed method can control keywords and their positions. The experimental results also demonstrate that controlling the keyword positions can generate summary texts that are closer to the user’s intent than baseline.

## introduction one of the challenges in text generation is to generate text that is consistent with the user's intent. many methods for specifying the keywords that should be included in the generated text to reflect the user's intent have been proposed. as for summarization, by providing the model with keywords that should be included in the summary, it is possible to generate summaries that focus on specific parts of the document @xcite @xcite . as for story generation, keywords are used to control the narrative storyline @xcite @xcite . as for other tasks, such as e-commerce generation, review generation, and question generation, keywords are also used to control text generation @xcite @xcite @xcite . in addition, more-advanced methods that specify the order of keywords to be included in the generated text to control the rough storyline have been proposed @xcite . the above-described methods, however, cannot generate texts that reflect more fine-grained intentions. specifically, the user may want to reflect the intended importance of each keyword in the generated text. an effective way to reflect the intended importance of each keyword is to adjust the position of keywords within the text. for example, important keywords such as topic words and eye-catching words can be placed at the beginning of the text to attract the reader's attention, while the keywords for supplementary information can be placed in the middle or later in the text. by controlling the specific position of each keyword according to its importance, it is possible to generate appropriate text for each situation. that is, controlling the specific position of keywords in the generated text is a challenge in terms of reflecting more-specific user intentions and generating texts that attract readers. however, as far as we know, no previous work has tackled this challenge. in this paper, we tackle a novel task of controlling keywords and the position of each keyword in text generation. inspired by previous work that controlled text attributes by using special tokens @xcite @xcite , we propose a task-independent method that uses special tokens to control text generation. specifically, the position of the keyword is specified by providing the model with a special token that represents the target relative position of the keyword (0-10%, 10-20%, etc.) and length of target text (20-24 words, 25-29 words, etc.). we use relative positions (rather than absolute positions) because it is more practical to specify relative positions such as "at the beginning," "in the middle," or "at the end" of the target text. moreover, length of the target text is controlled because text length is considered to be one of the important factors that users want to control when considering where to place keywords. during training of the model, the model is provided with control tokens, including keywords randomly extracted from the target text, the positions of each keyword, and the length of the target text. the model is trained with crossentropy loss in the same manner as conventional text generation; as a result, the model can learn the correspondence between the input control tokens and the target text. the proposed "task-independent text-generationcontrol method" ("proposed method" hereafter) was comprehensively evaluated by applying it to summarization and story-generation tasks. the results of the evaluation show that the proposed method can control keywords and their positions in both tasks (section 3.2). they also show that the proposed method can generate summary texts that are more similar to the gold summary than the baseline, indicating that text closer to the user's intent can be generated (section 3.3). case studies show that a model specifying keyword position can reflect the user's fine-grained intention (section 3.4). ## method 2.1 models a bart model @xcite is used for the summarization task, and a gpt model @xcite is used for the story-generation task. when the bart model is used, the source document is combined with the control tokens (i.e., keywords in the text to be generated, positions of each keyword, and length of the text to be generated) and given to the encoder as shown in figure 1 . when the gpt model is used, the control tokens are given to the decoder. as with regular text gen-eration using bart and gpt models, the model is trained to maximize the conditional probabilities p(y i |@xmath0, x) by using cross-entropy loss, where y denotes the target text and x denotes the input to the model, including the control tokens and the source document used in summarization task. ## conclusion a method for controlling keywords and the position of each keyword in generated text is proposed and evaluated experimentally by applying it to two tasks: summarization and story generation. the results of the evaluation show that the proposed method, which uses special tokens, can control the keyword positions in both tasks. they also show that the method can generate summary texts that are more similar to the gold summary than the baseline, and that finding indicates that text closer to the user's intent can be generated.