Generating Dialog Responses with Specified Grammatical Items for Second Language Learning (2023)

Abstract: This paper proposes a new second language learning task of generating a response including specified grammatical items. We consider two approaches: 1) fine-tuning a pre-trained language model (DialoGPT) by reinforcement learning and 2) providing a few-shot prompt to a large language model (GPT-3). For reinforcement learning, we examine combinations of three reward functions that consider grammatical items, diversity, and fluency. Our experiments confirm that both approaches can generate responses including the specified grammatical items and that it is crucial to consider fluency rather than diversity as the reward function.

## introduction the use of dialog systems for language learning has attracted attention. many studies have introduced dialog systems as training partners for language learners and verified their effectiveness. according to previous studies @xcite @xcite , the advantages of using dialog systems in language education include: they can be used regardless of time, i.e., are more available for learners, they can be easily integrated into chatbased applications that many people are familiar with, i.e., are more user-friendly, and they can be adapted to each learner using various information from chit-chat, i.e., are more supportive. needless to say, experiencing a substantial amount of production is critical in language acquisition. @xcite showed that even a very primitive rule-based chatbot like eliza has the potential to increase learner's sentence production. their experiments also revealed that learners adopted words that appeared in the chatbot's responses, suggesting that the expressions used by the dialog system had a positive impact on learners and that the system was effective in helping them learn unfamiliar words. considering these results, we propose a task of generating a response including the specified grammatical items. here, grammatical items refer to such as to the present perfect, subjunctive, and relative clauses. usually, they are gradually covered in a language learning course, typically through a school curriculum. such responses can naturally expose learners to a variety of uses of a specific item and can give them experience of how to use the item in a variety of topics and situations, based on their own past experiences evoked in the conversation. in turn, we expect the learners to use the exposed constructions in their own production more as the exposed uses are linked tightly to their memories by encountering usage examples through dialog based on their own experiences. the proposed task is formalized as follows. given @xmath0, c 2 , ..., c n ], a dialog context that is a sequence of n utterances between two interlocutors (the system and the learner), and i, a set of grammatical items specified to be included, the task is to generate r, a natural response that follows c n , on the condition that r includes an expression corresponding to each item @xmath1. to the best of our knowledge, this is the first work to tackle this generation task for language learning. to generate text that satisfies particular conditions, @xcite propose using auxiliary modules to guide pre-trained language models. keskar et al. (2019) propose training language models with control code. since these methods are based on supervised learning, they require annotated datasets. however, there is a lack of large labeled dialog datasets for grammatical items. in this paper, we examine two approaches for generating responses containing the specified grammatical items without a large labeled dataset: 1) rl-based generation: fine-tuning a pre-trained language model using reinforcement learning (rl), and 2) prompt-based generation: providing a large language model with prompt text with a task in-struction and a few examples. the experiments confirm both approaches are promising. ## method for the sake of simplicity, in this paper, we assume context c contains only the immediately previous utterance (@xmath2. we also limit the number of specified items to 1 (|i| = 1). ## experiment we verified the effectiveness of both rl-based and prompt-based approaches using three items in the score corpus @xcite : the present perfect, relational clause, and subjunctive. ## results the rl-based approach successfully improved g-ratio in all cases. although the dist.-2 values got lower than before training (baseline), this was expected in advance as the result of introducing a grammatical constraint in generation. in the rl-based approach, a higher dist.-2 tended to be obtained with the fluency reward function r f than with the diversity reward function r d except for the subjunctive, suggesting that the effect of r d was limited. the reasons for this may be as follows. even if sentences with a high dist.-2 are more likely to be generated, it does not necessarily reflect the diversity of the model overall, and if the input sentences in the batch are similar, dist.-2 in the output will naturally decrease, but the current reward function does not fully take this into account. in addition, taking fluency into account suppresses the abuse of fixed patterns (fixed patterns increase r g but decrease diversity). for all grammatical items tested, goal improved when the reward function for fluency, r f , was applied. in the prompt-based approach, g-ratio tended to be higher for inputs with both task instruction and shots. however, 3-shots sometimes gave worse results than 1-shot. this suggests that task instructions should be included in the input, but that increasing the number of shots may add noise or unintended bias to the language model, making it more difficult to obtain the desired output. comparing the two approaches, the promptbased one demonstrated higher diversity than the rl-based one, and a comparable g-ratio. though the goal scores for the rl-based approach were higher than those for the prompt-based approach, we must note that goal is favorable to the rlbased approach that, in this paper, uses the same dialogpt model as goal. as far as we manually compared the concrete responses from gpt-3 and dialogpt for a small number of randomly picked cases, we did not find significant differences. ## discussion even though we want to expose more instances of a particular item to a learner, it is not natural to include the item in every dialog response. therefore, we do not need to pursue 100% for g-ratio. we presented goal as a primary metric candidate for the proposed task. however, as noted in the previous section, it is not reliable when one wants to compare two results based on different language models. taking the similarity to the reference sentences into account is one direction to mitigate this issue. another strategy is combining goal with reference-free unsupervised dialog evaluation methods using follow-ups such as full @xcite . unlike goal, these evaluation methods do not measure the likelihood of the target utterances directly; they, however, still rely on a particular language model. a simple way to make this issue easier would be an ensemble approach using multiple language models or majority voting. considering the high diversity and the nature of training-free, so far the prompt-based approach seems to be advantageous, assuming the availability of a huge pre-trained model such as gpt-3. however, the rl-based approach may have merits in terms of its fine-grained, delicate, and implicit control than the prompt-based approach. (besides, dialogpt and gpt-2 did not work in the promptbased approach. see appendix c.) ## conclusion we have proposed a new task of generating a response including the specified grammatical items for language learners. we examined two approaches and found that both are feasible. future directions include the expansion of the grammatical items. to push this task to practical use, locating appropriate places in conversations to include the items is also important. this paper aimed to increase learners' exposure to specific grammatical items, but another inter-esting direction is generating preceding utterances that encourage or facilitate learners to use specific grammatical items in their next utterances.