Towards an open-domain chatbot for language practice (2022)

Abstract: State-of-the-art chatbots for English are now able to hold conversations on virtually any topic (e.g. Adiwardana et al., 2020; Roller et al., 2021). However, existing dialogue systems in the language learning domain still use hand-crafted rules and pattern matching, and are much more limited in scope. In this paper, we make an initial foray into adapting open-domain dialogue generation for second language learning. We propose and implement decoding strategies that can adjust the difficulty level of the chatbot according to the learner’s needs, without requiring further training of the chatbot. These strategies are then evaluated using judgements from human examiners trained in language education. Our results show that re-ranking candidate outputs is a particularly effective strategy, and performance can be further improved by adding sub-token penalties and filtering.

## introduction studies in second language acquisition have shown that interaction is an important aspect of language learning (e.g. @xcite @xcite . however, interaction typically involves one-on-one sessions with a teacher, which can be costly or may simply be unavailable to some learners. in addition, learners may experience language anxiety during interaction, which can be detrimental to learning @xcite . artificial dialogue systems provide an alternative way to learn through interaction. learners can chat with the system in their target language at their own convenience, without needing a teacher. existing systems typically rely on handcrafted rules, and require learners to practise within a specified context (e.g. shopping at a supermarket) (cf. @xcite . they are therefore quite limited in scope and require much manual work to anticipate possible responses. in our work, we leverage existing chatbot technology that can generate responses in virtually any topic @xcite . as a first step towards integrating this technology into language education, we experiment with ways to adjust the difficulty of chatbot messages to a specified level (e.g. one that matches the learner's proficiency level). our contributions are as follows: ## related work we provide an overview of four related topics: 1) dialogue systems; 2) decoding strategies for adding various desired attributes to text; 3) text simplification methods for transforming existing text (instead of generating text at a specified difficulty level); and 4) methods for predicting linguistic complexity. ## implementation we propose 5 different decoding strategies to adjust the chatbot difficulty to one of 6 cefr levels 1 . for our implementations, we used facebook's blender 2.7b (version 1 2 , generator model) @xcite as the basis, though our methods can be used or adapted to other language models that use beam search or sampling-based generation. for comparability, all strategies use top-k sampling 3 @xcite using @xmath0. we did not use the additional safety mechanisms (as described by @xcite ) to ensure fair comparison of results. some of our methods use regression, either during generation or beforehand. for all regression tasks described below, we use a continuous scale from 0 to 5 to denote cefr values, even though they are typically differentiated by qualitative properties (council of europe, 2020). this is because: a) we have limited training data, and a scalar 1 throughout this paper, we draw on the common european reference framework (cefr) (council of europe, 2020) to denote proficiency levels. an international standard for describing language ability, the cefr organises ability into 6 levels, beginning with a1, continuing to a2, b1, b2, c1, and ending with c2, representing mastery of a second language. ## evaluation for each of our 5 methods, we generated 300 selfchat dialogues using blender, where the chatbot talks to itself @xcite . each self-chat was generated using the settings for a specific cefr level: for example, method 1 at b1 level would only generate vocabulary at b1 level or below; method 3 at c1 level would re-rank outputs based on how close it is to c1 difficulty. then, to determine whether these methods are truly able to generate messages at the intended level, we recruited english language examiners to judge the true difficulty level of each self-chat. we chose self-chats rather than human-model chats (i.e. chats between a human and the language model) for three reasons: firstly, because we did not want the examiner's judgement of the chatbot output to be biased by the proficiency level of the user; secondly, because it is cheaper and less time consuming to generate self-chats; and finally, because second language users may struggle to communicate with the chatbot. additionally, previous work comparing self-chats to human-model chats found that they produced similar results @xcite . each self-chat consists of 18 messages, all prompted by an initial message, "hello!". an example of a generated self-chat can be found in the appendix. the 300 dialogues for each method are split evenly into 6 sets of 50, each set targeting a different cefr level. an additional 100 dialogues were generated without any modifications for comparison, resulting in an overall total of 1600 dialogues (see we recruited 10 english language examiners from cambridge university press & assessment. all 10 examiners were provided with a set of genrespecific descriptors adapted from the cefr foot_3 . in addition, to assess the general quality of the produced text, each message in the dialogue was labelled according to whether it was sensible and whether it was specific (following @xcite , as well as whether it was grammatical. examiners were given additional guidance on edge cases to support their application of these labels. each dialogue was annotated by at least 3 different examiners. for the final results, disagreements between examiners are resolved by taking the average of all annotations. the inter-annotator agreement for our cefr annotations is 0.79, measured with weighted fleiss' κ @xcite , and assuming equal distance between cefr levels. for the grammatical, sensible, and specific labels, we used gwet's ac 1 (gwet, 2014) 18 . the agreement scores are 0.62 for grammaticality labels, 0.23 for sensibleness labels, and 0.67 for specific labels. agreement in sensibleness is noticeably lower than the others: feedback from annotators suggested that sensibleness of a particular message is often unclear when the previous context already contained messages that were not sensible. experimental results from @xcite suggest that agreement scores may be higher if annotators are only asked to label single responses within a pre-written, sensible context. however, they also note that "final results are always aggregated labels", so the overall proportion of sensible for all 5 methods, proportions of grammatical, sensible, and specific messages are found to be statistically equivalent @xcite to the original (ϵ = 0.001, @xmath2.001). ## results and discussion our results are in for each of the methods, we compared the target cefr to the cefr determined by examiners -henceforth referred to as the true cefr. spearman's ρ and pearson's r show the correlation between the two, and mae is the mean absolute error, where an mae of 1 refers to the difference of 1 cefr level. %gramm., %sensible, and %specific refers to the percentage of grammatical, sensible, and specific responses out of all responses (excluding the original "hello!" prompt). from the correlation and mae scores, we can see that the re-ranking methods work best, with method 5 -reranking with sub-token penalties and filtering -achieving the strongest correlation and lowest mae between the target and true cefr. both vocabulary-based methods performed poorly, achieving almost no correlation and high mae scores. this is somewhat surprising, as one might expect vocabulary to be a key factor in determining text difficulty. we suspect that this is because many of the easier words in the evp also have more difficult word senses, but our method only considered the lowest cefr level. additionally, we looked at a set of 10 randomly sampled dialogues and counted 66 multi-word expressions (mwes) in total, averaging at 0.36 mwes per message. mwes might often be more difficult than their constituent words individually: for example, the idiom a cut above the rest consists of words that are individually simple, but the phrase itself is relatively complex. unfortunately, our vocabulary restriction methods are not able to account for this. ## conclusion and future work this paper presents an initial foray into using opendomain chatbots for language practice. we propose and compare 5 variants of decoding strategies to adjust the difficulty level of generated text. we then evaluate these methods on dialogue generation, and find that our re-ranking strategies significantly outperform vocabulary-based methods. our best variant achieved a spearman's ρ of 0.755 and pearson's r of 0.731. our current work only looks at self-chat difficulty from a teacher/examiner's perspective, which may not transfer well to interactive difficulty. it is also important to ensure that language learners would benefit from this endeavour. for our future work, we will directly engage with learners to investigate the utility and impact of chatbots on language learning. however, there are also areas where the chatbot needs to be significantly improved upon. for example, to cater for a1 learners, we need to be able to generate messages at a1 difficulty. this paper only looked at text complexity in terms of vocabulary, but it may also be possible to adjust the complexity by paraphrasing or altering the sentence structure. we also need to ensure that generated messages are grammatical, sensible, specific, and appropriate. there is ongoing research on grammatical error detection (cf. @xcite , toxic language detection (e.g. @xcite , and improving dialogue consistency (e.g. @xcite , which can be used to improve the chatbot. additionally, a language learning chatbot can be further augmented with other technologies to enhance the user experience, such as grammatical error correction tools, dictionary lookup, or personalisation mechanisms. however, it is not always clear what tools or mechanisms would best facilitate language learning: for example, immediate grammar correction could distract and cause communication breakdown @xcite . we leave this investigation to future research.