Controlled Language Generation for Language Learning Items (2022)

Abstract: This work aims to employ natural language generation (NLG) to rapidly generate items for English language learning applications: this requires both language models capable of generating fluent, high-quality English, and to control the output of the generation to match the requirements of the relevant items. We experiment with deep pretrained models for this task, developing novel methods for controlling items for factors relevant in language learning: diverse sentences for different proficiency levels and argument structure to test grammar. Human evaluation demonstrates high grammatically scores for all models (3.4 and above out of 4), and higher length (24%) and complexity (9%) over the baseline for the advanced proficiency model. Our results show that we can achieve strong performance while adding additional control to ensure diverse, tailored content for individual users.

## introduction recent advancement of transformer @xcite based pre-trained language models (lm) @xcite @xcite have resulted in unprecedented success in generating large amounts of fluent english text. one possible area where text generation can be applied is item generation for english language learning applications (llas). llas are popular apps used by millions of people all over the world. 2 these apps often include multiple choice items for vocabulary tests, flashcards, grammar lessons, and more. typically, such items are created manually @xcite or curated from crowd-sourced sentence database, e.g., tatoeba (settles et al., 2020). 3 {dog-arg0, chase-v, cat-arg1} {dog-arg1, chase-v, cat-arg0} ## data we employ two datasets for concept2seq generation. each instance in the datasets is a set of concepts paired with a sentence that contains those concepts. first, we use the commongen data which is based on existing caption corpora @xcite . from this dataset we use 71,408 concept/sentence pairs. although commongen is used in related concept2seq generation @xcite , since it is based on image captions, many samples are phrases (not sentences) and less diverse. thus, we also collect another dataset based on fourteen rel-evant vocabulary items for language learning that belong to different cefr levels. 6 sentences are collected from diverse sources such as the rocstories @xcite , tatoeba sentence database, and the google book corpus. 7 we first retrieved sentences containing the vocabulary items from these different sources and then extracted the concepts using the conceptnet knowledge-base by employing @xcite . we extract noun, verbs, and adjective tokens as concepts and keep only those sentences containing 2-5 concepts to keep consistency with commongen. this dataset is denoted as the vocabulary dataset and consists of an additional 218,997 concept/sentence pairs. in total, the commongen and vocabu-lary gives us a dataset of 290,399 pairs of concepts and sentences. to evaluate our generation models, we create two test sets to evaluate two different scenarios. our goal is to evaluate concept sets that occur frequently in the training data, as well those that occur rarely. in order to build these two types of test sets, we first generate the frequency counts of each concept over the entire dataset. we then calculate the frequency of a given concept set as the sum of the frequency counts of each concept within that set. we then sample 500 instances from both the com-mongen and vocabulary datasets from the top 10% highest frequency concept sets and 500 each from the bottom 10% frequency sets. we split the test data this way to evaluate two likely use cases. the lowest 10% aligns with the case where we have unseen concepts and want to generate something novel; the most frequent 10% matches the everyday use case where we generate from things we've seen before. this gives us a test set of 2000 total sample, half from commongen and half from vo-cabulary, additionally split into high frequency and low frequency concepts. from the remaining dataset we randomly use 90% as training and 10% as validation. ## methods here, we present our computational approaches for sentence generation using the concept2seq framework. at its base form, our task is to generate a sentence s that consists of sequence of tokens, @xmath0, ..., s m } using a list of concepts @xmath1, ..., c n }. using the standard autoregressive sequence-to-sequence architecture @xcite we model p θ (s | c) as follows: note, the resulting sentence s should contain relevant vocabulary from the concept set c, but is otherwise unconstrained. we use a pretrained bart model @xcite composed of a bidirectional encoder and an autoregressive decoder. in our simplest setup (called base), the input for bart is a set of concepts c and the output text s is a sentence that contains those concepts. for the base model, we fine-tune the bart-base model on our training data described above, and then generate sentences based on the test concepts. note, this is equivalent to the bart based experiment reported in lin et al. (2020). foot_4 ## results evaluating natural language generation tasks can be difficult, and some automatic metrics can be problematic @xcite . to overcome these difficulties, we use metrics specifically tailored to our task, as well as performing manual evaluation to get a concrete understanding of model performance. ## related work the concept2seq generation problem has been investigated in several recent studies. @xcite released the commongen dataset and generated sentences using various transformer models, @xcite have proposed prompting for generation, and @xcite have conducted instruction tuning for generation using concepts. our work is related to the above and our novelty is that we utilize this framework to generate lla items. although we did not experiment with the ordering of concepts similar to @xcite , our srl based generation in fact implicitly control the order of the concepts by offering specific grammar roles. in prior work on controllable generation, embedding vectors of the control variables were fed into the model to control the output @xcite , whereas our approach resembles recent efforts where the control variable is concatenated to the main input @xcite to control particular style, such as sentiment, style for chatbots, diverse story continuations and argument generation @xcite @xcite . ## conclusion and future work we proposed a type-controlled sentence generation framework for llas. we generate sentences (a) conditioned on the cefr levels to provide content for users/students who belong to different skill sets (e.g., beginner or proficient in english), and (b) conditioned with specific argument structures for grammar. in automatic evaluation, the srl model shows better coverage of input concepts than base, whereas human evaluation demonstrates high grammatically scores (3.4 and above) for all the models as well as high complexity for the cefr c2 model that was designed to generate complex sentences for proficient users. in future, we want to continue a couple of error analyses on the input as well as on the generated sentences. having taken into account that input data is pre-processed in several ways (e.g., concept extraction @xcite and analysis of semantic roles @xcite ), we want to select a small subset of data to determine whether such extraction has any error. likewise, we also want to employ expert content developers to analyze the results of the cefr predictor. finally, we plan to employ additional controls such as word senses to guide context specific generations.