Automated Sentence Generation for a Spaced Repetition Software (2024)

Abstract: This paper presents and tests AllAI, an app that utilizes state-of-the-art NLP technology to assist second language acquisition through a novel method of sentence-based spaced repetition. Diverging from current single word or fixed sentence repetition, AllAI dynamically combines words due for repetition into sentences, enabling learning words in context while scheduling them independently. This research explores various suitable NLP paradigms and finds a few-shot prompting approach and retrieval of existing sentences from a corpus to yield the best correctness and scheduling accuracy. Subsequently, it evaluates these methods on 26 learners of Danish, finding a four-fold increase in the speed at which new words are learned, compared to conventional spaced repetition. Users of the retrieval method also reported significantly higher enjoyment, hinting at a higher user engagement.

## introduction spaced repetition is a well-known learning technique that involves repeated exposure to learning material, usually at increasing intervals, which has been shown to enhance long-term retention (see section 2.1). usually, spaced repetition in language learning is done by repeating single words or whole sentences curated by humans. already a decade ago, the potential of computational linguistics for vocabulary learning was identified by @xcite : "there is so much more we could do these days by using corpora and computational linguistics know-how, to extract the to-be learned words from text and to display them with their context. hence, rather than having the user repeat single words (or word pairs) we could display them in various contexts (e.g. sentences), thereby making sure that the chosen ones correspond to the learners' level and interests.." developing a software system that automatically generates sentences for spaced repetition has the potential to provide learners with a more efficient learning experience by generating sentences with many words that are due for repetition, with more personalized and versatile tasks that make studying more enjoyable and engaging. furthermore, it could free up human language teachers to focus on in-person teaching instead of writing example sentences. this work introduces allai (automated language learning with ai), an application utilizing nlp to create such a sentence-based approach to spaced repetition. the app keeps track of the user's vocabulary and generates sensible sentences (spaced repetition "tasks") from only the subset of words of a language that the user knows and currently needs to repeat, with some minor amount of new words that make sense to learn. the user can then calibrate the spaced repetition of each word by answering which of the words in the sentence they correctly remembered. we then investigate the learning outcomes of using such a system compared to current solutions. as such, the main research questions are the following: the proposed system combines the following potential advantages over the conventional spaced repetition approaches mentioned in 2: the main contribution of this work is putting the current and soon-to-be due words of a spaced repetition system into context by investigating different methods of automating the forming of sentences with them. we also develop a metric for calculating the scheduling accuracy and select other metrics to assess the quality of the output sentences for the task. we compare a range of candidate methods and configurations that managed to return sensible sentences containing target words with regard to these metrics. we develop an application consisting of a front-end for the user to interact with the generated tasks and a back-end to do the spaced repetition scheduling and house the developed methods for sentence generation. finally, we test the real-world usefulness of two of the best-performing methods, a retrieval-based method and a gpt-3.5based method using few-shot prompting, in a user study, assessing learning outcomes and indicators of user engagement against a baseline similar to current spaced repetition practices. we implement and test the system in danish. still, it applies to any language in which the sentences are made up of words and is developed in such a way that it could teach a different language if the nlp component is swapped out, e.g. by translating the prompts of a prompting-based solution to a different language. ## comparing candidates methods for the sentence generation component this section describes our simulated study to narrow down the methods and configurations that could optimize the system's objective to two that can be tested in the user study. ## user study in addition to the two selected methods, a baseline method was developed to allow for comparison to the proposed methods in the user study. as the baseline, it was chosen to associate a set sentence (the one with the best bm25 score) with each word in the vocabulary, which is then shown when the word is due. the due word is specially marked and only it can be reported as remembered correctly or not for the spaced repetition. this mimics the common approach of putting a single word on the spaced repetition flashcard, accompanied by some example sentences, as identified in section 2.1, but is put into a comparable format to how the two selected methods are presented to the user. ## implications the results mean first and foremost, that using a sentence-based spaced repetition scheme should be preferred over using single-word spaced repetition, even when the single word is shown in the context of an example sentence. this will show users more vocabulary in less time, increasing efficiency and thus enjoyment. since a retrieval model is far less costly in terms of computing costs and there is light evidence that it is the more time-efficient and enjoyable option, it could be advisable to prefer retrieval over lmbased options, but this would have to be proven in a bigger trial to achieve significant results after bonferroni corrections (see limitation in section 7). on the other hand, even though this specific prompting-based lm method and configuration could not outperform retrieval, with the current rapid advancements in lm size and tasks they can perform through prompting, other lms e.g. gpt-4, which has substantially more parameters than gpt-3.5, could improve correctness and possibly number of due words in the prompt. while our experiments compared the proposed system to a conventional baseline under similar conditions and presentation, we can also compare the results to previous literature. @xcite studies learners' efficiency of learning lists of word pairs and mentions an average of 0.57 words per minute, with 0.34 recalled words after 42 days. thus, it seems that hybrid and retrieval groups with mean of 0.54 and 0.6 words per minute recalled after a few days had a higher efficiency than the results from thorndike's study, even though not directly comparable, since thorndike's study did not have the problem of time being wasted on previously known words, which we did not count for vocabulary growth. ## conclusion the aims of this work were first to identify nlp paradigms and configurations for sentence generation that can optimize spaced repetition timing and best avoid out-of-user-vocabulary words, while keeping the correctness of the generated sentences as high as possible, and then to quantify these methods' influence on user engagement and learning outcomes among language learners, compared to conventional approaches. two methods of achieving these goals were developed: one based on retrieval of suitable sentences from a corpus of high-quality sentences using many upcoming due words as queries, and the other was few-shot-prompting a plm to generate sentences from a subset of the due words. both methods were found to be able to form sentences comprised mostly of words from the user vocabulary, soon to be due and mostly correct, thereby reaching the objectives. while the retrieval method reached 100% correctness, the lm method optimized the spaced repetition scheduling even better but had worse correctness and had an unsolved problem with looping due to the treatment of lemmas, despite multiple countermeasures, making it unsuitable for deployment to users. a hybrid method switching between retrieval and lm generation could solve the looping problem while optimizing the research question's objectives. consequently, the hybrid and the retrieval method were compared to a baseline to answer the second research aim. it was found that the proposed sentence-based spaced repetition significantly increased learning outcomes (four-to-six-fold) compared to the baseline, primarily by increasing efficiency and vocabulary growth by showing more words more quickly, without decreasing the fraction of words remembered by learners. in the retrieval group, a significantly higher enjoyment was observed, possibly due to the higher efficiency, hinting at a higher user engagement. it can thus be concluded that it is beneficial to use the proposed sentence-based spaced repetition over the conventional approach and that the retrieval approach might be advisable over lm-based or hybrid approaches, but that a bigger trial comparing the two is necessary, and further developments, such as fixing problems with lemmatization and looping and higher correctness possibly achievable with newer language models could improve the results when using a more advanced lm based method in the future.