Low-Cost Generation and Evaluation of Dictionary Example Sentences (2024)

Abstract: Dictionary example sentences play an important role in illustrating word definitions and usage, but manually creating quality sentences is challenging. Prior works have demonstrated that language models can be trained to generate example sentences. However, they relied on costly customized models and word sense datasets for generation and evaluation of their work. Rapid advancements in foundational models present the opportunity to create low-cost, zero-shot methods for the generation and evaluation of dictionary example sentences. We introduce a new automatic evaluation metric called OxfordEval that measures the win-rate of generated sentences against existing Oxford Dictionary sentences. OxfordEval shows high alignment with human judgments, enabling large-scale automated quality evaluation. We experiment with various LLMs and configurations to generate dictionary sentences across word classes. We complement this with a novel approach of using masked language models to identify and select sentences that best exemplify word meaning. The eventual model, FM-MLM, achieves over 85.1% win rate against Oxford baseline sentences according to OxfordEval, compared to 39.8% win rate for prior model-generated sentences.

## introduction dictionary example sentences play a vital role in illustrating the meanings and usage of headwords for dictionary users. prior studies have found evidence supporting the importance of the quantity and quality of example sentences in improving learners' performance for receptive and productive language tasks @xcite ). creating and maintaining example sentences for dictionaries can be a labour-intensive task -the oxford dictionary of english @xcite , for example, aims to represent the current usage of †, ‡ equal contribution close to 100,000 english headwords. traditional efforts to improve productivity of example sentence creation focused on searching for sentences with headwords in existing electronic text corpus @xcite @xcite . however, such text corpora consists of content for myriad purposes, and may not always contain example sentences that best exemplify the meanings and usage of words to support language learning. to address this, recent works have demonstrated that language models can be trained to generate new sentences on unseen headwords @xcite , vastly expanding the search space that can be considered for crafting quality sentences. however, they often rely on custom-trained models and datasets annotated with word senses for the generation and evaluation of sentences, which can be a costly process. rapid advancements in foundational models (fms) @xcite now offer new possibilities for more flexible and creative generation of dictionary example sentences at low cost. closed and open-source large language models (llms) like claude and llama-2 have demonstrated an impressive capacity to generate fluent, coherent text while capturing nuances of style, tone and topic @xcite . their exposure to large corpora of linguistic data during pretraining @xcite and ability to follow specific instructions @xcite allow these models to perform well for new previously unseen instructional tasks @xcite . they offer the potential for dictionary example sentences to be generated and evaluated at scale without the use of specialised models or datasets. this paper examines low-cost, zero-shot methods for llms to automatically generate and evaluate dictionary example sentences. we start by defining the oxfordeval metric, which measures the win-rate of candidate dictionary example sentences when evaluated competitively by llms for quality against samples from the oxford dictionary. using human annotations on a subset of outputs, we validate that oxfordeval matches well with human preferences. with this validated automatic evaluation measure, we are then able to expansively test various state-of-the-art models including claude, llama-2 and mistral, and analyse how variations in generation methodology affect sentence quality. we show that llms can generate sentences that are preferred to oxford dictionary example sentences 83.9% of the time, while past model-generated sentences only have a win-rate of 39.8%. in addition, we develop a novel method of adapting pre-trained masked language models to generate measurements of how much a generated sentence exemplifies the meaning of a word. when this method is used to rerank a set of 5 sentences generated by llms for each word for use in the oxfordeval measurement, the win-rate further increases to 85.1%. we estimate that one end-to-end run of the generation, reranking, and evaluation steps for a set of 8000 word senses and definitions costs less than $50. put together, we hope the work provides a refreshed low-cost baseline for the development of methods to generate high quality dictionary example sentences for the benefit of language learners. ## related work recent work has demonstrated that language models can be trained on existing sets of headwords and example sentences from dictionaries and other corpora for automatic example sentence generation. @xcite and @xcite adapt encoder-decoder architectures trained on reference sentences for each word sense to generate new sentences on unseen headwords, with @xcite demonstrating the ability to control for length and lexical complexity. @xcite demonstrate that sentence generation can also be performed in a one-shot manner using just a reference sentence without word sense labels (which are less abundantly available), using an autoencoder trained to generate new sentences conditioned on latent sense representations. earlier studies of dictionary example sentences use a small-scale experimental setup to evaluate the usefulness of dictionary headwords with and without examples on learning outcomes @xcite . these methods measure the direct impact for learners, but are limited to small-scale experimental settings. later studies use specific model-based scores to automate the evaluation of sentence quality. @xcite compared the cosine similarity of pretrained bertlarge word embeddings @xcite of the target word w* between generated sentences and example sentences in a golden dataset. @xcite trained a bert model to evaluate definition and part-of-speech (pos) accuracy using triplets of (word, definition, example) and (word, pos, example) respectively, with positive examples from oxford dicionary and negative examples generated synthetically by replacing words, definitions, pos, and examples. however, these automatic methods evaluate narrow factors (word sense accuracy, pos accuracy and definition accuracy) that have not been shown to be correlated with human-validated preferences for the sentence as a whole. recent studies have demonstrated the ability of llms to evaluate free-form outputs of llms as instruction-following agents @xcite and chatbot assistants @xcite , with a high degree of correlation to human preference when performing pairwise comparison on the overall quality of outputs. however, they also warn of position bias, verbosity bias and self-enhancement bias @xcite . ## sentence generation in this section, we present the fm-mlm (foundational model -masked language model) method for generating quality dictionary example sentences at low cost. fm-mlm uses a three-step process. first, foundational llms are used to generate a set of candidate sentences. next, pre-trained masked language models are adapted with a novel method to score how much each candidate sentence exemplifies the meaning of the target word. finally, the sentence with the highest score is selected to be the final generated sentence for evaluation. ## main results sentences generated by the fm-mlm model achieve a win-rate of 85.1% when compared to example sentences in the oxford dictionary dataset, demonstrating the ability of fm-mlm to generate sentences that compete favorably against expert references. without the use of mlm exemplification scores to select sentences, the win-rate drops slightly to 83.9%. using mlm exemplification scores to select amongst samples in the oxford dictionary dataset gives a win-rate of 56.7%. these results demonstrate that the mlm exemplification scores can be used to identify sentences of higher quality from a set. cdeg-beam and cdeg-greedy sentences achieve a win-rate of 39.8% and 39.1% respectively. this is aligned with results re-ported in @xcite , where s2s @xcite and exmaker @xcite were found to have slightly lower human-annotated scores, in terms of semantic match and fluency, when compared against gold samples in the oxford dictionary and semcor dataset. we note that the win rate for claude instant (batched generation) over baseline oxford sentences based on 447 agreed human annotations was 76.5%, which is similar to the win-rate of 78.5% evaluated by the chosen llm evaluator shown in by examining the secondary metrics, we see that sentences generated by fm-mlm has an average (sd) of 16.9 (4.0) words per sentence and an average (sd) fkgl of 8.9 (3.2). these are lower than the sentences in the oxford dictionary dataset, which has an average (sd) of 18.7 (7.4) words and an average (sd) fkgl of 9.1 (3.9), indicating that the oxford dictionary dataset has a more diverse range of sentences that are also typically more advanced in nature. sentences generated by cdeg models tend to be shorter, simpler and more diverse in nature, with an average (sd) of 11.2 (1.3) words and average (sd) fkgl of 5.3 (2.9) for cdeg-beam. we also obverse that the mlm selection step tends to favor slightly longer sentences, with the average words and fkgl levels increasing slightly for fm-mlm as compared to fm only, and oxford (+mlm selection) compared to oxford (1st sentence). ## conclusion rapid advancements in llm research has led to the creation of powerful, general-purpose models that can be flexibly adapted for myriad tasks without the need for expensive customisation efforts. while previous works years before this paper had to rely on compute-intensive model training steps and scarce datasets of word-sense annotations, we are now able to study dictionary example sentence generation at a fraction of the cost. in this paper, we demonstrated the fm-mlm model, which uses foundational llms to generate dictionary example sentences, and then uses a novel adaptation of pretrained masked language models to provide a measure of how much they exemplify the meaning of words. with automatic evaluation methods of the overall sentence quality that are validated with human preference data (in the form of the oxfordeval metric), we are able to easily perform extensive ablation studies on how various modelling choices and hyperparameters affect the effectiveness of the model, which we note was difficult for past studies. sentences generated by the fm-mlm model achieve a winrate of 85.1% when evaluated competitively against typical sentences in the oxford dictionary dataset. the high win-rate indicates the potential of adapting the fm-mlm model in practice to generate example sentences for actual use by language learners. however, for safety reasons, we recommend that any real-world application should be carefully supervised and governed to manage risks and limitations which we discuss in the limitations section. we identify many potential areas for impactful future work. to improve sentence quality further, researchers can consider methods to customise and tune the foundational models for this specialised task. to work towards real-world use, we can also perform more rigourous and holistic evaluations on generated sentences for accuracy, fluency, and other factors of interest. evaluation can also be conducted on sets of multiple sentences, rather than single sentences as we have done in this study, in order to better capture factors such as sentence diversity that would apply for a real dictionary. lastly, the work can also be extended for other languages. we hope that our work enables these studies by providing a refreshed low-cost baseline reference for good quality sentence generation and evaluation that future researchers can build on.