Generating bilingual example sentences with large language models as lexicography assistants (2024)

Abstract: We present a study of LLMs’ performance in generating and rating example sentences for bilingual dictionaries across languages with varying resource levels: French (high-resource), Indonesian (mid-resource), and Tetun (low-resource), with English as the target language. We evaluate the quality of LLMgenerated examples against the GDEX (Good Dictionary EXample) criteria: typicality, informativeness, and intelligibility (Kilgarriff et al., 2008). Our findings reveal that while LLMs can generate reasonably good dictionary examples, their performance degrades significantly for lower-resourced languages. We also observe high variability in human preferences for example quality, reflected in low interannotator agreement rates. To address this, we demonstrate that in-context learning can successfully align LLMs with individual annotator preferences. Additionally, we explore the use of pre-trained language models for automated rating of examples, finding that sentence perplexity serves as a good proxy for “typicality” and “intelligibility” in higher-resourced languages. Our study also contributes a novel dataset of 600 ratings for LLM-generated sentence pairs, and provides insights into the potential of LLMs in reducing the cost of lexicographic work, particularly for low-resource languages.

## introduction example sentences in bilingual dictionaries play a crucial role in language learning, helping l2 speakers to understand the meaning of headwords (words that mark a separate entry in the dictionary), and their usage in context @xcite @xcite . what makes candidate sentences good as examples is the subject of linguistic research, with @xcite proposing the gdex (good dictionary example) framework, which qualifies good examples as typical ("exhibiting frequent and well-dispersed patterns of usage"), the extensive work required to come up with example sentences increases the cost of compiling lexicographic resources @xcite . this has prompted research into the automatic selection of example sentences from existing corpora @xcite . however, existing corpora might not always contain sentences that are suited to language learning, as their text can be overly complex, fail to further explain the meaning of the headword, or not be licensed for reproduction. as a result, researchers have begun exploring models tailored for the generation of dictionary example sentences from a headword and its dictionary definition @xcite . large language models (llms) trained on a wide range of texts @xcite ) might be well suited to formulate generic and informative example sentences that benefit language learning. in particular, their capacity to adapt to new, unseen tasks @xcite means that they might be well suited to generate sentences against specific criteria. however, questions about the quality of the sentences they generate, and their ability to understand what makes a good example, remain. in this paper, we review llms capability to generate and rate example sentences in a bilingual lexicography context, against the gdex criteria. we work with three language pairs, with english on the target side, and source sides that cover a range of language resource levels: french (high-resource), indonesian (mid-resource), tetun (low-resource). the paper makes the following contributions: ## background llms for synthetic data generation. while hallucinations can make llms unreliable for tasks that require factual accuracy @xcite , the text they generate can be of high quality, in some cases preferred over human-generated text by human annotators @xcite @xcite . llm generation of synthetic data has several downstream applications, including the creation of corpora for subsequent training of specialised models @xcite and the generation 1 https://github.com/raphaelmerx/llm-bilingual-examples ## llm generation of bilingual example sentences this section describes our methodology for generating bilingual example sentences using llms, and results from human annotation of these generated sentences. ## automated rating of example sentences beyond baseline performance across different resource levels, we evaluate how well llms can assess example quality. this could enable more efficient dictionary creation pipelines, where automated rating systems that align with human judgement could help filter and select the best examples from larger sets of generated candidates, reducing the need for extensive manual review. furthermore, reliable automated evaluation metrics could facilitate the development of self-improvement systems where llms learn from their own assessments to generate increasingly better examples. ## discussion our study provides several insights into the capabilities and limitations of llms for generating and evaluating bilingual dictionary examples. first, we demonstrate that llms are capable of producing reasonably good quality example sentences across multiple language pairs. however, there is a clear degradation in performance as we move from highresource languages like french to low-resource languages like tetun. the variability in output quality across languages underscores the need for careful evaluation and potential supplementary techniques when applying llms to lexicographic tasks, especially for less-represented languages. a notable challenge revealed in our study is the high variance in personal preferences for example sentence quality, as evidenced by low interannotator agreement rates. this variability poses difficulties in establishing a single, universally accepted metric for evaluating dictionary examples. however, our experiments with in-context learning demonstrate that llms can be successfully aligned with individual annotator preferences, even for low-resource languages like tetun. this finding suggests a promising avenue for tailoring llm outputs to specific lexicographic standards or individual annotator judgements, potentially facilitating the example generation and evaluation process. the low inter-annotator agreement observed in our study highlights the need for annotations from multiple annotators before drawing conclusions about the quality (or lack thereof) of example sentences. this multi-annotator approach can help capture a more comprehensive range of perspectives and mitigate individual biases. additionally, our findings, particularly for french where most gdex criteria were rated "yes" due to the high quality of generated sentences, suggest the need for finer measures of criteria to better capture nuanced levels of quality. we recommend developing more granular rating scales or additional sub-criteria, especially for high-resource languages where llms perform well. this refinement in evaluation methods could provide more discriminative assessments of llm-generated example sentences. ## conclusion we contribute a first evaluation of llm capability to generate bilingual example sentences, across languages of various resource levels. we show that although llms are capable of generating good bilingual example sentences on average, their performance degrades with language resource level. we further show that even when using a shared framework for sentence evaluation (gdex), annotators tend to disagree with each other on sentence quality, but that in-context learning can be leveraged to align llms with a specific annotator's ratings. our findings highlight the potential of llms in lowering the cost of lexicographic work, and their ability in aligning with human judgement in a field where human judgement can be highly variable. this is of particular value in low-resource lexicographic work, where lack of human resources may prevent the widespread compilation of lexicographic resources.