LLM -Generated Contexts to Practice Specialised Vocabulary: Corpus Presentation and Comparison (2024)

Abstract: This project evaluates the potential of LLM and dynamic corpora to generate contexts ai- med at the practice and acquisition of specialised English vocabulary. We compared reference contexts—handpicked by expert teachers—for a specialised vocabulary list to contexts generated by three recent large language models (LLM) of different sizes (Mistral-7B-Instruct, Vicuna-13B, and Gemini 1.0 Pro) and to contexts extracted from articles web-crawled from specialised websites. The comparison uses a representative set of length-based, morphosyntactic, semantic, and discourse- related textual characteristics. We conclude that the LLM-based corpora can be combined effectively with a web-crawled one to form an academic corpus characterised by appropriate complexity and textual variety.

## introduction the present study is part of a broader project conducted at université catholique de louvain that aims at leveraging natural language processing (nlp) tools to facilitate the acquisition of english for specific purposes (esp) vocabulary by providing multiple examples of its natural use in context. the project involves several university courses mapped to proficiency levels b1 to c1 and designed to teach esp to stem (science, technology, engineering, and mathematics) students. in these courses, students have to master predefined vocabulary lists comprised of both strictly scientific vocabulary (e.g. "a chemical") and other vocabulary that commonly appears in scientific contexts (e.g. "to assume"). a preliminary survey revealed that the target students currently study the lists as is, at best using flash cards. research has, however, shown the importance of regularly exposing learners to words in authentic and informative contexts @xcite @xcite . collecting authentic contexts is, unfortunately, very time-consuming for esp teachers, which generally impedes offering enough authentic contexts to students as support for vocabulary learning. this is why we intend to automatically retrieve and generate contexts of use for any target esp word, thereby significantly decreasing the burden in terms of both time and effort that currently results from the manual collection of context sentences. in this study, we explore two ways of collecting such contexts : firstly, their extraction from a large corpus made up of websites routinely exploited by esp teachers ; and secondly, their generation using three large language models (llms) : mistral-7b-instruct, vicuna-13b, and gemini 1.0 pro. more specifically, this paper analyses the linguistic characteristics -using standard readability-and stylometry-related variables -of sentences in our web-crawled and llm-generated corpora and their similarities and differences with a reference corpus. the last is made up of examples handpicked by esp teachers among vocabulary test materials used in university esp courses. ## methods in the current study, we collect and analyse two general types of contexts for esp vocabulary learning : generated by llms (mistral-instruct, vicuna, and gemini pro) and obtained through web-crawling. figure 1 illustrates the different steps of this procedure. the resulting corpora, described in section 3.1, are compared to a reference esp corpus associated with the same vocabulary items, handpicked by teachers on the basis of a set of stylistic features introduced in section 3.2. we hypothesise that the most adequate generation method produces contexts closest to the reference corpus in terms of stylistic characteristics. 3.1 derivation of the corpora 3.1.1 reference the reference corpus, consisting of 244 contexts, is crafted by esp teachers from université catholique de louvain and consists of a sample of study and examination materials used in the acquisition of vocabulary knowledge in esp courses. the provided contexts are typically one to three sentences long and reflect key pedagogical qualities as defined by the esp teachers, including appropriate length, topic relevance, authenticity, timeliness and level-appropriate vocabulary and grammar. the corpus's items are further classified as belonging to cefr levels b1 or b2 and to the fields of "agronomy" or "general science". ## results the results of our stylistic comparison are reported in ## discussion given the high value ranges and sd pertaining to the web-crawled corpus, as well as the tendency between its and llm-generated texts to diverge from the baseline in opposite directions, the two types of corpora can work together effectively within an educational framework to provide a variety of contexts for esp vocabulary. the web-crawling method is computationally efficient and tends to provide texts that are naturally close to the reference examples. in turn, the benefits of llm generation include sensitivity to the cefr level at hand and high malleability, including the potential to derive a large number of examples from a single one within a one-shot setting. ## conclusion and future directions this work discussed context corpora derived via two discrete nlp-based methods (web-crawling and generation by llms), which can be used to aid university students in the acquisition of esp vocabulary. the stylistic characteristics of the generated contexts were compared to a professionally crafted baseline of context examples through quantitative evaluation based on readability features. the comparison revealed higher similarity among different llm-generated corpora than between them and corpora of a different nature. the "gemini : one-shot" corpus was discovered to be globally closest to the reference, thereby showing that generation can be refined through prompt engineering. future experiments including few-shot use of only several high-quality examples independent of a vocabulary list can help mitigate the current limitation of reliance on a reference corpus. associated future plans include an evaluation of the pedagogic characteristics of contexts as opposed to their readability-based qualities. in more practical terms, pre-and post-tests of students' performance will be conducted in relation to the introduction of a large inventory of nlp-derived examples in the implied university courses.