Evaluating Vocabulary Usage in LLM s (2024)

Abstract: The paper focuses on investigating vocabulary usage for AI and human-generated text. We define vocabulary usage in two ways: structural differences and keyword differences. Structural differences are evaluated by converting text into Vocabulary-Managment Profiles, initially used for discourse analysis. Through VMPs, we can treat the text data as a time series, allowing an evaluation by implementing Dynamic time-warping distance measures and subsequently deriving similarity scores to provide an indication of whether the structural dynamics in AI texts resemble human texts. To analyze keywords, we use a measure that emphasizes frequency and dispersion to source ‘key’ keywords. A qualitative approach is then applied, noting thematic differences between human and AI writing.

## introduction we are navigating a transformative era, marked significantly by the integration of ai technologies into various aspects of daily life. this is particularly evident in the realm of language learning, where large language models (llms) have become instrumental. llms find application across diverse sectors including education, healthcare, and research, showcasing their versatility and impact @xcite . the role of llms in language acquisition and written composition deserves special attention; it is claimed they offer substantial benefits to learners through personalized learning experiences, interactive prompts for questions and examples, and feedback on writing @xcite . this highlights the potential of llms to enhance the efficacy of language learning strategies significantly. while the potential is certainly undeniable, a factor that is worth addressing is whether texts produced by llms -particularly in the form of examples generated in a learning environmentaccurately represent what a learner is likely to observe in a real-world scenario. in particular, we aim to gain a better understanding of whether llms generate text with respect to different registers in a fashion similar to humans. previous research (alafnan and mohdzuki, 2023; @xcite provides some insight into the perceived 'style' and characteristics of llm production. we narrow our scope to focus on attributes related to discourse and vocabulary, two adjacent concepts that we expect to differ by register. in particular, we are interested in how vocabulary is deployed within the structure of texts. anecdotally, llm text is often described as 'generic' or 'bland' in tone. thus, we were motivated to understand the extent to which such differences are linked to lexical diversity and the rate, or sequencing, with which new vocabulary is introduced. to achieve this, we investigate human authored and machine generated texts through vocabulary-management profiles (vmp). in their simplest interpretation, vmps provide a linear representation, that can be graphically illustrated, representing the rate of newly introduced vocabulary through the progression of a text. ## method we aim to investigate vocabulary usage through two distinct lenses. by implementing vocabulary-management profiles (vmps), we evaluate structural differences in writing patterns, shedding light on how texts from various sources and registers unfold. then, through word keyness analysis, we qualitatively examine words associated with specific sources and registers to grasp salient differences through vocabulary usage better. ## results and discussion initial observations comparing vmps, as illustrated in figure 3 , reveal notable differences across all conditions of varying window sizes for both registers, represented in scenarios of excluding common words (figure 6 ) and including common words (figure 7 ). ## conclusion this study combined more seasoned and newer approaches for evaluating vocabulary usage between human and ai-generated texts. we noted structural differences in text sources, particularly in how vmps respond to our research queries about discernible vocabulary patterns. using distributional moment features like mean and standard deviation; we pinpointed statistical disparities between groups under various conditions, such as window size and vocabulary inclusion. by converting dtw distances into self-similarity measures, we observed marked differences in distributions by register for specific sources. these measurable variations underscore the distinct structural patterns of vmps generated from different sources. further investigation, particularly in response to rq3, uncovered specific vocabulary that served as key indicators of thematic variations related to emotional tone. understanding these variations can help educators and language learners select materials that best align with their learning objectives. we envision a scenario where aspects of llm-produced text with lower mean vmps could be combined with derived word keyness features to seek out text samples that incorporate desired vocabulary and appropriate repetition, an advantage for learning new vocabulary.