How Do Large Language Models Evaluate Lexical Complexity? (2025)

Abstract: In this work, we explore the prediction of lexical complexity by combining supervised approaches and the use of large language models (LLMs). We first evaluate the impact of different prompting strategies (zero-shot, one-shot, and chain-of-thought) on the quality of the predictions, comparing the results with human annotations from the CompLex 2.0 corpus. Our results indicate that LLMs, and in particular gpt-4o, benefit from explicit instructions to better approximate human judgments, although some discrepancies remain. Moreover, a calibration approach to better align LLMs predictions and human judgements based on few manually annotated data appears as a promising solution to improve the reliability of the annotations in a supervised scenario.

## introduction the prediction of lexical complexity is an essential task for adapting linguistic content to the specific needs of learners and educational systems. such a task consists in predicting a numerical complexity score for a target word in a given sentence (thereafter an instance). data annotation plays a key role in this task, directly influencing the performance of supervised models. with the emergence of largescale language models (llms) the possibility of using automatically generated annotations raises new questions regarding the generalization and robustness of these models. in this work, we focus on measuring the similarities between human annotators and generative models (llms) by varying the prompts. the objective is to determine whether it is possible to use llms as reliable annotators by measuring their level of agreement with human annotations and by analyzing the distribution of the produced annotations. we also seek to identify new perspectives for improving the alignment between these two sources of annotations using a calibration model based on few manually annotated data. we specifically apply this approach in the context of a supervised model trained on llm-based annotated data to avoid the use of llms at prediction time, prioritizing time efficiency and energy conservation. all our experiments were performed on the complex 2.0 dataset @xcite for english. this dataset has the advantage of including the source individual human annotations that can be used for directly comparing human and llm annotations. the paper is organized as follows. section 2 presents related work with respect to lexical complexity prediction and data annotation using llms. next, section 3 describes the complex 2.0 dataset, the llm strategies to be tested as well as the supervised model used in the final experiments. then, sections 4 and 5 evaluate llms performance against human annotations. finally, section 6 explores supervised scenarios integrating a calibration model for llms. ## data and models this section will present the models and the data that will be used to (i) evaluate the performances of llms with respect to a gold standard and also with respect to individual human annotations; (ii) evaluate the impact of llms in a supervised scenario where the llms are only used to annotate the training dataset, in order to reduce the energy costs and improve response-time efficiency. ## evaluation of llms performances against human-based gold complexity scores in this section, we analyze the performances of the 21 llm systems derived from our three prompting strategies (section 3.2). we compare the predicted lexical complexity scores with the gold scores, that are, for each instance, the average of several individual human numerical annotations. ## comparison of llm predictions with individual human annotations in this section, we compare llm predictions with the individual human annotations. unfortunately, the way the dataset is annotated using the mturk platform with a limitation of 10 human annotations per instance makes a fair comparison difficult, whereas we have the predictions of all the llms per instance. although there are clear limitations in the various provided evaluations below due to this issue, the results will reveal some trends that will pave the way for other experiments. ## train supervised model in a real scenario, annotating an instance using 10 llms simultaneously would be very expensive in terms of time, money and energy cost compared to using a small supervised model. in this section, we train various supervised models on the complex 2.0 dataset, trying to take advantage of llms to annotate the training data, and therefore limiting their use to an offline setting. ## conclusion in this study, we explored the prediction of lexical complexity by using large language models (llms) with different prompting strategies (zero-shot, oneshot, chain-of-thought). our experiments show that adding structure and explicit examples significantly improves the models' ability to approach human judgments, with gpt-4o notably standing out with high correlations and better alignment with the reference annotations. the comparative analysis of predictions distributions and errors (mse) highlights significant variability between llm predictions and human evaluations. while some models (such as llama3 and mistral) manage to approach human performance in certain scenarios, others (such as qwen2) exhibit marked biases or excessive dispersion in their predictions. these findings underscore the importance of precise calibration and prompt optimization to fully leverage the capabilities of generative models. moreover, although training a supervised model on human annotations remains the performance benchmark (@xmath0,79, r 2 = 0, 62, @xmath1,0065), our results show that the use of a calibration model which integrates an optimized weighting of the llms' predictions yields significantly higher scores than simply averaging the llms predictions, with pearson coefficients reaching up to 0,75 and r 2 values of 0,57 with as few as 500 examples. this improvement, consistent across various subsets, confirms that calibration by stacking enables a better use of the combined richness of human annotations and automatic predictions, while drastically reducing the number of human annotations required. in brief, our work shows the potential of llms and in particular that of the calibration models for lexical complexity prediction. however, the variability observed in certain metrics, such as the mse, and the persistent gaps with human annotations call for continued optimization efforts, notably by refining prompting techniques and calibration strategies. future research could focus on improving the self calibration of generative models and adapting these approaches to other languages and educational contexts, in order to fully exploit the synergy between human annotations and automatic predictions.