W i C Evaluation in G alician and S panish: Effects of Dataset Quality and Composition (2025)

Abstract: This work explores the impact of dataset quality and composition on Word-in-Context performance for Galician and Spanish. We assess existing datasets, validate their test sets, and create new manually constructed evaluation data. Across five experiments with controlled variations in training and test data, we find that while the validation of test data tends to yield better model performance, evaluations on manually created datasets suggest that contextual embeddings are not sufficient on their own to reliably capture word meaning variation. Regarding training data, our results suggest that performance is influenced not only by size and human validation but also by deeper factors related to the semantic properties of the datasets. All new resources will be freely released.

## introduction lexical ambiguity (e.g., polysemous words conveying different senses depending on the context) is a central feature of natural languages, and its resolution remains a challenge for computational models, as distinguishing between different senses of a word can be difficult even for humans (bevilacqua et al., 2021). in nlp, one of the most widely used tasks to evaluate model performance in lexical disambiguation is word-in-context (wic) and its extensions to other languages (pilehvar and camacho-collados, 2019; raganato et al., 2020), where the goal is to determine whether a target word used in two sentences has the same meaning or not. these datasets are typically built using existing lexical resources, primarily wordnet and wiktionaries. as a result, their quality depends heavily on the coverage and reliability of these underlying sources, as well as on the specific methodology used to construct the wic instances. in this context, human performance on wic datasets varies considerably, e.g., around 75% in japanese and 76% in korean, compared to 97% in farsi, where sen-tence pairs were manually grouped by an annotator (raganato et al., 2020). @xcite in the case of languages such as galician and spanish, existing datasets have been automatically constructed using lexical resources with limited coverage, often relying on machine translation both in the development of the wordnets and in the generation of sentence examples (vázquez abuín and garcia, 2025). moreover, these datasets have neither been validated nor evaluated with respect to human performance, which makes it difficult to assess their overall quality and reliability. this work investigates the impact of dataset composition and quality on model performance in the wic task, focusing on both training and evaluation data. we begin by assessing the quality of existing test sets for galician and spanish, followed by a validation process to remove instances that are ambiguous for human annotators. additionally, we construct two new evaluation datasets foot_1 in which all sentences are manually authored and validated by experts-resources that are also useful for analyzing potential data contamination issues (sainz et al., 2023). using these resources, we design five experiments that show that while validating evaluation data tends to improve model performance, results on manually created datasets suggest that contextual embeddings alone are not sufficient to reliably capture word meaning variation. however, during training, data size and human validation seem to have a more limited effect, while other factors related to the semantic properties of the data (such as the well-known effect of word not seen during training) may have a more substantial impact. ## datasets and human annotation original datasets: we start by employing, to the best of our knowledge, the only available wic datasets for galician and spanish (vázquez abuín and garcia, 2025). these datasets were constructed following the same methodology as the original wic paper (pilehvar and camacho-collados, 2019), using wordnets from the multilingual central repository (mcr) (gonzalez-agirre et al., 2012). @xcite the galician dataset comprises 1500 training, 400 development, and 1400 test instances. for spanish, only two splits are available: 200 instances for training or development, and 800 for testing. in all cases, each instance consists of a pair of sentences containing the same word form, along with a binary label indicating whether the word has the same meaning in both contexts ('true') or not ('false'). it is worth mentioning that some instances of the galician datasets were translated from english or spanish using machine translation. while the translations were validated by native speakers, the original paper only mentions a manual review of the test data, but does not report any human evaluation of the final datasets. validation of the test sets: to assess the quality of the original automatically created dataset, we randomly selected 150 instances from the test sets, which were independently annotated by three experts, bilingual speakers of both languages. annotators performed the wic task by determining whether a target word carried the same meaning in two different contexts, assigning a binary label (0 for 'false', 1 for 'true'). no external resources were made available during the annotation process. the average agreement between each annotator was 71% and 63% for galician and spanish, respectively. @xcite ## results and discussion the best results for each model can be found at tables 4 and 5 , including the highest results for the baselines. @xcite focusing on the impact of the different test sets, we observe a slight performance improvement when using the human-validated test set in most of the models (exp1 vs. exp2), suggesting that human review contributes to the reduction of potential ambiguities and errors of the automatically generated test sets. however, when evaluating on the manually created and validated test sets, we observe a notable performance drop in both languages, particularly for the logistic regression classifiers, and to a lesser extent for the baselines (exp2 vs. exp3). this suggests that, despite being less ambiguous, the manually constructed datasets pose greater challenges for models relying on contextualized representations, as simpler cosine similarity-based methods outperform the classifiers. @xcite notably, the baseline results remain largely consistent across different training configurations (exp3-exp5 for galician and exp4-exp5 for spanish), as they share the same test set and the baselines are only minimally influenced by the training data. contributing factors may include the fact that these sentences were not seen during pretraining, unlike those derived from wordnet and other public resources, which were likely included in the models' pre-training data, and that they may also differ in nature from examples originating in such lexical resources. concerning the impact of incorporating humanvalidated data into the training (exp3 vs exp4), no remarkable changes in the overall performance were observed. however, a deeper analysis re- @xcite the complete results of the baselines are shown in tables 6 and 7 in appendix a while the full results by layers and cosine thresholds are reported in tables 8 and 9 in appendix b. finally, although the results from training models exclusively on a small validated dataset (which, in the case of spanish, is larger than the initial one) are not conclusive, it is noteworthy that in some cases this setup outperforms training on a larger corpus (exp3), even when including the validated data itself (exp4). in this respect, and given that the datasets include a range of semantic phenomena (e.g., homonymy, different types of polysemy), these results suggest that generalization may be hindered not just by data size or quality, but also by other factors such as the semantic relatedness between training and test instances or the presence or absence of regular polysemy patterns. ## conclusions and further work this paper presented an evaluation of the impact of dataset composition and quality on wic performance for galician and spanish. we began by assessing the quality of publicly available datasets for these languages, followed by a validation process to enhance the reliability of the test sets. in addition, we constructed new manually created datasets for both languages, also verified by expert annotators. to systematically examine the effects of data quality and composition, we conducted five experiments involving controlled variations in both training and test data. while models appear to handle many ambiguous cases in the automatically constructed datasets, they often struggle when evaluated on the manually created ones. this suggest that contextualized representations may not fully capture fine-grained sense distinctions, and that simpler methods based on cosine similarity can sometimes be more reliable. regarding training data, our findings suggest that performance depends not only on the amount and validation of data but also on deeper factors that deserve further analysis, such as the proportion of target words shared between training and test sets, or the semantic relatedness and distribution of polysemy patterns across datasets. it is worth noting that the models used in this study are general-purpose pretrained models, and