S en W i C h: Sense-Annotation of Low-Resource Languages for W i C using Hybrid Methods (2025)

Abstract: This paper addresses the critical need for high-quality evaluation datasets in low-resource languages to advance cross-lingual transfer. While cross-lingual transfer offers a key strategy for leveraging multilingual pretraining to expand language technologies to understudied and typologically diverse languages, its effectiveness is dependent on quality and suitable benchmarks. We release new sense-annotated datasets of sentences containing polysemous words, spanning nine low-resource languages across diverse language families and scripts. To facilitate dataset creation, the paper presents a demonstrably beneficial semi-automatic annotation method. The utility of the datasets is demonstrated through Word-in-Context (WiC) formatted experiments that evaluate transfer on these low-resource languages. Results highlight the importance of targeted dataset creation and evaluation for effective polysemy disambiguation in low-resource settings and transfer studies. The released datasets and code aim to support further research into fair, robust, and truly multilingual NLP.

## introduction cross-lingual transfer is a key strategy in modern nlp, particularly for low-resource languages, where training data is scarce. by leveraging multilingual pretraining, models can transfer taskspecific abilities from high-resource languages to low-resource ones, expanding access to language technologies for underrepresented communities @xcite @xcite . despite its promise, transfer learning is not universally effective across tasks or languages. studies on tasks like pos tagging, ner, nli, qa, and sentiment analysis @xcite @xcite @xcite , as well as polysemy disambiguation @xcite , show that cross-lingual transfer can be inconsistent and, in some cases, fail entirely. this is also true for generative models @xcite @xcite , with particularly poor performance in low-resource languages, highlighting the need for more robust and languageinclusive transfer. a main obstacle for transfer is the lack of highquality datasets in low-resource and typologically diverse languages. without these benchmarks, assessing transfer performance, let alone training models on target languages, remains a formidable challenge. this lacking is largely due to the scarcity of linguistic resources in low-resource languages. for instance, wiktionary contains over a million entries for german, english, french, chinese, and russian, but fewer than 100,000 for punjabi and marathi @xcite . this lack of resources underscores the urgent need for dedicated datasets to evaluate and refine transfer techniques for underrepresented languages, which this work addresses by developing a semiautomatic method for sense annotation in polysemy and generating resources in ten languages. we focus on the task of polysemy disambiguation, as it particularly challenges cross-lingual transfer by revealing structural and semantic differences between languages. while some nlp tasks, like sentiment analysis, rely on meaning preservation across languages, where direct translation can maintain performance, polysemy is highly language-specific @xcite , making it a rigorous test of a model's ability to generalize across languages. for example, the english word "movement" refers to both physical motion and a political or social movement. however, its polish translation, "ruch", also encompasses these two meanings, but additionally means "traffic", a sense not covered by the english word. conversely, "movement" in english can also refer to a section of a musical composition. polysemy disambiguation has long been considered a hallmark of human cognition and a central challenge in nlp @xcite . a model that can accurately distinguish between different senses of a word must capture linguistic subtleties, metaphorical meanings, and even emerging word usages, much like human speakers. thus, success in cross-lingual polysemy disambiguation would suggest a model's ability to generalize deep semantic understanding, beyond surface-level patterns in a single language. while many high-resource languages already benefit from sense-annotated datasets (see §2), lowresource languages remain largely unrepresented in this area. existing contextualized models can process polysemous words within downstream tasks @xcite , but sense disambiguation remains a major challenge across dozens of languages @xcite @xcite . beyond nlp, polysemy also presents difficulties in multimodal models, such as object detection systems, where the same word can refer to multiple visual categories @xcite . this suggests that solving polysemy is not just beneficial for language tasks but has broader implications for ai reasoning and multimodal understanding. our contributions despite extensive work on polysemy disambiguation in high-resource languages, datasets for low-resource languages remain scarce. we address this gap with the following contributions: ## sense-annotated datasets we introduce a sense-annotated corpus of sentences containing polysemous words covering ten low resource languages that span different language families and use different scripts: azerbaijani (turkic), kannada and telugu (dravidian), punjabi, marathi and urdu (indo-aryan), polish (slavic), swahili (afro-semitic), vietnamese (austroasiatic) and korean (koreanic). statistics for each language are presented in ## experiments to assess the quality of the datasets we created, and to demonstrate the need for proper evaluation in low-resource languages, we tested transfer in three transfer conditions, full-shot, zero-shot and mixed. the full-shot condition is mainly a sanity-check, and serves to evaluate the quality of the training set, as it does not test for transfer. in zero-shot, a model is fine-tuned on english (combined training data taken from the mcl @xcite and xl @xcite datasets, totaling 13.4k sentence pairs) and evaluated on each of our ten languages, which it was not fine-tuned on. in the mixed condition, a model is first fine-tuned on english, and then on the target language training data, evaluating on the target language. this allows us to investigate whether leveraging large amounts of data in a high-resource language can enhance full-shot performance on low-resource corpora. we use xlm-roberta @xcite ) due to its strong multilingual capabilities. the model is pretrained on 100 languages, including all those in our novel datasets. it has proven highly effective in embedding both high-and low-resource languages and is widely studied in cross-lingual transfer research @xcite , particularly in the context of polysemy disambiguation @xcite @xcite . for model fine-tuning, we follow @xcite and use a bi-encoder architecture that independently processes the two sentences containing the polysemous target word using a siamese network to generate two distinct vector representations (embeddings). the model outputs the cosine distance between the output embeddings of the two inputs, and, to collapse this to a binary label, a threshold is applied to decide if the words are classified as having the same sense. the model is trained to adapt embeddings and increase this distance when the target word has different meanings and decrease it when the meanings are the same in the two sentences by minimising contrastive loss. after training, we set the threshold for each model 6 results ## discussion in this work we present sense-annotated datasets across a diverse range of language families, providing valuable resources for linguistic and computational studies. punjabi, marathi, and urdu belong to the indo-aryan branch, enabling research on linguistic relatedness alongside the hindi wic dataset @xcite . telugu and kannada represent the dravidian family, while azerbaijani, swahili, vietnamese, polish, and korean extend coverage to additional linguistic groups. the dataset includes arabic-based (punjabi, urdu), devanagari (marathi), latin-based (azerbaijani, polish, swahili, vietnamese), hangul (korean), and brahmic scripts (kannada, telugu), facilitating research on script variation and its impact on nlp. by encompassing a broad linguistic spectrum, our dataset supports studies on linguistic relatedness, historical evolution, and polysemy disambiguation in low-resource settings. it serves as a foundation for evaluating and improving multilingual and cross-lingual transfer, particularly in tasks requiring deep semantic understanding. our experiments highlight the importance of language-specific resources. the unexpected finding that zero-shot xlm-r trained only on english outperformed full-shot models trained on the target language challenges assumptions about crosslingual transfer stability, emphasizing the need for dedicated evaluation datasets. manual annotation is essential yet laborintensive, particularly for low-resource languages. we introduce an automated method to identify sentences across all word senses, even when certain senses are sparsely represented. our quantitative results demonstrate the effectiveness of this approach in enhancing annotation efficiency and supporting sense-annotated dataset development. to encourage further research, we release our code on github: github.com/roksanagow/projecting_sentences. ## limitations the dataset remains relatively small, which may limit the generalizability of findings, particularly for full-shot experiments, where additional training data would likely improve performance. additionally, data imbalance across languages makes direct comparisons challenging without subsampling, which in turn reduces overall performance. even within a single language, the number of senses and sentences per word varies, further complicating evaluation. moreover, each language was sourced from different corpora, leading to potential inconsistencies in text style, domain coverage, and annotation quality. the evaluation setup also has certain constraints. train-dev-test splits were generated randomly (according to the algorithm specified in §4.2), and the prevalence of sentences corresponding to different words across splits could impact the results. furthermore, zero-shot evaluation was conducted only from english, leaving open questions about transfer from other high-resource languages and cross-lingual settings beyond english-centric transfer.