Common Ground, Diverse Roots: The Difficulty of Classifying Common Examples in S panish Varieties (2025)

Abstract: Variations in languages across geographic regions or cultures are crucial to address to avoid biases in NLP systems designed for culturally sensitive tasks, such as hate speech detection or dialog with conversational agents. In languages such as Spanish, where varieties can significantly overlap, many examples can be valid across them, which we refer to as common examples. Ignoring these examples may cause misclassifications, reducing model accuracy and fairness. Therefore, accounting for these common examples is essential to improve the robustness and representativeness of NLP systems trained on such data. In this work, we address this problem in the context of Spanish varieties. We use training dynamics to automatically detect common examples or errors in existing Spanish datasets. We demonstrate the efficacy of using predicted label confidence for our Datamaps (CITATION) implementation for the identification of hard-to-classify examples, especially common examples, enhancing model performance in variety identification tasks. Additionally, we introduce a Cuban Spanish Variety Identification dataset with common examples annotations developed to facilitate more accurate detection of Cuban and Caribbean Spanish varieties. To our knowledge, this is the first dataset focused on identifying the Cuban, or any other Caribbean, Spanish variety.

## introduction language reflects culture and identity, while also capturing subtle variations that shape communication. in natural language processing (nlp), it is crucial to account for these nuances, especially in language variety identification, where small shifts in meaning, often tied to cultural interpretations, can impact sensitive tasks like hate speech detection. expressions that may be benign in one dialect * these authors contributed equally. can be offensive in another, making accurate variety identification essential to prevent misclassifications and ensure culturally appropriate responses @xcite . in such tasks, "el auto está en el garaje" (the car is in the garage) variety 1: argentinian variety 2: cuban common example annotator 1: "auto" (automobile) is commonly used in argentina instead of "coche" (car). annotator 2: "garaje" (garage) is used in cuba, while "cochera" (carport) might be used elsewhere. cross-lingual models often struggle with these subtle cultural and linguistic distinctions, as the same formulation may carry vastly different meanings across varieties. language-specific models tend to perform better in such cases, as they are more sensitive to regional variations @xcite @xcite @xcite . however, distinguishing between closely related languages, dialects, and regional varieties of the same language is a key and difficult task in language identification @xcite @xcite . adding to this complexity is the issue of common examples -valid phrases across multiple dialects or varieties. overlooking these examples can result in biased classifications, especially in languages like spanish, where variety overlap is frequent. 1 despite this, many current datasets treat the identification of the language variety as a single label classification task, which overlooks this crucial aspect @xcite . current datasets for language variety identification often rely on manual annotations or automated methods such as geographic information @xcite @xcite or keyword-based classification @xcite . however, both approaches have limitations, and manually checking large datasets for common examples is challenging and costly @xcite . datamaps based on training dynamics @xcite , which track how the confidence of the model changes over epochs, have been used successfully to detect annotation errors and human label variation. these methods highlight which examples are consistently easy or difficult for the model, with hard examples often pointing to ambiguity or errors. we propose using training dynamics to detect common examples in language variety identification tasks. in 1 we show an example of these common examples. these are expected to be among the hard examples the model struggles with during training. by tracking the model's confidence in its predicted labels over multiple training epochs, rather than using gold labels, we aim to detect ambiguous instances that are hard for the model to classify consistently. our research addresses the following questions: to investigate these questions, we use two datasets: the spanish subset of dsl-tl dataset @xcite , which contains texts extracted from news articles, and a new dataset of cuban spanish varieties we collected from twitter. we adapt the datamaps technique by changing the way confidence and variability are calculated, allowing us to identify common examples. our results demonstrate the efficiency of this approach in detecting common examples in both datasets. our main contributions are as follows: common examples. the challenge of handling common examples that can be valid across multiple language varieties has been a recurring issue in language variety identification. traditional single-label classification often struggles to assign unique labels to common examples @xcite . addressing this challenge, @xcite introduced a third class specifically for common instances in their dsl-tl dataset for language variety identification. this dataset allowed the exploration of the impact of these ambiguous cases on model performance. the authors found that the models had difficulty distinguishing between common and dialectspecific examples. then, their results served as a baseline for the dsl-tl shared task at vardial 2023 @xcite . in the scope of this shared task, vaidya and kane (2023) introduced a two-stage multilingual dialect detection system. their approach first identifies the macro-language, followed by applying dialect-specific models to refine the classification. although this system performed well overall, it struggled with the common examples class, where it frequently misclassified examples due to the lack of clear dialect-specific markers. the spanish language, with its rich array of varieties, provides a particularly challenging landscape for variety identification due to the high similarity between varieties. @xcite noted that the prevalence of common examples in spanish is especially high. given the significant lexical and syntactical overlap among spanish varieties, sentences that can belong to more than one variety are frequent, making traditional classification approaches less reliable. the misclassification of these common instances not only introduces noise into the datasets but also impacts the overall performance of the models, as evidenced by the poor handling of argentine examples in @xcite . ## task definition: automatic common examples detection our main task is to identify common examples across similar language varieties. our proposed pipeline can be separated into two main stages: ## datasets in this section, we describe the datasets used for our analysis. we use an existing dataset dsl-tl and propose a new dataset cubanspvariety focused on the cuban spanish variety. ## error analysis to better understand our models' performance, we analyzed the errors for each dataset by counting the most frequent words in the top-500 non-common instances predicted by the dm mean-pred model (prediction errors). after removing stopwords and special tokens, we found that in the cubansp-variety dataset, the most frequent words were cuba and soscuba, directly tied to the cuban variety in this context. in contrast, the dsl-tl dataset showed common words like ha, pero, fue, and también, which do not indicate a specific variety. the topic bias in the cuban dataset can influence the model predictions, mainly when the examples contain keywords specific to the variety. this also explains why dm std-pred performs better for cubanspvariety, as these keywords in both classes make variability more significant than in dsl-tl. in the cubanspvariety dataset, figure 7 shows that about 67% of the top-500 non-common examples and 54% of the top-1000 non-common examples contained the word cuba, suggesting a strong influence on model behavior, given that only 33% of the total examples contain this word. additionally, we found that 63.31% of the top-500 errors in cubanspvariety were cases where only two annotators agreed on the label, and for the top-1000, this number was 57%. across all non-common instances, full agreement (three annotators) occurred in 57% of cases, indicating a clear link between annotation difficulty and model errors as shown in figure 8 . 0 250 500 750 1000 1250 1500 1750 n 0 100 200 300 400 500 count full_agreement partial_agreement figure 8: agreement index for error instances in top-n using dm mean score metric. another key point is understanding why the model fails to retrieve certain common examples. we focus on the last two common examples in the ranking for each dataset, using shapley additive explanations (shap) @xcite to analyze the model's behavior. shap is based on game theory and assigns importance scores to features, showing how much each feature influences the model's prediction. figure 9a presents the shap scores for the last two common examples in the dsl-tl dataset ranking. for the first example, the words argentina, rosario, and marcelo are the most influential for predicting the es-ar label. the first two refer to the country and one of its major cities, while marcelo is a common name in argentina. for the second example, auto (commonly used in argentina to mean "car," as opposed to coche in spain) is the most significant feature, followed by puerto and madero, a wellknown place in argentina. while named entities influence the first example, the second example, with the word auto, suggests a potential annotation error, as it points to the argentinian variety. in figure 9b , we provide the corresponding analysis for the cubanspvariety dataset. for the first example, the word buen (from the phrase buen día, which is used in spanish varieties other than the cuban one) is the most significant, along with argenzuela (a blend of argentina and venezuela), garchan, and tenés, which are characteristic of the argentinian variety. this example likely represents an annotation mistake. for the second example, the most influential words are profesional, partido, fidelidad, and obediencia, none of which are strong indicators of a specific variety. this suggests that common topics in cuban tweets may affect the model's prediction, potentially introducing biases into the classification process. regarding the named entities, we followed the precedent set by previous works in variety identification, such as the study introducing the dsl-tl dataset @xcite , retained named entities. consequently, we included them in our initial approach while emphasizing the importance of analyzing their influence. we agree that a set of experiments where we could switch the named entities with neutral entities (or even adversarial entities (eg switch soscuba with sosmexico) would be interesting. in our case, while evidence suggests that named entities contribute to model errors, our preliminary analysis demonstrates the model's robustness to their presence. for example, the sentence "mi mensaje para el pueblo de cuba emoji bandera cuba emoji :. ¡no están solos!. cuenten con nosotros para seguir apoyando su lucha por la libertad y la democracia. soscuba url" was ranked second using the datamaps mean approach. although it contained clear markers such as "cuba" and "soscuba," the model correctly identified it. this is not an isolated case, and further analysis of correctly classified examples can provide additional evidence of the system's robustness. ## conclusion in this work, we examine the effectiveness of datamaps methods in identifying common examples across closely related language varieties. our results demonstrate the value of training dynamics in detecting difficult examples early in the model's learning process, as reflected by the effectiveness of dm mean-pred across both datasets. this confidencebased approach consistently outperformed the variability-based method, suggesting that tracking model confidence over predicted labels offers a reliable way to identify common examples automatically across different domains. although the performance difference between variability-based and confidence-based approaches is less significant for theinformal dataset, the overall results indicate that confidence-based datamaps can be a powerful tool for improving data quality in different contexts. although these methods may not fully solve the challenges of variety and dialect annotation, they offer a promising step forward, particularly when combined with automatic techniques and targeted human annotation. we hope that this initial dataset, freely accessible under a cc-by-sa license upon publication, the first centered on cuban, a caribbean variety of spanish, will prove a valuable resource for future research on this topic.