Elote, Choclo and Mazorca: on the Varieties of S panish (2024)

Abstract: Spanish is one of the most widespread languages: the official language in 20 countries and the second most-spoken native language. Its contact with other languages across different regions and the rich regional and cultural diversity has produced varieties which divert from each other, particularly in terms of lexicon. Still, available corpora, and models trained upon them, generally treat Spanish as one monolithic language, which dampers prediction and generation power when dealing with different varieties. To alleviate the situation, we compile and curate datasets in the different varieties of Spanish around the world at an unprecedented scale and create the CEREAL corpus. With such a resource at hand, we perform a stylistic analysis to identify and characterise varietal differences. We implement a classifier specially designed to deal with long documents and identify Spanish varieties (and therefore expand CEREAL further). We produce varietal-specific embeddings, and analyse the cultural differences that they encode. We make data, code and models publicly available.

## introduction spanish was heavily expanded by overseas european conquest starting in the early-16th century. one of the main axes of the colonisation was the imposition of the spanish culture. as a result, nearly 70% of the current population in latin america is catholic and spanish is a global language (ammon, 2010) and the second largest native language over the world (+450m speakers), only after chinese @xcite . different from the latter, spanish is a sea-bound language and it is the official language in 20 countries across america, africa and europe, spreading over 11.7m km 2 . the diversity in terms of culture, geography and languages in contact (e.g., basque in northern spain, nahuatl in central mexico, quechua in the andes) have produced well-differentiated varieties foot_0 permeated by the lexicon and semantics from the relevant local languages. even with the apparent diversity, spanish is often considered a homogeneous language in most datadriven natural language processing (nlp) tasks and applications. a first reason is the difficulty to get data for all the varieties, also taking into account the lack of information about the origin of the data for lots of the existing corpora. a second reason is a popular motto in the nlp community: the more data the better. following this mantra, data in spanish is usually mixed together and, as a consequence, one might increase the performance on downstream tasks at the cost of erasing cultural differences. such differences can be reflected in textual corpora by the topics present, the linguistic forms used or even the writing style. at the corpus scale, the differences are evident, but in individual texts, specially short ones, differences might be non-existent. because of this reason, a text might belong to more than one variety but, the longer the text, the more likely its origin country shines. the register is relevant as well. for instance, newspapers tend to follow standard norms. under this point of view, the jungle of the web is a good source to get diverse data. still, one should acknowledge the presence of collaborative texts (i.e. the wikipedia) and media outlets for which no clear variety can be assigned. we develop a long-document classifier to extract and release cereal and cerealex, corpus del español real (extended), a new corpus from online data five times larger than the now corpus, to the best of our knowledge the largest available corpus of spanish varieties up to date. cerealex contains 35 b words in 41 m documents, 36 m of which include information about the country they were mazorca elote choclo figure 1 : common geographic (country-based) spanish linguistic zones as described by the real academia española. the size of the bubbles is proportional to the number of spanish speakers. the three words associated to corn included in the title of this work appear on the sea close to the regions where they are most used. it is larger than corpes -with 7 b words-and covers 21 countries. in this case, the proportion between continents is 78% america vs 22% europe. now is downloadable, upon the payment of a fee. another effort is that of @xcite , who collect 0.75 m geolocated tweets posted over a two-year span. they queried the twitter api with seed words representing concepts that are expressed in different ways across countries; e.g., corn is elote in mexico and most of central america (from the nahuatl elotitutl), choclo in most of south america (from the quechua chuqllu) and mazorca in colombia, cuba, and spain (from the arabic masúrqa). recently, @xcite collected 800 m tweets (11 b tokens) over a three-year span, 2016 to 2019, posted from 26 countries (including a few where spanish is not an official language). with them, they create 26 country-specific word embeddings, plus bert-like language models for the countries with the largest amounts of data. the models are publicly available, but the data is not. 4 smaller-scale corpora include those for the series of shared tasks on discriminating between similar languages (dsl), organised within the workshop on nlp for similar languages, varieties and dialects. the dsl collection @xcite contains individual sentences in spanish from ar- ## classification of long documents the documents in oscar have an average length of 900 words, significantly surpassing the 512 token limit from most lm-based classifiers which, despite this limit, achieve state-of-the-art results in many classification tasks. however, the differences across language varieties might not be present in short fragments, making it crucial to consider the whole contents of a document, regardless of length. therefore, we implement doctransformer, foot_6 a multi-class classifier on top of xlm-roberta large @xcite ) that considers the whole document, as shown in figure 2 . the extension to document-level is simple: the [cls] token is indeed an average of the [cls] tokens for each of the n fragments in the document. this is done online, which allows for the full document to take part in the classification and the backpropagation. the best practice to split the document is explored in the following. experimental settings. we conduct 3 tasks: 3c 3 classes: es, mx and mix 4c 4 classes: cl, es, mx and mix 5c 5 classes: ar, cl, es, mx and mix the tasks consider the top-k languages in terms of number of documents (see for the test set, we use a subset of the corpus with unknown origin. to assign the origin, we collect sports, fashion, travel, gastronomy and news sites without a country identifier in the url for the 24 varieties, and extract the documents available in cereal with the mix class. we build datasets with the same proportions as the validation sets for the 3 classification tasks. figure 3 depicts the data distribution for the 4c training set. notice that only half of the documents fit the 512 token limit. the average number of segments is 27, but a few surpass 100. both length and number of segments are relevant for the classifier. training models. we train the classifiers along three iterations for 3c and two iterations for 4c and 5c to explore a similar number of instances in the three cases. we first explore the relevance of going beyond the 512 input token limit of roberta with 5 models in 4c. after encoding, we split an input document d into 2, 3, 6, or 16 parts. the baseline is a single split with the first 512 tokens. with the 2-splits model, one considers up to 1,024 tokens, with 3 splits up to 1,536, and so on. we cover fully every d in the training set with 6 splits (max. 3,072 tokens per document). with the model with 16 splits, we are interested in checking if the length of the input fragments is relevant: the model with 6 and the model with 16 splits cover the same amount of text because the documents are limited to 3,000 words, but in the latter the [cls] tokens summarise information of less context (specifically, max. 62 words vs 167 in the 6-splits model). the splitting of d is done at the fragment level using @xmath0. if d has less or the same number of segments as the desired splits, each split contains a segment. if d has more divisions, we balance the number of segments across splits. if d is longer than the number of splits allowed, we first balance the number of segments and then these fragments are limited to the 512 limit each, so information is lost homogeneously across d and not gathered at the end, as the baseline system would do. figure 4 (top) shows the performance evolution for the 5 models for 4c as measured by accuracy. cerealex, classifying the unknown. finally, we apply the 3c, 4c and 5c classifiers with 2 splits to the 28 m documents of the unknown part of cereal. ## cultural aspects in embeddings we analyse the quality and effect of variety-specific word embeddings built from the cereal corpus. we estimate fasttext @xcite em-beddings as described in appendix d. for bilingual lexicon induction, we use fasttext pretrained english embeddings. 10 for comparison purposes, we use the twitter embeddings from @xcite . ## conclusions spanish is a language with large variations across territories. variations are reflected in written documents as the high accuracies of our classifiers show. the amount of data we gather in cereal and cerealex turns argentinian, chilean, colombian, mexican and peninsular spanish into highresourced varieties that do not need extra data to achieve competitive results in tasks such a bilingual lexicon induction. for other varieties, the addition of data from the richest counterparts is still needed. ongoing work is being devoted to study the topology of our variety-specific embedding spaces and to derive data-based phylogenetic trees for spanish. although currently the amount of documents in cereal that remains with unknown origin is relatively small (5 m documents out of 41 m), we plan to train classifiers for the lowest-resourced varieties. beside creating larger corpora, further work should also focus on the development of varietyspecific curated resources for nlp tasks evaluation. corpora, code and models are available for the research community in hope for fostering research and development of technology that reflects cultural variety beyond single languages.