Get to Know Your Parallel Data: Performing E nglish Variety and Genre Classification over M a C o C u Corpora (2023)

Abstract: Collecting texts from the web enables a rapid creation of monolingual and parallel corpora of unprecedented size. However, unlike manually-collected corpora, authors and end users do not know which texts make up the web collections. In this work, we analyse the content of seven European parallel web corpora, collected from national top-level domains, by analysing the English variety and genre distribution in them. We develop and provide a lexicon-based British-American variety classifier, which we use to identify the English variety. In addition, we apply a Transformer-based genre classifier to corpora to analyse genre distribution and the interplay between genres and English varieties. The results reveal significant differences among the seven corpora in terms of different genre distribution and different preference for English varieties.

## introduction collecting text corpora in an automatic manner, by crawling web pages, allows for quick gathering of large amounts of texts. with this approach, the macocu foot_0 project @xcite aims to provide some of the largest freely available monolingual and parallel corpora for more than 10 underresourced european languages. however, in contrast to manual text collection methods, the disadvantage of automatic methods is that both the corpora creators and the users do not know what the overall quality of the dataset is, and what type of texts the collections consist of @xcite . the macocu corpora address this issue by providing rich metadata, including information on source urls, paragraph quality, translation direction, english varieties, and genres. in this paper, we present two of the text classification methods, used to automatically enrich massive corpora with meaningful metadata: english variety classification and automatic genre identification. we show how they provide a better insight into the differences between corpora. there is limited research on the use of british and american english in the non-native englishspeaking countries. previous findings show that these english varieties are preferred to a different extent in different educational systems @xcite , translation services @xcite and on different national webs @xcite . however, to the best of our knowledge, there is no freely available classifier between american and british english which would allow easy identification of an english variety in large corpora, and thus allow for a corpus-based research of this phenomenon on a larger scale. to this end, we develop a fast and reliable classifier which is based on a lexicon of variety-specific spellings and words. in addition, we also compare the web corpora in terms of genres. genres are text categories which are defined considering the author's purpose, common function of the text, and the text's conventional form @xcite . examples of genres are news, promotion, legal, etc. in addition to providing a valuable insight into the dataset content, information about the genre of the document was shown to be beneficial for various nlp tasks, including pos-tagging @xcite , machine translation ( @xcite and automatic summarization @xcite . the main contributions of our paper are the following: the paper is organized as follows. we first present the related work on english variety identification and automatic genre identification in section 2. in section 3, we present the web corpora to which we apply the classifiers, described in section 4. the results in section 5 show that these approaches reveal important differences between the corpora. the paper concludes with section 6, where we summarize the main findings and present future work. ## related work diatopic variation, that is, variation among national varieties of the same language @xcite , can be approached similarly to variation between different languages. two main approaches in language identification of english varieties are 1) corpus-based text classification and 2) lexiconbased text classification. in corpus-based classification, researchers use datasets which have a known origin of the texts as a reference based on which the classifiers are trained and evaluated, while lexiconbased classifiers identify varieties based on a list of variety-specific words or spelling variants. most previous studies on identification of english varieties were corpus-based @xcite @xcite @xcite . the advantage of corpus-based classification is that as the model is trained on actual text collections, it could show the differences in the varieties as they appear "in the wild", and researchers do not need a profound knowledge of lexical differences between the varieties that linguists are aware of. to obtain reference datasets that are large enough to be used for training the model, researchers most often used or constructed web corpora @xcite , using the national top-level domains as indicators of the text origin (e.g., .uk for british english), journalistic corpora @xcite , national corpora @xcite , such as the british national corpus (bnc) @xcite , and/or social media corpora @xcite @xcite , consisting of texts from twitter and facebook, where the variety is assigned to texts based on the metadata about the post or its author. however, one of the major drawbacks of this approach is that it is assumed that the texts from a specific top-level domain or posted to social media from a certain location are written by a native speaker of this variety, while this is hard or impossible to verify. in addition, web, national and journalistic corpora can contain cross citations and republications (e.g., a british text that was republished by an american newspaper website and vice versa). this was revealed for the dsl corpus collection, used in the discriminating between similar languages (dsl) shared task 2014, where 25% of texts were discovered to be likely annotated with the wrong english variety @xcite . another drawback of the corpus-based approach is that training on text collections can introduce various bias into the classification task. as no parallel corpus of english varieties exists, the classification is based on two or more separate collections of texts. the datasets which represent each variety do not differ only in language specificities, but also in content and style. this hinders learning truly representative differences between the varieties, and the classification models might learn to differentiate between the datasets based on other differences, unrelated to varieties, such as topic @xcite . an alternative approach to the corpus-based classification is lexicon-based. it has been used by @xcite who devised a "variant pair" classifier based on the varcon lexicon of spelling variants @xcite . this approach does not introduce any biases, related to the corpora content, and the classification is explainable. however, its disadvantage is that as it relies on a specific list of words, if none of the words occur in a text, its variety is unknown, meaning that some texts in the corpus may remain unclassified. while research on automatic identification of english varieties is rather limited, automatic genre identification (agi) has been an established text categorization task ever since the advent of the world wide web. as genre information is very useful for obtaining better hits to a query in information retrieval tools, used on the web, there has been large interest for genre identification in the area of information retrieval (see @xcite ; @xcite ). in addition, with the emergence of technologies for automatic collection of text corpora, an interest for tools for agi emerged also in the field of corpora creation and curation. to this end, genre researchers devised sets of genre categories which aim to cover all of the diversity of texts found on the web, and provided manually annotated datasets (see @xcite ; @xcite ; @xcite ). classification of genres was shown to be a hard task as texts can display characteristics of multiple genres @xcite , and most genre classification models were not able to generalize outside of the dataset on which they were trained @xcite . however, recent advances in deep neural technologies led to a breakthrough in this field, and transformer-based language models @xcite , fine-tuned on manually-annotated genre datasets, showed the ability to identify genres in various web corpora and languages (see @xcite ; @xcite ). following encouraging results, transformer-based genre classifiers have started to be applied to web corpora to provide genre information as metadata. for instance, as part of newly available massive oscar web corpora, 351 million documents in 14 languages were enriched with genre information @xcite . ## datasets in this paper, we compare seven parallel web corpora, created in the scope of the ma-cocu project @xcite : croatian-english macocu-hr-en @xcite , slovene-english macocu-sl-en @xcite , bulgarian-english macocu-bg-en @xcite , macedonian-english macocu-mken @xcite , turkish-english macocutr-en @xcite , icelandic-english macocu-is-en @xcite and maltese-english macocu-mt-en @xcite corpus. the corpora were created by crawling the national top-level domains, e.g. the slovenian toplevel domain .si for the english-slovene dataset macocu-sl-en. important to note is that the crawl primarily focused on the top-level domain crawling, but was allowed to harvest data from generic domains (.com, .net etc.) if the domain proved to have enough data in the language being crawled. websites containing the target language and english were identified and processed with the bitextor foot_2 tool. ## conclusion in this paper, we introduce a freely-available english variety classifier for fast and reliable identification of british and american english. the corpus-based approaches to language variety classification can be impacted by topic-related or other biases, occurring due to differences between the corpora on which the model is trained. in contrast, our lexicon-based approach is based on a carefully selected lexicon of words which are confirmed by linguists to be variety-specific, making the results more reliable and explainable. we then show how the classifier can be used to obtain an insight into the characteristics of large parallel corpora, collected with automatic methods. we compare parallel web corpora from european national webs in seven languages. as all languages are paired with english, we obtained meaningful information on the differences between the corpora in terms of english varieties. the results revealed british english is prevalent in maltese, icelandic, slovene and croatian corpora, while corpora from the mace-donian, bulgarian and turkish national webs are more influenced by american english. a stark difference between the use of varieties was observed in the case of the maltese corpus, where a large majority of texts were written in british english and there were less than 10% of texts in american english. these results reflect the country's historical connection with the united kingdom, along with a significant presence of eu websites in the corpus, which have a policy of preferring british english. thus, we show how the classifier can be used for not only comparing corpora, but also obtaining insight into the use of english by native and non-native speaking content writers and translators. by making the classifier freely available, we hope to encourage analyses of the use of english and its varieties among teachers, translators and content creators in the fields of corpus linguistics, translation studies, linguistics and digital humanities. furthermore, we extend the comparison by automatically annotating the english texts from the parallel corpora with genre information. the results revealed significant differences between the corpora in terms of genre distribution. once again, the maltese corpus was shown to be more different than the others, consisting mostly of news and legal texts. news is also strongly present in macedonian and icelandic corpora, while slovene, croatian, bulgarian and turkish corpora constitute of large amounts of promotional texts. with the two classification approaches, we obtained valuable information on the characteristics of the datasets. as such datasets are often used for creation of machine translation systems, various nlp tools, as well as linguistic studies, it is crucial that the users are provided with the information on what types of texts and language varieties the datasets consist of. the macocu project will provide this information for all their datasets, covering 13 european under-resourced languages: albanian, bosnian, bulgarian, catalan, croatian, icelandic, macedonian, maltese, montenegrin, slovene, serbian, turkish and ukrainian. the datasets will be made freely available by june 2023. as the initial analysis of the english variety and genre distribution in corpora, presented in this paper, revealed that this information highlights important differences between the corpora, in the future, we plan to extend the analysis to all 13 newly available macocu corpora. furthermore, one important downstream task that we did not tackle in this work is the inspection of the impact of the variation in variety and genre on machine translation and other systems based on these and other datasets, which we also plan to analyse in future studies.