Towards better structured and less noisy Web data: Oscar with Register annotations (2022)

Abstract: Web-crawled datasets are known to be noisy, as they feature a wide range of language use covering both user-generated and professionally edited content as well as noise originating from the crawling process. This article presents one solution to reduce this noise by using automatic register (genre) identification -whether the texts are, e.g., forum discussions, lyrical or how-to pages. We apply the multilingual register identification model by Rönnqvist et al. (2021) and label the widely used Oscar dataset. Additionally, we evaluate the model against eight new languages, showing that the performance is comparable to previous findings on a restricted set of languages. Finally, we present and apply a machine learning method for further cleaning text files originating from Web crawls from remains of boilerplate and other elements not belonging to the main text of the Web page. The register labeled and cleaned dataset covers 351 million documents in 14 languages and is available at https://huggingface.co/datasets/TurkuNLP/register_oscar .

## introduction massive web-crawled datasets are widely used in natural language processing (nlp), for instance for training language models @xcite @xcite . however, the challenge with these crawled datasets is that they are typically very noisy. first of all, this noise originates from the lack of structure and metadatathe datasets don't include any information on the origin of the documents. this complicates their use, because language on the web varies extremely, ranging from toxic language, discussion forums and other user-generated content to professionallylike edited texts. second, the noisiness comes from the crawling process-despite the cleaning efforts, web-crawled data still contain remains of boilerplate and other elements not belonging to the main text, such as click here or read more. all these properties affect the automatic processing of text @xcite @xcite . the automatic identification of web genres or registers-whether the documents are, e.g., forum discussions, originally spoken, informative or narrative @xcite -would offer a solution to reduce the noisiness of web data and to add metadata on the origin of the documents. however, this has been a challenge. there are no gatekeepers ensuring that the users follow any conventions when writing on the web, and thus, web language use has been referred to as a jungle @xcite . the available register datasets, almost entirely focusing on english, have been restricted to only selected and well-defined registers, and they do not generalize to the entire web @xcite @xcite . recently, however, the corpus of online registers of english (core) @xcite sampled from the unrestricted open web has allowed the modeling of the full range of registers found in web-crawled datasets. furthermore, similarly register-annotated datasets in finnish, swedish and french @xcite have extended these possibilities to a multilingual setting @xcite . in this paper, we benefit from these advances and present register oscar, a version of the widely used oscar dataset @xcite to which we have automatically created register labels. furthermore, we introduce and apply a machine learning method for cleaning text files originating from web crawls-such as the oscar documentsto filter out noise left after boilerplate removal. register oscar covers 14 languages. to identify the document registers, we use the multilingual register model by @xcite based on four languages. to evaluate the model on the wider set of languages included in register oscar, we present new core-style annotated evaluation datasets in eight languages: arabic, catalan, chinese, hindi, indonesian, portuguese, spanish and urdu. we find that the zero-shot performance of the model on these culturally and linguistically different languages is 0.70 f1-score, similar to the previously reported zero-shot results. in sum, our main contributions are: the register annotations for oscar are available at https://huggingface.co/datasets/ turkunlp/register_oscar, and the new manually annotated register corpora and the text quality annotations used to train the cleaning system at https://github.com/turkunlp/ multilingual-register-labeling. the new manually annotated multilingual register corpora cover eight languages created as a part of the current study, the main objective being to allow for a more extensive evaluation of the register identification model on the oscar languages. the newly annotated datasets include culturally and linguistically varied languages. as register is deeply associated with the situational context of the text @xcite and, e.g., blogs can have very different characteristics in different cultures, this offers a unique chance to evaluate the robustness of the register model. the documents for the annotation were randomly sampled from a recent common crawl foot_0 dataset. the annotation was done using a custom annotation tool. most of the annotators have a background in linguistics or nlp. the annotators were given a detailed tutorial to the register scheme, see https://turkunlp.org/register -annotation-docs/. the annotations of the new datasets follow the hierarchical core register scheme consisting of eight main registers, tens of subregisters, and the category machine translated, see the newly annotated register corpora are described in ## methods the register labeling of the oscar documents is done using the master multilingual model by @xcite . the model is based on a fine-tuned xlm-r @xcite using french, finnish and english data, and is available at https://github.com/turkunlp/ multilingual-register-labeling. to account for hybrid documents (see section 2), the model is multi-label allowing to predict several registers for one document. the register model has been reported to achieve an f1-score of 0.77 on a multilingual dataset. furthermore, it outperforms also monolingual language-specific neural classifiers in these languages @xcite , and provides much higher performance than earlier systems based on svms or statistical techniques that additionally would not allow for the modeling of languages without training data @xcite . therefore, the use of the xlm-r is motivated in the current study despite the computational costs. finally, we also evaluate the performance of the xlm-r-based register identification model on the new multilingual register corpora. the cleaning of the oscar documents from remains of boilerplate and elements not belonging to the main text works on text files and is based on machine learning, unlike boilerplate removal that is typically rule-based and takes html as input. the pipeline consists of three steps. first, the data is run through a heuristic filtering script with language detection using langdetect to filter out e.g., documents that are less than 75 words long or have a high ratio of digit (> 0.075) or foreign characters (>0.02). second, an xlm-r @xcite classifier is trained to predict whether a document is machine generated or not, using data from our ongoing register annotation projects where machine translation and generation is one of the register categories. we optimize learning rate using a grid of rates between 1e-7..9e-5. third, we filter out lines, defined as sequences of characters separated by a line break, that do not belong to the main text of the document. this step uses the text quality annotations described in section 2 and includes two xlm-r models: a bagof-lines classifier to predict whether a line is main text content or not, and another one with an extra long short-term memory (lstm) layer to predict the line quality based on sequences of embeddings retrieved from the first model. we optimize the learning rate within the range of 1e-7..9e-5, and compare the performances of the first model to the entire architecture. ## conclusions in this paper, we have presented automatically produced register annotations for the widely used oscar dataset in 14 languages, and we have evaluated the register identification model against new datasets covering eight languages not included in the original model development. furthermore, we have described a machine-learning method for cleaning text data originating from web crawls, and we have applied the method to further clean the documents in the entire dataset. the evaluation showed that the performance of the register model is comparable to previously reported zero-shot results, although the newly annotated datasets feature linguistically and culturally diverse languages. this suggests that multilingual register identification can be used to provide structure and improve the usability of web-crawled data, where the content ranges from noisy user-generated text to professionally edited documents. the register annotations automatically produced in this study cover altogether eight registers and 351 million documents, available at https://huggingface.co/datasets /turkunlp/register_oscar. the new manually annotated register datasets and the text quality annotations used to develop the cleaning pipeline can be found at https://github.com/turkunlp/ multilingual-register-labeling.