Text Preprocessing and its Implications in a Digital Humanities Project (2021)

Abstract: This paper focuses on data cleaning as part of a preprocessing procedure applied to text data retrieved from the web. Although the importance of this early stage in a project using NLP methods is often highlighted by researchers, the details, general principles and techniques are usually left out due to consideration of space. At best, they are dismissed with a comment “The usual data cleaning and preprocessing procedures were applied”. More coverage is usually given to automatic text annotation such as lemmatisation, part-of-speech tagging and parsing, which is often included in preprocessing. In the literature, the term ‘preprocessing’ is used to refer to a wide range of procedures, from filtering and cleaning to data transformation such as stemming and numeric representation, which might create confusion. We argue that text preprocessing might skew original data distribution with regard to the metadata, such as types, locations and times of registered datapoints. In this paper we describe a systematic approach to cleaning text data mined by a data-providing company for a Digital Humanities (DH) project focused on cultural analytics. We reveal the types and amount of noise in the data coming from various web sources and estimate the changes in the size of the data associated with preprocessing. We also compare the results of a text classification experiment run on the raw and preprocessed data. We hope that our experience and approaches will help the DH community to diagnose the quality of textual data collected from the web and prepare it for further natural language processing.

## introduction in this paper we discuss our experience of preprocessing textual descriptions of cultural events collected from a number of diverse web sources for a dh project. the problems that we faced demonstrate that there is a big difference between the amount of raw data scraped from the web and the amount of data distilled from it through preprocessing for further application of natural language processing (nlp) methods. the web is a unique source of available information that, to an extent, reflects social and cultural trends, and it is increasingly used in sociological and cultural studies. however, it seems that preprocessing and filtering, necessary to ensure reliability of automatic analyses, creates an additional refraction of the original signal registered by the web. this paper offers a description of the data preprocessing stage in a dh project and reports its effects on the quality of text classification on the one hand, and on the size of data available for the analysis as compared to the original web-scraping results on the other hand. we used pre-trained elmo (embeddings from language model) models for text representation in the classification task. we aim to find a reasonable trade-off between quantity and quality, and try to salvage as many text instances as possible without compromising the performance of nlp methods. our specific tasks include: the data underlying this project includes texts in at least five languages, with a lot of cases of code-switching. however, for the purposes of the paper we focus on english and russian subsets of the data. ## related work the term 'data preprocessing' is used as an umbrella term to refer to a wide range of procedures, including cleaning, filtering, normalising, annotating and transforming raw data. it is not always clear which operations from this list are referred to in a particular description. for example, functions included in a keras preprocessing module convert files in a tree of folders into the keras internal dataset format by creating batches of texts and linking texts to labels, and possibly truncating text to a pre-set maximum length. it assumes that the text files in folders are already suitably prepared for machine learning experiments. in this paper, we focus on the steps that precede this sort of data transformation and have to do with ensuring structural completeness of the data, its cleaning and normalisation. the link between the quality of the data and the reliability of the research outcomes is well known. it is reflected in the popular 'garbage in, garbage out' wisdom shared by the research communities in data science and nlp. it was famously formulated by john sinclair, a widely-recognised authority in corpus building and corpus linguistics: "the beginning of any corpus study is the creation of the corpus itself ... the results are only as good as the corpus" @xcite . data collection and preprocessing is not only an important stage in a research project, it is also the most tedious and time-consuming step. a popular belief is that it claims up to 80% of the research time. according to a report by anaconda "2020 state of data science: moving from hype toward maturity", based on a survey, which attracted a total of 2,360 responses from researchers from over 100 countries says that "on average 45% of their time is spent getting data ready (loading and cleansing)". 1 based on the previous research, there is no getting away from this chore, especially if using bag-of-words representations. preprocessing is always dependent on the data and task at hand, and it is often so specific to a project that it is not described in detail. nevertheless, there have been numerous attempts at evaluating the effects of different preprocessing steps for different types of data and nlp tasks. @xcite present a survey of different sources of noise in texts and how these types of noise can be handled. the survey cites tech-1 https://www.anaconda.com/state-of-data-science-2020 niques such as automatic speech recognition, optical character recognition and machine translation to be the biggest culprits in terms of introducing noise. @xcite go into detail about preprocessing techniques used specifically for information extraction (ie) and information retrieval (ir). while this study utilises all the techniques mentioned previously, it focuses on different stemming approaches, as it is said to be one of the most beneficial preprocessing steps in terms of performance. in @xcite 'text preprocessing' refers to data handling operations such as stopwords filtering with stemming, feature selection, feature transformation and a novel transformation method based on classes of terms. the study proves that using the novel feature transformation method and ensembles of term weighting methods yields good classification results even with very small amounts of features. similarly, @xcite reports the impact of non-relevant word removal and lemmatisation/stemming in conjunction with data transformation steps in the context of text classification for two languages. using a support vector machine (svm) classifier to represent portuguese data from the law domain as well as english reuters texts, experiments were carried out on feature reduction and construction, feature subset selection and term weighting. in terms of feature reduction, three configurations were tested including different combinations of non-relevant word removal (including articles, pronouns, adverbs and prepositions), as well as lemmatisation and stemming. crucially, the authors found that for portuguese stop-word removal did yield better results, while for english stemming had to also be included to attain better results. @xcite propose similar methods for tf-idf based text classification, but also include the removal of html tags. more recently, hacohen-kerner et al. (2020) performed an extensive evaluation of common preprocessing steps before text classification. exploring the impact of all possible combinations of five to six basic preprocessing methods on four benchmark text corpora, the study tested three machine learning methods. the preprocessing methods included spelling correction, conversion to lower case, html tag removal, punctuation removal, stop-words removal and reduction of replicated characters. overall, it is determined that the best results could be achieved with a combination of preprocessing methods, while stop word removal to be the most effective preprocessing method that significantly improved text classification. however, the exact combination of methods is said to vary significantly across datasets, leading to the conclusion that every dataset has to be treated accordingly. the impact of noise on the outcomes of nlp tasks, implemented using bert-based deep learning methods, such as offensive language identification, informativeness identification and sentiment analysis, is explored in al @xcite . in line with the above studies, they limit the concept of noise to non-standard appearance of language units (casing, hashtags, code-switching, emoji, url and punctuation). they found that for some tasks these types of noise are actually useful, and filtering them out results in lower classification results. most of the related work is focused on handling text representations such as feature reduction and selection, with some emphases on stop-words removal and lemmatisation/stemming. however, most researchers worked with available datasets that have undergone some sort of preprocessing. in our setting, we dealt the immediate results of crawling and had to extract raw text from it. ## data origins and structure the textual data underlying this study is part of a dataset contracted by a research group for a dh project and collected by a data company following a technical assignment. the dataset includes tables, where each row represents a cultural event announced at one of the seven web services aggregating news about cultural life in various locations across the globe. the project is set within comparative cultural studies and focuses on the types of events advertised in russia and elsewhere in the world. that is why the sources of data include both international and national publishing platforms, social networks and distributors of information regarding cultural events. a cultural event is understood broadly as any social (public) activity in the spheres of contemporary arts, education, sports, music, theatre, including social activities arranged by restaurants, computer clubs, and private enthusiasts (yoga classes, guided tours, etc). there is an understandable imbalance between the number of cultural events coming from international and russian sources (see the records of events obtained from these web resources included city and date of the event and its textual description. besides, each event was characterised by the name of the website section, where it was published, if available (e.g. 'business', 'graphic design', 'lectures'). this raw data was supplemented with additional information about event locations. the diverging thematic groups of events from the websites were manually mapped into seven major categories. the enhanced data tables have 24 columns, which store objective properties of each location: unique event identifier, date, city, synthetic and raw semantic categories, geographic coordinates, country, region, population, social and economic development parameters of the area (gdp, human development index, economic complexity index). ## results and discussion to demonstrate the impact of preprocessing and annotation on the outcomes of the experiments, we compared the results of automatic classification on the texts at different stages of preprocessing (raw tokens, clean tokens, all lemmas, content lemmas). to this end, we selected a subset of 600 event descriptions (in russian) registered in tyumen (a big city in western siberia, russia). these datapoints were labelled with the names of the website sections where the respective event was announced (timepad and ministry of culture). the six categories used as class labels were 'theatre', 'psychology and self-development', 'business', 'sport', 'concerts', 'kids events' and had at least 30 observations in each category. the texts are represented using available elmo models @xcite , pre-trained on raw and lemmatised corpora matching the types of preprocessing we have in our data. each text received an averaged vector for all words in it. the elmo vectors (size 1024) were then transformed into 32-dimensional vectors with autoencoder module from the keras library for python. these representations were classified with a svm algorithm with the default scikitlearn parameters. the evaluation is performed in 5fold cross-validation setting. given the imbalance in our data, we used stratified folds and weighting for underrepresented classes. as can be seen from another alarming aspect of preprocessing is the portion of the original dataset that gets discarded during preprocessing. overall, for the dataset at hand, preprocessing claimed 45% of the original size in terms of the number of observations. importantly, preprocessing affected the ratio of events assigned to various categories. the diverging inventories of categories used for sectioning on the websites were manually mapped into seven thematic groups shown in figure 1 . figure 1 demonstrates the impact of preprocessing on the distribution of records across the seven thematic categories from all seven web resources. it can be seen that preprocessing resulted in a significant reduction in the 'art' and 'politics' categories (62.7 and 57.4% of registered events lost, respectively). most of the items lost in the 'arts' category had a link to a media object such as a picture in the 'text' field. it raises a question whether any multi-modal processing can be employed as a contingency measure. the top three web resources that lost most of the records in preprocessing were behance, timepad and ted, which were reduced by 81.9, 54.1 and 55.3% respectively. the highest integrity was characteristic for e-flux and ministry of culture; they had little noise and hardly lost any datapoints. note that in terms of absolute number of datapoints, meetup remained the biggest source of data, dwarfing e-flux into insignificance (with only 10,280 raw records and 10,081 after pre-processing, it is shown as a dot in figure 2 . for the tyumen sample, we are left with less than 50% (992 events) of the original number of observations (1864 events), given the preprocessing described above. the number of observations for manchester (for example) dropped by 35%. most of the datapoints are discarded due to the lack or insufficiency of text attributes associated with recorded events. interestingly, in another experiment we set the text lengths threshold to 10 words (as opposed to 5 here) and achieved the classification accuracy of over 90% on just 452 observations, represented as content lemmas, in the same experimental setup. on the one hand, it seems to indicate that a smaller, but higher-quality data returns better results, but on the other hand, can we claim that the results pertain to the original dataset? if half of the datapoints end up in the garbage bin, how can we say that the results reflect the content of the web sources used? our observations raise issues with the selection of web resources and data collection techniques. we have seen that some websites are more suitable as sources of textual data than others. ## conclusion the paper contains a detailed description of preprocessing steps that were taken to prepare textual data collected from seven web aggregators of cultural information for automatic analysis using nlp methods. we followed the best practices in the field, and expected that cleaner data with 15-40% smaller vocabulary would yield gains in the classifier performance proportional to the efforts invested. however, it seems that these efforts fall with the 80% of work that generates only 20% of profits, following the famous pareto principle. in the future, we plan to explore the interrelations between types of preprocessing and actual gains seen for different types of language representations, particularly embeddings and classification setups. what is the impact of preprocessing on parsing? which types of noise can actually be tolerated? one take-away from this project is that selection of web resources is important: some of them return cleaner data than others. it is true that data preprocessing takes most of the research time, too. when reporting the results, in addition to the inaccuracies of analysis, two caveats have to be made with regard to the amount of observations (datapoints) discarded during preprocessing.