Challenging A merica: Modeling language in longer time scales (2022)

Abstract: The aim of the paper is to apply, for historical texts, the methodology used commonly to solve various NLP tasks defined for contemporary data, i.e. pre-train and fine-tune large Transformer models. This paper introduces an ML challenge, named Challenging America (ChallAm), based on OCR-ed excerpts from historical newspapers collected from the Chronicling America portal. ChallAm provides a dataset of clippings, labeled with metadata on their origin, and paired with their textual contents retrieved by an OCR tool. Three, publicly available, ML tasks are defined in the challenge: to determine the article date, to detect the location of the issue, and to deduce a word in a text gap (cloze test). Strong baselines are provided for all three ChallAm tasks. In particular, we pre-trained a RoBERTa model from scratch from the historical texts. We also discuss the issues of discrimination and hate-speech present in the historical American texts.

## introduction the dominant approach in the design of current nlp solutions is (pre-)training a large neural language model, usually applying a transformer architecture, such as gpt-2, roberta or t5, and fine-tuning the model for specific tasks @xcite . the solutions are evaluated on benchmarks such as glue @xcite or superglue @xcite , which allow comparing the performance of various methods designed for the same purpose. an important feature of a good nlp benchmark is the clear separation between train and test sets. this requirement prevents data contamination, when the model (pre-)trained on huge data might have "seen" the test set in some form. the expansion of digital information is proceeding in two directions on the temporal axis. in the forward direction, new data are made publicly available on the internet every second. what is less obvious is that, in the backward direction, older and older historical documents are digitized and disseminated publicly. to the best of our knowledge, our paper introduces the first benchmark which serves to use and evaluate the "pre-train and fine-tune scenario" applied to a massive collection of historical texts. the very idea of building language models on historical data is not new. the google ngram viewer @xcite is based on large amounts of texts from digitized books. the corpus as a whole is not open for the nlp community -only raw n-gram statistics are available. the temporal information is crude (at best, the year of publication is given) and the corpus is heterogeneous (in fact, it is a dump of digitized books of any origin). in our research, we use one of the richest sources of homogeneous historical documents, chronicling america, a collection of digitized newspapers that cover the publication period of over 300 years (with significant coverage of 150 years), and design an nlp benchmark that may open new opportunities for the modeling of the historical language. recently, time-aware language models such as temporal t5 @xcite and tem-pobert @xcite have been proposed. they focus on modern texts dated yearly, whereas we extend language modeling towards both longer time scales and more fine-grained (daily) resolution, using massive amounts of historical texts. the contribution of this paper is as follows: these are the main features of the corpus: the corpus size is 74 gb (201 gb of total raw text), which is comparable with contemporary text data for training massive language models, such as gpt-2, roberta or t5; the corpus is free of spam and noisy data (although the quality of ocr processing varies); ## related machine learning datasets and challenges this section concerns ml challenges which deliver labeled ocr documents as training data, a definition of the processing task, and an evaluation environment to estimate the performance of uploaded solutions. more often than not, such challenges concern either layout recognition (localization of layout elements) or key information extraction (finding, in a document, precisely specified business-actionable pieces of information). layout recognition in japanese historical texts is described in @xcite . the authors use deep learning-based approaches to detect seven types of layout element categories: page frame, text region, text row, title region, etc. some key information extraction tasks are presented in @xcite . the two datasets described there contain, respectively, nda documents and financial reports from charity organizations. the tasks for the datasets consist in detecting data points, such as effective dates, interested parties, charity address, income, spending. the authors provide several baseline solutions for the two tasks, which apply up-to-date methods, pointing out that there is still room for improvement in the kie research area. a challenge that comprises both layout recognition and kie is presented in @xcite ) -the challenge is opened for the recognition of ocr-scanned receipts. in this competition (named icdar2019) three tasks are set up: scanned receipt text localization, scanned receipt ocr, and key information extraction from scanned receipts. a common feature of the above-mentioned challenges is the goal of retrieving information that is explicit in the data (a text fragment or layout coordinates). our tasks in challam go a step further: the goal is to infer the information from the ocr image rather than just retrieve it. similar challenges for two out of the three tasks introduced in this paper have been proposed before for the polish language: the training sets for those challenges were purely textual. here, we introduce the challenges with the addition of original images (clippings), though we do not use graphical features in baselines yet. ## chronicling america in 2005 a partnership between the national endowment for the humanities and the library of congress launched the national digital newspaper program, to develop a database of digitized documents with easy access. the result of this 15-year effort is chronicling america -a website foot_0 which provides access to selected digitized newspapers, published from 1690 to the present. the collection includes approximately 140 000 bibliographic title entries and 600 000 library holdings records, converted to the marcxml format. the portal supports an api which allows accessing of the data in various ways, such as the json format, bulkdata (bulk access to data) or linked data, foot_1 or searching of the database with the opensearch protocol. 3 . the accessibility of data in various forms makes chronicling america a valuable source for the creation of datasets and benchmarks. the portal serves as a resource for various research activities. cultural historians may track performances and events of their interest in a resource which is easily and openly accessible, as opposed to commercial databases or "relatively small collections of cultural heritage organizations whose online resources are isolated and difficult to search" @xcite . the database enables searching for the first historical usages of word terms. for instance, thanks to the chronicling america portal, it was discovered in @xcite that the term "fake news" was first used in 1889 in the polish newspaper ameryka. the resource is helpful in research aiming to improve the output of the ocr process. the authors of @xcite study ocr errors occurring in several digital databases -including chronicling america -and compare them with human-generated misspellings. the research results in several suggestions for the design of ocr post-processing methods. the implementation of an unsupervised approach in the correction of ocr documents is described in @xcite . two million issues from the chronicling america collection of historic u.s. newspapers are used in a sequence-to-sequence model with attention. chronicling america is a type of digitized resource that may be of wide use for both humanities and computational research. we prepared datasets and challenges based on the data from the chronicling america resource. we hope that our initiative will bring about research that will facilitate the development of ml-based processing tools, and consequently increase access to digitized resources for the humanities. an example of an ml tool based on chronicling america is described in @xcite . the task was to predict bounding boxes around various types of visual content: photographs, illustrations, comics, editorial cartoons, maps, headlines and advertisements. the training set was crowd-sourced and included over 48k bounding boxes for seven classes. using a pre-trained faster-rcnn detection object, the researchers achieved an average accuracy of 63.4%. both the training set and the model weights file are publicly available. still, it is difficult to estimate the value of the results achieved without any comparison with other models trained on the same data. in our proposal we go a step further. we provide and make freely available training data from chronicling america for three ml tasks. for each task we develop and share baseline solutions. alternative solutions can be submitted to the gonito foot_3 evaluation platform @xcite to be evaluated automatically and compared against our baselines. ## data processing the pdf files were downloaded from chronicling america and processed using a pipeline primarily developed for extracting texts from polish digital libraries @xcite . firstly, the metadata (including url addresses for pdf files) were extracted by a custom web crawler and then normalized; for instance, titles were normalized using regular expressions (e.g. the bismarck tribune. the procedure is executed in a continuous manner to allow the future processing of new files that are yet to be digitized and made public by the chronicling america initiative. this solution requires a future-proof procedure for splitting and preparing data for machine-learning challenges. for instance, the assignment of documents to the training, development and test sets should not change when the raw data set is expanded. such a procedure is described in section 6. ## data for unsupervised training the state of the art in most nlp tasks is obtained by training a neural-network language model on a large collection of texts in an unsupervised manner and fine-tuning the model on a given downstream task. at present, the most popular architectures for language models are transformer @xcite models (earlier, e.g. word2vec @xcite or lstm models @xcite ). the data on which such models are trained are almost always modern internet texts. the high volume of texts available at chronicling america, on the other hand, makes it possible to train large transformer models for historical texts. using a pre-trained language model on a downstream task bears the risk of data contamination -the model might have been trained on the task test set and this might give it an unfair edge (see @xcite for a study of data contamination in the case of the gpt-3 model when used for popular english nlp test sets). this issue should be taken into account from the very beginning. in our case, we release foot_5 a dump of all chronicling america texts (for pre-training language models), but limited only to the 50% of texts that would be assigned to the training set (according to the md5 hash). this dump contains all the texts, not just the excerpts described in section 6.2. as the size of the dump is 74.0g characters, it is on par with the text material used to train, for instance, the gpt-2 model. we also release a roberta base challam model trained on the text corpus. the model was trained from scratch, i.e. it was not based on the weights of the original roberta model @xcite . the bpe dictionary was also induced anew. two versions of the roberta challam model were prepared: one foot_6 was trained with temporal metadata encoded as a prefix of the form year: yyyy, month: mm, day: dd, weekday: wd, another, foot_7 for comparison, without such a prefix. the challam models have the same number of parameters as the original roberta base (125m). each model was trained on two tesla v100 32gb gpus for 9 days. ## procedure for preparing challenges we created a pipeline that can generate various machine learning challenges. the pipeline input should consist of djvu image files, text (ocr image), and metadata. our main goals are to keep a clear distinction between dataset splits and to assure the reproducibility of the pipeline. this allows potential improvement to current challenges and the generation of new challenges without dataset leaks in the future. we achieved this by employing stable pseudo-randomness by calculating an md5 hash on a given id and taking the modulo remainder from integers from certain preset intervals. these pseudo-random assignments are not dependent on any library, platform, or programming language (using a fixed seed for the pseudo-random generator might not give the same guarantees as using md5 hashes), so they are easy to reproduce. this procedure is crucial to make sure that challenges are future-proof, i.e.: ## challenging america tasks in this section, we describe the three tasks defined in the challenge. they are released on the gonito evaluation platform, which enables the calculation of metrics both offline and online, as well as the submission of solutions. an example of text from an excerpt given in those tasks is shown in figure 1b . ## results strong baselines for all three tasks are available at the gonito evaluation platform. the baselines (see tables 2 and 3 ) include, for each model, its score in the appropriate metric as well as the git sha1 reference code in the gonito benchmark (in curly brackets). reference codes can be used to access any of the baseline solutions at http: //gonito.net/q . we distinguish between self-contained submissions, which use only data provided in the task, and non-self-contained submissions, which use external data, e.g. publicly available pre-trained transformers. our baselines take into account only textual features. more detailed analysis of the baseline performance is given in appendix c. the current top performing models have the most difficulty with texts which (1) are older, (2) contain ocr noise, (3) come from less popular locations (especially, in the west). ## ethical issues we share the data from chronicling america, following the statement of the library of congress: "the library of congress believes that the newspapers in chronicling america are in the public domain or have no known copyright restrictions." 12 historical texts from american newspapers may be discriminatory, either explicitly or implicitly, particularly regarding race and gender. recent years have seen research on the detection of discriminatory texts. in @xcite adversarial training is used to mitigate racial bias. in @xcite the authors "take an unsupervised approach to identifying gender bias against women at a comment level and present a model that can 12 https://chroniclingamerica.loc.gov/about surface text likely to contain bias." the most recent experiments on the topic ( @xcite , ( aluru et al., 2020) ) result in re-trained bert models for abusive language detection in english. we use one of them, @xcite , to detect the abusive texts in the challam dataset. we tagged items that either (1) are marked as abusive speech by dehatebert with the probability greater than 0.75 or (2) contain words from a list of blocked words. the fraction of detected texts was 2.04-2.40 % (depending on the challenge and set). the tags along with the probabilities are available in the hate-speech-info.tsv files for each test directory. note that temporal and geospatial metadata might constitute useful features in future work on better detection of hate speech in historical texts. ## conclusions this paper has introduced a challenge based on ocr excerpts from the chronicling america portal. the challenge consists of three tasks: guessing the publication date, guessing the publication location, and filling a gap with a word. we propose baseline solutions for all three tasks. chronicling america is an ongoing project, as we define our challenge in such a way that it can easily evolve in parallel with the development of chronicling america. firstly, any new materials appearing on the portal can be automatically incorporated into our challenge. secondly, the challenge is open for five yet undefined ml tasks. if the newspaper edition contains no such excerpts, we reject it. if there is more than one such excerpt, we select one excerpt using a stable pseudo-random procedure based on the newspaper edition id. this procedure produces text excerpts with images consisting of ocr texts only. the excerpts are downsized to reduce the size to an appropriate degree to maintain good quality. we do not pre-process images in any other way, so excerpts may have different sizes, height-to-width ratios, and colors.