Batavia asked for advice. Pretrained language models for Named Entity Recognition in historical texts. (2021)

Abstract: Pretrained language models like BERT have advanced the state of the art for many NLP tasks. For resource-rich languages, one has the choice between a number of language-specific models, while multilingual models are also worth considering. These models are well known for their crosslingual performance, but have also shown competitive in-language performance on some tasks. We consider monolingual and multilingual models from the perspective of historical texts, and in particular for texts enriched with editorial notes: how do language models deal with the historical and editorial content in these texts? We present a new Named Entity Recognition dataset for Dutch based on 17th and 18th century United East India Company (VOC) reports extended with modern editorial notes. Our experiments with multilingual and Dutch pretrained language models confirm the crosslingual abilities of multilingual models while showing that all language models can leverage mixed-variant data. In particular, language models successfully incorporate notes for the prediction of entities in historical texts. We also find that multilingual models outperform monolingual models on our data, but that this superiority is linked to the task at hand: multilingual models lose their advantage when confronted with more semantical tasks.

## introduction pretrained language models @xcite @xcite have recently advanced the state-of-the-art in many nlp tasks, providing deep contextual language representations and ease of deployment. bert @xcite has proven particularly successful, combining the attention-based contextual representations of transformers @xcite with a simple fine-tuning procedure, allowing to deploy quickly to any sequence or document classification task. bert has given birth to a myriad of variants, differing by their training procedure, model size or language. resource-rich languages in particular are spoilt for choice. dutch for instance has two main monolingual models: the bert-based bertje @xcite and @xcite , itself based on roberta @xcite , a revision of bert's training procedure. but multilingual models like mbert @xcite and xlm-r @xcite ) are also applicable. these models, which are trained on 104, respectively 100 languages at once, perform well on crosslingual transfer @xcite @xcite . @xcite notably show that mbert learns generic representations over different input languages and scripts. this makes multilingual models particularly interesting for historical texts, not only because these contain noncontemporary language, but also because they exhibit language variation due to language change over long periods of time or unstandardized spelling. but what when historical texts are accompanied by modern notes? how do monolingual and multilingual models compare on the historical and editorial parts of such texts, and on their combination? if we only consider modern notes, it is not given that monolingual models outperform multilingual models on their training language. notwithstanding possible differences in genre, the nature of the linguistic task at hand is an important factor in determining whether a monolingual or multilingual model is more appropriate. @xcite and @xcite show for dutch that mbert is competitive with monolingual models on pos tagging (on universal dependencies pos), while dutch models perform better at semantic tasks like semantic role labelling or language-specific tasks like agreement resolution. for named entity recognition (ner), results appear mixed, as both bertje and robbert perform in-between mbert and a version optimized by @xcite . how semantic or language-specific does a task need to be for monolingual models to perform better than multilingual models in historical texts with editorial notes? in this article, we consider these questions in the context of ner in 17th and 18th century dutch texts. we introduce a new ner dataset based on the general letters of the united east india company (voc) foot_0 . these official letters were written to the board of the voc between 1610 and 1795. they report on the activities of the voc (trade, conflicts, intelligence) in and around indonesia and up to japan and south africa. between 1960 and 2017, a selection of this corpus, spanning the time period 1610-1767, was transcribed and edited by the huygens institute for the history of the netherlands (huygens ing) and its predecessors. annotating entities in these letters is part of a larger effort to facilitate historical research on these texts. our dataset foot_1 consists of 24.5k entities, with a repartition of 40%/60% between historical text and editorial notes, where the historical text covers the period 1621 to 1760. we introduce a semantic component in this dataset by distinguishing metonymical uses of locations (which represent almost half of all entities). this allows us to compare two ner variants: a standard one, where locations form a single entity type; and a more semantic one, where models additionally must distinguish locations in agent-like roles, as in for instance, batavia asked for advice. we compare the performance of monolingual and multilingual models on both ner variants through bertje @xcite , rob-bert @xcite , mbert @xcite and xlm-r @xcite . on standard ner, we find that multilingual models outperform monolingual models on both historical text and modern notes. in crosstext finetuning and evaluation, we confirm the strong crosslingual transfer ability of multilingual models in the context of historical language variation. we also show that notes and historical text are complementary for ner, as all pretrained models, multilingual and monolingual, benefit from their combination. on the more semantic-oriented ner task, we find that monolingual and multilingual models perform com-petitively, with robbert and mbert performing best. this confirms the importance of the semantic nature of a task as a factor in choosing between monolingual and multilingual models. ## annotations the labelset consists of five base labels-loc, org, per, rel and shp-for the entity types locations, organisations, persons, religions and ships. following @xcite , this labelset is extended with secondary labels of the form x deriv or x part for expressions derived from entity names by grammatical derivation (as with for instance, the location banda and the derived bandanezen foot_2 ) or composition (as with java and javakoffie foot_3 ). the labelset is extended with a gpe label (geopolitical entities) to distinguish metonymical use of location names. ## results we first present results on standard ner, comparing the in-text (text or notes) and crosstext performance of monolingual and multilingual models, and assessing the reciprocal contribution of text and notes. we end with results on semantic-oriented ner. ## conclusion we have introduced a new dutch dataset for named entity recognition, consisting of early modern dutch voc letters and modern edito-rial notes. comparing monolingual and multilingual pretrained language models, we confirm the stronger crosslingual abilities of multilingual models, while showing that both monolingual and multilingual models can leverage on mixed language variants at fine-tuning. for named entity recognition, pretrained language models are notably able to leverage on notes to improve text predictions. we have further shown that multilingual pretrained language models not only perform better on the historical part of this dataset, but also on modern dutch notes. however, the superiority of multilingual models on modern dutch is very much linked to the task at hand, as orienting the data to more semantic distinctions can turn the tables for monolingual models.