P enn- H elsinki Parsed Corpus of Early M odern E nglish: First Parsing Results and Analysis (2022)

Abstract: The Penn-Helsinki Parsed Corpus of Early Modern English (PPCEME), a 1.7-million-word treebank that is an important resource for research in syntactic change, has several properties that present potential challenges for NLP technologies. We describe these key features of PPCEME that make it challenging for parsing, including a larger and more varied set of function tags than in the Penn Treebank, and present results for this corpus using a modified version of the Berkeley Neural Parser and the approach to function tag recovery of Gabbard et al. (2006). While this approach to function tag recovery gives reasonable results, it is in some ways inappropriate for span-based parsers. We also present further evidence of the importance of in-domain pretraining for contextualized word representations. The resulting parser will be used to parse Early English Books Online, a 1.5 billion word corpus whose utility for the study of syntactic change will be greatly increased with the addition of accurate parse trees.

## introduction the penn-helsinki parsed corpus of early modern english (ppceme) @xcite consists of over 1.7 million words of text from 1500 to 1712, manually annotated for phrase structure. it belongs to a family of treebanks of historical english @xcite @xcite and other languages @xcite @xcite with a shared annotation philosophy and similar guidelines across languages, which form the basis for reproducible studies of syntactic change @xcite @xcite @xcite . ppceme is of interest for nlp researchers as well as linguists since it differs from the penn treebank (ptb) @xcite along several dimensions, and it is not obvious that parsing technologies mostly based on the ptb can transfer over successfully. while the text in the ptb is rigidly standardized, subject to editorial guidelines, the source text in ppceme is more chaotic, with a lack of standardization of spelling and punctuation. the annotation style is similar to that of the ptb, but it has more finely-grained part of speech (pos) tags, some common and important structures (e.g., coordination) are annotated differently than in the ptb, and perhaps most significantly, it makes far greater use of function tags. there has however been relatively little work in the nlp community @xcite @xcite using ppceme or its sister corpora, the penn parsed corpus of middle english, 2nd edition (ppcme2) and the penn parsed corpus of modern british english (ppcmbe) 1 . in this paper, we present first results for both parsing accuracy and function tag recovery for ppceme, pointing out some limitations for the latter. our parser is a slightly modified version of the self-attentive neural constituency parser introduced by @xcite . while constituentparsing technology has seen notable gains in recent years, to our knowledge there has no work on recovering function tags within this framework, which is crucial for the larger context of this work, discussed below in this introduction. we use an approach to function tag recovery used earlier with the ptb @xcite and show that while it gives reasonable results, it is inefficient and inappropriate for contemporary span-based constituency parsers, especially as the number of function tags increases. many of our decisions are informed by the larger context of this work: parsing the early english books online (eebo) corpus @xcite . while ppceme has been used for building models of syntactic change, its usefulness for this work is limited by the fact that various phenomena of interest are too sparse within it to support reliable statistical modeling of language change (e.g. distinguishing between competition @xcite and drift @xcite ). in contrast, eebo covers the same time period, 1475 to 1700, with 1.5 billion words of text -a difference of three orders of magnitude. however, eebo's potential as a resource for linguistic research remains unrealized because it is not linguistically annotated and its size renders manual annotation infeasible. our goal therefore is the creation of a parsed version of eebo consistent with the structures in ppceme. this parsed version of eebo will be used for linguistic research on language change by searching for structures of interest based on the phrase structure and pos tags. the parsing model trained on ppceme will be used to parse eebo so that it too can be used for queries for structures of linguistic interest, although at a much larger scale than previously possible. for more details, we refer to @xcite . this broader concern impacts our design choices for the parser; for instance, our decision to work on constituency parsing rather than dependency parsing -linguistic queries depend on phrase structure trees. likewise, since the function tags and pos tags are used by the linguistic queries, it critical that the parser be able to generate this information. it also informed the development of our tokenizer for eebo. while in this paper we use standard metrics for parser evaluation, we also refer to @xcite for a discussion of an alternative method of evaluation based on precision and recall for the specific queries. ## previous work the ppceme is one of several corpora designed for historical linguistics research that share annotation styles and design decisions. others include the second edition of the penn-helsinki parsed corpus of middle english @xcite ) (ppcme2) and the penn parsed corpus of modern british english @xcite . while there has been much linguistics research based on these corpora, there is relatively little previous nlp work using them. @xcite described parsing ppcmbe, while @xcite and @xcite focused on postagging, the former on ppcme2, and the latter on ppceme and ppcmbe. this previous work has discussed characteristics of these corpora that differ from that of the ptb and the potential impact on models trained on these corpora. a focus has been on how to transform the annotation to be closer to that of the ptb, such as by transforming the phrase structure @xcite or by mapping the ppcme2 pos tag set into that of ptb @xcite . ## major differences between ppceme and ptb as mentioned in the introduction, the ppceme annotation style differs from that of ptb in certain respects. we describe here two of the most significant differences in more detail as well as how they influenced our preprocessing decisions. we also give examples in section 3.3 of the spelling variations present in the early modern english data. ## elmo embeddings trained on eebo in recent years, contextualized word embeddings such as elmo @xcite and bert @xcite have driven significant improvements on downstream nlp tasks, including pos tagging and parsing. due to the significant overhead involved in training these representations, researchers often use pretrained models distributed by large companies, sometimes fine-tuned to the domain of interest. although this often produces perfectly satisfactory results, in cases of significant mismatch between a test domain and standard training domains -usually sources such as text scraped from wikipedia, bookscorpus @xcite , news text from common crawl @xcite , and discussion forums @xcite ) -pretraining on the novel domain yields significant improvements @xcite @xcite . because of the orthographic and syntactic differences between early modern english and contemporary english mentioned in section 3, our current work involves exactly such a mismatch, and so we pretrained elmo embeddings on eebo. 7 we used the same model configuration as @xcite for 11 epochs foot_7 . pretraining was performed using the tensorflow implementation maintained by al-lennlp foot_8 using the default model configuration. we then integrated the resulting embeddings, which have 1,024 dimensions, into the parser model (see section 6). further details regarding the pretraining may be found in appendix c. ## model and evaluation 6.1 parser architecture we use the parsing model of @xcite , which represents a constituency tree t as a set of labeled spans (i, j, l) where i and j are the beginning and ending positions of the span and l its label. each tree is assigned a score s(t ), which is decomposed as a sum of per-span scores: the per-span scores s(i, j, l) are themselves assigned using a neural network that takes a sequence of per-word embeddings as input, processes these embeddings using a transformer-based encoder @xcite , and then produces a span score from an mlp classifier @xcite . the highest-scoring valid tree is found using a variant of the cky algorithm. pos tags are recovered using a separate classifier operating on top of the encoder output, which is jointly optimized with the span classifier. for more details, see @xcite . we consider four approaches for producing the word embeddings that serve as input to the encoder: bert-base contextual word embeddings are computed using bert @xcite by retaining the output of the last layer for the final subword unit of each token. we use the bert-base model distributed via the transformers python package foot_9 , which is pre-trained on 3.3 billion words of modern english. elmo (orig) as above, but using elmo @xcite . we use the elmo-original model distributed by allennlp foot_10 , which was pre-trained on 1 billion words of modern english. elmo (eebo) elmo embeddings pre-trained on early modern english from eebo. uses the same model architecture as elmo (orig). char-lstm tokens are represented as the concatenation of the outputs of a bidirectional character-level lstm for the final character of the token. all four of these embeddings types are character or subword based, which is particularly helpful for the ppceme corpus due to the spelling variations. the subword units for bert however are derived from its modern english training material. our implementation is based on version 0.2.0 of the berkeley neural parser foot_11 with modifications to allow the use of elmo. 13 . we train each of the 8 models (one for each cross-validation split) for 50 epochs, using the evalb @xcite score on the dev section as our criterion for saving the best model. for additional details regarding training and hyperparameters, see appendix d. ## pretraining comparison experiments for each of the three versions of the training and evaluation data, the best f1 scores are obtained using elmo (eebo), followed by bert, elmo (orig), and, char-lstm. the fact that bert is consistently outperformed by elmo (eebo) is particularly interesting given that (a) bert in general outperforms elmo when trained on similar material, (b) bert in this case was trained on three times as much material as elmo, and (c) the bert model parameters were fine-tuned, while the elmo parameters were not. this result underlines the importance of pretraining on in-domain materials, a fact that has also been observed for nlp tasks in biomedical @xcite , financial @xcite , and legal @xcite domains. overall, we note that the scores are a few points lower than the current state of the art for ptb (96.38% f1 on the test section @xcite ). as @xcite point out, all of the english historical corpora lack certain brackets present in ptb (base-nps and vps) that are relatively "easy to get", and this tends to adversely affect their parsing scores. specifically, they find that it impacts the f1 score for ppcmbe by about 2.5% absolute, an effect we expect to carry over to ppceme. ## part-of-speech results while the evalb scores are of greater importance, pos accuracy also matters as the linguistic queries we plan to run on the eventual parsed eebo sometimes refer explicitly to pos tags. we therefore consider the pos results in detail. as is apparent from as the distribution of pos tags is highly unbalanced, we also break down the 98.30% score in ## out-of-vocabulary span labels and resource inefficiency as shown in as discussed in section 6.1, the parser uses a classifier to produce the most likely label for a span. the parser determines the label vocabulary at training time, by collecting all the possible nonterminal labels. there are two complications in this determination of the label vocabulary. first, for unary branches, the parser combines the different nonterminals into a single label. for example, for the structure (np (cp ...)), which is a np with a single child, a cp, the label is np::cp. at parse time, if this label is the most likely for a span, then it is split up again into a parent np and child cp node. second, since function tags are integrated into the label vocabulary, they multiply out over the cases with unary branching. for example, with the structure (np (cp-frl ...)), for a np with a free relative cp child, the value in the label vocabulary will be np::cp-frl. but there can also be instances (np-sbj (cp-frl ...)), (np-ob1 (cp-frl ...)), resulting in the entries np-sbj::cp-frl and np-ob1::cp-frl in the label vocabulary. the label vocabulary used by the model is the "collapsed" value for the train section. this increase of 687.88 resource requirements during training, and we found that the training time could increase much of the increased resource requirements is not required, though. the "test" rows show the corresponding numbers for the eight test sections, and the "both" row shows the label values that are in both the train and test sections. even without function tags, an average of only 111.12 of the labels in the training section occur in the test section, while the test section has an average of 6.38 labels that are not in the training section (117.50 -111.12). the inefficiency worsens considerably with more function tags. for ftags-31, an average of only 278.62 of the 937.88 labels in the training section occur in the test section, and the test section has an average of 21.26 labels that are not in the training section (299.88 -278.62). if a label is in the test section but not in the label vocabulary determined by the train section, the parser will not correctly predict spans with that label. they are in effect out-of-vocabulary span labels. while the parser score is not damaged more, due to the infrequency of these oov labels, this shows the limits of the current approach. we consider these current results to be a baseline for future improvement. ## function tag analysis to explore these results in more detail, we pick one cell in the table -elmo eebo with 31 tags, for the test sections -and look at the individual scores for the 31 function tags that make up the 95.55% result, as shown in examining the results for the tags included in ftags-31 but not in ftags-10, the main cause for the drop in score is the spe tag for direct speech, which is one of the most common tags, with an average 3.91% frequency, but with an average score of 46.08% f1. 17 the ppceme lacks the consistent clues for direct speech (such as quotation marks) that would be available in newswire text adhering to modern punctuation guidelines. for example, the sentence and he shall be welcome has an ip-mat-spe (matrix ip, direct speech) as the root of the entire tree in gold annotation, while in the parser output it is ip-mat. the parser (or for that matter, a person) cannot know that the sentence is direct speech without looking at the wider context, which is a conversation spanning a few sentences. the underlying problem here is related to that of the first example of a pos error in section 8. future work to recover the direct speech information will need to examine discourse contexts beyond just single sentences in isolation, as the parser does now. the other scores show some variance in accuracy, although some of the most common tags (sbj, sub, mat) have the highest scores. considering the tags as groups, there are some drops in scores among some of the tags for cp or ip constituents (e.g. purpose clauses). these require more anal-ysis, although in general the question of whether the function tag recovery is satisfactory can only be answered with reference to the larger aim of this work (that is, by determining if the tags are accurate enough to ensure accurate results from subsequent linguistic searches on automatically parsed material). ## conclusion and future work in this work we have presented the first parsing results on the ppceme, with a focus on challenges posed by its extensive set of function tags. adapting an earlier approach to function tag recovery works reasonably well, while we point out some challenges for future work concerning improvements in accuracy and resource efficiency. we have also demonstrated the continued importance of in-domain pretraining, as the parser configuration using elmo trained on in-domain eebo outperforms the configuration using bert trained on modern english. we performed crossvalidation to ensure that the results were robust and not an accident of a particular split. future work will proceed along a number of lines. regarding the function tags, an obvious approach is to integrate a separate function tag classifier into the parser separate from one for the bare nonterminals and to combine them to produce the full nonterminal labels. we will also evaluate the function tags by their utility for linguistic searches on automatically parsed material, as mentioned at the end of section 10.