Chunking Historical G erman (2021)

Abstract: Quantitative studies of historical syntax require large amounts of syntactically annotated data, which are rarely available. The application of NLP methods could reduce manual annotation effort, provided that they achieve sufficient levels of accuracy. The present study investigates the automatic identification of chunks in historical German texts. Because no training data exists for this task, chunks are extracted from modern and historical constituency treebanks and used to train a CRF-based neural sequence labeling tool. The evaluation shows that the neural chunker outperforms an unlexicalized baseline and achieves overall F-scores between 90% and 94% for different historical data sets when POS tags are used as feature. The conducted experiments demonstrate the usefulness of including historical training data while also highlighting the importance of reducing boundary errors to improve annotation precision.

## introduction the analysis of linguistic phenomena in historical language requires large amounts of annotated data. for example, to study the development of syntactic phenomena like object order or extraposition in german, syntactically annotated texts from all relevant time periods are needed. to date, however, only very few historical corpora provide annotations beyond the morpho-syntactic level, thus limiting syntactic research to qualitative studies on small data sets. using nlp methods for the automatic creation of relevant annotations could support the annotation process and reduce the necessary manual effort for quantitative studies. but the application of standard tools to historical data faces a variety of challenges, as there is less or no training data, the data is less standardized, etc. the present study investigates the automatic recognition of chunks in historical german. section 2 gives a short introduction to the chunking task and explains peculiarities about chunking german concerning complex pre-nominal modification. section 3 presents previous approaches to automatic chunking, which have not yet been applied to historical data, likely because no manually annotated data is available. in this study, to address the lack of chunked historical data, chunks are extracted from modern and historical constituency treebanks. section 4 describes the training data as well as the additional test data sets before section 5 introduces the selected methods for automatic chunking: a regular expression-based baseline and a neural crf chunker. finally, section 6 details the evaluation process and presents the results, followed by a conclusion in section 7. ## chunking (german) chunking is also referred to as partial or shallow parsing. the concept of chunks was introduced by @xcite , who defines them as non-recursive phrases from a sentence's parse tree ending with the head of the phrase. according to this definition, a chunk may contain chunks of other types but not of the same type, and post-nominal modifiers start a new chunk. example (1) shows the annotation of an english sentence following abney's chunk definition: defining chunks this way makes them suitable for the automatic annotation with sequence labeling methods and is especially useful for tasks that do not require a complete syntactic analysis but profit from an easy and fast annotation, e.g. agreement checking in word processors @xcite . furthermore, it may serve as a basis for deeper syntactic analyses (cf. @xcite @xcite and thus could build the foundation for the automatic syntactic annotation of historical data. however, applying the standard definition of chunks is problematic when chunking german because of possibly complex pre-nominal modification. the phrase in example (3) violates abney's chunk definition due to the embedded noun chunk and, when annotated according to the conllstyle definition, it would contain an article der that is separated from its noun chunk as in example (4). the his son loving father 'the father who loves his son' @xcite (4) der [nc seinen sohn] [nc liebende vater] while in some german corpora, these stranded tokens are left unannotated, e.g. dereko @xcite , @xcite introduce a special category for stranded material, marked with an initial 's', e.g. snc for a stranded noun chunk. they also suggest including the head noun chunk in the prepositional chunk while leaving post-nominal modifiers separate. in the following, their approach is adopted for chunking german. of the eleven original chunk types from the conll-2000 shared task, four main types are considered in this study: noun chunks (nc), prepositional chunks (pc), adjective chunks (ac), and adverb chunks (advc), and, in addition, stranded noun (snc) and prepositional chunks (spc). example (5) shows the annotation of a sentence from an 1871 newspaper taken from one of the historical data sets in this study. for better readability, the relation of stranded articles to their respective noun chunks is indicated by subscripts. as chunks of a given type can only contain certain part-of-speech sequences, most of the studies use pos tags as features. however, lexicalization of models can also improve chunking results (cf. @xcite and current contextual word representations already seem to have some awareness of shallow syntactic structures like chunks @xcite . in general, @xcite find that pos tags are most relevant if the training data is small, while words become more helpful with increasing amounts of data, and a combination of both features yields the best results. for evaluation, most studies still use the data set from the conll-2000 shared task (sang and buchholz, 2000), i.e. wsj data from the penn treebank, and written news data also serves as the evaluation basis for most studies on german. however, when @xcite compare tools on english conll-2000 data with their performance on twitter data, they find that for standard toolkits, f 1 -scores decrease by 17 to 38 percentage points to 45%-54% on social media text. a similar drop in performance might also occur for other non-standard data like historical language and would underline the importance of methods and models that are specifically tailored to a particular language variety. but to date, there has only been a small number of studies on the automatic syntactic analysis of historical german, all of which have to deal with a lack of syntactically annotated historical data. in the absence of a gold standard, some studies develop rule-based approaches, e.g. @xcite for topological field identification in middle high german. but without the possibility for evaluation, the accuracy of such systems remains unclear. other studies try to compensate for the lack of historical data by falling back on modern german. @xcite approximates historical language by removing punctuation and capitalization from a modern german news corpus. using crfs, he tries to identify segments of increasing length, chunks, clauses, and sentences, in this artificial data set and concludes that smaller units are easier to identify. for chunking, he reports an f 1 -score of 93.3%, but since capitalization and punctuation are not the only differences between modern and historical german, it is unclear how well these results generalize to real historical data. nevertheless, the exploitation of modern data can be conducive for automatically annotating historical language by reducing the need for large annotated historical data sets. as a previous study has shown, models trained on modern newspaper text can successfully be transferred to historical german with f 1 -@xmath0, 2020). ## data as already mentioned, most german corpora and especially historical corpora do not offer a manual chunk annotation that could be used for training and evaluating automatic models. however, @xcite notice that chunks can be extracted directly from constituency trees by converting the lowest phrasal projections with lexical content to chunks. using this method, they automatically transform the constituency annotations from the tüba-d/z treebank @xcite into chunks. the resulting corpus foot_0 comprises 3,816 newspaper articles with more than 100k sentences and almost 2m tokens. in total, it contains over 743k instances of the six chunk types considered in the present study. since the extracted chunks might be influenced by the structure of the constituency trees and, hence, may differ between treebanks with different syntactic annotation schemes, a second german treebank is included in the present study. the tiger corpus @xcite foot_1 contains about 50k sentences with about 888k tokens from 2,263 german news articles, but the annotation of certain syntactic phenomena deviates significantly from those in the tüba-d/z corpus @xcite . most notably, the tiger treebank includes discontinuous annotations. therefore, all sentences must be linearized first foot_2 before chunks of the six different types can be extracted from the constituency trees similar to the procedure described by @xcite . besides accounting for possible influences of the annotation scheme on the extracted chunks, including the tiger treebank offers another advantage: while annotated historical data sets rarely exist for syntactic annotation tasks, there are two treebanks for historical german, which are annotated according to the tiger scheme and thus, fortunately, can also be used for chunk extraction. the mercurius corpus @xcite foot_3 contains semi-automatic annotations of approximately 8k sentences with 187k tokens from newspaper text from the 16 th and 17 th centuries. the second treebank, ref.up, is a subcorpus of the reference corpus of early new high german @xcite foot_4 and includes annotations of 26 documents with 21k sentences and 500k tokens from different language areas from the 14 th to 17 th century. like with the tiger corpus, the constituency trees from both historical treebanks must be linearized before chunks can be extracted from them. in total, the two corpora contain about 67k chunks and over 205k chunks of the six relevant types, respectively. while the tiger corpus is already provided with a training, development, and test section, the other three corpora were split into a training (80%), development (10%), and test set (10%) for this study. also, the historical pos tagsets in the mercurius and ref.up treebanks were mapped to the german standard tagset stts @xcite . compared to previous studies on historical data, the two modern and historical treebanks form a solid basis for training and evaluating automatic chunking methods on historical german. however, @xcite notes that distributional differences between training and test data can be even more problematic for chunking performance than noise in the data itself. therefore, three additional data sets from a previous study (ortmann, 2020), 6 which are unrelated to the training data, are used for evaluation. the first data set is a collection of about 550 sentences with 7.6k tokens from five modern registers with a varying degree of formality: wikipedia articles, fiction texts, christian sermons, ted talk subtitles, and movie subtitles. in total, the modern data set contains about 2.8k chunks of the six types and is used to test the applicability of annotation methods to non-newspaper registers. the two other data sets comprise historical data from two different corpora. the hipkon corpus @xcite ) contains 342 manually annotated sentences from 53 sermons from the 12 th to the 18 th century. originally, the corpus only includes a partial annotation of chunks, which was completed for the present study. also, the mapping of the historical pos tags to stts tags from ortmann (2020) was used. the second historical data set consists of 600 sentences with 18.5k tokens from 29 texts from the german text archive dta @xcite . the texts were published in a variety of genres foot_6 from the 16 th to the 20 th century and were manually enriched with chunks for this study, using the corrected pos tags and sentence boundaries from ortmann (2020). ## methods as detailed in section 3, various methods have been applied to the automatic recognition of chunks in modern text. in the present study, two different approaches are tested: an unlexicalized regular expression-based chunker, which serves as a baseline, and a neural state-of-the-art sequence labeling tool. the regular expression-based approach is comparable to the finite-state chunkers mentioned in section 3. for this study, a simple regexp chunker as implemented in the nltk foot_8 is used, which successively applies a set of manually created context-sensitive regular expressions to an input pos sequence to identify non-recursive, nonoverlapping chunks of the six different types. the neural sequence labeling tool ncrf++ (yang and zhang, 2018) foot_9 achieves state-of-theart results for several tasks, including chunking. on the english conll-2000 data, the best model reaches an f 1 -score of 95% @xcite . the toolkit consists of a three-layer architecture with a character sequence layer, a word sequence layer, and a crf-based inference layer. while the regexp chunker relies on expert knowledge in the form of manually compiled rules, ncrf++ must be trained on annotated data to perform the task. for this study, the tool is trained on the two different modern treebanks: model news1 is trained on the tüba-d/z training set, and model news2 on the tiger training set. also, the two historical treebanks are used to train a joined model hist, which might be more suitable for the analysis of historical data and its peculiarities. finally, since the historical data sets are smaller than the modern training sets, a model news2+hist is trained on a combination of the modern and historical treebanks that follow the same annotation scheme. during training, the tool is provided with the corresponding development data and each of the models is trained with and without pos tags as an additional feature. since current contextual word representations seem to be aware of shallow syntactic structures @xcite , each model is also trained with glove embeddings pretrained on german wikipedia. 11 to ensure comparability, all models are trained with the same default settings. 12 while the news2 and hist training sets only contain annotations of the six chunk types considered in this study, the news1 model is trained on all chunk types included in the tüba-d/z corpus, although only the six types described in section 2 are evaluated here. for each token, both selected methods, i.e. the regexp chunker and the ncrf++ toolkit, output the single most likely chunk label encoded as a bio tag. ## evaluation and results to assess the performance of the automatic methods introduced in the previous section, their output is compared to the gold standard annotation. as already mentioned, every token is annotated with a bio tag, i.e. either b-xc (beginning of chunk), i-xc (inside chunk), or o (outside chunk). however, the number of tokens inside and outside of chunks provides little information about the quality of the automatic chunk annotation. instead, it is of interest whether the boundaries of chunks align between automatic and gold annotation. therefore, the evaluation is carried out chunk-wise instead of token-wise and each chunk in the gold 11 glove embeddings trained on german wikipedia and provided by deepset, https://deepset.ai/ german-word-embeddings 12 the experiments of @xcite suggest that the default combination of character cnn, word lstm, and a crf-based inference layer gives the best result for the chunking task with good model stability for random seeds (mean f1: 94.86 ± 0.14). however, the present study is only a first investigation of chunking historical german and further experiments should be conducted to test for model stability and to explore fine-tuning of parameters for optimal results. the evaluation shows that the regexp parser, which operates on pos tags only, reaches f 1scores between 85% and 92% for all data sets, setting a high baseline for the task. the best results are achieved for the modern non-newspaper data and the hipkon corpus. the ncrf++ models outperform this baseline by several percentage points on each data set, achieving f 1 -scores between 90% and 97%. the recall lies between a similar problem with named entity evaluation ( https://nlpers.blogspot.com/2006/08/ doing-named-entity-recognition-dont. html ). the problem with double penalties when using f-scores has also been recognized in the literature. for example, in the context of word tokenization, @xcite show that precision favors under-splitting systems, suggesting that recall, i.e. the proportion of correctly segmented units, gives a more realistic impression of system performance and should be used as the only evaluation metric. however, for tasks that require segmentation and labeling such as chunking or ner, almost correct chunks/entities may still provide useful information for certain purposes. thus, the more fine-grained distinction of errors and adjusted calculation of precision and recall seem appropriate for a thorough evaluation of these annotations. the modern newspaper data is analyzed with the highest f 1 -scores of 97% and 95% respectively. unsurprisingly, models trained on the training section of the same corpus perform better on the test data than models trained on another data set. this may be a result of distributional differences between data sets @xcite but could, in part, also be due to differences between the constituency trees from which the chunks were extracted. the results for the modern non-newspaper data are slightly lower than for the news corpora with a maximum f 1 -score of 94%. interestingly, the overall f 1 -scores are higher for the more informal registers than for the formal ones. probably, informal sentences are generally easier to chunk because they contain more simple (noun) chunks and less pre-nominal modification. while models purely based on words still perform well on the modern data, pos tags prove to be especially relevant for the historical data. even the hist model must be complemented with (modern) pre-trained word embeddings for acceptable performance on the historical corpora, possibly reflecting problems with the non-standardized spelling in historical german. for the mercurius and ref.up corpora, the hist model with pos and word embeddings achieves the best results with f 1 -scores of about 93%, followed by the news2+hist model. for the hipkon corpus, the news2+hist model with pos reaches the highest f 1 -score of 94.5%, closely followed by the news2 model. the dta data is analyzed with the highest f 1 -score of 90.4% by the hist model with pos and word embeddings, followed by the news2+hist and the news1 models with f 1scores of about 90% as well. these results are in line with the observations of ortmann (2020) that models trained on modern news data can successfully be transferred to historical german with overall f 1 -@xmath1. however, the since, in these cases, the embedded noun chunk cannot be recognized based on stts pos tags, a morphological analysis is necessary to distinguish structures with a pre-nominal genitive from prepositional chunks with a post-modifying noun chunk. when the genitive form is not syncretized, i.e. the word form differs from the morphological realization in other cases like nominative or dative, lexicalized models could, in theory, identify the correct structure. but as stranded chunks constitute only about one percent of all chunks in the data sets, there is not enough training data to recognize them reliably. finally, ## conclusion the present study has investigated the automatic recognition of chunks in historical german. to address the main problem of analyzing historical language, namely a lack of manually annotated data for training and evaluation, chunks of six different types were derived from modern and historical constituency treebanks. using the extracted chunks, the state-of-the-art neural sequence labeling tool ncrf++ was trained on modern news articles, early new high german corpora, as well as a combination of modern and historical data. the evaluation has shown that models that include pos tags as features can be transferred successfully from modern to historical language, with f 1 -@xmath2, thereby outperforming a regular expression-based baseline. by adding historical training data, the results can be improved further, yielding f 1 -scores between 90.4% and 94.5% for the different historical corpora. regarding the evaluation of chunks, the present study has argued for a distinction between different types of errors that are commonly penalized as multiple errors in a single unit. an analysis of the occurring error types showed that the majority of errors are boundary errors, meaning that the system identified the chunks, but the boundaries do not exactly match those in the gold standard. since this type of error can be considered less severe than pure false positives or negatives, the presented results give a more realistic impression of the actual system performance. future studies should focus primarily on a reduction of incorrect chunk boundaries to increase the annotation precision, as well as further investigate and improve the analysis of stranded chunks and complex pre-nominal modification in (historical) german.