Automatic Topological Field Identification in (Historical) G erman Texts (2020)

Abstract: For the study of certain linguistic phenomena and their development over time, large amounts of textual data must be enriched with relevant annotations. Since the manual creation of such annotations requires a lot of effort, automating the process with NLP methods would be convenient. But the required amounts of training data are usually not available for non-standard or historical language. The present study investigates whether models trained on modern newspaper text can be used to automatically identify topological fields, i.e. syntactic structures, in different modern and historical German texts. The evaluation shows that, in general, it is possible to transfer a parser model to other registers or time periods with overall F1-scores >92%. However, an error analysis makes clear that additional rules and domain-specific training data would be beneficial if sentence structures differ significantly from the training data, e.g. in the case of Early New High German.

## introduction to study the development of language over time, sufficient amounts of textual data from different time periods need to be enriched with linguistic annotations. for example, to investigate the historical development of certain syntactic phenomena like extraposition or object order in the middle field of the german sentence, annotated corpora from all relevant language stages, e.g. middle high german, early new high german, and modern german, would be needed. however, since the creation of annotations requires a lot of manual effort, historical corpora are rarely annotated with linguistic information beyond the morpho-syntactic level like sentence or clause structure. this limits investigations of syntactic change to qualitative studies on small data sets, often with limited statistical significance. complementing the manual approaches with quantitative studies on large amounts of annotated texts could validate their results as well as unveil new patterns in the data. to reduce the annotation effort required for the application of quantitative methods, there is a growing interest in the use of nlp methods to automate the annotation task. but the necessary amounts of training data usually do not exist for non-standard or historical language. the present study investigates whether modern newspaper training data can be used to automatically identify topological fields, i.e. syntactic structures, in various modern and historical german texts. the remainder of this paper is structured as follows: section 2 covers the theoretical background of the study and gives a short introduction to the topological field model before section 3 summarizes previous approaches to the automatic identification of topological fields. section 4 describes the data sets used in this study and section 5 explains the selected approach for the automatic topological field identification. in section 6 the evaluation results are presented, including a detailed error analysis, followed by a conclusion in section 7. ## topological field model the topological field model @xcite is a widely used theory-neutral framework for the description of syntactic structures in german sentences. while german is considered to have a relatively free word order, the topological fields provide a clear structure on the clause level. in german, there are three different clause types, which are characterized by the position of the finite verb. ## related work there has been a number of different approaches to the automatic identification of topological fields in german. the first studies @xcite @xcite used rule-based approaches, implemented with finite-state cascades, to identify the sentence brackets and, based on this, the other topological fields. for this rule-based approach, @xcite report an overall f 1 -score of about 87%. @xcite show that for sentence brackets, i.e. fields that contain a very restricted 'this means that one can gather more and more information around a hypothesis or a point of view that strengthens it, but does not definitely prove it.' figure 2 : example sentence from the present study with nested topological fields. set of elements, such rule-based systems can yield competitive results. for the identification of more complex topological fields and embedded clauses, though, using (probabilistic) parsers seems more promising: @xcite train a non-lexicalized chart parser on a probabilistic context-free grammar and achieve labeled recall and precision values of about 93%. @xcite describes a bi-directional bottom-up parsing approach for non-recursive topological field recognition, resulting in an overall f 1 -score of about 95%. de kok and hinrichs (2016) treat topological field annotation as a sequence labeling task. they use a bi-directional lstm and achieve an overall accuracy of 97% for non-recursive topological field identification. for recursive topological field annotation, cheung and penn (2009) apply the berkeley parser @xcite and report an f 1 -score of 95% on the tüba-d/z corpus and 91% on the negra corpus. they observe the best results for sentence brackets with f 1 -@xmath0. f 1 -scores of about 95% or more are also achieved for coordinations and the pre-and middle field. the post-field is recognized less reliably with about 83% and left dislocation with only 7%. all of these approaches focus on standard german (newspaper) text. to date, there has only been one attempt to automatically identify topological fields in historical data. using conll-rdf and sparql, @xcite implement a deterministic rule-based parser for topological field identification in middle high german. it relies on grammars and expert knowledge and makes use of the manual annotations provided in the reference corpus of middle high german (rem). however, in the absence of a manual gold standard annotation, the accuracy of the parser is not evaluated and thus remains unclear. ## data although the topological field model is widely used for the description of german syntactic structures, only few corpora actually provide topological field annotations. the tüba-d/z corpus @xcite foot_0 is the largest available data set, consisting of 3,816 german newspaper articles that are manually annotated with pos tags and topological fields. discounting headlines and other fragments, which do not receive a topological field annotation, it contains 92,505 sentences with 606,755 fields. for this study, the corpus is split into a training (80%), development (10%), and test set (10%). most of the studies described in section 3 use previous versions of this corpus for training and/or evaluation. to investigate how well the automatic identification of topological fields can be transferred to other domains, the present study includes two additional data sets for modern german. the tüba-d/s corpus @xcite consists of 14 spontaneous speech dialogues from a business context, which were manually transcribed and annotated with pos tags and topological fields. discounting fragments, the data set comprises 19,523 sentences with 107,432 fields. the data set of collection of five different written registers: wikipedia articles, fiction texts, christian sermons, ted talk subtitles, and movie subtitles. the data is provided with manually annotated pos tags and was enriched with topological fields for this study. without fragments, it consists of 462 sentences with 3,054 fields. besides the modern data, the present study also includes two historical german corpora to assess whether topological fields can be identified automatically in texts from different time periods, without any historical training data available. the hipkon corpus @xcite ) contains sermons from the 12th to the 18th century and offers manual annotations for 342 sentences from the entire time span (except 15th century). because hipkon was created for the investigation of post-fields, only sentences with a postfield are annotated. for the present study, these sentences were manually enriched with topological fields, yielding a total amount of 1,857 fields. as hipkon is the only corpus annotated with a custom pos tagset specifically for historical data, for this study it was mapped to the german standard tagset stts @xcite . the second historical corpus, the german text archive dta (bbaw, 2019), is provided with automatically generated linguistic annotations, including sentence boundaries and pos tags. for this study, 414 sentences from 29 texts published between 1562 and 1912 were selected and annotated with a total number of 4,828 topological fields. the dta sample includes texts from a variety of genres: five newspaper texts and three texts each from the genres funeral sermon, language science, medicine, gardening, theology, chemistry, law, and prose. for every genre, the texts were randomly selected from three (five) different centuries. since the pos tagging and sentence segmentation accuracies in the sample were considered too low to use them as an evaluation basis, pos tags and sentence boundaries were manually corrected during topological field annotation. 3 ## topological field identification the best results for recursive topological field identification are, so far, reported by cheung and penn (2009), who apply the unlexicalized latent variable-based berkeley parser @xcite foot_3 to the identification of topological fields in german newspaper text. in the present study, their approach is transferred to different data sets, including modern non-standard and historical german texts. to train the berkeley parser, the tüba-d/z training data is converted to a treebank format. only sentences with a gold standard annotation are used for training. to ensure the applicability to different data sets, the model cannot be based on word forms, which differ significantly between modern and historical writings. instead, the pos tags, which are consistent across data sets, are taken to form the basic text. to meet the required input format of the parser without supplying word forms, the topological field annotations must be modified. a top-level sentence node is added and artificial pre-terminal nodes are inserted where necessary so that each preterminal corresponds to exactly one terminal symbol as it would be the case for words and pos tags in a standard syntax tree. the parser is trained with default options foot_4 using the larger topological field tagset of the tüba-d/z corpus @xcite during training, which is then mapped to the simple scheme as described in section 2 for evaluation. @xcite note that this strategy of training on more fine-grained categories and evaluating on a coarser tagset can improve the accuracy of topological parsing. to run the java-based berkeley parser, it is invoked in interactive mode via the command line and always returns the single best parse. ## evaluation and results for the evaluation of the automatic topological field identification, the parser output is compared to the gold standard annotation and labeled precision and recall are calculated. here, true positives are fields that cover the correct span of tokens and are labeled with the correct field tag. the evaluation only considers the token span covered by a field, independently of possibly intermediate embedded fields. punctuation at the edge of fields is removed before evaluation. if fields in the parser output have incorrect boundaries or do not exist at all in the gold standard, they are counted as false positives. if a field is present in the gold standard, but there is no corresponding field in the parser output, this is counted as a false negative. only sentences for which there is a gold annotation are evaluated. newspaper except for post-fields and left dislocations, all fields in the tüba-d/z test data are recognized with f 1 -scores between 90% and 99%. the sentence brackets are identified with the highest accuracy, followed by the pre-and middle field and coordinations. for all fields (except ko and lv), at least 40% of the false positives have incorrect boundaries but overlap with the gold annotation. this value is highest for the middle field, where 82% of the false positives only have incorrect boundaries. this can, for example, be the case if the right sentence bracket is empty and the parser regards the middle and post-field as a single field, resulting in a false positive middle field and a false negative post-field. in total, four out of five sentences from this data set are analyzed without any error. on average, those sentences are ten words shorter than sentences containing errors. errors mostly occur with elliptical constructions, fragments, and parenthetical phrases as well as sentence structures that are uncommon in standard written german and therefore rare in the training data. this observation is in alignment with @xcite , who also identify parentheticals as the main error cause in their study. further error sources are quotes and reported (direct) speech, as well as left dislocations and post-fields without internal structure. overall, the newspaper data is annotated with high accuracy, reproducing the results of prior studies. spoken while the sentence brackets and pre-and middle fields are recognized with f 1 -@xmath1, only two thirds of the coordinations and post-fields are identified correctly in the spoken data. left dislocations are recognized with an f 1 -score of only 18.5%. again, many false positives overlap with the gold standard annotation, especially in the case of pre-(59%) and middle fields (80%), which often erroneously stretch across left dislocations or post-fields, in turn leading to low recall values for the latter fields. almost two thirds of all sentences in the spoken data set are analyzed without errors. on average, these sentences contain nine words less than incorrect sentences. errors mostly result from the divergence between spoken and written language structures, for instance incomplete utterances, repeated words, or unrelated clauses and fragments in a single sentence. still, it can be stated that, despite the differences between written training and spoken test data, the majority of the fields is recognized with fairly high accuracy and, if similar data should be processed automatically, using part of the spoken data as additional training resource could further improve the results for this text type. ## conclusion the present study has investigated the automatic identification of topological fields in different modern and historical german texts using only modern newspaper text as training data. the evaluation has shown that, in general, transferring a model from modern newspaper data to other registers or time periods is possible. using the berkeley parser, different non-standard and spoken modern data sets as well as sermons from the 12th to the 18th century can be analyzed automatically with overall f 1 -@xmath2. for the most common fields like sentence brackets or the middle field, the accuracy can be considered sufficient for qualitative and quantitative research based on the automatic field identification. however, additional rules and especially additional training data for specific data sets could be very beneficial if sentence structures or the distribution of fields differ substantially from modern newspaper language. the evaluation has shown that texts from the early new high german period, in particular, often exhibit such complex structures that they are hard to process even for human annotators. as a result, the parser only reaches an overall f 1 -score of 84% on the dta data set. future work has to unveil whether time-or genre-specific training data can improve these results and enable a reliable identification of all topological fields in various text types from all time periods. since creating such training resources is effortful and time-consuming, the presented automatic analyses could be used for pre-annotation, subsequently improving their accuracy by adding further training material.