Cross-Lingual Constituency Parsing for M iddle H igh G erman: A Delexicalized Approach (2023)

Abstract: Constituency parsing plays a fundamental role in advancing natural language processing (NLP) tasks. However, training an automatic syntactic analysis system for ancient languages solely relying on annotated parse data is a formidable task due to the inherent challenges in building treebanks for such languages. It demands extensive linguistic expertise, leading to a scarcity of available resources. To overcome this hurdle, cross-lingual transfer techniques which require minimal or even no annotated data for low-resource target languages offer a promising solution. In this study, we focus on building a constituency parser for Middle High German (MHG) under realistic conditions, where no annotated MHG treebank is available for training. In our approach, we leverage the linguistic continuity and structural similarity between MHG and Modern German (MG), along with the abundance of MG treebank resources. Specifically, by employing the delexicalization method, we train a constituency parser on MG parse datasets and perform cross-lingual transfer to MHG parsing. Our delexicalized constituency parser demonstrates remarkable performance on the MHG test set, achieving an F1-score of 67.3%. It outperforms the best zero-shot cross-lingual baseline by a margin of 28.6% points. The encouraging results underscore the practicality and potential for automatic syntactic analysis in other ancient languages that face similar challenges as MHG.

## introduction constituency parsing, which involves analyzing the grammatical structure of sentences and identifying the hierarchical relationships between words, plays a crucial role in linguistic research, especially for the analysis of ancient languages that are no longer spoken. its significance extends beyond linguistic analysis, serving as a building block for various natural language processing (nlp) applications, such as information extraction @xcite , sentiment analysis @xcite , question answering @xcite , etc. however, ancient languages lack large labeled and unlabeled corpora @xcite and treebanks suitable for parser training are seldom available. this scarcity of resources can be attributed to two reasons. firstly, ancient languages usually have a dearth of digital text resources. secondly, the construction of a treebank for an ancient language requires substantial linguistic expertise and manual effort. nonetheless, the continuity in the process of language evolution gives rise to linguistic similarities between ancient languages and their corresponding modern counterparts @xcite . cross-lingual transfer techniques @xcite ) are trained on high-resource languages and require little or no annotated data from low-resource target languages. they can effectively be applied to languages with similar sentence structure and word order. hence, they can be a viable solution to this challenge. in this work, we focus on building a constituency parser for middle high german (mhg). mhg is a historical stage of the german language that was spoken between 1050 and 1350. it is the linguistic predecessor of modern german (mg). both languages have many similarities in word formation and grammatical features, e.g., similar word order patterns and inflectional systems @xcite . the availability of mhg parse trees is extremely limited. the deutsche diachrone baumbank (german diachronical treebank, ddb) @xcite comprises merely around 100 manually annotated parse trees, encompassing less than 3000 tokens. these resources are far from what is required to train an automatic syntactic analysis system, and are only suitable for use as test sets. on the other hand, there is an abundance of treebank resources available for mg, in particular the tiger treebank @xcite . hence, we capitalize on the structural similarity between mhg and mg, as well as the rich mg treebank resources in order to develop a cross-lingual delexicalized constituency parsing model that we can directly apply to mhg sentences. in the delexicalized approach, the parsing model operates on part-of-speech (pos) sequences rather than token sequences. we accomplish this by training a cross-lingual parser using pos sequences from high-resource source languages as input. subsequently, we utilize this trained parser to directly parse pos sequences of low-resource target languages @xcite . in our work, we first train a delexicalized constituency parsing model on a delexicalized mg treebank. in order to parse mhg sentences with this model, we need to annotate them first with the pos tags used in the mg treebank. to this end, we train a pos tagger on an mhg corpus which has been manually annotated using a pos tag set similar, but not identical to the mg tag set. we em-ploy a pos mapper to replace the mhg tags by the corresponding mg tags, ensuring the uniformity of the model's inputs across the two languages, which is a prerequisite of the delexicalization method. the experimental results show that our delexicalized constituency parser substantially outperforms all other zero-shot cross-lingual parsing baselines, achieving an f1-score of 67.3% on the mhg parse test set. the delexicalization method is particularly wellsuited for languages which (1) lack treebank resources, (2) possess sufficient annotated data for training pos taggers, and (3) exhibit syntactic similarities with a high-resource language. our investigation of this realistic scenario shows the feasibility of automatic syntactic analysis for an ancient language. the subsequent sections of this paper are organized as follows. in sec. 2, we discuss related work. sec. 3 gives an overview of our research languages and the available corpora. the delexicalization method employed in our approach is detailed in sec. 4. sec. 5 describes our experimental setup, and in sec. 6, we analyze the results. we conclude in sec. 7. ## related work cross-lingual transfer learning the fundamental principle underlying cross-lingual transfer is that the processing of source and target languages uses a shared input representation, which can be either discrete or continuous. the delexicalization method is based on a shared discrete input representation, i.e., pos tags. other discrete representation types include glossed words @xcite and grounding texts in multilingual knowledge bases @xcite . continuous cross-lingual representation spaces emerged with advancements in neural networks. typical examples are cross-lingual word embeddings @xcite and sentence embeddings @xcite . the emergence of massively multilingual transformers @xcite , which are jointly pretrained on multilingual corpora, introduces a novel pattern of zero-shot crosslingual transfer learning. in this paradigm, a pretrained multilingual model is finetuned on a downstream nlp task dataset of a source language. the finetuned multilingual model is then directly applied to target language data for the same task @xcite @xcite . neural constituency parsing recent advances in constituency parsing have witnessed a growing emphasis on harnessing neural network representations, making a shift from the previously prominent role of grammars, whose relevance has gradually diminished. @xcite propose a spanbased constituency parsing system specifically designed to leverage the powerful representation capabilities of the bidirectional long short-term memory (lstm) networks @xcite . in this method, an input sentence is represented as a set of spans, and each span is assigned a score. the best-scoring parse tree is computed using dynamic programming techniques. they combine smaller spans into larger spans until the entire sentence is covered. subsequently, several variations of the span-based method have been proposed, e.g. approaches replacing the inference algorithm with chart-based methods @xcite , using character-level representations instead of wordlevel representations @xcite , and replacing lstms with self-attention modules @xcite . @xcite take advan-tage of the newly developed pretrained language models (plms) and use bert @xcite to compute the span representations, resulting in enhanced performance. @xcite improve the runtime complexity of constituency parsing to linear time by reducing parsing to tagging. ## languages and corpora the ancient language which we study in this paper is middle high german (mhg). mhg and modern german (mg) are stages of the same germanic language family, representing different historical periods. mhg emerged during the middle ages in the german-speaking regions of central europe. it was primarily used in literary and administrative contexts and played an important role in medieval literature, including epic poems such as the nibelungenlied and minnesang (courtly love poetry) @xcite . linguistic considerations of mhg mhg has a phonetic system that included a set of vowel and consonant sounds. the pronunciation and sound patterns differ from those of mg, but some mhg words are still recognizable in mg. mhg has a more complex grammatical system, such as a more extensive case system with different noun and adjective declensions. besides, verb conjugation has more intricate forms and patterns @xcite . in terms of orthography, the spelling and writing conventions of mhg are different from mg. for example, ü, the umlaut of u, is usually written iu in mhg. the transition from mhg to mg was a gradual process, occurring over several centuries. mg can be considered the linguistic descendant of mhg, with linguistic changes and developments shaping the language over time. ## methods in our work, we focus on developing a constituency parser for mhg. in the previous section, we reviewed annotated resources available for mhg and mg. basically, we have ample treebank resources for mg and plenty of pos-tagged texts for mhg, whereas the treebank resources for mhg are extremely limited. given the resource availability for mg and mhg along with the linguistic connection between the two languages, employing a cross-lingual constituency parsing approach utilizing delexicalization proves to be an effective solution. as figure 1 shows, the delexicalized model is trained on the delexicalized inputs of mg. in the inference stage, the delexicalized parser is directly applied to mhg pos sequences. the delexicalization method requires that mhg and mg share the same set of pos tags. the final constituency parser for mhg (the right side of figure 1 comprises three modules: (1) the delexicalized parser, (2) the mhg pos tagger, and (3) the pos mapper from mhg to mg. in the next section, we describe the delexicalized parsing system in more detail. ## experiments we begin by training benepar on the delexicalized tiger treebank for mg. then we annotate the sentences of the small ddb treebank for mhg with rnntagger and map the hits tags that it returns to stts tags. finally, we parse the pos tag sequences with the trained parser. ## conclusion in conclusion, our study presents an effective crosslingual constituency parsing approach for ancient languages, specifically focusing on the parsing of middle high german (mhg) sentences. through the utilization of delexicalization and and the similarities between mhg and modern german (mg), we have developed a delexicalized parser based on the rich treebank resources of mg, which demonstrates remarkable performance in parsing mhg sentences. our experimental results showcase the efficacy of the delexicalized approach, outperforming existing baselines and achieving substantial improvements in parsing accuracy. these findings highlight the practicality and promise of our approach for parsing historical and ancient languages, addressing the challenges posed by limited annotated data and linguistic variations.