Parser Evaluation for Analyzing S wedish 19th-20th Century Literature (2023)

Abstract: In this study, we aim to find a parser for accurately identifying different types of subordinate clauses, and related phenomena, in 19th–20th-century Swedish literature. Since no test set is available for parsing from this time period, we propose a lightweight annotation scheme for annotating a single relation of interest per sentence. We train a variety of parsers for Swedish and compare evaluations on standard modern test sets and our targeted test set. We find clear trends in which parser types perform best on the standard test sets, but that performance is considerably more varied on the targeted test set. We believe that our proposed annotation scheme can be useful for complementing standard evaluations, with a low annotation effort.

## introduction dependency parsers can be useful tools for analyzing large text materials, and as such can enable large-scale studies within many scientific disciplines. modern parsers can achieve very high scores on standard test sets, at least for languages with large treebanks, but these test sets are often limited to only a few domains, and typically to publication-level modern language, such as news or wikipedia. for more challenging text types, for instance, noisy data like twitter or historical texts, parsers typically perform considerably worse even for high-resource languages. parsers are typically evaluated on a treebank that is split into training, development, and test sets. this can overestimate the parser performance, since parsers are then trained on data that matches its test set in all relevant aspects, such as genre, time period, and annotation style. furthermore, parser evaluation is typically done using metrics that give a holistic score for the full tree, such as (un)labeled attachment score. in many real-world scenarios, such as ours, we are not interested in the full tree, but in a subset of relations. this study is part of a larger project with the overall aim to identify and explore language change in swedish literature during the period 1800-1930. during the 19th century, the swedish language changed in several aspects. this change includes various linguistic levels and also involves lexical aspects. overall, the changes led to a smaller difference between spoken and written swedish since the written language moved closer to the spoken vernacular (see section 3). the goal of the project is to cover morphological, syntactical, and lexical changes. in this paper, however, we focus only on syntactic aspects, focusing on subordinate clauses. the changes in the 19th century resulted in a less complex languagenot least as far as subordinate clauses and related phenomena are concerned. to enable large-scale analysis of subordinate clauses, we require a highquality parser for our target domain, swedish literary novels and short stories from 1800-1930. in this paper, we explore whether parsers can be evaluated for this domain, without requiring a large manual annotation effort. to evaluate a parser for a new text type and task, as in our case 19th century literature with a focus mainly on subordinate clauses, we would ideally like to have an annotated treebank for the target text type. however, this is a human annotation task that is time-consuming, and thus costly, and which requires an expert on dependency grammar. for many practical projects, this is not feasible. we propose a lightweight annotation task for our target task, which consists of only annotating one type of phenomenon per sentence. the focus is on four phenomena related to subordinate clauses, for which we annotate a small targeted test set for our target text type. for comparison, we also evaluate we compare several variants of three generations of parsers trained on different subsets of the universal dependencies (ud) treebanks @xcite , and evaluate them on ud, both with holistic metrics and for a subset of relations of interest, as well as on our targeted test set. on the ud test sets we see clear trends that a modern transformer-based parser is better than bilstmand svm-based parsers, and that it is better to train on several north germanic languages than only on swedish. however, on our new targeted test set, the results are more mixed, and we see less clear trends, which is in line with earlier work for german @xcite . we think that our targeted test set is able to give a complementary view to standard evaluations, but that the sampling procedure can be improved. in section 2 we review related work, followed by a description of our project focused on swedish language change in section 3. in section 4 we describe the data and in section 5 we describe the parsers evaluated, including the multilingual training setup. we present the results in section 6, discuss them in section 7, and finally, we conclude in section 8. ## related work dependency parsers have continuously developed, from 'old school' parsers like maltparser @xcite and mstparser @xcite based on classical machine learning, like support vector machines, to modern neural parsers. many of the first strong neural parsers were based on recurrent neural networks, as most of the best parsers in the conll 2017 shared task on dependency parsing @xcite . next, models based on deep contextualized embeddings have been taking over, and most strong parsers today are based on fine-tuning contextualized models like bert @xcite or xlm-r @xcite , e.g. machamp @xcite and trankit @xcite . the standard way to evaluate dependency parsers is by calculating holistic metrics such as labeled attachment score (las), which measures the percentage of words which gets both their head word and label correct. there are, however, examples of more detailed evaluations (e.g. @xcite @xcite , focusing on aspects such as arc and sentence lengths, non-projective dependencies, and scores for specific pos-tags and dependency relations. the overall conclusion is typically that different parser types have different strengths, e.g. that graph-based parsers tend to perform better than transition-based parsers on long-distance @xcite . as far as we are aware, there are no datasets and evaluations like our proposal, focused on a single relation per sentence. highly relevant to our study is the work of @xcite , who evaluate a set of six parsers for digital humanities research, focusing on german novels and academic texts. like us, they are also interested in specific relations, for instance, related to speaker attribution, and not only in holistic evaluation. unlike us, they perform a full dependency tree annotation effort for three sample texts. in addition, they do not include any neural parsers in their evaluation. they find that several parsers do well on the holistic metrics, but that the results are considerably worse for several of the specific relations of interest, such as appositions, and that it is not always the overall strongest parser that is the best choice for a specific relation. @xcite performed a detailed evaluation on parsing german 17th-century literature, for which he annotated two excerpts of text with full dependency annotations. again, no neural parsers were included in the study, which found a drop compared to in-domain results, but where the relative performance of the two parsers evaluated was consistent on different metrics, possibly because of the large difference in performance between them. swedish literary texts from different eras have been analyzed for different purposes before, requiring taggers and/or parsers. @xcite aims to characterize differences between dialogue and narrative in contemporary fiction, whereas @xcite analyze prose rhythm in a novel from 1940. however, in none of these studies, the choice of tagger and/or parser is motivated. there have also been some earlier smallerscale studies focusing on the transition towards a more colloquial written swedish. for instance, language development in swedish literature during the 19th century has been explored, but only on a small scale focusing on individual authors (e.g. @xcite . ## language change in 19th century swedish this study is part of a larger project with the overall aim to identify and explore language change in swedish literature during the period 1800-1930. in the history of the swedish language, this period is characterized by modernization in the sense that the written language was influenced by the spoken vernacular. in this process of modernization, fictional prose is of certain interest since it has been suggested that linguistic change spread from literary dialogue @xcite . by investigating a corpus of literary texts the project will not only contribute with a more detailed account of language change in 19th-century swedish but also address the question of how linguistic change increased in the community. the modernization of the swedish written language during the 19th century affected several linguistic aspects. as for the lexicon, it is wellknown that formal function words were replaced by colloquial counterparts. much attention has also been devoted to the loss of verbal agreement, i.e. the use of the vernacular singular variant in both singular and plural. on the syntactic level, @xcite has shown a remarkable change in sentence length during the end of the 19th century. engdahl's study focuses on non-fictional prose, periodicals from 1878 to 1950, but his results call for a more detailed account of syntactic complexity during the period, and hence we will focus on subordinate clauses and phenomena related to them in this paper. for this study, we have chosen to focus on three types of subordinate clauses, based on ud dependency labels, and one phenomenon related to subordinate clauses: (i) relative clauses (relcl), (ii) cleft constructions (cleft), foot_0 (iii) clausal complements not determined by obligatory control (ccomp), and (iv) auxiliary drop (no-aux). whereas the first three types can be used in order to measure syntactic complexity, auxiliary drop has been suggested to mark written style, and hence almost never occur in spoken language (cf. @xcite . since auxiliary drop of finite verbs is restricted to subordinate clauses in swedish, we have included it as related to subordinate clauses. in this study, we only include auxiliary drop that occurs in clausal complements, ccomp. examples of the selected clause types are shown in ## data in this section, we will describe the existing data from ud and the new targeted dataset we constructed for this project ## parsers in order to investigate how well the different types of evaluation work, we explore three generations of parsers. as a baseline, we use the easily accessible swepipe with its provided model for swedish. we also use two generations of neural parsers, uuparser and machamp, for which we also experiment with multilingual parsing. we train each model three times with different random seeds and report average scores. ## results tables 5 and 6 show results from the standard and targeted evaluations for swepipe, uuparser m with machamp k pos-tags and machamp k trained with kb-bert. in all tables, we mark the three best results for each metric in bold. while our focus is on swedish, which is reported in this section, we also report results with machamp for the additional languages used for training our parsing models in appendix a. in ## discussion an important question is whether the parser performance on our target task is good enough to use for our study of change in the swedish written language. overall, both machamp and uuparser have good precision for all our relations of interest, always scoring above 90, and reaching scores above 96 for some parsers for each relation type. the recall, however, is considerably lower. this means that the instances of each relation type the parser finds are mostly good, but it does miss a substantial part of relevant instances, especially given the fact that all examples are sampled from a parser, and we might have missed additional instances. the recall is highest for relcl, where it is well above 80 for several of the models both for machamp and uuparser. this approaches a level that is usable for our end project, of finding syntactic features in 18th-19th-century literature, and tracking them over time. other relation types have a more mixed performance, as cleft, for which uuparser m trained on norswe and scand performs very well, with a recall of over language groups. the other groups exhibit largely the same trends. on the standard evaluation, machamp is clearly overall better than uuparser, training on scand is better than training only on swedish, kb-bert is better than mbert for machamp, and uuparser is better with machamp tags than with swepipe tags. for our targeted test sets, however, we see fewer clear trends, and there is much more variation among the systems. machamp k and uuparser m tend to perform better than their counterparts, and the multilingual models may have a small advantage over the swedish-only models. swepipe clearly seems to fall behind the other parsers on precision, whereas its high recall can be explained by the sampling procedure. a side-effect of our study is that we have found that machamp k trained on scand or northg is a very strong parser for modern swedish as measured by the ud test sets. our targeted test set does suffer from an issue with sampling from only two parsers, which affects its recall mainly for swepipe, but also for uuparser s . we believe uuparser m is less affected since it relies on a different set of pos-tags. the dataset is also relatively small, especially for the cleft relation. however, we think it still contributes to showing that when selecting a parser for a particular target task and text type, we cannot rely solely on evaluation scores on standard test sets, as also shown by @xcite . even if we focus on the f1-score for the relations of interest in lines-m, rather than on the full tree, we see no clear similarity of parser ranking to the evaluation of the same relation types in our targeted test set. to further investigate whether this type of test set can indeed be useful, we would need to perform further analysis. it would be interesting to learn more about where the main improvements shown on ud evaluation for a parser like machamp k actually occurs. we also think it would be useful to consider the sampling for the test set, specifically to also annotate some raw text, in order to find out what type of instances are not identified by any of our parsers. another issue that we did not yet explore, is whether parsing performance varies over the time period in question. ## conclusion we describe a study of swedish dependency parsers with the goal of tracking changes in the use of certain types of subordinate clauses and related phenomena in swedish literature from 1800-1930. since standard test sets do not cover this time period or genre, and we did not have the resources to perform a full annotation of dependency trees, we propose a smaller-scale annotation task, focusing on single relation types. we evaluated a set of parsers on ud and on our targeted test set. while there was a clear and relatively consistent order between the parsers on the ud evaluation, the performance was more mixed on our targeted test set, without a clear overall best parser across relation types. we believe that our proposed annotation scheme can be useful in complementing standard evaluations, with a low annotation effort, but that more analysis is needed.