Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021)

Abstract: Text generation systems are ubiquitous in natural language processing applications. However, evaluation of these systems remains a challenge, especially in multilingual settings. In this paper, we propose L’AMBRE – a metric to evaluate the morphosyntactic well-formedness of text using its dependency parse and morphosyntactic rules of the language. We present a way to automatically extract various rules governing morphosyntax directly from dependency treebanks. To tackle the noisy outputs from text generation systems, we propose a simple methodology to train robust parsers. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.

## introduction a variety of natural language processing (nlp) applications such as machine translation (mt), summarization, and dialogue require natural language generation (nlg). each of these applications has a different objective and therefore task-specific evaluation metrics are commonly used. for instance, reference-based measures such as bleu @xcite , meteor @xcite and chrf @xcite are used to evaluate mt, rouge @xcite is a metric widely used in summarization, and various task-based metrics are used in dialogue @xcite . regardless of the downstream application, an important aspect of evaluating language generation systems is measuring the fluency of the generated text. in this paper, we propose a metric that can be used to evaluate the grammatical well-formedness of text produced by nlg systems. 2 our metric is referenceless and is based on the grammatical rules of the language, thereby enabling fine-grained identification and analysis of which grammatical phenomena the nlg system is struggling with. although several referenceless metrics for evaluating nlg models exist, most use features of both the input and output, limiting their applicability to specific tasks like mt or spoken dialogue @xcite . with the exception of the grammaticality-based metric of @xcite , these metrics are derived from simple linguistic features like misspellings, language model scores or parser scores, and are not indicative of specific grammatical knowledge. in contrast, there has recently been a burgeoning of evaluation techniques based on grammatical acceptability judgments for both language models @xcite @xcite and mt systems @xcite @xcite . however, these methods require an existing model to score two sentences that are carefully crafted to be similar, with one sentence being grammatical and the other not. these techniques are usually tailored towards specific downstream systems. additionally, they do not consider the interaction between multiple mistakes that may occur in the process of generating text (e.g., an incorrect word early in the sentence may trigger a grammatical error later in the sentence). most of these methods, with the exception of @xcite , focus only on english or translation to/from english. in this paper, we propose l'ambre, a metric that both evaluates the grammatical well-formedness of text in a fine-grained fashion and can be applied to text from multiple languages. we use widely available dependency parsers to tag and parse target text, and then compute our metric by identifying language-specific morphosyntactic errors in text (a schematic overview is outlined in figure 1 ). our measure can be used directly on text generated from a black-box nlg system, and allows for decomposing the system performance into individual grammar rules that identify specific areas to improve the model's grammaticality. l'ambre relies on a grammatical description of the language, similar to those linguists and language educators have been producing for decades when they document a language or create teaching materials. specifically, we consider rules describing morphosyntax, including agreement, case assignment, and verb form selection. following @xcite , we describe a procedure to automatically extract these rules from existing dependency treebanks ( §3) with high precision. 3 when evaluating nlg outputs, adherence to these rules can be assessed through dependency parses (figure 1 ). however, off-the-shelf dependency parsers are trained on grammatically sound text and are not well-suited for parsing ungrammatical (or noisy) text (hashemi and hwa, 2016) such as that generated by nlg systems. we propose a method to train more robust dependency parsers and morphological feature taggers by synthesizing morphosyntactic errors in existing treebanks ( §4). our robust parsers improve by up to 2% over offthe-shelf models on synthetically noised treebanks. finally, we field test l'ambre on two nlp tasks: grammatical error identification ( §5) and machine translation ( §6). our metric is highly correlated with human judgments on mt outputs. we also showcase how the interpretability of our approach can be used to gain additional insights through a diachronic study of mt systems from the conference on machine translation (wmt) shared tasks. the success of our measure depends heavily on the quality of dependency parses: we discuss potential limitations of our approach based on the grammar error identification task. ## l'ambre: linguistically aware morphosyntax-based rule evaluation in this section, we present l'ambre, a metric to gauge the morphosyntactic well-formedness of generated natural language sentences. our metric assumes a machine-readable grammatical description, which we define as a series of languagespecific rules g @xmath0, r 2 , . . . , r n }. we also assume that dependency parses of every grammatical sentence adhere to these rules. 4 given a text, we compute a score by verifying the satisfiability of all applicable morphosyntactic rules from the grammatical description. similar to standard metrics for evaluating nlg, our scoring framework allows for computing scores at both segment-level and corpus-level granularities. segment level: computing l'ambre first requires segmentation, tokenization, tagging, and parsing of the corpus. 5 given the tagged dependency tree for a segment of text and a set of rules in the language, we identify all rules that are applicable to the segment. we then compute the percentage of times that each such rule is satisfied within the segment, based on the parser/tagger annotations. the final score is a weighted average of the scores of individual rules. 6 our score lies between [0,1], where 1 and 0 represent that rules are perfectly satisfied or not satisfied at all respectively. consider the example sentence (s.2) from figure 1 . of the five agreement rules, two rules, number agreement between pron (ich) and aux (werde), and case agreement between adj (lange) and noun (bücher) are not satisfied. both rele-vant case assignment rules between, pron (ich) and aux (werde), and noun (bücher) and verb (lesen) are satisfied. thus, the overall score is 0.71 (5/7). this example showcases how l'ambre is inherently interpretable: given a segment (s.2), we can immediately identify that it is grammatically sound with respect to case assignment, but contains two errors in agreement. corpus level: to compute l'ambre at corpuslevel, we accumulate the satisfiability counts for each rule over the entire corpus and report the macro-average of the empirical satisfiability of each applicable rule. this is different from a simple average of segment-level scores and is a more reliable score as it allows the comparison of performance by rule over the entire corpus. ## creating a grammatical description in linguistics, grammars of languages are typically presented in (series of) books, describing in detail the rules governing the language through free-form text and examples (see @xcite ; @xcite for grammatical agreement). 7 however, to be able to use such descriptions in our metric, we require them to be concise and machine-readable. we build upon chaudhary et al. (2020) that constructed first-pass descriptions of grammatical agreement from syntactic structures of text, in particular, dependency parses. 8 in general, rules based on a complete formalized grammar govern several aspects of language generation, including syntax, morphosyntax, morphology, morphophonology, and phonotactics. in this work, we focus on agreement, case assignment, and verb form choice. ## parsing noisy text within our evaluation framework, we rely on parsers to generate the dependency trees of potentially malformed or noisy sentences from nlg systems. however, publicly available parsers are typically trained on clean and grammatical text from ud treebanks, and may not generalize to noisy inputs @xcite @xcite . therefore, it is necessary to ensure that parsers are robust to any morphology-related errors in the input text. ideally, the tagger should accurately identify the morphological features of incorrect word forms, while the dependency parser remains robust to such noise. to this end, we present a simple framework for evaluating the robustness of pre-trained parsers to such noise, along with a method to train the robust parsers necessary for our application. ## does l'ambre capture grammaticality? before deploying l'ambre on automatically generated text, we need to ensure that our approach is indeed able to identify syntactic ill-formedness. grammar error correction (gec) datasets are an ideal test bed. in its original formulation, the gec treebank overall treebank altered forms overall treebank altered forms. (las) (las) feature (acc.) feature (acc.) czech-pdt 89.7 -2 -0.8 81.7 -15 -4.3 95.1 -3.1 -1.3 96.5 -39.7 -24.1 greek-gdt 88.6 -0.6 -0.07 85.5 -6.3 -1.7 94.9 -1.3 -0.5 97.4 -35.6 -16.7 turkish-imst 62.5 -2.7 -1.7 63.0 -31.6 -18.4 91.6 -2 -0.6 90.3 -77.1 -39.9 ∠ ∠ ∠ original robust figure 4: our robust parsers reduce the errors on the noisy evaluation set (example over three treebanks) compared to the original pre-trained ones. the baseline axis in each plot corresponds to the performance on the clean evaluation set. our models are more robust on both parsing (las) and morphological feature prediction. we report results both over the whole treebank and over only the erroneous tokens. evaluation: to evaluate the effectiveness of l'ambre, we run it on the training foot_10 splits of the german and russian gec datasets. gec corpora typically annotate single words or phrases as errors (and provide a correction); in contrast, we only identify errors over a dependency link, which can then be mapped over to either the dependent or head token. this difference is not trivial: a subjectverb agreement error, for instance, could be fixed by modifying either the subject or the verb to agree with the other constituent. to account for this discrepancy, we devise a schema to ensure the proper computation of precision and recall scores. first, we detect any errors at a given token by evaluating all the valid l'ambre rules between the curren token and its dependency neighbors (head, dependents). if there is a gold error at the current token, we consider it a true positive or false negative depending on whether or not we detect the error. for false positive cases, we divide the score between the current token and the neighbor via the erroneous dependency link (see algorithm 1 in a.3). analysis: in both languages, we find agreement rules to be of higher quality than case and verb form assignment ones. this phenomenon is more pronounced in german where many case assignment rules are lexeme-dependent, as discussed in §3. importantly, our proposed robust parsers lead to clear gains in error identification recall, compared to the pre-trained ones ("original" vs. "robust" in to understand where l'ambre fails, we man-ually inspected a sample of false positives. first, we notice that tokens with typos are often erroneously tagged and parsed. our augmentation is only equipped to handle (correctly spelled) morphological variants. additionally applying a spell checker might be beneficial in future work. second, we find that german interrogative sentences and sentences with more rare word order (e.g., object-verb-subject) are often incorrectly parsed, leading to misidentifications by l'ambre. in the training portion of the german hdt treebank, 76% of the instances present the subject before the verb and the object appears after the verb in 62% of the sentences. questions and subordinate clauses that follow the reverse pattern (ovs order) make a significant portion of the false positives. last, we find that morphological taggers exhibit very poor handling of syncretism (i.e., forms that have several possible analyses), often producing the most common analysis regardless of context. for example, nominative-accusative syncretism is well documented in modern german feminine nouns @xcite . german auxiliary verbs like werden ('will') that share the same form for 1st and 3rd person plurals, are almost always tagged with the 3rd person. as a result, our method mistakenly identifies correct pronoun-auxiliary verb subject dependency constructions as violations of the rule r agree (pron, aux, subj) → person, as the pron and aux are tagged with disagreeing person features (1st and 3rd respectively). by manually correcting for this issue over our german rules (by specifically discounting such cases) we improve l'ambre's precision by almost 7 percentage points ("robust++" in comparison with other metrics: we also compare l'ambre to other metrics that capture fluency and/or grammatical well-formedness, namely perplexity as computed by large language models and the grammaticality-based metric (gbm) of @xcite . to provide a fair comparison of l'ambre, perplexity and gbm, we reformulate the gei task into an acceptability judgment task. specifically, we check if the metrics' score the grammatical target sentence higher than the ungrammatical source sentence in the gei test split for russian and german. to compute the perplexity scores, we use transformer-based lms @xcite . gbm relies on the open-source language tool @xcite , which is a widelyused rule-based proofreading software to detect sentence-level errors. 16 gbm measures error count rate as 1 -#errors #tokens . for additional details on the task setup, we refer the readers to a.4 in appendix. our findings are two-fold. first, perplexity performs better than the other metrics. however, perplexity cannot provide any error diagnosis, so it by itself is not useful for providing feedback to a user. second, l'ambre is better at capturing morpho-syntactic rules necessary for grammatical correctness (especially in russian), while gbm is better at other fluency-related aspects. while both l'ambre and gbm are interpretable, l'ambre's ud-based rule construction makes it easier to extend it to new languages. 17 for complete results, refer to in our gei analysis, we utilized the russian and german gec corpora for evaluating the quality of l'ambre. in future work, it would be interesting to expand the analysis to datasets from other languages, czech @xcite and ukrainian @xcite . ## evaluating nlg: a machine translation case study grammaticality measures, including l'ambre, can be useful across nlg tasks. here, we chose mt due to the wide-spread availability of (humanevaluated) system outputs in many languages. in addition to bleu, chrf and t-bleu foot_13 are commonly used to evaluate translation into morphologically-rich languages @xcite @xcite . evaluating the well-formedness of mt outputs has previously been studied @xcite . recent wmt shared tasks included special test suites to inspect linguistic properties of systems @xcite @xcite , which construct an evaluation set of contrastive source sentence pairs (typically english). while such contrastive pairs are very valuable, they only implicitly evaluate well-formedness and require access to underlying mt models to score the contrastive sentences. in contrast, l'ambre explicitly measures well-formedness, without requiring access to trained mt models. for evaluating mt systems, we use the data from the metrics shared task in wmt 2018 and 2019 @xcite . this corpus includes outputs from all participating systems on the test sets from the news translation shared task @xcite . our study focuses on systems that translate from english to morphologically-rich target languages: czech, estonian, finnish, german, russian, and turkish. we used all relevant languages from the wmt shared task except for lithuanian and kazakh, which lack reasonable quality parsers. ## conclusion and future work in this paper, we introduce l'ambre, a framework to evaluate grammatical acceptability of text by verifying morphosyntactic rules over dependency parse trees. we present a method to automatically extract such rules for many languages along with a method to train robust parsing models which facilitate better verification of these rules on natural language text. we demonstrate the practical application of l'ambre on the popular generation task of machine translation, focusing on translation into morphologically-rich languages. directions for future work include (1) incorporating additional morphosyntactic rules (e.g., word order), automatically extracted or hand-crafted ones such as those in @xcite and (2) building more robust parsers and morphological taggers that are aware of the dependency structure of the sentence.