An Error Analysis Framework for Shallow Surface Realization (2021)

Abstract: The metrics standardly used to evaluate Natural Language Generation (NLG) models, such as BLEU or METEOR, fail to provide information on which linguistic factors impact performance. Focusing on Surface Realization (SR), the task of converting an unordered dependency tree into a well-formed sentence, we propose a framework for error analysis which permits identifying which features of the input affect the models’ results. This framework consists of two main components: (i) correlation analyses between a wide range of syntactic metrics and standard performance metrics and (ii) a set of techniques to automatically identify syntactic constructs that often co-occur with low performance scores. We demonstrate the advantages of our framework by performing error analysis on the results of 174 system runs submitted to the Multilingual SR shared tasks; we show that dependency edge accuracy correlate with automatic metrics thereby providing a more interpretable basis for evaluation; and we suggest ways in which our framework could be used to improve models and data. The framework is available in the form of a toolkit which can be used both by campaign organizers to provide detailed, linguistically interpretable feedback on the state of the art in multilingual SR, and by individual researchers to improve models and datasets.1

## introduction surface realization (sr) is a natural language generation task that consists in converting a linguistic representation into a well-formed sentence. sr is a key module in pipeline generation models, where it is usually the last item in a pipeline of modules designed to convert the input (knowledge graph, tabular data, numerical data) into a text. while end-to-end generation models have been proposed that do away with such pipeline architecture and therefore with sr, pipeline generation models @xcite @xcite have been shown to perform on a par with these end-to-end models while providing increased controllability and interpretability (each step of the pipeline provides explicit intermediate representations that can be examined and evaluated). as illustrated in, for example, @xcite , @xcite , and @xcite , sr also has potential applications in tasks such as summarization and dialogue response generation. in such approaches, shallow dependency trees are viewed as intermediate structures used to mediate between input and output, and sr permits regenerating a summary or a dialogue turn from these intermediate structures. finally, multilingual sr is an important task in its own right in that it permits a detailed evaluation of how neural models handle the varying word order and morphology of the different natural languages. while neural language models are powerful at producing high quality text, the results of the multilingual sr tasks @xcite clearly show that the generation, from shallow dependency trees, of morphologically and syntactically correct sentences in multiple languages remains an open problem. as the use of multiple input formats made the comparison and evaluation of existing surface realisers difficult, @xcite and @xcite organized the sr shared tasks, which provide two standardized input formats for surface realizers: deep and shallow dependency trees. shallow dependency trees are unordered, lemmatized dependency trees. deep dependency trees include semantic rather than syntactic relations and abstract over function words. while the sr tasks provide a common benchmark on which to evaluate and compare sr systems, the evaluation protocol they use (automatic metrics and human evaluation) does not support a detailed error analysis. metrics (bleu, dist, nist, meteor, ter) and human assessments are reported on the system level, and so do not provide a detailed feedback for each participant. neither do they give information about which syntactic phenomena impact performance. in this work, we propose a framework for error analysis that allows for an interpretable, linguistically informed analysis of sr results. while shallow surface realization involves both determining word order (linearization) and inflecting lemmas (morphological realization), since inflection error detection is already covered in morphological shared tasks @xcite , we focus on error analysis for word order. motivated by extensive linguistic studies that deal with syntactic dependencies and their relation to cognitive language processing @xcite @xcite , we investigate word ordering performance in sr models given various tree-based metrics. specifically, we explore the hypothesis according to which these metrics, which provide a measure of the sr input complexity, correlate with automatic metrics commonly used in nlg. we find that dependency edge accuracy (dea) correlates with bleu, which suggests that dea could be used as an alternative, more interpretable, automatic evaluation metric for surface realizers. we apply our framework to the results of two evaluation campaigns and demonstrate how it can be used to highlight some global results about the state of the art (e.g., that certain dependency relations such as the list dependency have low accuracy across the board for all 174 submitted runs). we indicate various ways in which our error analysis framework could be used to improve a model or a dataset, thereby arguing for approaches to model and dataset improvement that are more linguistically guided. finally, we make our code available in the form of a toolkit that can be used both by campaign organizers to provide a detailed feedback on the state of the art for surface realization and by researchers to better analyze, interpret, and improve their models. ## related work there has been a long tradition in nlp exploring syntactic and semantic evaluation measures based on linguistic structures @xcite @xcite @xcite . in particular, dependency-based automatic metrics have been developed for summarization @xcite @xcite and machine translation @xcite . relations between metrics were also studied: dang and @xcite found that automatic metrics perform on a par with the dependency-based metric of @xcite while evaluating summaries. the closest research to ours, which focused on evaluating how dependency-based metrics correlate with human ratings, is @xcite , who showed that syntactic-based metrics perform equally well as compared to automatic metrics in terms of their correlation with human judgments for a german surface realizer. researchers, working on sr and word ordering, have been resorting to different metrics to report their models' performance. @xcite , @xcite , @xcite , @xcite , and @xcite used bleu; @xcite parsed their outputs and calculated the uas parsing metric; @xcite used kendall correlation together with edit-distance to account for english word order. similarly, @xcite used spearman correlation between produced and gold word order for a dozen of languages. @xcite , in their ccg-based realization, calculated average dependency lengths between grammar-generated sentences and gold standard. @xcite and narayan and @xcite proposed an error mining algorithm for generation grammars to identify the most likely sources of failures, when generating from dependency trees. their algorithm mines suspicious subtrees in a dependency tree, which are likely to cause errors. @xcite drew attention to their model performance for non-projective sentences. @xcite assessed their binary classifier for word ordering using the accuracy of predicting the position of a dependent with respect to its head, and a sibling. @xcite showed that, for their system, error rates correlate with word order freedom, and reported linearization error rates for some frequent dependency types. in a similar vein, @xcite looked at their system performance in terms of dependency relations, which shed light on the differences between their non-delexicalized and delexicalized models. in sum, multiple metrics and tools have been developed by individual researchers to evaluate and interpret their model results: dependencybased metrics, correlation between these metrics and human ratings, performance on projective vs. non-projective input, linearization error rate, and so forth. at a more global level, however, automatic metrics and human evaluation continue to be massively used. in this study, we gather a set of linguistically informed, interpretable metrics and tools within a unified framework, apply this framework to the results of two evaluation campaigns (174 participant submissions) and generally argue for a more interpretable evaluation approach for surface realizers. ## framework for error analysis our error analysis framework gathers a set of performance metrics together with a wide range of tree-based metrics designed to measure the syntactic complexity of the sentence to be generated. we apply correlation tests between these two types of metrics and mine a model output to automatically identify the syntactic constructs that often co-occur with low performance scores. ## data and experimental setting we apply our error analysis methods to 174 system outputs (runs) submitted to the shallow track of sr'18 and sr'19 shared tasks @xcite . for each generated sentence in the submissions, we compute the metrics described in the preceding section as follows. computing syntactic complexity metrics. tree-based metrics, dependency relation entropy and projectivity are computed on the gold parse trees from universal dependencies v2.0 and v2.3 (nivre et al., 2017) for sr'18 and sr'19, respectively. following common practice in dependency linguistics computational studies, punctuation marks were stripped from the reference trees (based on punct dependency relation). if a node to be removed had children, these were assigned to the parent of the node. ## error analysis we analyze results focusing successively on: treebased syntactic complexity (are sentences with more complex syntactic trees harder to generate?), projectivity (how much does non-projectivity impact results?), entropy (how much do word order variations affect performance?), dea and error mining (which syntactic constructions lead to decreased scores?). ## conclusion we presented a framework for error analysis that supports a detailed assessment of which syntactic factors impact the performance of surface realisation models. we applied it to the results of two sr shared task campaigns and suggested ways in which it could be used to improve models and datasets for shallow surface realisation. more generally, we believe that scores such as bleu and, to some extent, human ratings do not provide a clear picture of the extent to which sr models can capture the complex constraints governing word order in the world natural languages. we hope that the metrics and tools gathered in this evaluation toolkit can help address this issue.