Comparison by Conversion: Reverse-Engineering UCCA from Syntax and Lexical Semantics (2020)

Abstract: Building robust natural language understanding systems will require a clear characterization of whether and how various linguistic meaning representations complement each other. To perform a systematic comparative analysis, we evaluate the mapping between meaning representations from different frameworks using two complementary methods: (i) a rule-based converter, and (ii) a supervised delexicalized parser that parses to one framework using only information from the other as features. We apply these methods to convert the STREUSLE corpus (with syntactic and lexical semantic annotations) to UCCA (a graph-structured full-sentence meaning representation). Both methods yield surprisingly accurate target representations, close to fully supervised UCCA parser quality—indicating that UCCA annotations are partially redundant with STREUSLE annotations. Despite this substantial convergence between frameworks, we find several important areas of divergence.

## comparing meaning representations several symbolic meaning representations (mrs) support human annotation of text with broad coverage @xcite . to date, it is still not completely clear, for all frameworks, what linguistic semantic phenomena they encode, and how it compares to the content represented by the others. it therefore behooves us to develop a firm linguistic understanding of mrs. in particular: are they merely a coarsening and rearranging of syntactic information, such as is encoded in universal dependencies (ud; @xcite ? to what extent do they take lexical semantic properties into account? what does this suggest about the potential for exploiting simpler or better-resourced linguistic representations for improved mr parsing? intuitively, we ask whether: sentence-level mr ? = syntax + lexical semantics to address this question, we examine ucca, a document-level mr often used for sentence-level semantics (see §2.1). @xcite began to examine the relation of ucca to syntax, contributing a corpus with gold standard ud and ucca parses, heuristically aligning them, and quantifying the correlations between syntactic and semantic labels. conversely, @xcite provided some initial evidence that other mrs can be brought to bear on the ucca parsing task via multitask learning, but left the details of the relationship between representations to latent (and opaque) parameters of neural models. in this paper, we aim to close the gap between the two previous investigations by (1) building an interpretable rule-based system to convert from shallower representations (syntax and lexical semantic units/tags) into ucca, forcing us to be linguistically precise about what ucca captures and how it "decomposes"; and (2) training top-performing supervised parsers in a delexicalized setting with only syntactic and lexical semantic features, as a data-driven mapping corroborating the rule-based approach. we perform our analysis on the reviews section of the english web treebank @xcite , which has been manually annotated with ud and ucca; and streusle for lexical semantics ( §2). although at present we only have the necessary evaluation data for english, the linguistic representations we examine have been applied to multiple languages ( §2). our approach can thus be applied crosslinguistically with minimal adaptation. our contributions are: ## representations under consideration the increasing interest in semantic representation and parsing, and the partial overlap in content between the different frameworks @xcite , is a main motivation for our inquiry into content differences between ud and streusle, and ucca. we expect our inquiry to be relevant to other schemes, both in developing a general methodology, and in the insights gathered. for example, besides streusle, ud also serves as the backbone of the decomp scheme @xcite , and so information as to its semantic content is important there as well. argument structural phenomena are at the heart of many mrs, which provide further motivation for empirical studies to the extent lexical semantics and syntax can encode them. ## first conversion approach: rule-based ucca parsing from syntax and lexical semantics here we describe a system to produce ucca analyses for text, given ud syntactic graphs and streusle lexical semantic annotation. this is a rule-based converter that inspects the ud structure in tandem with streusle annotations to build an ucca parse. 6 an analysis of the converter's successes and failures ( §7) will, in turn, reveal the similarities and differences between the schemes. what follows is an overview of the algorithm; futher details about the rules are given in appendix a. 7 mwes and unanalyzable units. based on streusle mwe annotation, we group together text tokens into single unanalyzable semantic units when they are annotated as strong mwes (we do not use weak mwes, as ucca does not encode them), except for light verbs and annotations that would lead to cycles. mwe-internal dependency edges are discarded so they will not be processed later. punctuation is marked u. mappings between units and dependency nodes are maintained. for example, in this is one of the worst places i have stayed, we cut out[sic] stay short and went to the mulberry. (reviews-023620-0001), cut. . . short is an unanalyzable unit according to the gold ucca annotation, but the syntactic relation between cut and short, namely xcomp, does not indicate that; only few xcomps, in general, correspond to ucca unanalyzable units @xcite . in streusle, however, cut. . . short is annotated as a strong mwe, whose lexcat is v.vid (idiomatic verb). our rules create an unanalyzable unit covering this phrase, which matches the gold ucca annotation in this case. the same is true for blue cross foot_7 in figure 2 . ## second conversion approach: delexicalized supervised ucca parsing previous work tackled the ucca parsing task using supervised learning. in order to complement and validate the analysis of the rule-based converter, we compare its findings to a delexicalized supervised parser, that can be seen as inducing a converter from data. by removing all word and lemma features from these parsers, and instead adding features based on gold ud and streusle annotations, we obtain supervised "converters", which can be used for data-driven analysis and complement the rules. tupa. this ucca parser @xcite is based on a transition-based algorithm with a neural network transition classifier, using a bilstm for encoding input representation, with word, lemma, and syntactic features embedded as real-valued features. we add the supersense and lexcat from streusle as embedding inputs to the tupa bilstm (concatenated with existing inputs). for prepositions, we add both the scene role and function (see §2). 9 hit-scir parser. this is a transition-based parser for several mr frameworks, including ucca @xcite . it achieved the highest average score in the conll 2019 shared task @xcite . 5 experiments data. we use the reviews section from ud 2.6 english_ewt @xcite , with lexical semantic annotations from streusle 4.4 @xcite , 11 and with ucca graphs from ucca_english-ewt v1.0.1 @xcite . 12 we use the standard train/development split for this dataset, and do not use the test split to avoid over-analyzing it, although all datasets contain annotations for it too. the data statistics are listed in rule-based converters. we evaluate the rule-based converter ( §3), as well as the syntax-based converter from @xcite , which uses the ud tree and a majority-based category mapping (based on the most common ucca category in the training set for each ud relation). this converter is oblivious to lexical semantics. ## results rules with gold ud and streusle close the gap between the syntax-based converter and parsers with word information, reaching the same primary labeled f1 as tupa with word features. this is surprising (since supervised parsers are known to usually outperform rule-based ones), and suggests that the training data (see ## analysis the delexicalized parsers' confusion matrix (in appendix c) is similar. 15 note that we consulted the training set iteratively while developing the rules, addressing many recurring issues that would show up as prominent confusions. 16 we proceed with an extensive error analysis of the converter, to point out similarities and delineate remaining divergences, which we stipulate constitute content differences between ucca and the combination of syntax and lexical semantics from ud and streusle. figure 3 shows gold annotation and the converter's predictions. ## conclusion we have presented an extensive analysis of the similarities and differences between streusle and ucca on the ewt reviews corpus, assisted by two complementary methods: manual rule-based conversion, and delexicalized parsing. both approaches arrived at similar results, showing that the conversion between the frameworks can be moderately accurate, while also revealing important divergences, namely distinctions made in ucca but not in streusle: semantic relation between nouns in compounds, adverbial and linkage usage of adverbs, and the scene-evoking status of nouns, possessives and adjectives, among others. enriching supervised parsers with lexical semantic features improves parsing performance when using gold input. while this paper focuses on analysis, future work will investigate using predicted features with a parser/tagger @xcite . this approach is expected to improve parsing performance and robustness, demonstrating the utility of linguistically-informed approaches in complementing general supervised semantic parsers.