Can predicate-argument relationships be extracted from UD trees? (2021)

Abstract: In this paper we investigate the possibility of extracting predicate-argument relations from UD trees (and enhanced UD graphs). Con- cretely, we apply UD parsers on an En- glish question answering/semantic-role label- ing data set (FitzGerald et al., 2018) and check if the annotations reflect the relations in the resulting parse trees, using a small number of rules to extract this information. We find that 79.1% of the argument-predicate pairs can be found in this way, on the basis of Ud- ify (Kondratyuk and Straka, 2019). Error anal- ysis reveals that half of the error cases are at- tributable to shortcomings in the dataset. The remaining errors are mostly due to predicate- argument relations not being extractible algo- rithmically from the UD trees (requiring se- mantic reasoning to be resolved). The parser itself is only responsible for a small portion of errors. Our analysis suggests a number of improvements to the UD annotation schema: we propose to enhance the schema in four ways, in order to capture argument-predicate relations. Additionally, we propose improve- ments regarding data collection for question answering/semantic-role labeling data.

## introduction universal dependencies (ud), can be seen as a compromise, a balancing act between six principles, referred to as manning's law @xcite : however, certain syntactic dependencies relevant to semantics are not included in the original formulation of ud. for example, a word may be the subject of two conjoined verbs, but in ud the subject is only connected to one of the verbs. to discover that the word is the subject of two verbs it has to be inferred from the conjunction. however, this creates unnecessary burdens for models using the ud schema. the enhanced ud schema (eud) @xcite includes such edges, with the aim to make semantics more explicit. recently there has been a surge of interest and development of eud, spurred on by its applicability on semantic downstream tasks such as information extraction @xcite . research into eud has also be facilitated recently by two shared tasks on eud parsing @xcite , which has resulted in a mix of machine learning and rule-based approaches for producing eud graphs. we come back to an evaluation of the eud schema in section 5.1. the support provided by ud w.r.t. downstream nlu tasks raises the question of how much "semantics" ud actually contains, or better put, how much semantic reasoning can one perform by using just the information provided by ud. this is also related to the question of whether ud dependencies should be seen as semantic, syntactic, or maybe something between the two. to some extent all three possibilities have been considered. one way to approach this question is to check the amount of semantic knowledge that ud exhibits, explicitly or implicitly, in relation to specific semantic tasks or features. @xcite argues that the way to see ud is as a representation "for" semantics, not "of" semantics. under this view, ud can be seen as a kind of scaffolding where some proper semantic backbone will be built upon. again, however, this begs the question of the nature of the scaffolding. @xcite claims that ud has implicit semantic role information and also shows that their enhanced version, which, as they argue, mirror semantic relations more closely, perform better than normal ud in an event extraction task involving a model that extracts dependency features from different parses. previous research has shown the opposite to be the case, i.e. ud performing better than the enhanced version in this task @xcite ; @xcite , even though these pieces of work are not directly tested on enhanced ud, but on previous related efforts to expand basic ud @xcite . ud has been also criticized by researchers working in theoretical linguistics @xcite . according to them, ud fails to observe manning's first desideratum because "ud annotation choices are not satisfactory on linguistic analysis grounds because they result from a mixture of semantic and syntactic criteria". lastly, one could argue that approaches that attempt to combine ud with an explicit logical semantics interface implicitly assume that ud is syntactic and/or missing crucial semantic information. in this paper, we propose a way to test the semantic capabilities of ud parsers for english by using their output to infer answers in a question-answering task. more precisely, what we want to investigate is the question of whether predicateargument relations are correctly captured by ud parsers. we believe that this is an important question to be posed, because, if this is the case and there is enough ground/scaffolding, then a more fine-grained semantic representation may be build on top of ud (for example, some correspondence between ud syntactic trees and logical semantics). a related question is to what extent enhanced dependencies are better, if at all, in precisely encoding predicate-argument relations. ## dataset we perform experiments on the questionanswering/semantic-role-labeling dataset of @xcite , which is based on the work of @xcite , simply referred to as "qa-srl" below. the rationale is that, in the qa-srl dataset, question-answers pairs are directly concerning predicate-argument structures. each question has a passage which it refers to. for example, the dataset might contain the passage "un published a report" together with the question "what did something publish?". the answers are provided by annotators selecting a contiguous span of text in the passage which answers the question, in this case the object "a report". the dataset contains passages from 3 domains in english: wikipedia, wikinews and science, with questions and answers generated by crowdsourcing. for each verbal predicate in the passage, questions about one of the arguments are constructed by the annotators using question templates. in total the dataset contain 265156 valid questions over 76397 passages. the qa-srl dataset also contains an automatically generated dataset. however, we have not included this part and only consider the crowdsourced part. ## task and method the most obvious way to test whether ud parsers can correctly identify the semantic arguments of verbs would be to map the form of a qa-srl question to an ud role, then retrieve the subtree of the argument from the ud tree and check if it matches the human annotations. unfortunately it is not easy to map the argument types of the qa-srl dataset to ud roles. one difficulty is the mismatch of passive and active voice between questions and answers. another problem is that the non-subject ud roles (obj/obl/advcl/etc) are in n-to-n correspondence with the qa-srl argument types @xcite . converting these relationship to a functional mapping would require the use of some statistical model to extract these features from the sentence. using a statistical model would make unclear whether it is ud that captures argument-predicate relationships, or the model. thus, to keep the method simple we resort to checking if the ud trees obtained from a parser contains the annotated qa-srl argument. to avoid the question of which semantic role should be extracted, we check if any of the children of the verb matches the answer. we make two further amendments to the task: 1. we enhance ud trees with eud arcs and 2. we check for arguments in the parent position. the second amendment helps with cases when the sentence has the form of a copula or when the verb plays the role of adjectival phrase. for example, given the passage "paleontologists are interested in fossils" and the question "who is interested in something?", then one should be able to recover "paleontologists" as an argument. however, in the ud tree, "paleontologists" is the parent of "interested". likewise, given "the observed animals were tortoises." and the question "what was observed?" should point to "animals"; which is the parent of "observed" in the ud tree. the first amendment is to use the eud schema rather than plain ud. while the state-of-the-art ud parsers do not provide this information, it is possible to automatically add most eud edges using a number of rules @xcite . thus our pipeline consists in first running a plain ud parser, we test both the stanza parser @xcite and the udify parser @xcite , and then we apply the following enhancements to the ud trees, using the system developed in (ek and bernardy, 2020): ## results and analysis in this section we present the results obtained from extracting predicate-argument relations, and provide an analysis of the errors observed. ## suggested improvements to annotation schemes in this section we leverage our understanding of eud and qa-srl, and provide advice to creators of datasets featuring either annotation schema. ## related work in addition to our suggestions, there has been several other proposals to extend syntactic dependency trees to more explicitly cover semantic phenomena, including the work of @xcite , already discussed in the introduction. additionally, @xcite notably propose additions to the eud schema mainly focusing on extracting the arguments of non-finite verbs and dealing with syntactic alterations in a french treebank. the universal decompositional semantics project @xcite is another attempt at extending the ud framework to cover semantic phenomena. they develop the semantic proto-role labeling protocol (spr1 and spr2), to find proto-semantic roles by decomposing semantic roles such as "agent" into more finegrained properties. working more generally on dependency trees, @xcite develop a framework to enhance dependency trees such that semantic propositions are more easily recoverable which includes a similar propagation of subjects and objects as in eud. however they do not appear to take any special note of purpose clauses or topicalization. ## conclusion and future work we have found that a state-of-the-art ud parser such as udify only fails to produce a semantically correct ud trees in rare cases. if we exclude difficulties in handling quotes, only 8 cases out of 100 errors are imputable to the parser. however, in a lot of cases the semantic relationship cannot possibly be present in the ud format, due to its tree structure. to express this, enhancing the structure with additional arcs is needed. some of those arcs can be found by algorithmic means (as listed in section 3), boosting the accuracy by a several points, see while the goals of the qa-srl appear to align perfectly with ours, and the annotation for qa-srl was both effective and relatively cheap, we notice some shortcomings in the annotations (section 4.3). sometimes annotators get something wrong because of a tricky phenomena or they are presented with a badly formulated question about the passage. we have proposed a number of strategies to improve data collection for future similar datasets (section 5.2). another point to consider is that it is much cheaper to annotate qa-srl than full eud parse trees. therefore qa-srl could be a proxy for training eud parsers on predicate-argument structures, together with for example multi-task learning. that is, in addition to training a system to predicting arcs, the system would be optimized on selecting the spans of text corresponding to the arguments of predicates.