How Much of Enhanced UD Is Contained in UD ? (2020)

Abstract: In this paper, we present the submission of team CLASP to the IWPT 2020 Shared Task on parsing enhanced universal dependencies. We develop a tree-to-graph transformation algorithm based on dependency patterns. This algorithm can transform gold UD trees to EUD graphs with an ELAS score of 81.55 and a EULAS score of 96.70. These results show that much of the information needed to construct EUD graphs from UD trees are present in the UD trees. Coupled with a standard UD parser, the method applies to the official test data and yields and ELAS score of 67.85 and a EULAS score is 80.18.

## introduction universal dependencies (ud) is a syntactic annotation schema focusing on representing shallow syntactic dependencies between words. one of the goals of ud has been to use it in semantic downstream tasks, such as event extraction @xcite or negation resolution @xcite among others. in general, ud can be used as a shallow representation of argument structures, which are useful in a wide array of semantic tasks. however, the ud format restricts the shape of dependencies to a tree structure. this can be limiting because semantics dependencies can in principle exhibit any graph structure. to remedy this situation, the enhanced universal dependencies (eud) schema was proposed @xcite . the goal of the schema is to make certain implicit dependencies explicit, such as conjoined subjects and objects. the enhanced dependencies include additional edges between words, as well as augmented labels. for example in enhanced dependencies, the conjunction relation also include what type of conjunction is used, e.g. and, or, but and so on. in this paper, we present our submission for the iwpt 2020 shared task on parsing enhanced universal dependencies from annotated ud treebanks. the task target treebanks from 17 different languages where the majority of the languages are indo-european with five notable exceptions, tamil (dravidian), arabic (semitic), finnish and estonian (uralic). the goal of the task to produce valid eud graphs, given raw text as input. in this context, this paper proposes to test the following hypothesis: that is, if (h1) holds, then it is possible to map, algorithmically, ud trees to eud graphs. thus, we set out to implement such an algorithm. concretely, we construct a tree-to-graph transformation, which recognizes patterns in the (basic) universal dependencies to derive enhanced dependencies. this experiment provides a lower bound on the amount of eud information which can be extracted from raw ud information, for representative inputs. in other words, we measure how much of eud is contained in ud. this becomes a lower bound, because a better algorithm can always be conceived, and do better. conversely, additionally, by running our algorithm after a state-of-the-art basic ud parser, we will provide a baseline for the eud reconstruction task. the enhanced dependency parsing is evaluated using two metrics, elas and eulas. elas calculate the f 1 -score over both enhanced arcs and labels (for example: "and" in the label "conj:and"). eulas on the other hand disregard label enhancements and calculate the f 1 -score over the enhanced edges. to illustrate, given the gold label "conj:and" and a system prediction of "conj:or" (or "conj", i.e. if nothing is appended to the label), elas will count this as an error while eulas won't. our results show that (h1) is vindicated: we achieve an elas score of 81.55 and eulas of 96.70 on human-annotated dependency trees. as a baseline for the shared task, our method is also effective, achieving an elas score of 67.85 and a eulas score of 80.18. (thus losing about 15 percentage points when going through a machinegeneration phase to obtain ud trees.) ## method in essence, our method is to apply, as far as possible, the tree-to-graph recipes provided by @xcite to transform basic ud trees into eud graphs. ## results we present our official results for the shared task in the scoring for unlabeled edges is typically around 80%. the biggest outlier being tamil, at 55.32%. the scoring for labeled edges is around 12 points lower, with more variation in scores. as explained in the introduction, to isolate the accuracy of the tree-to-graph transformation from the performance of the underlying ud parser, we also apply it to the human-annotated dependency trees. we test our approach on both the development and test data. the results from this experiment are shown in the scoring for enhanced edges and unenhanced labels (eulas) is typically above 95%, with little variation. tamil is no longer an outlier. the scoring for enhanced edges and labels can be classified into two categories depending on the language. in one comparing to get a further sense on how much our algorithm is sensitive to the quality of the input raw ud trees, we can compare the score that it obtains when fed the gold (raw) ud trees and the trees provided by stanza for the same text. the results are shown in we find that using our algorithm with stanza trees only marginally increase the performance. given that we know that our trees are effective for gold trees, this corroborates the suspicion that while having a state-of-the-art performance in dependency parsing, stanza produces trees that have a significant number of errors for the paths relevant to eud. if the ud trees were of good quality for the parts that are relevant for eud, we would see a larger increase in performance based on the potential of our algorithm. to some extent, this indicates that when developing a eud parser, the best approach may be to parse ud and eud in parallel, and not sequentially. this modus operandi may help the parser to produce fewer ud errors for the parts relevant to eud. ## discussion/analysis we can draw the following conclusions from our experiments: this shortcoming explains much of the discrepancy between the eulas and the elas scores. we note however that (not enhanced) ud is incapable of expressing the difference between the structure of the following two sentences: in ( 1 ) "a movie" is an object of a verb ("watching") which is conjoined with another verb ("reading"), but it applies only to a single verb. in (2), we also have a conjunction between verbs, but "fruits" is the object of both verbs. yet, the ud structure is the same for both sentences. and thus, our algorithm cannot recognize the difference between the two trees. one solution is to use eud, but another solution would be to use parse trees, which can make the grouping explicit. figuring out which case applies depends on the semantics and pragmatics. she was cleaning and eating fruit nsubj nsubj root aux cc conj:and obj obj figure 8 : "she was cleaning and eating fruit" even though we have shown that ud contains most information to recover eud, eud annotations do have some additional value in certain cases. however, because they have a more rich structure (graph vs. tree), it may be that eud is more suited as inputs to machine-learning systems. our systems can help to test this hypothesis by inserting it (or not) in the training pipeline of a state-of-theart eud parser. if the eud parser performs just as well with reconstructed eud data compared to human-constructed eud data, then we would know that manual eud annotations are not necessary. it is likely that we would observe a middle ground situation, where reconstructed eud data helps, but does not supplant human-constructed eud. in such a situation, our system can be useful as a bootstrapping tool. (we note however that excellent eud parsers are not available just yet; in fact they are the purpose of the iwpt task which our system is entering.) in addition to bootstrapping, a run of our system can provide a baseline for such systems. indeed, while performing well on gold data, our approach is transparent (only 5 patterns are applied) and efficient, requiring a few seconds to generate enhanced graphs from a treebank. future work in addition to the shortcoming about case labeling mentioned above, our system struggles with labels that include multi-word tokens. for example, a valid adverbial clause modifier is so that. currently our system is not able to identify these. to incorporate this our model would either need to utilize statistical or deep learning methods, or look for children with the "fixed" relation. another case that our system is not capable of handling is ellipsis. the problem of ellipsis is difficult, especially in a tree-rewriting approach. we decided not to tackle this problem and instead plan on using a deep learning parser for this task in future work. yet our main venue for future work is combining our simple, yet effective method, with deep learning for the problematic cases that are out of reach. these cases include ellipsis, multi-word tokens, objects of conjoined verbs.