UD on2: a library for manipulating U niversal D ependencies trees (2020)

Abstract: UDon2 is an open-source library for manipulating dependency trees represented in the CoNLL-U format. The library is compatible with the Universal Dependencies. UDon2 is aimed at developers of downstream Natural Language Processing applications that require manipulating dependency trees on the sentence level (in addition to other available tools geared towards working with treebanks).

## introduction universal dependencies (ud) is a framework unifying ways of annotating grammar for different human languages @xcite . to date, the ud community has produced more than 150 treebanks in 90 languages and a number of ud-compatible tools for processing data. most of the available tools focus on working with treebanks, e.g. annotating textual data, validating existing treebanks or making simple edits. however, many downstream natural language processing (nlp) applications require researchers to manipulate individual dependency trees. for instance, finding all subordinate clauses in the sentence might help in performing text simplification, finding all objects connected to a verb in the passive form might be useful for creating a list of candidate referents for co-reference resolution, and being able to remove certain subtrees might assist in generating reading-comprehension questions. some of those tasks are easy to achieve with some simple scripting, but such ad-hoc solutions become difficult to maintain over time. furthermore, they tend to lack speed and hinder large-scale experimentation, since they are typically written in high-level programming languages in presence of time pressure. to aid the community in solving these tasks, we present udon2 -a library for manipulating ud dependency trees optimized for querying. udon2 has a user-friendly api allowing to perform routine tasks with only a couple of lines of code. for instance, finding all nominal objects in singular requires only a code snippet below. udon2 is an open-source library written in c++ with python bindings, combining the speed of c++ and the flexibility and ease-of-use of python. udon2 is hosted on github (the source code is available at https://github.com/udon2/udon2 ), and everyone is welcome to contribute. ## example use cases udon2 operates on dependency trees for individual sentences. preparing a raw text for downstream applications requires segmenting it into sentences and then parsing every sentence to get its dependency tree stored in conll-u format. the result of reading a conll-u file is an instance of the node class representing a 'root' pseudonode of the dependency tree. the dependent of the 'root' pseudonode will be later referred to as a root word. in the section below, we present possible use cases along with the manipulations available for a generic node instance n, and exemplify using the dependency tree in figure 1 with its root word study being denoted as r. @xcite and visualized using udon2. ## related work currently available ud processing tools for python are geared towards working with treebanks and making batch manipulations and edits. udpipe @xcite is a library written in c++ with bindings to other programming languages. udpipe provides a trainable pipeline which performs sentence segmentation, tokenization, pos-tagging, lemmatization and dependency parsing. the library provides no built-in support for manipulations on dependency trees. a similar functionality is also provided by the stanza package @xcite . depedit @xcite is a configurable tool for manipulating dependency trees in the conll-u format. the manipulations are specified in the configuration file using regular expressions for selecting nodes of interest, and a custom syntax for specifying relations between the nodes, and actions to perform on the matched nodes. the tool is geared towards performing batch operations and thus operations like querying to get a list of matching nodes for performing further manipulations, getting a text of the subtree induced by the node or implementing convolution tree kernels are impossible to achieve, to the best of our knowledge. udapi @xcite is one such framework providing the ability to parse dependency trees, visualize them, convert between different representation formats (conll-u, sdparse and visl-cg), applying batch queries and edits to treebanks, and validate the format and contents of treebanks. udapi is available as a command line tool, and has apis for java, python and perl. one of the reviewers has brought to our attention that udapi is capable of performing directly (or gives a possibility to implement) the same transformations as udon2. two smaller packages, pyconll foot_3 and conllu foot_4 , provide an interface to the conll-u annotation scheme without the possibility of visualization, but with a possibility to reimplement the same transformations as in udon2. in order to compare the last three mentioned packages, we provide the benchmark results for udon2 and udapi in ## discussion and conclusions most of the current ud-compatible tools are focused on treebank developers, whereas udon2 aims at helping researchers explore the use of dependency trees for downstream applications, and hence is optimized mostly for querying and interacting with conll-u files. to the best of our knowledge, udon2 is the first package providing the possibility to both perform manipulations on dependency trees, perform advanced transformations (such as grct, pct or lct), and compute convolution tree kernels. udon2 provides a superior performance on the majority of the benchmarks, except for read and write. the reason is that these two benchmarks require using python's for-loops for c++ objects, requiring a lot of type conversions between python and c++. udon2 tries to avoid this by offering various query methods for common tasks, where looping is done in c++ as well (e.g. text, relchain, load and save benchmarks), which brings evident performance gains. optimizing udon2 for working better with python's loops is an ongoing work and contributions are welcome. we hope that udon2 is going to aid researchers in experimenting with dependency trees, and that it will be expanded with the help of the ud community.