Syntactic Nuclei in Dependency Parsing – A Multilingual Exploration (2021)

Abstract: Standard models for syntactic dependency parsing take words to be the elementary units that enter into dependency relations. In this paper, we investigate whether there are any benefits from enriching these models with the more abstract notion of nucleus proposed by Tesnière. We do this by showing how the concept of nucleus can be defined in the framework of Universal Dependencies and how we can use composition functions to make a transition-based dependency parser aware of this concept. Experiments on 12 languages show that nucleus composition gives small but significant improvements in parsing accuracy. Further analysis reveals that the improvement mainly concerns a small number of dependency relations, including nominal modifiers, relations of coordination, main predicates, and direct objects.

## introduction a syntactic dependency tree consists of directed arcs, representing syntactic relations like subject and object, connecting a set of nodes, representing the elementary syntactic units of a sentence. in contemporary dependency parsing, it is generally assumed that the elementary units are word forms or tokens, produced by a tokenizer or word segmenter. a consequence of this assumption is that the shape and size of dependency trees will vary systematically across languages. in particular, morphologically rich languages will typically have fewer elementary units and fewer relations than more analytical languages, which use independent function words instead of morphological inflection to encode grammatical information. this is illustrated in figure 1 , which contrasts two equivalent sentences in english and finnish, annotated with dependency trees following the guidelines of universal dependencies (ud) @xcite , which assume words as elementary units. an alternative view, found in the seminal work of @xcite , is that dependency relations hold between slightly more complex units called nuclei, semantically independent units consisting of a content word together with its grammatical markers, regardless of whether the latter are realized as independent words or not. thus, a nucleus will often correspond to a single word -as in the english verb chased, where tense is realized solely through morphological inflection -but it may also correspond to several words -as in the english verb group has chased, where tense is realized by morphological inflection in combination with an auxiliary verb. the latter type is known as a dissociated nucleus. if we assume that the elementary syntactic units of a dependency tree are nuclei rather than word forms, then the english and finnish sentences will have the same dependency trees, visualized in figure 2 , and will differ only in the realization of their nuclei. in particular, while nominal nuclei in finnish are consistently realized as single nouns inflected for case, the nominal nuclei in english involve standalone articles and the preposition from. in this paper, we set out to investigate whether research on dependency parsing can benefit from making explicit use of tesnière's notion of nucleus, from the point of view of accuracy, interpretability and evaluation. we do this from a multilingual perspective, because it is likely that the effects of introducing nuclei will be different in different languages, and we strongly believe that a comparison between different languages is necessary in order to assess the potential usefulness of this notion. we are certainly not the first to propose that tesnière's notion of nucleus can be useful in parsing. one of the earliest formalizations of dependency grammar for the purpose of statistical parsing, that of @xcite , had this notion at its core, and @xcite presented a conversion of the penn treebank of english to tesnière style the dog chased the cat from the room det noun verb det noun adp det noun @xmath0: word-based dependency trees for equivalent sentences from english (top) and finnish (bottom). the dog chased the cat from the room koira jahtasi kissan huoneesta nsubj nsubj obj obj obl obl figure 2: nucleus-based dependency trees for equivalent sentences from english (top) and finnish (bottom). on the parsing side, we will follow a transitionbased approach, which can relatively easily be extended to include operations that create representations of syntactic nuclei, as previously shown by @xcite , something that is much harder to achieve in a graph-based approach. ## related work dependency-based guidelines for syntactic annotation generally discard the nucleus as the basic syntactic unit in favor of the (orthographic) word form, possibly with a few exceptions for fixed multiword expressions. a notable exception is the three-layered annotation scheme of the prague dependency treebank @xcite , where nucleus-like concepts are captured at the tectogrammatical level according to the functional generative description @xcite . @xcite propose a syntactic analysis model for latvian based on the x-word concept analogous to the nucleus concept. in this grammar, an x-word acts as a non-terminal symbol in a phrase structure grammar and can appear as a head or dependent in a dependency tree. @xcite compare this model to the original dependency formalism of @xcite . finally, as already mentioned, @xcite develop an algorithm to convert english phrase structure trees to tesnière style representations. when it comes to syntactic parsing, järvinen and tapanainen (1998) were pioneers in adapting tesnière's dependency grammar for computational processing. they argue that the nucleus concept is crucial to establish cross-linguistically valid criteria for headedness and that it is not only a syntactic primitive but also the smallest semantic unit in a lexicographical description. as an alternative to the rule-based approach of @xcite , @xcite defined a generative statistical model for nucleus-based dependency parsing, which however was never implemented. the nucleus concept has affinities with the chunk concept found in many approaches to parsing, starting with @xcite , who proposed to first find chunks and then dependencies between chunks, an idea that was generalized into cascaded parsing by @xcite among others. it is also clearly related to the vibhakti level in the paninian computation grammar framework @xcite . in a similar vein, @xcite use cascaded chunking for dependency parsing of japanese, @xcite show that base-np chunking can significantly improve the accuracy of dependency parsing for thai, and durgar el-kahlout et al. @xcite show that chunking improves dependency parsing of turkish. @xcite study the importance of chunking in the transfer parsing model between @xcite show that np chunks are informative for universal part-of-speech tagging and dependency parsing. in a more recent study, de lhoneux et al. (2019b) investigate whether the hidden representations of a neural transition-based dependency parser encodes information about syntactic nuclei, with special reference to verb groups. they find some evidence that this is the case, especially if the parser is equipped with a mechanism for recursive subtree composition of the type first proposed by @xcite and later developed by @xcite and de @xcite . the idea is to use a composition operator that recursively combines information from subtrees connected by a dependency relation into a representation of the new larger subtree. in this paper, we will exploit variations of this technique to create parser-internal representations of syntactic nuclei, as discussed in section 4. however, first we need to discuss how to identify nuclei in ud treebanks. this definition of nucleus in turn depends on what we define to be functional relations. for this study, we assume that the following 7 ud relations foot_3 belong to this class: the inclusion of the cc relation among the nucleusinternal relations is probably the most controversial decision, given that tesnière treated coordination (including coordinating conjunctions) as a third type of grammatical relation -junction (fr. jonction) -distinct from both dependency relations and nucleus-internal relations. however, we think coordinating conjunctions have enough in common with other function words to be included in this preliminary study and leave further division into finer categories for future work. 5 given the definition of nucleus in terms of functional ud relations, it would be straightforward to convert the ud representations to dependency trees where the elementary syntactic units are nuclei rather than words. however, the usefulness of such a resource would currently be limited, given that it would require parsers that can deal with nucleus recognition, either in a preprocessing step or integrated with the construction of dependency trees, and such parsers are not (yet) available. moreover, evaluation results would not be comparable to previous research. therefore, we will make use of the nucleus concept in ud in three more indirect ways: ## syntactic nuclei in transition-based dependency parsing a transition-based dependency parser derives a dependency tree from the sequence of words forming a sentence @xcite @xcite . the parser constructs the tree incrementally by applying transitions, or parsing actions, to configurations consisting of a stack s of partially processed words, a buffer b of remaining input words, and a set of dependency arcs a representing the partially constructed dependency tree. the process of parsing starts from an initial configuration and ends when the parser reaches a terminal configuration. the transitions between configurations are predicted by a history-based model that combines information from s, b and a. for the experiments in this paper, we use a version of the arc-hybrid transition system initially proposed by @xcite , where the initial configuration has all words w 1 , . . . , w n plus an artificial root node r in b, while s and a are empty. 8 there are four transitions: shift, left-arc, right-arc and swap. shift pushes the first word b 0 in b onto s (and is not permissible if b 0 = r). left-arc attaches the top word s 0 in s to b 0 and removes s 0 from s, while right-arc attaches s 0 to the next word s 1 in s and removes s 0 from s. swap, finally, moves s 1 back to b in order to allow the construction of non-projective dependencies. 9 our implementation of this transition-based parsing model is based on the influential architecture of @xcite , which takes as input a sequence of vectors x 1 , . . . , x n representing the input words w 1 , . . . , w n and feeds these vectors through a bilstm that outputs contextu-alized word vectors v 1 , . . . , v n , which are stored in the buffer b. parsing is then performed by iteratively applying the transition predicted by an mlp taking as input a small number of contextualized word vectors from the stack s and the buffer b. more precisely, in the experiments reported in this paper, the predictions are based on the two top items s 0 and s 1 in s and the first item b 0 in b. in a historical perspective, this may seem like an overly simplistic prediction model, but recent work has shown that more complex feature vectors are largely superfluous thanks to the bilstm encoder @xcite . the transition-based parser as described so far does not provide any mechanism for modeling the nucleus concept. it is a purely word-based model, where any more complex syntactic structure is represented internally by the contextualized vector of its head word. specifically, when two substructures h and d are combined in a left-arc or right-arc transition, only the vector v h representing the syntactic head is retained in s or b, while the vector v d representing the syntactic dependent is removed from s. in order to make the parser sensitive to (dissociated) nuclei in its internal representations, we follow de @xcite and augment the right-arc and left-arc actions with a composition operation. the idea is that, whenever the substructures h and d are combined with label l, we replace the current representation of h with the output of a function f (h, d, l). we can then control the information flow for nuclei and other constructions through the definition of f (h, d, l). ## experiments in the previous sections, we have shown how syntactic nuclei can be identified in the ud annotation and how transition-based parsers can be made sensitive to these structures in their internal representations through the use of nucleus composition. we now proceed to a set of experiments investigating the impact of nucleus composition on a diverse selection of languages. ## conclusion we have explored how the concept of syntactic nucleus can be used to enrich the representations of a transition-based dependency parser, relying on ud treebanks for supervision and evaluation in experiments on a wide range of languages. we conclude that the use of composition operations for building internal representations of syntactic nuclei, in particular the technique that we have called soft composition, can lead to small but significant improvements in parsing accuracy for nucleus-external relations, notably for relations of coordination, direct objects, nominal modifiers and main predicates. for some languages, further improvements can be obtained by applying composition also to the nucleus-external relations themselves (generalized composition). in future work we want to study the behavior of different types of nuclei in more detail, in particular how the different internal relations of nominal and verbal nuclei contribute to overall parsing accuracy. we also want to analyze the variation between different languages in more detail and see if it can be explained in terms of typological properties.