Extracting Headless MWE s from Dependency Parse Trees: Parsing, Tagging, and Joint Modeling Approaches (2020)

Abstract: An interesting and frequent type of multi-word expression (MWE) is the headless MWE, for which there are no true internal syntactic dominance relations; examples include many named entities (“Wells Fargo”) and dates (“July 5, 2020”) as well as certain productive constructions (“blow for blow”, “day after day”). Despite their special status and prevalence, current dependency-annotation schemes require treating such flat structures as if they had internal syntactic heads, and most current parsers handle them in the same fashion as headed constructions. Meanwhile, outside the context of parsing, taggers are typically used for identifying MWEs, but taggers might benefit from structural information. We empirically compare these two common strategies—parsing and tagging—for predicting flat MWEs. Additionally, we propose an efficient joint decoding algorithm that combines scores from both strategies. Experimental results on the MWE-Aware English Dependency Corpus and on six non-English dependency treebanks with frequent flat structures show that: (1) tagging is more accurate than parsing for identifying flat-structure MWEs, (2) our joint decoder reconciles the two different views and, for non-BERT features, leads to higher accuracies, and (3) most of the gains result from feature sharing between the parsers and taggers.

## introduction headless multi-word expressions (mwes), including many named entities and certain productive constructions, are frequent in natural language and are important to nlp applications. in the context of dependency-based syntactic parsing, however, they pose an interesting representational challenge. dependency-graph formalisms for syntactic structure represent lexical items as nodes and headdominates-modifier/argument relations between lexical items as directed arcs on the corresponding pair of nodes. most words can be assigned clear linguistically-motivated syntactic heads, but several frequently occurring phenomena do not easily fit into this framework, including punctuation, coordinating conjunctions, and "flat", or headless mwes. while the proper treatment of headless constructions in dependency formalisms remains debated @xcite , many well-known dependency treebanks handle mwes by giving their component words a "default head", which is not indicative of a true dominance relation, but rather as "a tree encoding of a flat structure without a syntactic head" (de marneffe and nivre, 2019, pg. 213). fig. 1 shows an example: the headless mwe mellon capital has its first word, mellon, marked as the "head" of capital. despite the special status of flat structures in dependency tree annotations, most state-of-theart dependency parsers treat all annotated relations equally, and thus do not distinguish between headed and headless constructions. when headless-span identification (e.g., as part of namedentity recognition (ner)) is the specific task at hand, begin-chunk/inside-chunk/outside-chunk (bio) tagging @xcite is generally adopted. it is therefore natural to ask whether parsers are as accurate as taggers in identifying these "flat branches" in dependency trees. additionally, since parsing and tagging represent two different views of the same underlying structures, can joint decoding that combines scores from the two modules and/or joint training under a multitask learning (mtl) framework derive more accurate models than parsing or tagging alone? to facilitate answering these questions, we introduce a joint decoder that finds the maximum sum of scores from both bio tagging and parsing decisions. the joint decoder incorporates a special deduction item representing continuous headless spans, while retaining the cubic-time efficiency of projective dependency parsing. the outputs are consistent structures across the tagging view and the parsing view. we perform evaluation of the different strategies on the mwe-aware english dependency corpus and treebanks for five additional languages from the universal dependencies 2.2 corpus that have frequent multi-word headless constructions. on average, we find taggers to be more accurate than parsers at this task, providing 0.59% (1.42%) absolute higher f1 scores with(out) pretrained contextualized word representations. our joint decoder combining jointly-trained taggers and parsers further improves the tagging strategy by 0.69% (1.64%) absolute. this corroborates early evidence @xcite ) that joint modeling with parsing improves over ner. we also show that neural representation sharing through mtl is an effective strategy, as it accounts for a large portion of our observed improvements. our code is publicly available at https://github.com/tzshi/flat-mwe-parsing . ## background on headless structures a (multi-word) headless construction, or flat structure, is a span of lexical items that together reference a single concept and where no component is a syntactically more plausible candidate for the span's head than any other component. examples are boldfaced in the following english sentences. a. acl starts on july 5, 2020. b. my bank is wells fargo. c. the candidates matched each other insult for insult. @xcite (1)a and (1)b show that dates and many named entities can be headless constructions, suggesting that they are frequent. indeed, in the mwe-aware english dependency corpus @xcite , nearly half of the sentences contain headless constructions, 75% of which are named entities. for comparison, (2) shows examples of non-flat mwes, which are also interesting and important, but they are beyond the focus of our paper. a. congressman at large @xcite [@xmath0. i have moved on. [verb-particle construction, @xmath1. i take your argument into account. @xcite [light-verb construction, @xmath2 tamil), ur_udtb (urdu), vi_vtb (vietnamese) 0.00 0.00 overall, more than 20% of the treebanks in the ud 2.2 collection have flat structures in more than 20% of their training-set sentences. 4 therefore, a parsing approach taking into account the special status of headless structural representations can potentially benefit models for a large number of languages and treebanks. ## three approaches we first present the standard approaches of edgefactored parsing ( §3.2) and tagging ( §3.3) for extracting headless spans in dependency trees, and then introduce a joint decoder ( §3.4) that finds the global maximum among consistent (tree structure, tag sequence) pairs. ## related work syntactic analysis in conjunction with mwe identification is an important line of research @xcite . the span-based representations that form the basis of phrase-structure trees (as opposed to dependency trees) are arguably directly compatible with headless spans. this motivates approaches using joint constituency-tree representations based on context-free grammars @xcite and tree substitution grammars @xcite . @xcite add new phrasal nodes to denote named entities, enabling statistical parsers trained on this modified representation to produce both parse trees and named entity spans simultaneously. le @xcite use dual decomposition to develop a joint system that combines phrase-structure parsers and taggers for compound recognition. these ap-proaches do not directly transfer to dependencybased representations since dependency trees do not explicitly represent phrases. in the context of dependency parsing, @xcite report that mwe annotations have a large impact on parsing. they find that the dependency parsers are more accurate when mwe spans are not unified into single lexical items. similar to the phrase-structure case, candito and constant (2014) consider mwe identification as a side product of dependency parsing into joint representations. this parse-then-extract strategy is widely adopted @xcite @xcite . @xcite introduce additional parameterized scoring functions for the arc labelers and use global decoding to produce consistent structures during arc-labeling steps once unlabeled dependency parse trees are predicted. our work additionally proposes a joint decoder that combines the scores from both parsers and taggers. alternative approaches to graph-based joint parsing and mwe identification include transition-based @xcite and easy-first @xcite dependency parsing. these approaches typically rely on greedy decoding, whereas our joint decoder finds the globally optimal solution through dynamic programming. our work only focuses on a subset of mwes that do not have internal structures. there is substantial research interest in the broad area of mwes @xcite including recent releases of datasets @xcite , editions of shared tasks @xcite and workshops @xcite . we leave it to future work to extend the comparison and combination of taggers and dependency parsers to other mwe constructions. ## conclusion and further directions our paper provides an empirical comparison of different strategies for extracting headless mwes from dependency parse trees: parsing, tagging, and joint modeling. experiments on the mwe-aware english dependency corpus and ud 2.2 across five languages show that tagging, a widely-used methodology for extracting spans from texts, is more accurate than parsing for this task. when using bi-lstm (but not bert) representations, our proposed joint decoder reaches higher f1 scores than either of the two other strategies, by combining scores of the two different and complementary representations of the same structures. we also show that most of the gains stem from a multi-task learning strategy that shares common neural representations between the parsers and the taggers. an interesting additional use-case for our joint decoder is when a downstream task, e.g., relation extraction, requires output structures from both a parser and a tagger. our joint decoder can find the highest-scoring consistent structures among all candidates, and thus has the potential to provide simpler model designs in downstream applications. our study has been limited to a few treebanks in ud partially due to large variations and inconsistencies across different treebanks. future community efforts on a unified representation of flat structures for all languages would facilitate further research on linguistically-motivated treatments of headless structures in "headful" dependency treebanks. another limitation of our current work is that our joint decoder only produces projective dependency parse trees. to handle non-projectivity, one possible solution is pseudo-projective parsing @xcite . we leave it to future work to design a non-projective decoder for joint parsing and headless structure extraction.