Weakly Supervised Headline Dependency Parsing (2022)

Abstract: English news headlines form a register with unique syntactic properties that have been documented in linguistics literature since the 1930s. However, headlines have received surprisingly little attention from the NLP syntactic parsing community. We aim to bridge this gap by providing the first news headline corpus of Universal Dependencies annotated syntactic dependency trees, which enables us to evaluate existing state-of-the-art dependency parsers on news headlines. To improve English news headline parsing accuracies, we develop a projection method to bootstrap silver training data from unlabeled news headline-article lead sentence pairs. Models trained on silver headline parses demonstrate significant improvements in performance over models trained solely on gold-annotated long-form texts. Ultimately, we find that, although projected silver training data improves parser performance across different news outlets, the improvement is moderated by constructions idiosyncratic to outlet.

## introduction english news headlines are written to convey the most salient piece of information in an article in as little space as possible. this makes them an attractive target for information extraction systems, and other nlp applications that operate on the most salient information in a news article. headlines have been the target for many nlp tasks including semantic clustering @xcite , multi-document summarization @xcite , and sentiment/stance classification @xcite . however, while headlines often present the most salient information from an article, brevity introduces its own obstacles. english news headlines are barbara bush hospitalized in houston propn propn verb adp propn nsubj flat root case obl eight jailed on various charges num verb adp adj noun root acl case amod obl seoul shares fall propn verb noun nsubj root obj figure 1 : example english news headlines with parses and pos tags generated by stanza @xcite . mispredicted relations and labels in red. the text shown is after truecasing, before being fed to stanza. the structure of this register is determined primarily by typographical constraints along with the various functions that headlines serve, including summarization and eliciting reader interest @xcite . headlinese syntax deviates from long-form news body through such features as a preference for atypical word senses and terms, frequent omission of determiners and auxiliaries, the acceptability of nominal and adverbial phrases, as well as multiple independent phrases in a single headline (decks). figure 1 presents a sample of english headlines exhibiting some of these properties, and parse errors made by a strong english dependency parser, stanza @xcite . while the syntax of english headlines deviates significantly from article body text, there has been little work in evaluating and developing classical nlp pipeline models for this register. news headline-related nlp tasks, such as headline gener-ation @xcite @xcite and classification @xcite , do not rely on syntactic annotations like pos tags, syntactic or semantic parses. this is by design, as the oftentimes poor performance of existing syntactic parsers on headlinese has impeded their application to tasks such as sentence compression @xcite . in this work, we take a step towards improving headline dependency parsers by releasing the first english news headline treebank annotated according to universal dependency (ud) typology. we present the first quantitative evaluation of existing dependency parsers on english headlinese, and we propose a method for generating weak supervision for headline dependency parsers inspired by cross-lingual annotation projection @xcite . contributions 1. we release the first english headline treebank of 1,055 manually annotated and adjudicated universal dependency (ud) syntactic dependency trees, the english headline treebank (eht), to encourage research in improving nlp pipelines for english headlinese. foot_0 2. we establish baselines on the eht evaluation set with existing state-of-the-art parsers. our experiments confirm prior observations that existing syntactic parsers perform poorly on headlinese @xcite . this paper is structured as follows: section 2 presents prior work on linguistic analyses of headlinese and their treatment in the nlp community; 2 background linguistic analysis of headlines english news headlines are known for their compressed telegraphic style, constituting a unique register known as headlinese @xcite . through a manual corpus analysis of over 1,800 headlines from two british newspapers, @xcite finds that headlinese shares some syntactic features with "ordinary" english language, but there also exist a number of features peculiar to headlinese. these include the validity of nominal and adverbial headlines, lack of determiners, omission of auxiliaries and copulas, and use of the present tense to denote urgency of the event. nevertheless, these syntactic hallmarks of headlinese vary across country of publication (ehineni, 2014), news outlet @xcite , and time period @xcite @xcite , making development of a strong, general english headline parser particularly challenging. headline nlp in spite of the clear evidence that headlinese differs significantly from standard written english syntax, there has been scant work on building traditional nlp pipeline components for headlines. this has limited the linguistic features that nlp researchers can extract from headlines, and subsequently limited the analyses that can be performed on them. for instance, @xcite reports that poor parsing accuracy for headlines impedes their use of headline parses in alignment with a body sentence. this is not to say that headlines have been ignored as an object of study by the community. tasks such as headline generation, compression, and news summarization are all well-studied problems, partly because they circumvent the need for annotation of linguistic structure @xcite @xcite @xcite . other studied tasks include emotion identification/senti-dataset headlines tokens eht gsc 600 5,017 nyt 455 3,986 silver projection 48,633 395,237 ## the english headline treebank here we describe the compilation and characteristics of our evaluation set, the english headline treebank (eht). ## generating silver data by projecting from lead sentences while eht is suitable for evaluating parser performance on english news headlines, 1,055 headlines is much less data than is typically used for training a syntactic parser. for comparison, the ewt contains more than 15 times as many tokens as the eht. on the other hand, it may be data-inefficient to manually annotate a training set of tens of thousands of headlines, since english news headlines constitute a different register of written english, not a different language. although certain constructions are idiosyncratic to headlines, one can often expand a headline to a well-formed sentence in the news body register, as words are frequently omitted to produce a headline @xcite . this section describes an algorithm algorithm 1 algorithm for projecting a parse tree from a news article lead sentence s to a headline h, which is a subsequence of s. definitions n(s) : set of nodes in tree s, each corresponding to a word in the sentence or the dummy root n(h) ⊆ n(s) : set of nodes in tree h function extractsubtree(s, h) n' ← ∅ subset of nodes to be returned for all @xmath0.t. both the head and tail of the relation are in our approach is based on the key observation that headlines convey similar semantic content as the bodies and they typically share many local substructures. lead sentences, often the first sentence in an article, serve a similar function as news headlines in grabbing reader attention and stating essential facts about news events; lead sentences are sometimes direct expansions of the headlines. consequently, the pairs of lead sentence and headline have been used to automatically construct examples for sentence compression @xcite . algorithm 1 projects the dependency tree annotation from a news article lead sentence to a headline, where the headline is a (possibly non-contiguous) subsequence of the lead sentence. the main idea of this algorithm is to prune the lead sentence's dependency tree until it only contains those tokens in the headline. when a token from the lead sentence is missing in the headline, but it has children appearing in both strings, we promote its first child to preserve connectivity. for example, the following sentence snippet contains an extra "promised" than the corresponding headline: and our algorithm promotes "release" to be the new root of the tree for the headline: we use algorithm 1 to construct a silverannotated corpus of headline dependency trees from headline-lead sentence pairs in the gsc corpus. our silver corpus contains 48,633 headlines that satisfied the subsequence constraint, the same magnitude as the ewt and significantly larger than our manually-annotated eht. of these, 8,633 were held out as a development set, with the remaining 40,000 used for training. ## training headline parsers we vary two main dimensions during parser training: training data selection and data combination methods. we consider the ewt, the projected gsc headline data described in section 3, and a combination of both as different training sets. we experiment with three different ways of combining training sets: a) simply concatenate the two cor-pora; b) use a multi-domain foot_6 model with shared feature extractors, but independent parameters for the parsing modules in each domain @xcite ; and c) first train on the gold-standard ewt corpus, and subsequently finetune on the silverannotated gsc headline corpus. model our model architecture follows the deep biaffine parser @xcite , using a pre-trained bert @xcite as a feature extractor. this architecture underlies many stateof-the-art dependency parsers (e.g., @xcite and the winning solutions in recent runs of iwpt shared tasks @xcite . model and implementation details are provided in appendix c. ## intrinsic parser performance intrinsic parser performance is shown in the discrepancy in baseline performance between nyt and gsc (85.49% vs. 60.60% las) can be attributed to the fact that nyt headlines exhibit a closer distribution of relation types to ewt than gsc headlines (figure 2 ). many nyt headlines already constitute a well-formed body sentence, albeit without final punctuation. this is further supported by the fact that only training on projected gsc parse trees significantly improves performance on gsc (89.09% las) while actually hurting nyt performance, with a slight drop to 84.75% las. however, training on both ewt and projected gsc improves parser performance across both domains. we found that training a multi-domain model performed about as well as concatenating ewt and projected gsc training data. ultimately, we found that a pipelined finetuning scheme -first training on ewt, then silver projected gsc headlines -yielded a strong parser across both domains (las of 87.13% on nyt and 90.08% on gsc). ## extrinsic evaluation in addition to intrinsic evaluation of parsers, we also evaluate these models downstream. we perform an extrinsic evaluation using the state-of-theart syntax-based predpatt openie system @xcite , and evaluate extracted tuples using the protocol and error typology taken from @xcite . as predpatt relies solely on a ud parse and pos tag sequence to extract candidate tuples, this constitutes a direct downstream evaluation of more accurate headline parses. ## further related work headline syntactic processing perhaps the two most relevant works are the recently published posh @xcite and goodnewseveryone corpora @xcite . posh is a dataset of pos-tagged english news headlines, without gold dependency parse annotations. good-newseveryone, on the other hand, contains thousands of emotion-bearing headlines labeled for semantic roles (srl). in goodnewseveryone, the relationships between identified actor, target, and predicate are solely determined by their roles. collecting dependency parse annotations is much more involved than either pos tagging or srl, as dependency parses require identifying deep relationships between individual words that are not solely derived from their types. that said, the release of both of these corpora underscores the importance of headlines as an object of study in nlp, and the desire for richer linguistic annotations. low resource syntactic processing low resource syntactic parsing is typically motivated by the need to develop a parser for languages with scant gold supervision @xcite . @xcite employ a similar, yet more involved method of annotation projection to project parser predictions from a high-resource language to a lowresource language. as we are not projecting across languages, and we restrict our parallel text to cases where a headline is a subsequence of the lead sentence, we rely on heuristics to repair the projected dependency parse. dependency parsers and treebanks for tweets are similar in spirit to the current work @xcite @xcite . unlike tweebank, we chose not to develop our own annotation scheme, but rather annotate under the ud schema. ud is sufficiently expressive for annotating headlines, and allows us to leverage multiple domains for training a parser. ## conclusion in this work, we describe the first gold-annotated evaluation set for english headline ud dependency parsing, the eht. we hope this data will encourage further research in improving dependency parsers for overlooked registers of english. in addition, we hope that the development of accurate headline dependency parsers will result in stronger performance at existing headline understanding and processing tasks, and enable more subtle linguistic analysis, such as identification of "crash blossom" news headlines.