Cross-Register Projection for Headline Part of Speech Tagging (2021)

Abstract: Part of speech (POS) tagging is a familiar NLP task. State of the art taggers routinely achieve token-level accuracies of over 97% on news body text, evidence that the problem is well understood. However, the register of English news headlines, “headlinese”, is very different from the register of long-form text, causing POS tagging models to underperform on headlines. In this work, we automatically annotate news headlines with POS tags by projecting predicted tags from corresponding sentences in news bodies. We train a multi-domain POS tagger on both long-form and headline text and show that joint training on both registers improves over training on just one or naïvely concatenating training sets. We evaluate on a newly-annotated corpus of over 5,248 English news headlines from the Google sentence compression corpus, and show that our model yields a 23% relative error reduction per token and 19% per headline. In addition, we demonstrate that better headline POS tags can improve the performance of a syntax-based open information extraction system. We make POSH, the POS-tagged Headline corpus, available to encourage research in improved NLP models for news headlines.

## introduction news headlines were identified as a unique register of written language at least as far back as @xcite , and the term headlinese was coined specifically to refer to the unnatural register used in headlines. hallmarks of headlinese include frequent omission of articles and auxiliary verbs, stand-alone nominals and adverbials, and infinitival forms of the main verb when referring to the future tense @xcite . in spite of the clear differences in syntax between news headlines and bodies, the nlp community has invested little effort in building headline-specific models for predicting syntactic annotations such as part of speech (pos) tags and dependency parse trees. this limits the use of syntax-based models on headlines. for example, the open domain information extraction (open ie) system, predpatt @xcite , cannot be expected to perform well on headlines, if existing models cannot accurately tag headlines with pos tags and dependency relations. more broadly, headline processing provides an important signal for downstream applications such as summarization @xcite , sentiment/stance classification @xcite , semantic clustering @xcite , and information retrieval @xcite , among other applications. in this work we take a first step toward developing strong nlp models for headlines by focusing on improving pos taggers. we propose a projection technique (inspired by work on cross-lingual projection @xcite ) to generate silver training data for each headline based on pos tags from the lead sentence of the news body. to facilitate evaluation, we crowdsource and manually adjudicate gold pos tag annotations for 5,248 headlines from the google sentence compression corpus @xcite . to the best of our knowledge, this is the first headline dataset with gold-annotated pos tags. we evaluate a range of neural pos taggers trained on long-form text, silver-labeled headlines, and the concatenation of these datasets, and find that multi-domain models which train a separate decoder layer per domain outperform models trained on data from either headlines or long-form text. we finally show that more accurate pos taggers directly lead to more precise open ie extractions, yielding an extracted tuple precision of 56.9% vs. 29.8% when using a baseline pos tagger trained on the english web treebank alone. we release our gold-annotated data as posh, the pos-tagged headline corpus, at https://zenodo.org/ record/5495668 . japanese firms push posh car showrooms twinkies are back but smaller world briefing -africa: angola: new bid to end war ebola fear stalks home hunt for quarantined now released madonna addicted to sweat dance plugs toronto condos: mortgages figure 1 : examples of headlines exhibiting register-specific phenomena: article/auxiliary omission, multiple "decks" or independent segments, and garden paths following from the need to fit a large amount of information in a small space. contributions 1. a gold-annotated evaluation set of over 5,000 english headlines to promote stronger nlp tools for news headlines. the proposed projection technique and models can also be applied to other sequence tagging tasks such as chunking or named entity recognition. these are also applicable in other domains where one has access to long-form sentence and a parallel reduced sentence (e.g., simplified english @xcite or writing from english as a second language students @xcite ). ## data we rely on two main sources of data for training and evaluating headline pos taggers: version 2.6 of the universal dependencies english web treebank in this work, we only consider the ud pos tag set. ud has become a dominant tag set for tagging and parsing, with many treebanks available across many languages @xcite . in addition, we chose to annotate headlines with ud tags as the coarser granularity made it easier for non-experts to label. we leave experiments with finer representation granularity as future work, but do not expect the choice of representation to alter the underlying conclusions. ## methods here we describe the architecture of the pos taggers and our approach for learning a headline pos tagger without direct supervision. ## experiments models are trained on either ewt, gscproj, or both training sets. we evaluate all models on the gsch test set according to token and headline accuracy, unless otherwise stated. we also consider training multi-domain models on the three auxiliary datasets as well as ewt and gscproj, to explore the value added by training on more long-form text data. for the ewt-trained baseline taggers, we insert a final period to gsch headlines before running inference so as to mitigate the mismatch between the training and test data. ## related work most nlp work on news headlines has focused on the problem of headline generation @xcite @xcite @xcite or summarization @xcite . there has also been work on training headline classification models to label headlines by their expressed emotion @xcite , stance with respect to an issue @xcite , framing of/bias towards a political issue @xcite , or the category or value of the news article (di @xcite . however, we are not aware of work to develop headline-specific models for predicting traditional linguistic annotations such as pos tags or dependency trees, although there has been work on adapting machine translation models to headlines @xcite . as we show in this work, and has been observed in the past @xcite , a model trained on non-headline-domain data will not necessarily perform well on headlines. our work is analogous to work on constructing pos taggers or syntactic parsers for less traditional domains such as tweets @xcite @xcite . unlike this line of work, we did not have to craft a unique tag set for headlines, as headlines are not rife with the typographical errors and atypical constructions of tweets. at the same time, we do not presume a gold-annotated headline training set, but use projection to construct a silver-labeled training set. data is annotated data purely for the purpose of evaluation. ## conclusion this work is a first step towards developing stronger nlp tools for news headlines. we show that training a tagger on headlines with projected pos tags results in a far stronger model than taggers trained on gold-annotated long-form text. this suggests that more expensive syntactic annotations, such as dependency trees, may also be reliably projected onto headlines, obviating the need for gold dependency annotations when training a headline parser. although this work is focused on learning strong headline pos taggers, the projection technique described here can be adapted to train other strong headline sequence taggers; for example, training a headline chunker or named entity tagger on iob tags projected from the lead sentence. projection could potentially be applied to generate silverlabeled data for other domains such as simplified english (e.g., aligned sentences from simplified to original wikipedia @xcite ) and other languages.