News Headline Grouping as a Challenging NLU Task (2021)

Abstract: Recent progress in Natural Language Understanding (NLU) has seen the latest models outperform human performance on many standard tasks. These impressive results have led the community to introspect on dataset limitations, and iterate on more nuanced challenges. In this paper, we introduce the task of HeadLine Grouping (HLG) and a corresponding dataset (HLGD) consisting of 20,056 pairs of news headlines, each labeled with a binary judgement as to whether the pair belongs within the same group. On HLGD, human annotators achieve high performance of around 0.9 F-1, while current state-of-the art Transformer models only reach 0.75 F-1, opening the path for further improvements. We further propose a novel unsupervised Headline Generator Swap model for the task of HeadLine Grouping that achieves within 3 F-1 of the best supervised model. Finally, we analyze high-performing models with consistency tests, and find that models are not consistent in their predictions, revealing modeling limits of current architectures.

## introduction headlines are a key component in everyday news consumption. as the first piece of text the user interacts with when learning about a story, the headline can play many roles, including: summarize the main points of the story, promote a particular detail, and convince the reader to choose one source over another @xcite . news aggregators amass content from many disparate news sources and have become popular, in part because they offer news readers access to diverse sources @xcite . @xcite find that news aggregators help news readers access content they are unfamiliar with, and potentially on opposite sides of the political spectrum. at the heart of a news aggregator is the ability to group relevant content together, to support a reader in finding varying views and angles on the news. natural language understanding (nlu) has seen rapid progress in recent years. the creation of multi-task benchmarks such as the general language understanding evaluation collection (glue), paired with fast-paced progress in transformer-based architectures has led to models outperforming human baseline performance on many tasks, such as paraphrase identification @xcite , semantic similarity @xcite , and extractive question-answering (qa) @xcite . this success has led to the questioning of the composition of benchmarks, and the subsequent creation of ever-more challenging datasets, for example by increasing the diversity of texts in textual entailment datasets @xcite , or introducing unanswerable questions in qa datasets @xcite . ## headline grouping dataset we now present the headline grouping dataset. we describe the dataset of news articles we collected for annotation, our annotation procedure, an analysis of the resulting dataset, and the challenges we propose to the community. ## results on the challenges in ## analysis of model consistency checking whether deep learning models are consistent across predictions has recently become a subject of interest, for example with qa systems with text @xcite and image @xcite inputs. we analyze model consistency by probing the electra finetune + time model, which achieves highest performance in terms of f-1 score. we propose a commutative test and transitive test, both illustrated in figure 5 . in order to evaluate consistency across training runs, we trained six versions of the electra finetune + time model with the same hyper-parameters. because each training run processes through the data in a different order, the models are distinct from each other. with regard to performance, the models perform very similarly, achieving within 0.01 f-1 of each other on the development and test sets. ## conclusion in this work we present the new task of headline grouping (hlg) a new challenging nlu task, with an accompanying dataset (hlgd). even though state-of-the-art nlu models have achieved close to human performance on many nlu tasks, we show that there is a considerable gap between best model performance (0.75 f-1) and human performance (about 0.9 f-1) on hlgd. we therefore propose this dataset as a challenge for future nlu benchmarks. we propose to repurpose a headline generator for the task of headline grouping, based on prompting it for the likelihood of a headline swap, and achieve within 3 f-1 of the best supervised model, paving the way for other unsupervised methods to repurpose generators for nlu. analysis of models on hlgd reveals that they are not consistent in trivial ways, suggesting further improvements needed to nlu models. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems, pages 5998-6008. nguyen xuan vinh, julien epps, and james bailey. 2010. information theoretic measures for clusterings comparison: variants, properties, normalization and correction for chance. the journal of machine learning research, 11:2837-2854. adina williams, nikita nangia, and samuel bowman. 2018. a broad-coverage challenge corpus for sentence understanding through inference. in naacl, pages 1112-1122. sander wubben, antal van den bosch, emiel krahmer, and erwin marsi. 2009. clustering and matching headlines for automatic paraphrase acquisition. in proceedings of the 12th european workshop on natural language generation (enlg 2009), pages 122-125. w. xu, alan ritter, chris callison-burch, w. dolan, and yangfeng ji. 2014. extracting lexically divergent paraphrases from twitter. transactions of the association for computational linguistics, 2:435-448.