Catching Attention with Automatic Pull Quote Selection (2020)

Abstract: To advance understanding on how to engage readers, we advocate the novel task of automatic pull quote selection. Pull quotes are a component of articles specifically designed to catch the attention of readers with spans of text selected from the article and given more salient presentation. This task differs from related tasks such as summarization and clickbait identification by several aspects. We establish a spectrum of baseline approaches to the task, ranging from handcrafted features to a neural mixture-of-experts to cross-task models. By examining the contributions of individual features and embedding dimensions from these models, we uncover unexpected properties of pull quotes to help answer the important question of what engages readers. Human evaluation also supports the uniqueness of this task and the suitability of our selection models. The benefits of exploring this problem further are clear: pull quotes increase enjoyment and readability, shape reader perceptions, and facilitate learning. Code to reproduce this work is available at https://github.com/tannerbohn/AutomaticPullQuoteSelection .

## introduction in a way, a pq is like clickbait, except that it is not lying to people. discovering what keeps readers engaged is an important problem. we thus propose the novel task of automatic pull quote (pq) selection accompanied with a new dataset and insightful analysis of several motivated baselines. pqs are graphical elements of articles with thought provoking spans of text pulled from an article by a writer or copy editor and presented on the page in a more salient manner @xcite , such as in figure 1 . pqs serve many purposes. they provide temptation (with unusual or intriguing phrases, they make strong entrypoints for a browsing reader), emphasis (by reinforcing particular aspects of the article), and improve overall visual balance and excitement @xcite . pq frequency in reading material is also significantly related to information recall and student ratings of enjoyment, readability, and attractiveness @xcite . the problem of automatically selecting pqs is related to the previously studied tasks of headline success prediction @xcite , clickbait identification @xcite @xcite , as well as key phrase extraction @xcite and document summarization @xcite . however, in sections 5.4 and 5.5 we provide experimental evidence that performing well on these previous tasks does not translate to performing well at pq selection. each of these types of text has a different function in the context of engaging a reader. the title tells the reader what the article is about and sets the tone. clickbait makes unwarranted enticing promises of what the article is about. key phrases and summaries help the reader decide whether the topic is of interest. and pqs provide specific intriguing entrypoints for the reader or can maintain interest once reading has begun by providing glimpses of interesting things to come. with their unique qualities, we believe pqs satisfy important roles missed by these popular existing tasks. ## related work in this section, we look at three natural language processing tasks related to pq selection: (1) headline quality prediction, (2) clickbait identification, and (3) summarization and keyphrase extraction. these topics motivate the cross-task models whose performance on pq selection is reported in section 5.4. ## models we consider four groups of approaches for the pq selection task: (1) handcrafted features (section 3.1), (2) n-gram features (section 3.2), (3) sbert embeddings combined with a progression of neural architectures (section 3.3), and (4) cross-task models (section 3.4). as discussed further in section 4, these approaches aim to determine the probability that a given article sentence will be used for a pq. ## experimental setup to support the new task of automatic pq selection, we both construct a new dataset and describe a suitable evaluation metric. ## experimental results we present our experimental results and analysis for the four groups of approaches: handcrafted features (section 5.1), n-gram features (section 5.2), sbert embeddings combined with a progression of neural architectures (section 5.3), and cross-task models (section 5.4). we also perform human evaluation of several models (section 5.5). appendix a contains implementation details of our models, and appendix c includes examples of pq sentences selected by several models on various articles. ## conclusion in this work we proposed the novel task of automatic pull quote selection as a means to better understand how to engage readers. to lay foundation for the task, we created a pq dataset and described and benchmarked four groups of approaches: handcrafted features, n-grams, sbert-based embeddings combined with a progression of neural architectures, and cross-task models. by inspecting results, we encountered multiple curious findings to inspire further research on pq selection and understanding reader engagement. there are many interesting avenues for future research with regard to pull quotes. in this work we assume that all true pqs in our dataset are of equal quality, however, it would be valuable to know the quality of individual pqs. it would also be interesting to study how to make a given phrase more pq-worthy while maintaining the original meaning.