# Text Summarization

## Summary


On the web, everyone can be a publisher. We're already seeing vast amounts of information being published daily in the form of restaurant/movie/book reviews, blogs, status updates, and more. In addition, traditional print publications (newspapers, magazines, technical journals, whitepapers) are also available online. It's impossible for anyone to keep track of recent publications even if limited to one domain. This is where text summarization can help. 

A summary, created automatically by algorithms, typically contains the most important information. The summary should be mindful of the reader and the communication goals. It may also help the reader decide if the original text is worth reading in full. The summary can also help improve document indexing for information retrieval. An automated summary is often less biased than a human-written summary.

## Discussion

### What are some real-world applications of text summarization?

Here are some everyday examples of text summarization: news headlines, outlines for students, movie previews, meeting minutes, biographies for resumes or obituaries, abridged versions of books, newsletter production, financial research, patent research, legal contract analysis, tweeting about new content, chatbots that answer questions, email summaries, and more. 

When Google Search presents search results, some entries are accompanied by auto-generated summaries. Google may be leveraging a knowledge graph for this purpose. Google's approach to summarization is mainly entity centric. Summarization extends to timelines and events about entities. 

Doctors write long medical notes containing nutritional information for pregnant mothers. When these were reduced to short crisp summaries, pregnant mothers found them a lot easier to understand. 


### Which are the main approaches to text summarization?

With **extractive summarization**, summary contains sentences picked and reproduced verbatim from the original text. With **abstractive summarization**, the algorithm interprets the text and generates a summary, possibly using new phrases and sentences.  

Extractive summarization is data-driven, easier and often gives better results. Abstractive summarization is how humans tend to summarize text but it's hard for algorithms since it involves semantic representation, inference and natural language generation. Often abstractive summarization relies on text extracts. 

For extraction, sentences are scored and those with highest scores are selected. Scoring criteria may include word frequencies, location heuristics, sentence similarity, rhetorical relations, and semantic roles. 

Typically an intermediate representation is used to select relevant summary content. With **topic representation**, the intent is to identify the main topics in the text. Topic words, word frequencies (including TF-IDF), clustering, LSA and LDA have been applied to summarization. With **indicator representation**, a feature set is used to rank and select sentences. Examples of this approach are graph-based methods and machine learning. 


### What are the challenges and requirements of multi-document summarization?

The pipeline for multi-document summarization (MDS) has the same basic steps as for single-document summarization (SDS): content selection, information ordering, and sentence realization. However, MDS has some unique challenges:  

    + **Redundancy**: A single document has far less redundancy than a topically-related group of documents. Summary shouldn't repeat similar sentences. *Maximal Marginal Relevance (MMR)* is a scoring system to penalize similar sentences.
    + **Temporal Ordering**: A stream of news articles might be reporting the unfolding of an event. Summary should order them correctly and be sensitive to later developments overriding earlier ones.
    + **Cohesion and Coreference**: Both are important for information ordering. Sometimes cohesion might demand a certain ordering but cause coreference problems, such as a person's shortened name appearing before the full name.
    + **Compression Ratio**: Summarization becomes more difficult when more compression is demanded.MDS may cluster similar documents and passages. Summary should include sufficient context and right level of detail. Factual inconsistencies across documents can be reported. Finally, users must be allowed to filter out irrelevant content, dig deeper into the the sources via attribution, or compare related passages across documents. 


### How does text summarization vary across domains or contexts?

Summarization must tune its output to each domain or context. For example, summarization of a news article would involve different considerations from that of a corporate sales report. 

General text summarization techniques might not do well for specific domains. Summarizers therefore might wish to use domain-specific knowledge. For legal document summarization, *CaseSummarizer* is a tool. In biomedical domain, summaries are created of literature, treatments, drug information, clinical notes, health records, and more. 

Summarizing scientific literature is a challenge due to length, complexity, and structure (tables and figures). *IBM Science Summarizer* is a tool that IBM created to summarize computer science publications. It extracts domain-specific entities of types task, dataset and metric. 

Often there are extra clues about what might be important in a document. Summarization can use these for content selection. For example, comments and discussions on a blog post point to interesting content segments. Likewise, citations in scientific papers are useful pointers. For web summarization, it's possible to look at other pages linking to a particular page and determine the most suitable sentences. 


### How has machine learning been applied to text summarization?

The common ML approach is to view text summarization as a classification problem. Algorithm is trained in a supervised manner on original text, an extractive summary and a set of features. Algorithm learns to classify sentences as either summary sentences or non-summary sentences. 

Classifiers could be based on naive-Bayes, decision trees, SVM, HMM, and CRF. Often each sentence is classified independently of others. However, since HMM and CRF capture dependencies, they outperform other techniques.  

The problem with supervised algorithms is in creating labelled data for training. This problem is worse for MDS. In a semi-supervised approach, a small amount of labelled data is used along with much larger amount of unlabelled data. The algorithm learns iteratively by classifying some unlabelled data in each iteration. 


### Could you describe neural network architectures for text summarization?

The typical approach is to do **sequence-to-sequence modelling** since input is a sequence of words and the summary is also a sequence of words. In an encoder-decoder architecture, the encoder uses LSTM to give an input representation. The decoder is also an LSTM that generates the output sequence. An attention layer between the encoder and the decoder helps in determining the most relevant words for the summary.  

Seq2seq models, LSTMs and attention layers have made abstractive summarization possible, even if they're not yet state-of-the-art compared to extractive summarization methods. These models are trained **end-to-end** without bothering to model each step of a traditional summarization pipeline. They also don't need access to specialized vocabulary or do pre-processing. This end-to-end approach has been applied successfully to short output sequences, such as news headlines or short email responses. 

In a **pointer-generator** network, a generator provides new words whereas a pointer copies words from source text. Seq2seq models often produce repetitive sentences. A **coverage model** avoids repetitions. 

Fernandes et al. showed that sequence encoders with a graph component does better at capturing long-distance relationships. 


### How do I evaluate text summarization algorithms?

Human evaluation is the simplest. In 2004, **Recall-Oriented Understudy for Gisting Evaluation (ROUGE)** was created to automate evaluation by comparing against hand-crafted summaries. ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S, and ROUGE-SU are some metrics in this family.  

Different people produce different summaries of the same text. Meaning shared across different human summaries is called Summary Content Unit (SCU). With a focus on meaning, **Pyramid Method** evaluates a summary using SCUs. 

While there's no universal system of metrics, text summarizers are typically evaluated based on TREC, DUC and MUC systems. DUC (2001-2007) became a summarization track in TAC (2008-). 

Datasets for supervised training of MDS algorithms are not common. For summarizing a single or a few documents, commonly used datasets are Gigaword, CNN/DailyMail, TAC (2008-2011) and DUC (2003-2004). ELI5 and WikiSum can be used for longform question answering and MDS respectively. Opinosis is a dataset of 51 article-summary pairs. 

Released in 2018, Cornell Newsroom is the largest dataset for training and evaluating summarization systems. Spanning 1998-2017 and containing 1.3 million articles, it's been collected from newsrooms of 38 major publications. Summaries are obtained from search and social metadata.


### What are some useful resources for text summarization?

Pengfei Liu has curated a useful list of datasets, research papers, and groups researching on text summarization. 

In Python, Gensim has a module for text summarization, which implements *TextRank* algorithm. An original implementation of the same algorithm is available as PyTextRank package. PyTeaser is a Python implementation of Scala's TextTeaser. 

Back in 2016, Google released a baseline TensorFlow implementation for summarization. 

## Milestones

Apr  
1958

Luhn makes use of **word frequencies** to determine sentences most significant for summarization. Frequently occurring words close to one another suggest significant sentences. Thresholds are set to ignore most frequent and least frequent words. For example, in biology, the word 'cell' is too common and can be ignored. Luhn's algorithm, extractive in nature, is simple in that it doesn't merge word variations (differ, different, differently). 

Apr  
1969

In addition to word frequencies, Edmundson makes use of pragmatic or cue words, title and heading words, and structural indicators such as sentence location. He notes that these improve text extraction. Example cue words are 'significant', 'impossible' and 'hardly'. They're classified are positively relevant, negatively relevant and irrelevant. He hypothesizes that significant sentences or paragraphs occur very early and very late in the section or document. He also observes that future algorithms must consider language syntax and semantics. Statistical evidence alone is inadequate. 

1995

Kupiec et al. implements a **supervised machine learning** algorithm based on the **naive-Bayes classifier**. Algorithm is trained on hand-selected extracts. The features considered include sentence length cut-off, fixed-phrase, paragraph, thematic word, and uppercase word. For example, the model ignores short sentences. It picks out thematic words, proper names and acronyms. Words such as 'conclusions', 'summary' or 'discussion' are more likely to be in the summary. 

Dec  
1997

For his PhD thesis on text summarization, Marcu takes inspiration from Rhetorical Structure Theory (RST). He looks at the **rhetorical relation** between two non-overlapping text spans called nucleus and satellite. Examples of such relations are justification, evidence, restatement, and concession. Text is decomposed into smaller units connected by rhetorical relations. In the example, *Justification* is the relation between Mars weather and its distant orbit. 

Apr  
2000

Radev et al. propose **centroid-based summarization** for multi-document summarization. Similar documents and sentences are grouped into clusters. Each cluster may represent a different sub-topic. Cluster centroid is a pseudo document representative of the cluster. Summary would include sentences similar to the centroids.  

Oct  
2000

Since RST is limited to single documents, Radev introduces **Cross-document Structure Theory (CST)** for multi-document summarization. He proposes multi-document graphs as a useful abstraction to represent relations at word, phrase, paragraph and document levels. He identifies 24 cross-document relations, such as Identity (same text), Subsumption (one sentence is contained in another), and Follow-up (additional information reflecting new developments). Summarization is done in four steps: clustering, document structure analysis, link analysis, and personalized graph-based summarization. 

May  
2004

Barzilay and Lee propose a domain-sensitive **content model**. They use **Hidden Markov Model (HMM)** in which domain topics are the states and generates sentences relevant to that topic. State transitions model topic change. An n-gram model is used to generate sentences. This model jointly learns both content selection and information ordering. 

Jul  
2004

Inspired by Google's PageRank algorithm, Mihalcea proposes *TextRank*, a **graph-based algorithm**. Each sentence is a node in the graph. Edges correspond to sentence similarities using a metric such as cosine similarity. A weighted graph is constructed from the text. A ranking algorithm (such as HITS, POS or PageRank) is run on the graph. Graph nodes with the best scores are selected for the summary. 

2006

Wu proposes **event-based summarization**. Event terms could be verbs (incorporate) or action nouns (incorporation). Event elements are typically named entities (Person, Organisation, Location, Time). Document is represented as an event map on which PageRank algorithm is employed. The work of Li et al. is also event-based and it looks at intra-event and inter-event relevance. 

Sep  
2015

Rush et al. apply **neural networks for abstractive summarization**. Previous work on abstractive summarization relied on linguistic constraints or syntactic transformations. The proposed approach applies a neural language model along with an attention-based input encoder. They experiment with three different encoders: bag-of-words, convolutional (TDNN) and attention-based. The model using attention-based encoder performs best. Experiments are limited to headline generation based on only the first sentence. The model is trained on English Gigaword corpus. This work is improved by many others in 2016. 

Aug  
2016

Nallapati et al. use an **attentional encoder-decoder RNN** for abstractive summarization. Input embedding is feature-rich with word, POS, NER, TF, and IDF. A pointer-generator model handles rare or OOV words. The attention mechanism is hierarchical at word and sentence levels. Since existing datasets are limited to single sentence summaries, they present a new dataset from CNN/DailyMail news stories with an average of 53 words and 3.72 sentences in the summaries. This work establishes a baseline for abstractive summarization of long texts. 

Jan  
2018

As an exercise in multi-document summarization, Liu et al. attempt to **generate Wikipedia articles**. In the extractive stage, they select the most important content tokens. For the abstractive stage, they use a scalable decoder-only transformer architecture in which input and output sequences are combined into a single sequence. To make it scale for longer sequences, they introduce memory-compressed attention and local attention. The final model has five layers alternating between memory-compressed and local attention.  

Oct  
2019

Fan et al. show that using **knowledge graph representations** of the text as input to a seq2seq model gives better performance. The graph is linearized before it's given to a transformer encoder. Graph construction involves merging nodes and resolving coreferences. 

Sep  
2019

Liu proposes *BERTSUM*, a modification of BERT for summarization. The model encodes multiple sentences as a single input sequence. Interval segment embeddings are use to distinguish the sentences. For fine-tuning and capturing document-level features, he tries different summarization layers: simple classifier, RNN, inter-sentence transformer. He finds that two-layer inter-sentence transformer performs best.