README.md

---
language: en
tags:
- Summarization
license: apache-2.0
datasets:
- scientific_papers
- big_patent
- cnn_corpus
- cnn_dailymail
- xsum
- MCTI_data
thumbnail: https://github.com/Marcosdib/S2Query/Classification_Architecture_model.png
---

![MCTIimg](https://antigo.mctic.gov.br/mctic/export/sites/institucional/institucional/entidadesVinculadas/conselhos/pag-old/RODAPE_MCTI.png)


# MCTI Text Automatic Text Summarization Task (uncased) DRAFT

Disclaimer: The Brazilian Ministry of Science, Technology, and Innovation (MCTI) has partially supported this project.

## According to the abstract of the literature review,

- We provide a literature review about Automatic Text Summarization systems. We consider a citation-based approach. We start with some popular and well-known 
  papers that we have in hand about each topic we want to cover and we have tracked the "backward citations" (papers that are cited by the set of papers we 
  knew beforehand) and the "forward citations" (newer papers that cite the set of papers we knew beforehand). In order to organize the different methods, we 
  present the diverse approaches to ATS guided by the mechanisms they use to generate a summary. Besides presenting the methods, we also present an extensive 
  review of the datasets available for summarization tasks and the methods used to evaluate the quality of the summaries.  Finally, we present an empirical 
  exploration of these methods using the CNN Corpus dataset that provides golden summaries for extractive and abstractive methods.

This model was an end result of the above-mentioned literature review paper, from which the best solution was drawn to be applied to the problem of 
summarizing texts extracted from the Research Financing Products Portfolio (FPP) of the Brazilian Ministry of Science, Technology, and Innovation (MCTI). 
It was first released in [this repository](https://huggingface.co/unb-lamfo-nlp-mcti), along with the other models used to address the given problem.

## Model description

This Automatic Text Summarization (ATS) Model was developed in the Python language to be applied to the Research Financing Products 
Portfolio (FPP) of the Brazilian Ministry of Science, Technology, and Innovation. It was produced in parallel with the writing of a 
Systematic Literature Review paper, in which there is a discussion concerning many summarization methods, datasets, and evaluators
as well as a brief overview of the nature of the task itself and the state-of-the-art of its implementation.

The input of the model can be either a single text, a data frame or a CSV file containing multiple texts (in the English language) and its output
is the summarized texts and their evaluation metrics. As an optional (although recommended) input, the model accepts gold-standard summaries 
for the texts, i.e., human-written (or extracted) summaries of the texts which are considered to be good representations of their contents. 
Evaluators like ROUGE, which in its many variations is the most used to perform the task, require gold-standard summaries as inputs. There are, 
however, Evaluation Methods that do not depend on the existence of a golden summary (e.g. the cosine similarity method, the Kullback Leibler
Divergence method) and this is why an evaluation can be made even when only the text is taken as input to the model.

The text output is produced by a chosen method of ATS which can be extractive (built with the most relevant sentences of the source document) 
or abstractive (written from scratch in an abstractive manner). The latter is achieved by means of transformers, and the ones present in the
model are the already existing and vastly applied BART-Large CNN, Pegasus-XSUM and mT5 Multilingual XLSUM. The extractive methods are taken from
the Sumy Python Library and include SumyRandom, SumyLuhn, SumyLsa, SumyLexRank, SumyTextRank, SumySumBasic, SumyKL and SumyReduction. Each of the
methods used for text summarization will be described individually in the following sections.

## Methods

Since there are many methods to choose from in order to perform the ATS task using this model, the following table presents useful information
regarding each of them, such as what kind of ATS the method produces (extractive or abstractive), where to find the documentation necessary for 
its implementation and the article from which it originated.

| Method                 | Kind of ATS | Documentation   | Source Article |
|:----------------------:|:-----------:|:---------------:|:--------------:|
| SumyRandom             | Extractive  |   [Sumy GitHub](https://github.com/miso-belica/sumy/) | None (picks out random sentences from source text) |
| SumyLuhn               | Extractive  |   Ibid.         | [(Luhn, 1958)](http://www.di.ubi.pt/~jpaulo/competence/general/%281958%29Luhn.pdf) |
| SumyLsa                | Extractive  |   Ibid.         | [(Steinberger et al., 2004)](http://www.kiv.zcu.cz/~jstein/publikace/isim2004.pdf) |
| SumyLexRank            | Extractive  |   Ibid.         | [(Erkan and Radev, 2004)](http://tangra.si.umich.edu/~radev/lexrank/lexrank.pdf) |
| SumyTextRank           | Extractive  |   Ibid.         | [(Mihalcea and Tarau, 2004)](https://web.eecs.umich.edu/~mihalcea/papers/mihalcea.emnlp04.pdf) |
| SumySumBasic           | Extractive  |   Ibid.         | [(Vanderwende et. al, 2007)](http://www.cis.upenn.edu/~nenkova/papers/ipm.pdf) |
| SumyKL                 | Extractive  |   Ibid.         | [(Haghighi and Vanderwende, 2009)](http://www.aclweb.org/anthology/N09-1041) |
| SumyReduction          | Extractive  |   Ibid.         | None. |
| BART-Large CNN         | Abstractive | [facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn)                    | [(Lewis et al., 2019)](https://arxiv.org/pdf/1910.13461) |
| Pegasus-XSUM           | Abstractive | [google/pegasus-xsum](https://huggingface.co/google/pegasus-xsum)                            | [(Zhang et al., 2020)](http://proceedings.mlr.press/v119/zhang20ae/zhang20ae.pdf) |
| mT5 Multilingual XLSUM | Abstractive | [csebuetnlp/mT5_multilingual_XLSum](https://huggingface.co/csebuetnlp/mT5_multilingual_XLSum)| [(Raffel et al., 2019)](https://www.jmlr.org/papers/volume21/20-074/20-074.pdf?ref=https://githubhelp.com) |


## Limitations

- The only language supported by the model is English
- For texts summarized by transformers, the size of the original text is limited by the maximum number of tokens supported by the transformer
- No specific training is done for the application of the model and only pretrained transformers are used (e.g. BART is trained with CNN Corpus and Pegasus with XSum)
- There is a difference in the quality of the results depending on the sort of text which is being summarized. BART, for example, having been trained with a dataset in the news domain, will be better at summarizing news in comparison to scientific articles

### How to use

Initially, some libraries will need to be imported in order for the program to work. The following lines 
of code, then, are necessary:

```python
import threading
from alive_progress import alive_bar
from datasets import load_dataset
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import shutil
import regex
import os
import re
import itertools as it
import more_itertools as mit

```
If any of the above-mentioned libraries are not installed in the user's machine, it will be required for 
him to install them through the CMD with the command:

```python
>>> pip install [LIBRARY]

```

To run the code with a given corpus' of data, the following lines of code need to be inserted. If one or multiple
corpora, summarizes, and evaluators are not to be applied, the user has to comment the unwanted option.

```python
if __name__ == "__main__":

    corpora = [
        "mcti_data",
        "cnn_dailymail",
        "big_patent",
        "cnn_corpus_abstractive",
        "cnn_corpus_extractive",
        "xsum",
        "arxiv_pubmed",
    ]

    summarizers = [
        "SumyRandom",
        "SumyLuhn",
        "SumyLsa",
        "SumyLexRank",
        "SumyTextRank",
        "SumySumBasic",
        "SumyKL",
        "SumyReduction",
        "Transformers-facebook/bart-large-cnn",
        "Transformers-google/pegasus-xsum",
        "Transformers-csebuetnlp/mT5_multilingual_XLSum",
    ]

    metrics = [
        "rouge",
        "gensim",
        "nltk",
        "sklearn",
    ]

    ### Running methods and eval locally

    reader = Data()
    reader.show_available_databases()
    for corpus in corpora:
        data = reader.read_data(corpus, 50)
        method = Method(data, corpus)
        method.show_methods()
        for summarizer in summarizers:
            df = method.run(summarizer)
            method.examples_to_csv()
            evaluator = Evaluator(df, summarizer, corpus)
            for metric in metrics:
                evaluator.run(metric)
                evaluator.metrics_to_csv()
            evaluator.join_all_results()
```

### Preprocessing

The preprocessing of given texts is done by the clean_text method present in the Data class. It removes (or normalizes) from the text 
elements that would make it difficult for them to be summarized, such as jump lines, double quotations and apostrophes.

```python
class Data:
    def _clean_text(self, content):
        if isinstance(content, str):
            pass
        else:
            content = str(content)
        # strange jump lines
        content = re.sub(r"\.", ". ", str(content))
        # trouble characters
        content = re.sub(r"\\r\\n", " ", str(content))
        # clean jump lines
        content = re.sub(r"\u000D\u000A|[\u000A\u000B\u000C\u000D\u0085\u2028\u2029]", " ", content)
        # Replace different spaces
        content = re.sub(r"\u00A0\u1680​\u180e\u2000-\u2009\u200a​\u200b​\u202f\u205f​\u3000", " ", content)
        # replace multiple spaces
        content = re.sub(r" +", " ", content)
        # normalize hiphens
        content = regex.sub(r"\p{Pd}+", "-", content)
        # normalize single quotations
        content = re.sub(r"[\u02BB\u02BC\u066C\u2018-\u201A\u275B\u275C]", "'", content)
        # normalize double quotations
        content = re.sub(r"[\u201C-\u201E\u2033\u275D\u275E\u301D\u301E]", '"', content)
        # normalize apostrophes
        content = re.sub(r"[\u0027\u02B9\u02BB\u02BC\u02BE\u02C8\u02EE\u0301\u0313\u0315\u055A\u05F3\u07F4\u07F5\u1FBF\u2018\u2019\u2032\uA78C\uFF07]", "'", content)
        return content
```

There is a preprocessing done specifically in the dataset provided by MCTI, which aims at specifically adjusting the csv file for it to
be properly read. Lines in which data is missing are removed, for example.

```python
with alive_bar(len(texts), title="Processing data") as bar:
  texts = texts.dropna()
  texts[text_key] = texts[text_key].apply(lambda x: " ".join(x.split()))
  texts[title_key] = texts[title_key].apply(lambda x: " ".join(x.split()))              
```

## Datasets

In order to evaluate the model, summaries were generated by each of its summarization methods, which
used as source texts documents achieved from existing datasets. The chosen datasets for evaluation were the following:

- **Scientific Papers (arXiv + PubMed)**: [Cohan et al. (2018)](https://arxiv.org/pdf/1804.05685) found out that there were only
datasets with short texts (with an average of 600 words) or datasets with longer texts with
extractive human summaries. In order to fill the gap and provide a dataset with long text
documents for abstractive summarization, the authors compiled two new datasets with scientific
papers from arXiv and PubMed databases. Scientific papers are specially convenient given the
desired kind of ATS the authors mean to achieve, and that is due to their large length and
the fact that each one contains an abstractive summary made by its author – i.e., the paper’s
abstract.
- **BIGPATENT**: [Sharma et al. (2019)](https://arxiv.org/pdf/1906.03741) introduced the BIGPATENT dataset that provides goods
examples for the task of abstractive summarization. The data dataset is built using Google
Patents Public Datasets, where for each document there is one gold-standard summary which
is the patent’s original abstract. One advantage of this dataset is that it does not present
difficulties inherent to news summarization datasets, where summaries have a flattened discourse
structure and the summary content arises at the beginning of the document.
- **CNN Corpus**: [Lins et al. (2019)](https://par.nsf.gov/servlets/purl/10185297) introduced the corpus in order to fill the gap that most news
summarization single-document datasets have fewer than 1,000 documents. The CNN-Corpus
dataset, thus, contains 3,000 Single-Documents with two gold-standard summaries each: one
extractive and one abstractive. The encompassing of extractive gold-standard summaries is
also an advantage of this particular dataset over others with similar goals, which usually only
contain abstractive ones.
- **CNN/Daily Mail**: [Hermann et al. (2015)](https://proceedings.neurips.cc/paper/2015/file/afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf) intended to develop a consistent method for what
they called ”teaching machines how to read”, i.e., making the machine be able to comprehend a
text via Natural Language Processing techniques. In order to perform that task, they collected
around 400k news from the newspapers CNN and Daily Mail and evaluated what they considered
to be the key aspect in understanding a text, namely the answering of somewhat complex
questions about it. Even though ATS is not the main focus of the authors, they took inspiration
from it to develop their model and include in their dataset the human-made summaries for each
news article.
- **XSum**: [Narayan et al. (2018)](https://arxiv.org/pdf/1808.08745) introduced the single-document dataset, which focuses on a
kind of summarization described by the authors as extreme summarization – an abstractive
kind of ATS that is aimed at answering the question “What is the document about?”. The data
was obtained from BBC articles and each one of them is accompanied by a short gold-standard
summary often written by its very author.

## Evaluation results

Each of the datasets' documents was summarized through every summarization method applied in the code and evaluated 
in comparison with the gold-standard summaries. The following tables provide the results for evaluation methods applied 
to the many datasets:

### Table 1: Scientific Papers (arXiv + PubMed) results
<img src="https://huggingface.co/unb-lamfo-nlp-mcti/NLP-ATS-MCTI/resolve/main/results_arxiv_pubmed.png" width="800">

### Table 2: BIGPATENT results
<img src="https://huggingface.co/unb-lamfo-nlp-mcti/NLP-ATS-MCTI/resolve/main/results_bigpatent.png" width="800">

### Table 3: CNN Corpus results
<img src="https://huggingface.co/unb-lamfo-nlp-mcti/NLP-ATS-MCTI/resolve/main/results_cnn_corpus.png" width="800">

### Table 4: CNN + Daily Mail results
<img src="https://huggingface.co/unb-lamfo-nlp-mcti/NLP-ATS-MCTI/resolve/main/results_cnn_dailymail.png" width="800">

### Table 5: XSum results
<img src="https://huggingface.co/unb-lamfo-nlp-mcti/NLP-ATS-MCTI/resolve/main/results_xsum.png" width="800">

## BibTeX entry and citation info

```bibtex
@misc{https://doi.org/10.48550/arxiv.2301.03403,
  doi = {10.48550/ARXIV.2301.03403},
  
  url = {https://arxiv.org/abs/2301.03403},
  
  author = {Cajueiro, Daniel O. and Nery, Arthur G. and Tavares, Igor and De Melo, Maísa K. and Reis, Silvia A. dos and Weigang, Li and Celestino, Victor R. R.},
  
  keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
```

<a href="https://huggingface.co/exbert/?model=bert-base-uncased">
	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
</a>