README.md

---
title: Sem-nCG
tags:
- evaluate
- metric
description: "Sem-nCG (Semantic Normalized Cumulative Gain) Metric evaluates the quality of predicted sentences 
(abstractive/extractive) in relation to reference sentences and documents using Semantic Normalized Cumulative Gain 
(NCG). It computes gain values and NCG scores based on cosine similarity between sentence embeddings, leveraging a 
Sentence-BERT encoder. This metric is designed to assess the relevance and ranking of predicted sentences, making it 
useful for tasks such as summarization and information retrieval."
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
authors:
- user: nbansal
---

# Metric Card for Sem-nCG

## Metric Description
Sem-nCG (Semantic Normalized Cumulative Gain) metric evaluates system-generated summaries (`predictions`) by comparing 
them with ground truth reference summaries (`references`) and input documents (`documents`). It computes the Semantic 
Normalized Cumulative Gain (NCG) scores based on sentence embeddings, which assess the quality of summaries by 
evaluating the relevance of predicted sentences to the reference and input document sentences.


## How to Use

Before using this metric, you need to install the dependencies:
```bash
pip install -U evaluate sentence-transformers nltk
```

#### Python Usage
```python
from evaluate import load
predictions = [
    "This is a prediction1 sentence 1. This is a prediction1 sentence 2.", 
    "This is a prediction2 sentence 1."
]
references = [
    "This is a reference1 sentence 1. This is a reference1 sentence 2.",
    "This is a reference2 sentence 1. This is a reference2 sentence 2."
]
documents = [
    "This is a document1 sentence 1. This is a document1 sentence 2. This is a document1 sentence 3.",
    "This is a document2 sentence 1. This is a document2 sentence 2."
]
model_name = "all-MiniLM-L6-v2"
metric = load("nbansal/semncg", model_name=model_name)  # model_name is optional. Default=all-MiniLM-L6-v2
mean_score, scores = metric.compute(predictions=predictions, references=references, documents=documents)
print(f"Mean SemnCG: {mean_score}")
```

First step is to initialize the metric as `metric = load("nbansal/semncg", model_name=model_name)` where `model_name` is 
the sentence embedding model. The default value is `all-MiniLM-L6-v2`.

To `compute` the Sem-nCG scores, you need to provide three mandatory arguments:
 - `predictions` - List of predictions
 - `references` - List of references
 - `documents` - List of input documents

Sem-nCG also accepts several optional arguments:
 - `tokenize_sentences (bool)`: Flag to indicate whether to tokenize the sentences in the input documents. Default: True
 - `pre_compute_embeddings (bool)`: Flag to indicate whether to pre-compute embeddings for all sentences. Default=False
 - `k (int)`: The rank threshold used for evaluating gains (typically top-k sentences). Default is 3.
 - `gpu (Union[bool, str, int, List[Union[str, int]]])`: Whether to use GPU, CPU, or multiple processes for computation.
 - `batch_size (int)`: Batch size for encoding. Default is 32.
 - `verbose (bool)`: Flag to indicate verbose output. Default is False.
 - `debug (bool)`: Flag to return detailed debug information including ranked gains. Default is False.

Refer to the inputs descriptions for more detailed usage as follows:
```python
import evaluate
metric = evaluate.load("nbansal/semncg")
print(metric.inputs_description)
```

### Output Values

The output is a tuple containing:

- Mean Sem-nCG score: float: The average Sem-nCG score. 
- scores: List[Union[float, RankedGains]]: List of Sem-nCG scores or RankedGains objects for each document.


## Extensions
The current implementation supports any model from Huggingface/SentenceTransformer that is compatible with 
SentenceTransformer, such as `all-mpnet-base-v2` or `roberta-base`. You can extend the metric with more models by 
extending the `Encoder` base class in the `encoder_models.py` file.

## Deviations from Published Methodology
In our implementation, we expand upon the methodology presented in the original paper, which focused solely on 
extractive model summaries. The primary approach in the paper involved ranking sentences in the source document based on
ground-truth reference sentences. The Normalized Cumulative Gain (NCG) score was computed using the formula:

```ncg = $\frac{\text{cumulative gain}}{\text{ideal cumulative gain}}$```

Key deviations in our implementation from the paper include:
1. **Inclusion of Abstractive Model Summaries:** Unlike the paper, which exclusively considered extractive model 
summaries, our implementation supports both extractive and abstractive summarization models. 
2. **Enhanced Calculation of NCG Scores:** For both extractive and abstractive summaries, we compute rankings based on 
both the reference/ground truth (`gt_gain`) and predicted summaries (`pred_gain`). The NCG score is calculated using the
method shown below:
```python
def compute_ncg(pred_gains, gt_gains, k: int) -> float:
    gt_dict = dict(gt_gains)
    gt_rel = [v for _, v in gt_gains[:k]]
    model_rel = [gt_dict[position] for position, _ in pred_gains[:k]]
    return sum(model_rel)/sum(gt_rel)
```
 
This approach allows us to evaluate summarization quality across both extractive and abstractive methods, providing a 
more comprehensive assessment than the original methodology.


## Citation
```bibtex
@inproceedings{akter-etal-2022-revisiting,
    title = "Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than {ROUGE}?",
    author = "Akter, Mousumi  and
      Bansal, Naman  and
      Karmaker, Shubhra Kanti",
    editor = "Muresan, Smaranda  and
      Nakov, Preslav  and
      Villavicencio, Aline",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.findings-acl.122",
    doi = "10.18653/v1/2022.findings-acl.122",
    pages = "1547--1560",
    abstract = "It has been the norm for a long time to evaluate automated summarization tasks using the popular ROUGE metric. Although several studies in the past have highlighted the limitations of ROUGE, researchers have struggled to reach a consensus on a better alternative until today. One major limitation of the traditional ROUGE metric is the lack of semantic understanding (relies on direct overlap of n-grams). In this paper, we exclusively focus on the extractive summarization task and propose a semantic-aware nCG (normalized cumulative gain)-based evaluation metric (called Sem-nCG) for evaluating this task. One fundamental contribution of the paper is that it demonstrates how we can generate more reliable semantic-aware ground truths for evaluating extractive summarization tasks without any additional human intervention. To the best of our knowledge, this work is the first of its kind. We have conducted extensive experiments with this new metric using the widely used CNN/DailyMail dataset. Experimental results show that the new Sem-nCG metric is indeed semantic-aware, shows higher correlation with human judgement (more reliable) and yields a large number of disagreements with the original ROUGE metric (suggesting that ROUGE often leads to inaccurate conclusions also verified by humans).",
}
```

## Further References
 - [Paper](https://aclanthology.org/2022.findings-acl.122/)
 - [Video](https://underline.io/lecture/50182-findings-revisiting-automatic-evaluation-of-extractive-summarization-task-can-we-do-better-than-rougequestion)