File size: 7,207 Bytes
4695532
f036197
 
 
 
4695532
47f92c7
4695532
 
90d2f63
d9bff68
 
 
90d2f63
d9bff68
 
 
aac1aaa
 
4695532
 
dfd7508
f036197
 
dfd7508
90d2f63
 
f036197
 
42c888f
5c29286
 
955a39f
5c29286
 
f9cbf95
90d2f63
 
 
935fa6b
 
90d2f63
 
935fa6b
 
90d2f63
6ca6a6d
90d2f63
42c888f
 
90d2f63
f036197
3d8a6b1
f9cbf95
 
 
 
 
42c888f
251bfda
4f33285
2cc2c31
 
3116618
 
 
 
 
42c888f
6ca6a6d
 
42c888f
 
6ca6a6d
 
42c888f
dfd7508
 
 
 
 
f036197
 
42c888f
680431d
42c888f
 
 
f036197
dfd7508
251bfda
42c888f
251bfda
 
 
 
 
 
f036197
42c888f
 
431fa5b
dfd7508
431fa5b
dfd7508
f036197
dfd7508
f036197
dfd7508
f036197
dfd7508
f036197
 
90d2f63
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f036197
 
90d2f63
dfd7508
23122f2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
---
title: SemF1
tags:
- evaluate
- metric
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
description: >-
  SEM-F1 metric leverages the pre-trained contextual embeddings and evaluates
  the model generated semantic overlap  summary with the reference overlap
  summary. It evaluates the semantic overlap summary at the sentence level and 
  computes precision, recall and F1 scores.

  Refer to the paper `SEM-F1: an Automatic Way for Semantic Evaluation of
  Multi-Narrative Overlap Summaries at Scale`  for more details. 
authors:
- user: nbansal
---

# Metric Card for Sem-F1

## Metric Description
Sem-F1 metric leverages the pre-trained contextual embeddings and evaluates the model generated semantic overlap 
summary with the reference overlap summary. It evaluates the semantic overlap summary at the sentence level and 
computes precision, recall and F1 scores.

## How to Use

Before using this metric, you need to install the dependencies - 
```bash
pip install -U evaluate sentence-transformers scikit-learn nltk
```

#### Python Usage
```python
from evaluate import load
predictions = [
    "I go to School. You are stupid.",
    "I love adventure sports.",
]
references = [
    "I go to School. You are stupid.",
    "I love outdoor sports.",
]
metric = load("nbansal/semf1")
results = metric.compute(predictions=predictions, references=references)
for score in results:
    print(f"Precision: {score.precision}, Recall: {score.recall}, F1: {score.f1}")
```


Sem-F1 takes 2 mandatory arguments:  

- `predictions` - List of predictions. Format varies based on `tokenize_sentences` and `multi_references` flags.
- `references`: List of references. Format varies based on `tokenize_sentences` and `multi_references` flags.

Sem-F1 also accepts multiple optional arguments:


- `model_type (str)`: Model to use for encoding sentences. Options: ['pv1' ([paraphrase-distilroberta-base-v1](https://huggingface.co/sentence-transformers/paraphrase-distilroberta-base-v1)), 'stsb' ([stsb-roberta-large](https://huggingface.co/sentence-transformers/stsb-roberta-large)), 'use' ([Universal Sentence Encoder](https://huggingface.co/sentence-transformers/use-cmlm-multilingual)) (Default)]. Furthermore, you can use any model on Huggingface/SentenceTransformer that is supported by SentenceTransformer 
  such as `all-mpnet-base-v2` or `roberta-base`.
- `tokenize_sentences (bool)`: Flag to indicate whether to tokenize the sentences in the input documents. Default: True.
- `multi_references (bool)`: Flag to indicate whether multiple references are provided. Default: False.
- `gpu (Union[bool, str, int, List[Union[str, int]]])`: Whether to use GPU, CPU or multiple-processes for computation.
- `batch_size (int)`: Batch size for encoding. Default: 32.
- `verbose (bool)`: Flag to indicate verbose output. Default: False.

Refer to the inputs descriptions for more detailed usage as follows:

```python
import evaluate
metric = evaluate.load("nbansal/semf1")
print(metric.inputs_description)
```


[//]: # (*List all input arguments in the format below*)

[//]: # (- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*)

### Output Values
List of `Scores` dataclass corresponding to each sample - 

 - `precision: float`: Precision score, which ranges from 0.0 to 1.0.
 - `recall: List[float]`: Recall score corresponding to each reference
 - `f1: float`: F1 score (between precision and average recall).


## Extensions
Currently, we have only implemented the 3 encoders* that we experimented with in our 
[paper](https://aclanthology.org/2022.emnlp-main.49/). Furthermore, you can use any model on 
Huggingface/SentenceTransformer that is supported by SentenceTransformer such as `all-mpnet-base-v2` or `roberta-base`.

If you want to use your own encoder model, either make sure that is supported by `SentenceTransformer`. Or if it's a 
completely new architecture, it can easily with extended for more models by extending the `Encoder` base class (Refer to
`encoder_models.py` file).

`*` *In out paper, we used the Tensorflow [version](https://www.tensorflow.org/hub/tutorials/semantic_similarity_with_tf_hub_universal_encoder)
of the USE model, however, in our current implementation, we used [PyTorch version](https://huggingface.co/sentence-transformers/use-cmlm-multilingual).* 

[//]: # (*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*)

[//]: # (### Examples)

[//]: # (*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*)

[//]: # (## Limitations and Bias)

[//]: # (*Note any known limitations or biases that the metric has, with links and references if possible.*)

## Citation
```bibtex
@inproceedings{bansal-etal-2022-sem,
    title = "{SEM}-F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale",
    author = "Bansal, Naman  and
      Akter, Mousumi  and
      Karmaker Santu, Shubhra Kanti",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-main.49",
    doi = "10.18653/v1/2022.emnlp-main.49",
    pages = "780--792",
    abstract = "Recent work has introduced an important yet relatively under-explored NLP task called Semantic Overlap Summarization (SOS) that entails generating a summary from multiple alternative narratives which conveys the common information provided by those narratives. Previous work also published a benchmark dataset for this task by collecting 2,925 alternative narrative pairs from the web and manually annotating 411 different reference summaries by engaging human annotators. In this paper, we exclusively focus on the automated evaluation of the SOS task using the benchmark dataset. More specifically, we first use the popular ROUGE metric from text-summarization literature and conduct a systematic study to evaluate the SOS task. Our experiments discover that ROUGE is not suitable for this novel task and therefore, we propose a new sentence-level precision-recall style automated evaluation metric, called SEM-F1 (Semantic F1). It is inspired by the benefits of the sentence-wise annotation technique using overlap labels reported by the previous work. Our experiments show that the proposed SEM-F1 metric yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.",
}
```

## Further References
 - [Paper](https://aclanthology.org/2022.emnlp-main.49/)
 - [Presentation Slides](https://auburn.box.com/s/rs5p7sttaonbvljnq0i5tk7xxw0vonn3)
 - [Video](https://youtu.be/f8wrtOd-2tc)