File size: 6,345 Bytes
faed0cf
5db2699
 
 
 
 
faed0cf
 
 
 
 
 
 
 
 
 
b693c4e
faed0cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5db2699
 
 
 
 
faed0cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5db2699
faed0cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6d7b7a7
 
 
2e4d776
6d7b7a7
faed0cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6d7b7a7
 
 
 
 
 
5db2699
6d7b7a7
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
---
'[object Object]': null
license: apache-2.0
language:
- en
pipeline_tag: summarization
---

# Model Card for Pegasus for Claim Summarization

<!-- Provide a quick summary of what the model is/does. -->

This model can be used to summarize noisy claims on social media into clean and concise claims which can be used for downstream tasks in a fact-checking pipeline.

# Model Details

This is the fine-tuned D PEGASUS model with 'No Preprocessing (NP)' detailed in Table 2 in the paper.
This was the best performing model at the time of experimentation.

## Model Description

<!-- Provide a longer summary of what this model is. -->

- **Developed by:** Varad Bhatnagar, Diptesh Kanojia and Kameswari Chebrolu
- **Model type:** Summarization
- **Language(s) (NLP):** English
- **Finetuned from model:** https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4

## Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** https://github.com/varadhbhatnagar/FC-Claim-Det
- **Paper:** https://aclanthology.org/2022.coling-1.259/

## Tokenizer

Same as https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4


# Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

## Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

English to English summarization on noisy fact-checking worthy claims found on social media.

## Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

Can be used for other tasks in a fact-checking pipeline such as claim matching and evidence retrieval.

# Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

As the [Google Fact Check Explorer](https://toolbox.google.com/factcheck/explorer) is an ever growing and evolving system, the current Retrieval@k results may not exactly match
those in the corresponding paper as those experiments were conducted in the month of April and May 2022.

# Training Details

## Training Data

<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[Data](https://github.com/varadhbhatnagar/FC-Claim-Det/blob/main/public_data/released_data.csv)

## Training Procedure 
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
Finetuning the pretrained Distilled PEGASUS model on the 567 pairs released in our paper.

### Preprocessing

No preprocessing of input is done while fine-tuning this model.

# Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
Retrieval@5 and Mean Reciprocal Recall scores are reported.

## Results

Retrieval@5 = 34.91
MRR = 0.3

Further details can be found in the paper.

# Other Models from same work

[DBART](https://huggingface.co/varadhbhatnagar/fc-claim-det-DBART)

[T5-Base](https://huggingface.co/varadhbhatnagar/fc-claim-det-T5-base)

# Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**
```
@inproceedings{bhatnagar-etal-2022-harnessing,
    title = "Harnessing Abstractive Summarization for Fact-Checked Claim Detection",
    author = "Bhatnagar, Varad  and
      Kanojia, Diptesh  and
      Chebrolu, Kameswari",
    booktitle = "Proceedings of the 29th International Conference on Computational Linguistics",
    month = oct,
    year = "2022",
    address = "Gyeongju, Republic of Korea",
    publisher = "International Committee on Computational Linguistics",
    url = "https://aclanthology.org/2022.coling-1.259",
    pages = "2934--2945",
    abstract = "Social media platforms have become new battlegrounds for anti-social elements, with misinformation being the weapon of choice. Fact-checking organizations try to debunk as many claims as possible while staying true to their journalistic processes but cannot cope with its rapid dissemination. We believe that the solution lies in partial automation of the fact-checking life cycle, saving human time for tasks which require high cognition. We propose a new workflow for efficiently detecting previously fact-checked claims that uses abstractive summarization to generate crisp queries. These queries can then be executed on a general-purpose retrieval system associated with a collection of previously fact-checked claims. We curate an abstractive text summarization dataset comprising noisy claims from Twitter and their gold summaries. It is shown that retrieval performance improves 2x by using popular out-of-the-box summarization models and 3x by fine-tuning them on the accompanying dataset compared to verbatim querying. Our approach achieves Recall@5 and MRR of 35{\%} and 0.3, compared to baseline values of 10{\%} and 0.1, respectively. Our dataset, code, and models are available publicly: https://github.com/varadhbhatnagar/FC-Claim-Det/.",
}
```

# Model Card Authors

Varad Bhatnagar

# Model Card Contact

Email: varadhbhatnagar@gmail.com

# How to Get Started with the Model

Use the code below to get started with the model.

```
from transformers import PegasusForConditionalGeneration, PegasusTokenizerFast
tokeizer = PegasusTokenizerFast.from_pretrained('varadhbhatnagar/fc-claim-det-DPEGASUS')
model = PegasusForConditionalGeneration.from_pretrained('varadhbhatnagar/fc-claim-det-DPEGASUS')

text ='world health organisation has taken a complete u turn and said that corona patients neither need isolate nor quarantine nor social distance and it can not even transmit from one patient to another'
tokenized_text = tokeizer.encode(text, return_tensors="pt")

summary_ids = model.generate(tokenized_text,
                                  num_beams=6,
                                  no_repeat_ngram_size=2,
                                  min_length=5,
                                  max_length=15,
                                  early_stopping=True)

output = tokenizer.decode(summary_ids[0], skip_special_tokens=True) 
```