|
--- |
|
'[object Object]': null |
|
license: apache-2.0 |
|
language: |
|
- en |
|
pipeline_tag: summarization |
|
--- |
|
|
|
# Model Card for Pegasus for Claim Summarization |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
This model can be used to summarize noisy claims on social media into clean and concise claims which can be used for downstream tasks in a fact-checking pipeline. |
|
|
|
# Model Details |
|
|
|
This is the fine-tuned D PEGASUS model with 'No Preprocessing (NP)' detailed in Table 2 in the paper. |
|
This was the best performing model at the time of experimentation. |
|
|
|
## Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **Developed by:** Varad Bhatnagar, Diptesh Kanojia and Kameswari Chebrolu |
|
- **Model type:** Summarization |
|
- **Language(s) (NLP):** English |
|
- **Finetuned from model:** https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4 |
|
|
|
## Model Sources |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** https://github.com/varadhbhatnagar/FC-Claim-Det |
|
- **Paper:** https://aclanthology.org/2022.coling-1.259/ |
|
|
|
## Tokenizer |
|
|
|
Same as https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4 |
|
|
|
|
|
# Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
## Direct Use |
|
|
|
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> |
|
|
|
English to English summarization on noisy fact-checking worthy claims found on social media. |
|
|
|
## Downstream Use |
|
|
|
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app --> |
|
|
|
Can be used for other tasks in a fact-checking pipeline such as claim matching and evidence retrieval. |
|
|
|
# Bias, Risks, and Limitations |
|
|
|
<!-- This section is meant to convey both technical and sociotechnical limitations. --> |
|
|
|
As the [Google Fact Check Explorer](https://toolbox.google.com/factcheck/explorer) is an ever growing and evolving system, the current Retrieval@k results may not exactly match |
|
those in the corresponding paper as those experiments were conducted in the month of April and May 2022. |
|
|
|
# Training Details |
|
|
|
## Training Data |
|
|
|
<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
|
[Data](https://github.com/varadhbhatnagar/FC-Claim-Det/blob/main/public_data/released_data.csv) |
|
|
|
## Training Procedure |
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
Finetuning the pretrained Distilled PEGASUS model on the 567 pairs released in our paper. |
|
|
|
### Preprocessing |
|
|
|
No preprocessing of input is done while fine-tuning this model. |
|
|
|
# Evaluation |
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
Retrieval@5 and Mean Reciprocal Recall scores are reported. |
|
|
|
## Results |
|
|
|
Retrieval@5 = 34.91 |
|
MRR = 0.3 |
|
|
|
Further details can be found in the paper. |
|
|
|
# Other Models from same work |
|
|
|
[DBART](https://huggingface.co/varadhbhatnagar/fc-claim-det-DBART) |
|
|
|
[T5-Base](https://huggingface.co/varadhbhatnagar/fc-claim-det-T5-base) |
|
|
|
# Citation |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
**BibTeX:** |
|
``` |
|
@inproceedings{bhatnagar-etal-2022-harnessing, |
|
title = "Harnessing Abstractive Summarization for Fact-Checked Claim Detection", |
|
author = "Bhatnagar, Varad and |
|
Kanojia, Diptesh and |
|
Chebrolu, Kameswari", |
|
booktitle = "Proceedings of the 29th International Conference on Computational Linguistics", |
|
month = oct, |
|
year = "2022", |
|
address = "Gyeongju, Republic of Korea", |
|
publisher = "International Committee on Computational Linguistics", |
|
url = "https://aclanthology.org/2022.coling-1.259", |
|
pages = "2934--2945", |
|
abstract = "Social media platforms have become new battlegrounds for anti-social elements, with misinformation being the weapon of choice. Fact-checking organizations try to debunk as many claims as possible while staying true to their journalistic processes but cannot cope with its rapid dissemination. We believe that the solution lies in partial automation of the fact-checking life cycle, saving human time for tasks which require high cognition. We propose a new workflow for efficiently detecting previously fact-checked claims that uses abstractive summarization to generate crisp queries. These queries can then be executed on a general-purpose retrieval system associated with a collection of previously fact-checked claims. We curate an abstractive text summarization dataset comprising noisy claims from Twitter and their gold summaries. It is shown that retrieval performance improves 2x by using popular out-of-the-box summarization models and 3x by fine-tuning them on the accompanying dataset compared to verbatim querying. Our approach achieves Recall@5 and MRR of 35{\%} and 0.3, compared to baseline values of 10{\%} and 0.1, respectively. Our dataset, code, and models are available publicly: https://github.com/varadhbhatnagar/FC-Claim-Det/.", |
|
} |
|
``` |
|
|
|
# Model Card Authors |
|
|
|
Varad Bhatnagar |
|
|
|
# Model Card Contact |
|
|
|
Email: varadhbhatnagar@gmail.com |
|
|
|
# How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
``` |
|
from transformers import PegasusForConditionalGeneration, PegasusTokenizerFast |
|
tokeizer = PegasusTokenizerFast.from_pretrained('varadhbhatnagar/fc-claim-det-DPEGASUS') |
|
model = PegasusForConditionalGeneration.from_pretrained('varadhbhatnagar/fc-claim-det-DPEGASUS') |
|
|
|
text ='world health organisation has taken a complete u turn and said that corona patients neither need isolate nor quarantine nor social distance and it can not even transmit from one patient to another' |
|
tokenized_text = tokeizer.encode(text, return_tensors="pt") |
|
|
|
summary_ids = model.generate(tokenized_text, |
|
num_beams=6, |
|
no_repeat_ngram_size=2, |
|
min_length=5, |
|
max_length=15, |
|
early_stopping=True) |
|
|
|
output = tokenizer.decode(summary_ids[0], skip_special_tokens=True) |
|
``` |
