|
--- |
|
tags: |
|
- generated_from_trainer |
|
metrics: |
|
- accuracy |
|
model-index: |
|
- name: PubMedBERT-MNLI-MedNLI |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# PubMedBERT-MNLI-MedNLI |
|
|
|
This model is a fine-tuned version of [PubMedBERT](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext) on the [MNLI](https://huggingface.co/datasets/multi_nli) dataset first and then on the [MedNLI](https://physionet.org/content/mednli/1.0.0/) dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.9501 |
|
- Accuracy: 0.8667 |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
The model can be used for NLI tasks related to biomedical data and even be adapted to fact-checking tasks. It can be used from the Huggingface pipeline method as follows: |
|
|
|
|
|
```python |
|
from transformers import TextClassificationPipeline, AutoModel, AutoTokenizer, AutoModelForSequenceClassification |
|
model = AutoModelForSequenceClassification.from_pretrained("pritamdeka/PubMedBERT-MNLI-MedNLI", num_labels=3, id2label = {1: 'entailment', 0: 'contradiction',2:'neutral'}) |
|
tokenizer = AutoTokenizer.from_pretrained("pritamdeka/PubMedBERT-MNLI-MedNLI") |
|
pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer, return_all_scores=True, device=0, batch_size=128) |
|
|
|
pipe(['ALDH1 expression is associated with better breast cancer outcomes', |
|
'In a series of 577 breast carcinomas, expression of ALDH1 detected by immunostaining correlated with poor prognosis.']) |
|
``` |
|
|
|
The output for the above will be: |
|
|
|
```python |
|
[[{'label': 'contradiction', 'score': 0.10193759202957153}, |
|
{'label': 'entailment', 'score': 0.2933262586593628}, |
|
{'label': 'neutral', 'score': 0.6047361493110657}], |
|
[{'label': 'contradiction', 'score': 0.21726925671100616}, |
|
{'label': 'entailment', 'score': 0.24485822021961212}, |
|
{'label': 'neutral', 'score': 0.5378724932670593}]] |
|
``` |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 32 |
|
- eval_batch_size: 32 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 20.0 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Accuracy | |
|
|:-------------:|:-----:|:----:|:---------------:|:--------:| |
|
| 0.5673 | 1.42 | 500 | 0.4358 | 0.8437 | |
|
| 0.2898 | 2.85 | 1000 | 0.4845 | 0.8523 | |
|
| 0.1669 | 4.27 | 1500 | 0.6233 | 0.8573 | |
|
| 0.1087 | 5.7 | 2000 | 0.7263 | 0.8573 | |
|
| 0.0728 | 7.12 | 2500 | 0.8841 | 0.8638 | |
|
| 0.0512 | 8.55 | 3000 | 0.9501 | 0.8667 | |
|
| 0.0372 | 9.97 | 3500 | 1.0440 | 0.8566 | |
|
| 0.0262 | 11.4 | 4000 | 1.0770 | 0.8609 | |
|
| 0.0243 | 12.82 | 4500 | 1.0931 | 0.8616 | |
|
| 0.023 | 14.25 | 5000 | 1.1088 | 0.8631 | |
|
| 0.0163 | 15.67 | 5500 | 1.1264 | 0.8581 | |
|
| 0.0111 | 17.09 | 6000 | 1.1541 | 0.8616 | |
|
| 0.0098 | 18.52 | 6500 | 1.1542 | 0.8631 | |
|
| 0.0074 | 19.94 | 7000 | 1.1653 | 0.8638 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.22.0.dev0 |
|
- Pytorch 1.12.1+cu113 |
|
- Datasets 2.4.0 |
|
- Tokenizers 0.12.1 |
|
|
|
|
|
## Citing & Authors |
|
|
|
<!--- Describe where people can find more information --> |
|
|
|
If you use the model kindly cite the following work |
|
|
|
``` |
|
@inproceedings{deka-etal-2023-multiple, |
|
title = "Multiple Evidence Combination for Fact-Checking of Health-Related Information", |
|
author = "Deka, Pritam and |
|
Jurek-Loughrey, Anna and |
|
P, Deepak", |
|
booktitle = "The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks", |
|
month = jul, |
|
year = "2023", |
|
address = "Toronto, Canada", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2023.bionlp-1.20", |
|
pages = "237--247", |
|
abstract = "Fact-checking of health-related claims has become necessary in this digital age, where any information posted online is easily available to everyone. The most effective way to verify such claims is by using evidences obtained from reliable sources of medical knowledge, such as PubMed. Recent advances in the field of NLP have helped automate such fact-checking tasks. In this work, we propose a domain-specific BERT-based model using a transfer learning approach for the task of predicting the veracity of claim-evidence pairs for the verification of health-related facts. We also improvise on a method to combine multiple evidences retrieved for a single claim, taking into consideration conflicting evidences as well. We also show how our model can be exploited when labelled data is available and how back-translation can be used to augment data when there is data scarcity.", |
|
} |
|
|
|
``` |
