|
--- |
|
language: en |
|
tags: |
|
- summarization |
|
- medical |
|
library_name: transformers |
|
pipeline_tag: summarization |
|
--- |
|
|
|
# Automatic Personalized Impression Generation for PET Reports Using Large Language Models ๐โ |
|
|
|
**Authored by**: Xin Tie, Muheon Shin, Ali Pirasteh, Nevein Ibrahim, Zachary Huemann, Sharon M. Castellino, Kara Kelly, John Garrett, Junjie Hu, Steve Y. Cho, Tyler J. Bradshaw |
|
|
|
## ๐ Model Description |
|
|
|
This is the fine-tuned PEGASUS model for summarizing findings in PET reports. |
|
|
|
To check our fine-tuned large language models (LLMs) for PET report summarization: |
|
- [BERT2BERT-PET](https://huggingface.co/xtie/Clinicallongformer2roberta-PET-impression) |
|
- [BART-PET](https://huggingface.co/xtie/BART-PET-impression) |
|
- [BioBART-PET](https://huggingface.co/xtie/BioBART-PET-impression) |
|
- [PEGASUS-PET](https://huggingface.co/xtie/PEGASUS-PET-impression) |
|
- [T5v1.1-PET](https://huggingface.co/xtie/T5v1.1-PET-impression) |
|
- [Clinical-T5-PET](https://huggingface.co/xtie/ClinicalT5-PET-impression) |
|
- [Flan-T5-PET](https://huggingface.co/xtie/Flan-T5-PET-impression) |
|
- [GPT2-XL-PET](https://huggingface.co/xtie/GPT2-PET-impression) |
|
- [OPT-1.3B-PET](https://huggingface.co/xtie/OPT-PET-impression) |
|
- [LLaMA-LoRA-PET](https://huggingface.co/xtie/LLaMA-LoRA-PET-impression) |
|
- [Alpaca-LoRA-PET](https://huggingface.co/xtie/Alpaca-LoRA-PET-impression) |
|
|
|
## ๐ Abstract |
|
|
|
Purpose: To determine if fine-tuned large language models (LLMs) can generate accurate, personalized impressions for whole-body PET reports. |
|
|
|
Materials and Methods: Twelve language models were trained on a corpus of PET reports using the teacher-forcing algorithm, with the report findings as input and the clinical impressions as reference. An extra input token encodes the reading physicianโs identity, allowing models to learn physician-specific reporting styles. Our corpus comprised 37,370 retrospective PET reports collected from our institution between 2010 and 2022. To identify the best LLM, 30 evaluation metrics were benchmarked against quality scores from two nuclear medicine (NM) physicians, with the most aligned metrics selecting the model for expert evaluation. In a subset of data, model-generated impressions and original clinical impressions were assessed by three NM physicians according to 6 quality dimensions and an overall utility score (5-point scale). Each physician reviewed 12 of their own reports and 12 reports from other physicians. Bootstrap resampling was used for statistical analysis. |
|
|
|
Results: Of all evaluation metrics, domain-adapted BARTScore and PEGASUSScore showed the highest Spearmanโs ฯ correlations (0.568 and 0.563) with physician preferences. Based on these metrics, the fine-tuned PEGASUS model was selected as the top LLM. When physicians reviewed PEGASUS-generated impressions in their own style, 89% were considered clinically acceptable, with a mean utility score of 4.08/5. Physicians rated these personalized impressions as comparable in overall utility to the impressions dictated by other physicians (4.03, P=0.41). |
|
|
|
Conclusion: Personalized impressions generated by PEGASUS were clinically useful, highlighting its potential to expedite PET reporting. |
|
|
|
[Read the full paper](https://arxiv.org/abs/2309.10066) |
|
<!-- Link to our Arxiv paper --> |
|
|
|
## ๐ Usage |
|
|
|
```bash |
|
finetuned_model = "xtie/PEGASUS-PET-impression" |
|
tokenizer = AutoTokenizer.from_pretrained(finetuned_model) |
|
model = AutoModelForSeq2SeqLM.from_pretrained(finetuned_model, ignore_mismatched_sizes=True).eval() |
|
|
|
findings_info = |
|
""" |
|
Description: PET CT WHOLE BODY |
|
Radiologist: James |
|
Findings: |
|
Head/Neck: xxx Chest: xxx Abdomen/Pelvis: xxx Extremities/Musculoskeletal: xxx |
|
Indication: |
|
The patient is a 60-year old male with a history of xxx |
|
""" |
|
|
|
inputs = tokenizer(findings_info.replace('\n', ' '), |
|
padding="max_length", |
|
truncation=True, |
|
max_length=1024, |
|
return_tensors="pt") |
|
input_ids = inputs.input_ids.to("cuda") |
|
attention_mask = inputs.attention_mask.to("cuda") |
|
outputs = model.generate(input_ids, |
|
attention_mask=attention_mask, |
|
max_new_tokens=512, |
|
num_beam_groups=1, |
|
num_beams=4, |
|
do_sample=False, |
|
diversity_penalty=0.0, |
|
num_return_sequences=1, |
|
length_penalty=2.0, |
|
no_repeat_ngram_size=3, |
|
early_stopping=True |
|
) |
|
# get the generated impressions |
|
output_str = tokenizer.decode(outputs, |
|
skip_special_tokens=True) |
|
``` |
|
|
|
|
|
### ๐ Performance Metrics |
|
|
|
For detailed evaluation results, please refer to our paper. |
|
- **ROUGE-1**: 53.8 |
|
- **ROUGE-2**: 30.9 |
|
- **ROUGE-L**: 40.0 |
|
- **BLEU**: 24.7 |
|
- **BERTScore**: 0.744 |
|
|
|
### ๐ก Highlights |
|
|
|
- The fine-tuned large language model provides clinically useful, personalized impressions based on PET findings. |
|
- To our knowledge, this is the first attempt to automate impression generation for whole-body PET reports. |
|
|
|
### ๐ฅ๏ธ Hardware |
|
|
|
The models were trained on NVIDIA A100 GPUs. |
|
|
|
--- |
|
|
|
## ๐ Additional Resources |
|
- **Finetuned from model:** [Google's PEGASUS Large](https://huggingface.co/google/pegasus-large) |
|
- **Codebase for training and inference:** [GitHub Repository](https://github.com/xtie97/PET-Report-Summarization) |
