README.md

---
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: PegasusMedicalSummary
  results: []
widget:
- text: "PREOPERATIVE DIAGNOSIS: Chronic obstructive pulmonary disease (COPD).POSTOPERATIVE DIAGNOSIS: COPD.PROCEDURE: Bilateral video-assisted thoracoscopic lung volume reduction surgery (LVRS).ANESTHESIA: General anesthesia with single-lumen endotracheal tube.INDICATIONS FOR PROCEDURE: This 65-year-old female patient presented with severe COPD symptoms, including dyspnea and decreased exercise tolerance. After thorough evaluation and discussions of available treatment options, the decision for bilateral LVRS was made in order to improve lung function and quality of life.PROCEDURE IN DETAIL: Informed consent was obtained after explaining the risks and benefits of the procedure. The patient was placed in a lateral decubitus position, and general anesthesia was induced. Bilateral LVRS was performed using video-assisted thoracoscopic techniques. Intraoperatively, attention was given to minimize bleeding and ensure proper lung tissue removal. The patient tolerated the procedure well, and postoperative care instructions were provided."
  example_title: "Example 1"
- text: 'PREOPERATIVE DIAGNOSIS: Coronary artery disease.POSTOPERATIVE DIAGNOSIS: Coronary artery disease.PROCEDURE: Coronary artery bypass grafting (CABG) surgery.ANESTHESIA: General anesthesia with cardiopulmonary bypass.INDICATIONS FOR PROCEDURE: This 60-year-old male patient presented with significant coronary artery disease, with multiple vessels showing significant stenosis on angiography. After a thorough evaluation of his condition and considering the extent of the disease, the decision was made to proceed with CABG surgery to improve blood flow to the heart muscle.PROCEDURE IN DETAIL: After obtaining informed consent and ensuring adequate preoperative preparations, the patient was brought to the operating room. General anesthesia was induced, and cardiopulmonary bypass was established. The bypass grafts were harvested, and the stenotic coronary arteries were bypassed using appropriate grafts. Hemostasis was ensured, and the patient was weaned off cardiopulmonary bypass. The patient was transferred to the intensive care unit for postoperative monitoring and recovery. Postoperative care instructions were provided to the patient and family members.'
  example_title: "Example 2"
- text: 'PREOPERATIVE DIAGNOSIS: Lumbar disc herniation.POSTOPERATIVE DIAGNOSIS: Lumbar disc herniation.PROCEDURE: Minimally invasive lumbar microdiscectomy.ANESTHESIA: General anesthesia with endotracheal intubation.INDICATIONS FOR PROCEDURE: This 42-year-old male patient presented with radiating low back pain and leg numbness, along with positive imaging findings of a lumbar disc herniation. After conservative treatment failed to provide relief, the decision was made to proceed with a minimally invasive microdiscectomy to alleviate the symptoms.PROCEDURE IN DETAIL: The patient was positioned prone on the operating table, and general anesthesia was administered. A small incision was made, and using fluoroscopic guidance, the herniated disc material was carefully removed. The surgical site was inspected for any bleeding or complications before closure. The patient was awakened from anesthesia without any immediate postoperative complications. Postoperative instructions were given regarding activity restrictions and pain management.'
  example_title: "Example 3"
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# PegasusMedicalSummary

### Authors
This model was created by [mereshd](https://huggingface.co/mereshd), [renegarza](https://huggingface.co/renegarza) and [jasmeeetsingh](https://huggingface.co/jasmeeetsingh).

This model is a fine-tuned version of [google/pegasus-xsum](https://huggingface.co/google/pegasus-xsum) on the [MTSamples](https://mtsamples.com/) dataset.
It achieves the following results on the evaluation set:
- Loss: 0.1438
- Rouge1: 0.4318
- Rouge2: 0.2525
- Rougel: 0.3524
- Rougelsum: 0.3525
- Gen Len: 55.882

### Project Purpose

Our goal is to deliver an effective summarization solution aimed at making doctor discharge notes more structured and comprehensive.
A physician's job goes far beyond saving lives, doctors are also responsible for providing a comforting environment for their patients.
With that in mind, while accommodating in a high-stress environment it is difficult to follow a structure and formulate notes with universal interpretability in mind.
This leads to long and convoluted discharge documentation that becomes very tedious to leverage and understand. Our model is a product that will alleviate 
a significant amount of discomfort when creating and utilizing physician notes, which ultimately will lead to more fluid workflows and increased convenience for healthcare providers.

### Intended Use
#### Model
We leveraged Google's Pegasus abstractive text summarization to generate summaries of the discharged transcriptions included in the MTSamples dataset. 
This was later utilized to prompt the Transformer's Masked Language Modeling(MLM) functionality to train the model to generate meaningful text with better structure and organization than the original.

#### Use Cases
This model allows for the efficient summarization of complexly documented doctor notes. It provides instant access to insight with proper semantic cues in place. Additionally, Data Engineers that work with patient electronic records consistently spend an excessive amount of time parsing through the unstructured discharge notes format to accomplish their tasks.
The solution will be instrumental for agents who are not directly facing patients but hold back-end roles that are also of immense importance.

#### Limitations & Future Aspirations
With an increased amount of data, more deliberate results might be achieved through more training. Synthetic transcriptions could be created with GPT models to in turn train on. Also, further improvements on the model's summarization capabilities have been considered.
One of which is implementing summarization based on clustered titles within the discharge notes. The feature would allow for easier traversal through partitioned summarization and result in better structure.

#### Training and evaluation data
The generated summaries were assigned to the original transcription and after splitting the data into the train and test sets, the table was converted into a json file.
The structure allowed us to effectively train the model on the premise of transcription to summarization prompts. After all the metrics were evaluated, a number of medical transcriptions were generated through
generative transformers to summarize and upon testing the model performed well.  

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 4
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:-------:|
| 6.5172        | 1.0   | 999  | 0.1784          | 0.4161 | 0.2373 | 0.3388 | 0.3384    | 52.102  |
| 0.3174        | 2.0   | 1999 | 0.1550          | 0.4236 | 0.2434 | 0.343  | 0.3428    | 54.458  |
| 0.2632        | 3.0   | 2999 | 0.1462          | 0.4269 | 0.2467 | 0.3465 | 0.3464    | 55.503  |
| 0.2477        | 4.0   | 3996 | 0.1438          | 0.4318 | 0.2525 | 0.3524 | 0.3525    | 55.882  |


### Framework versions

- Transformers 4.28.1
- Pytorch 2.0.0+cu117
- Datasets 2.11.0
- Tokenizers 0.13.3