mia-project-2025/T5-base-feature-extraction-Xsum

T5-base Fine-tuned on XSum using Feature Extraction Transfer Learning Technique

This repository contains a PyTorch implementation for fine-tuning the flax-community/t5-base-cnn-dm model on the XSum dataset.
The task is extreme summarization — generating a concise one-sentence summary from a full BBC news article.

---

Final Results

After 12 epochs of fine-tuning, the model achieved the following ROUGE scores on the validation set:

• ROUGE-1: 35.28%
• ROUGE-2: 14.18%
• ROUGE-L: 28.91%
• ROUGE-Lsum: 28.91%

Average training loss after the final epoch: 1.2611

---

Dataset

• Name: XSum (EdinburghNLP/xsum)
• Task: Single-sentence abstractive summarization.
• Size: ~226k BBC news articles.
• Splits:
  • Train: 204,045
  • Validation: 11,332
  • Test: 11,334
• Fields:
  • document: news article text (input)
  • summary: single-sentence summary (target)
  • id: article identifier

---

Training Configuration

• Model: flax-community/t5-base-cnn-dm
• Framework: PyTorch + Hugging Face Transformers
• Optimizer: AdamW (learning rate = 5e-5)
• Batch size: 8
• Max input length: 512 tokens
• Max target length: 64 tokens
• Epochs: 12
• Device: CUDA
• Evaluation: ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum

---