flan-t5-small-sentence-compression

Full fine-tune of google/flan-t5-small for English sentence compression, trained on sentence-transformers/sentence-compression as a sequence-to-sequence task (text → simplified).

Model

Parameter	Value
Base model	google/flan-t5-small
Architecture	T5 encoder-decoder (T5ForConditionalGeneration)
Parameters	76,961,152
Fine-tuning	Full fine-tuning (no adapters, no quantization)
Input format	compress sentence: {text}
Max source length	256 tokens
Max target length	64 tokens

Data

Parameter	Value
Dataset	sentence-transformers/sentence-compression (train split only)
Mapping	text → simplified
Original examples	180,000
Removed (missing / empty / duplicate pairs)	0 / 0 / 0
Train examples	171,000
Validation examples	9,000
Split	train_test_split(test_size=0.05, seed=42)
Train/validation source overlap	0
Source token length (mean / p99 / max)	38.8 / 81 / 1212
Target token length (mean / p99 / max)	11.2 / 23 / 43
Truncated sources / targets	0.04% / 0.00%

The dataset has no dedicated test split; all reported metrics are on the held-out validation split.

Training

Parameter	Value
Epochs	3
Optimizer	adamw_torch_fused
Learning rate	0.0003
LR schedule	linear, warmup ratio 0.05
Weight decay	0.01
Per-device batch size	32
Gradient accumulation	1
Effective batch size	32
Optimizer steps	16,032
Precision	BF16 mixed precision
Seed	42
Best checkpoint selection	eval_loss = 0.7334
Final train loss	0.8378
GPU	NVIDIA GeForce RTX 5060 Ti (15.9 GB)
Peak VRAM (reserved)	6.52 GB
Training time	0.72 h
Throughput	199.2 samples/s
Software	PyTorch 2.11.0+cu128, CUDA 12.8, Transformers 5.17.0

Evaluation

Validation split, 9,000 examples. Generation: num_beams=4, do_sample=False, max_new_tokens=64, length_penalty=1.0, no_repeat_ngram_size=3. ROUGE is the mean per-example F1 (0-100) with Porter stemming.

Metric	Value
ROUGE-1	77.68
ROUGE-2	63.47
ROUGE-L	77.35
Avg. source length	24.89 words / 34.94 tokens
Avg. reference length	6.96 words / 10.11 tokens
Avg. generated length	7.28 words / 10.44 tokens
Compression ratio (generated/source, words)	0.325
Compression ratio (generated/source, tokens)	0.330
Shorter than source	99.72%
Approximately same length (±10% words)	0.28%
Longer than source	0.00%
Exact copies of source	0.03%

Base model comparison

Same 1,046 validation examples, same input format and generation parameters.

Model	ROUGE-1	ROUGE-2	ROUGE-L	Avg. generated words	Compression ratio (words)
Base google/flan-t5-small	47.11	29.83	46.44	21.14	0.874
Fine-tuned	78.59	64.59	78.29	7.20	0.319

Decoding settings

1,000 validation examples.

Generation parameters	ROUGE-1	ROUGE-2	ROUGE-L	Avg. generated words
do_sample=False, max_new_tokens=64, num_beams=1	78.07	64.07	77.77	6.91
do_sample=False, max_new_tokens=64, num_beams=4	78.60	64.56	78.30	7.19
do_sample=False, max_new_tokens=64, num_beams=4, no_repeat_ngram_size=3	78.60	64.56	78.30	7.19

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/flan-t5-small-sentence-compression"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "Major League Baseball Commissioner Bud Selig will be speaking at St. Norbert College next month."
inputs = tokenizer("compress sentence: " + text, return_tensors="pt", truncation=True, max_length=256)
outputs = model.generate(**inputs, num_beams=4, do_sample=False, max_new_tokens=64, length_penalty=1.0, no_repeat_ngram_size=3)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The compress sentence: prefix is required; the model was trained only with this input format.

Limitations

• English only.
• References in the training data are headline-style compressions of news sentences; outputs follow that style (average 7.3 words).
• Inputs longer than 256 tokens (including the prefix) are truncated.
• ROUGE measures lexical overlap with a single reference; it does not measure factual consistency.