flan-t5-base-text-paraphraser

Text paraphraser: google/flan-t5-base fine-tuned (full fine-tuning, standard seq2seq cross-entropy) on the paranmt5m configuration of redis/langcache-sentencepairs-v3 (ParaNMT-5M sentence pairs). No other data was used. Training used a seeded random subset of 1,000,000 pairs out of 5,041,448 usable training pairs.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/flan-t5-base-text-paraphraser"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "paraphrase: the weather was extremely cold yesterday ."
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, num_beams=8, num_return_sequences=3, max_new_tokens=128)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

Input format: paraphrase: <sentence> (prefix is required; also stored in paraphraser_config.json). Default generation config: beam search, num_beams=4, max_new_tokens=128.

Training data

• Source: redis/langcache-sentencepairs-v3, config paranmt5m (single train split, all label=1).
• Direction: sentence1 → sentence2.
• Text is lowercased and PTB-tokenized in the source corpus; it was not re-cased or detokenized.
• Cleaning: null/empty removal, whitespace normalization, PTB quote tokens ( `` / '') mapped to " (the T5 vocabulary has no backtick), control-character / U+FFFD / no-letter filtering, removal of pairs identical after normalization or differing only in case/punctuation, exact pair deduplication, word-length-ratio filter (>3.0 when the longer side has ≥10 words), token-length filter (128 source incl. prefix / 128 target). No truncation.
• Split: deterministic, grouped by normalized source sentence (seed 42); validation/test pairs sharing any normalized sentence with train were moved to train, test pairs sharing a sentence with validation were dropped. Exact pair leakage: none; sentence-level (loose) leakage: none.

Stage	Pairs
Raw paranmt5m train split	5,370,126
− non_positive_label	0
− null_values	0
− non_string_values	0
− empty_strings	0
− malformed_replacement_character	127
− malformed_control_characters	3,012
− malformed_no_letters	399
− source_equals_target_after_normalization	876
− trivial_variant_case_or_punctuation_only	68,851
− exact_duplicate_pairs	157,615
− exceeds_token_limits_128_128	319
− word_length_ratio_above_3.0	17,356
− test pairs overlapping validation (dropped)	18
Usable pairs	5,121,553

Split	Pairs	Share
train	5,041,448	98.44%
validation	39,837	0.78%
test	40,268	0.79%

Training procedure

Setting	Value
Base model	google/flan-t5-base (247,577,856 parameters)
Training examples	1,000,000
Epochs	2.00
Optimizer steps	62,500
Per-device batch × grad. accumulation	32 × 1
Effective batch size	32
Learning rate / schedule	3e-05 / linear, warmup ratio 0.05
Weight decay	0.01
Optimizer	adamw_torch_fused
Precision	bf16 (TF32 on)
Max source / target tokens	128 / 128
Checkpoint selection	best eval_ibleu = 14.8031
Hardware	NVIDIA GeForce RTX 5060 Ti (15.9 GB)
Peak GPU memory (allocated)	12.36 GB
Training time	3.42 h

Evaluation

Generation with beam search (num_beams=4); held-out splits of the processed data. BLEU: sacreBLEU corpus BLEU (13a). ROUGE: rouge_score F-measure, no stemming. BERTScore: roberta-large, baseline-rescaled. Copy rate: output equal to the input after NFKC/whitespace normalization. Distinct-n: corpus unique/total n-grams. iBLEU = 0.8·BLEU(ref) − 0.2·BLEU(source). The copy-source baseline outputs the input unchanged.

Evaluation metrics not found.

Examples (random test samples)

Qualitative examples not found.

Limitations

• Trained on lowercased, PTB-tokenized text; outputs follow that format (lowercase, spaces around punctuation).
• ParaNMT pairs were produced by back-translation; some references are loose or change meaning, and the model inherits this noise.
• Sentence-level model: inputs up to 128 tokens; training sentences have at most ~30 words.
• English only. Not evaluated for factual consistency on long or domain-specific text.
• Automatic metrics against a single reference underestimate valid paraphrases.

Citation

ParaNMT-50M: Wieting & Gimpel, 2018, ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations. FLAN-T5: Chung et al., 2022, Scaling Instruction-Finetuned Language Models.