LED-Large for Amazon All_Beauty Multi-Review Summarization
A fine-tuned allenai/led-large-16384 model for product-level multi-review summarization in the Amazon Reviews 2023 All_Beauty category.
The model takes multiple customer reviews for one product and generates a single abstractive summary that captures the main review consensus, recurring strengths, and meaningful complaints.
Input: product title + multiple review blocks
Output: one product-level summary
The product's numeric rating is not generated by the model. It is computed separately from the original review ratings.
Quick Start
Fastest option: open
AllBeauty_LED_Large_Inference.ipynbif you want a ready-to-run example without reading the full training notebook.
1. Install dependencies
pip install -U transformers torch sentencepiece
2. Load the model
import torch
from transformers import AutoTokenizer, LEDForConditionalGeneration
MODEL_ID = "vltruong01/amazon-all-beauty-led-large-summarization"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = LEDForConditionalGeneration.from_pretrained(
MODEL_ID,
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()
3. Prepare the input
The model was trained on inputs with this structure:
Product: Example Beauty Product
[Review 1]
Rating: 5/5
Title: Works very well
Text: I have used this product for several weeks and it works well...
[Review 2]
Rating: 4/5
Title: Good but expensive
Text: The quality is good, although the price is higher than expected...
[Review 3]
Rating: 2/5
Title: Not for me
Text: I experienced irritation and stopped using it...
Keep each review as a complete block whenever possible.
4. Generate a summary
LED uses sparse attention. The inference code below reproduces the global-attention pattern used in training: the first token, the product-title line, each review header, and each rating line receive global attention.
import re
import torch
MAX_SOURCE_LENGTH = 4096
def build_global_attention_mask(source, input_ids, attention_mask, tokenizer):
global_attention = torch.zeros_like(input_ids)
# Always give global attention to the first non-padding token.
global_attention[:, 0] = 1
lines = source.splitlines()
important_prefixes = ("Product:", "[Review ", "Rating:")
important_texts = [
line.strip()
for line in lines
if line.strip().startswith(important_prefixes)
]
for text in important_texts:
token_ids = tokenizer(
text,
add_special_tokens=False,
)["input_ids"]
if not token_ids:
continue
first_token = token_ids[0]
matches = (
(input_ids[0] == first_token)
& (attention_mask[0] == 1)
).nonzero(as_tuple=False)
for match in matches:
global_attention[0, match.item()] = 1
return global_attention
source = '''Product: Example Beauty Product
[Review 1]
Rating: 5/5
Title: Works very well
Text: I have used this product for several weeks and it works well.
[Review 2]
Rating: 4/5
Title: Good but expensive
Text: The quality is good, although the price is higher than expected.
[Review 3]
Rating: 2/5
Title: Not for me
Text: I experienced irritation and stopped using it.
'''
inputs = tokenizer(
source,
return_tensors="pt",
max_length=MAX_SOURCE_LENGTH,
truncation=True,
padding=False,
)
input_ids = inputs["input_ids"].to(device)
attention_mask = inputs["attention_mask"].to(device)
global_attention_mask = build_global_attention_mask(
source,
input_ids,
attention_mask,
tokenizer,
).to(device)
with torch.inference_mode():
generated = model.generate(
input_ids=input_ids,
attention_mask=attention_mask,
global_attention_mask=global_attention_mask,
num_beams=6,
max_new_tokens=192,
no_repeat_ngram_size=3,
repetition_penalty=1.05,
early_stopping=True,
)
summary = tokenizer.decode(
generated[0],
skip_special_tokens=True,
)
print(summary)
Inference Notebook
If you only want to use the trained model, open:
AllBeauty_LED_Large_Inference.ipynb
The notebook is intentionally much shorter than the training notebook and contains only the steps required for inference:
install dependencies
↓
load tokenizer + fine-tuned LED-Large
↓
paste one product's review collection
↓
build structured global attention
↓
generate one product-level summary
↓
optionally compute the numeric rating separately
The inference notebook reproduces the same structural global-attention pattern used during training.
Global attention is assigned to:
- the first source token,
- the
Product:line, - every
[Review N]header, - every
Rating:line.
This matters because LED uses sparse attention for long inputs. These structural anchors help the model connect product identity, review boundaries, and ratings across a long multi-review document.
The notebook also keeps the numeric product rating outside the model. LED generates only the summary.
What the Model Does
The model performs multi-review aggregation, not single-review summarization.
It is intended to learn patterns such as:
- what customers most consistently agree on,
- recurring positive experiences,
- recurring negative experiences,
- meaningful minority complaints,
- the overall product-level impression.
The model should not simply concatenate or extract individual review sentences.
Dataset
Training data comes from:
vltruong01/amazon-all-beauty-led-summarization
The dataset was derived from Amazon Reviews 2023 All_Beauty product reviews.
Each example contains fields such as:
parent_asin
product_title
rating
num_reviews
source
target_summary
target_word_count
target_qc_ok
source_title_mismatch
mismatch
Before training, rows failing quality control or marked as source/title mismatches were excluded.
The final training experiment used:
| Split | Examples |
|---|---|
| Train | 1,567 |
| Validation | 196 |
| Held-out test | 196 |
| Total clean examples | 1,959 |
The split was deterministic with seed 42.
Synthetic Reference Summaries
The target summaries used for training are synthetic reference summaries generated by a teacher model, not human-written gold summaries.
They were produced under constraints intended to:
- preserve product identity,
- summarize the review consensus,
- retain recurring strengths,
- retain meaningful minority complaints,
- avoid unsupported claims,
- avoid numeric rating generation,
- produce one natural paragraph.
This distinction is important when interpreting the evaluation results below: automatic metrics compare LED outputs against these synthetic teacher-generated references.
Training Configuration
| Setting | Value |
|---|---|
| Base model | allenai/led-large-16384 |
| Maximum source length | 4096 tokens |
| Maximum target length | 192 tokens |
| Epochs | 3 |
| Train batch size / device | 1 |
| Eval batch size / device | 1 |
| Gradient accumulation | 8 |
| Effective batch size | 8 |
| Learning rate | 1e-5 |
| Weight decay | 0.01 |
| Warmup ratio | 0.05 |
| Label smoothing | 0.05 |
| Precision | FP16 |
| Optimizer | AdamW |
| Gradient checkpointing | Enabled |
| Beam size | 6 |
| No-repeat n-gram size | 3 |
| Repetition penalty | 1.05 |
| Seed | 42 |
This was a full fine-tuning of LED-Large, not LoRA.
Global attention
Sparse global attention was assigned to:
- the first source token,
- the product-title line,
- the first token of every
[Review N]block, - every
Rating:line.
This gives LED explicit anchors for the product identity and review boundaries across long inputs.
Held-Out Test Results
Evaluation was performed on the 196-example held-out test split.
| Metric | Score |
|---|---|
| ROUGE-1 | 0.4806 |
| ROUGE-2 | 0.1607 |
| ROUGE-L | 0.3264 |
| ROUGE-Lsum | 0.3263 |
| BERTScore Precision | 0.9074 |
| BERTScore Recall | 0.9052 |
| BERTScore F1 | 0.9062 |
| Test loss | 2.6579 |
Validation loss for the final run:
2.6452
Automatic metrics are useful for reproducibility, but they do not fully measure factual consistency, consensus weighting, or product-identity preservation.
Qualitative Behavior
A manual audit of the 196 held-out generations found several strengths:
- summaries generally reflect the overall review consensus,
- product identity is usually preserved,
- outputs are concise and readable,
- recurring positive and negative opinions are often combined effectively.
Observed limitations include:
- repetitive summary openings,
- occasional local word or phrase repetition,
- product or specification drift,
- occasional polarity or factual drift,
- isolated opinions sometimes being promoted to consensus,
- minority complaints sometimes being over- or under-weighted.
These limitations are important for downstream use.
Example Application
A simple application can combine the deterministic product rating with the generated review summary:
Rating: 4.3/5
Summary: Customers generally appreciate ...
The rating should be computed from the original review ratings in application code.
Do not ask LED to generate the numeric rating.
Reproduce the Training
The full experiment notebook is included in this repository:
AllBeauty_LED_Large_Training.ipynb
The training workflow is:
prepared product-level review dataset
↓
quality filtering
↓
deterministic train / validation / test split
↓
LED-Large tokenization
↓
structured global attention
↓
full fine-tuning
↓
best-model selection
↓
held-out generation
↓
ROUGE + BERTScore + qualitative diagnostics
Experiment artifacts are stored under:
artifacts/
Typical files include:
experiment_config.json
final_metrics.json
split_manifest.json
test_predictions.csv
Repository Structure
amazon-all-beauty-led-large-summarization/
├── README.md
├── AllBeauty_LED_Large_Training.ipynb
├── AllBeauty_LED_Large_Inference.ipynb
├── config.json
├── generation_config.json
├── model.safetensors
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── merges.txt
├── vocab.json
└── artifacts/
├── experiment_config.json
├── final_metrics.json
├── split_manifest.json
└── test_predictions.csv
The model files are stored at the repository root so the model can be loaded directly with:
LEDForConditionalGeneration.from_pretrained(
"vltruong01/amazon-all-beauty-led-large-summarization"
)
Which File Should I Use?
| Goal | File |
|---|---|
| Run the model on your own review collection | AllBeauty_LED_Large_Inference.ipynb |
| Load the trained model programmatically | repository root |
| Reproduce training and evaluation | AllBeauty_LED_Large_Training.ipynb |
| Inspect the experiment configuration | artifacts/experiment_config.json |
| Inspect final metrics | artifacts/final_metrics.json |
| Inspect exact split membership | artifacts/split_manifest.json |
| Inspect held-out model generations | artifacts/test_predictions.csv |
Expected Input
Use the same structured format as training:
Product: <product title>
[Review 1]
Rating: <rating>/5
Title: <review title>
Text: <review text>
[Review 2]
Rating: <rating>/5
Title: <review title>
Text: <review text>
For best results:
- keep the product title at the beginning,
- preserve complete review blocks,
- include review rating lines,
- keep the full input within 4096 tokens,
- avoid mixing reviews from different products.
Expected Output
The intended output is:
- one paragraph,
- product-level rather than review-by-review,
- focused on recurring customer opinions,
- balanced across positives and meaningful complaints,
- no bullet list,
- no numeric product rating.
Intended Use
Suitable uses include:
- research on long-context review summarization,
- product-level multi-review summarization,
- comparison of LED against text-only or multimodal variants,
- studying consensus aggregation from many customer reviews,
- educational and academic experiments.
Limitations
This model has several important limitations:
- It is trained only on the Amazon All_Beauty domain.
- Training targets are synthetic teacher-generated references rather than human-written gold summaries.
- The model can hallucinate or distort product details.
- It can occasionally reverse or flatten sentiment.
- It may over-emphasize isolated opinions or miss minority complaints.
- Summary phrasing can become repetitive.
- Inputs longer than the configured context budget require review selection or truncation before inference.
- Automatic metrics do not guarantee factual faithfulness.
For high-stakes or user-facing deployment, summaries should be treated as generated text and checked against the source reviews.
Related Dataset
Training dataset:
vltruong01/amazon-all-beauty-led-summarization
Source-only review dataset:
vltruong01/amazon-all-beauty-led-reviews
Citation and Attribution
Base model:
allenai/led-large-16384
Dataset source:
- Amazon Reviews 2023, All_Beauty category
This repository is provided for research and educational use.
- Downloads last month
- 24
Model tree for vltruong01/amazon-all-beauty-led-large-summarization
Base model
allenai/led-large-16384