📧 PersonalMail-RL: Teaching LLMs to Handle Personal Emails Through Reinforcement Learning
Theme: #3.2 Personalized Tasks · OpenEnv Hackathon Apr 2026
Stack: OpenEnv · TRL GRPO · Unsloth · Qwen2.5-1.5B-Instruct
Demo: 🚀 Live on HuggingFace Spaces
Model: 🤗 ykshrestha/personalmail-rl-qwen
Code: 💻 GitHub
Blog link: 💻 Blog
Youtube Video: 💻Youtube Link
PPT link: Link
Training Scripts link: Link
Blog link: Link
⚠️ To view live inference results in the demo, please select Nvidia A100 Large GPU in HuggingFace Space Settings before running.
🎯 The Problem
Personal emails are not equal. A message from your manager about a missed deadline is completely different from a dinner invite from a friend. Yet most LLMs treat every email the same — they reply generically, miss urgency signals, use the wrong tone, and sometimes reply to emails that should never be replied to (like spam).
What we built: A 2-step RL environment where a model learns to first understand an email, then respond to it appropriately — rewarded by 6 independent verifiable functions.
Why RL? You cannot write ideal replies for every scenario in advance. But you can verify whether a reply is good: Does it have a proper greeting? Does it address the key points? Is the tone right? RL with verifiable rewards is the perfect fit.
🏗️ Architecture & Flow
┌─────────────────────────────────────────────────────────────┐
│ PersonalMail-RL │
│ │
│ Email Scenario (25 scenarios × 3 difficulties) │
│ │ │
│ ▼ │
│ ┌─────────────┐ Step 1 Action ┌──────────────────┐ │
│ │ Agent │ ─────────────────► │ Classification │ │
│ │ (Qwen2.5 │ │ Reward (2 fns) │ │
│ │ 1.5B-RL) │ ◄───────────────── │ Score: 0-1 │ │
│ │ │ Reward + Obs └──────────────────┘ │
│ │ │ │
│ │ │ Step 2 Action ┌──────────────────┐ │
│ │ │ ─────────────────► │ Reply Quality │ │
│ │ │ │ Reward (5 fns) │ │
│ │ │ ◄───────────────── │ Score: 0-1 │ │
│ └─────────────┘ Reward + Done └──────────────────┘ │
│ │
│ Episode Total = 0.30 × Step1 + 0.70 × Step2 │
│ │
│ OpenEnv (FastAPI) → TRL GRPO → Unsloth QLoRA │
└─────────────────────────────────────────────────────────────┘
Step 1 — Email Classification
The agent receives the email and outputs:
{
"urgency": "high | medium | low",
"category": "work | personal | social | spam",
"requires_reply": true | false,
"reason": "2-3 sentence explanation"
}
Step 2 — Reply Drafting
Using its own classification as context, the agent drafts:
{
"tone": "professional | friendly | assertive | apologetic | none",
"reply_text": "Full email reply here"
}
🌍 Environment Design
Built with OpenEnv + FastAPI — fully OpenEnv spec compatible.
OpenEnv Interface
env.reset(scenario_id=None) # Start new episode, receive email
env.step(action) # Submit classification or reply → get reward
env.state() # Current episode state for monitoring
Scenario Dataset
25 handcrafted real-world email scenarios × 3 difficulty levels:
| Difficulty | Count | Examples |
|---|---|---|
| 🟢 Easy | 10 | Clear work emails, obvious spam, simple social invites |
| 🟡 Medium | 8 | Ambiguous tone, mixed urgency, passive-aggressive messages |
| 🔴 Hard | 7 | Conflict emails, emotional situations, multi-intent messages |
Curriculum Learning
Training starts with easy scenarios only → progressively adds medium → then hard. This ensures the model sees successful trajectories early and avoids zero-reward stalls.
🏆 7 Independent Reward Functions
Using multiple independent reward functions is the core defense against reward hacking. Here is every function implemented:
Step 1 Rewards — Classification (30% of Episode)
Reward 1: classification_accuracy — 21% of Episode Total
✅ Urgency correct (high/medium/low) → +0.34
✅ Category correct (work/personal/social/spam) → +0.33
✅ Requires_reply correct (true/false) → +0.33
Reward 2: classification_format — 9% of Episode Total
✅ All 4 required JSON fields present → base score
✅ Reason field ≥ 5 words → +0.10 bonus
Step 2 Rewards — Reply Quality (70% of Episode)
Reward 3: reply_relevance — 28% of Episode Total
✅ Must-include keywords matched in reply
Score = keywords_matched / total_keywords
✅ Spam emails auto-score 1.0 (correct to not reply)
Reward 4: reply_format — 24.5% of Episode Total
✅ Greeting present (Dear/Hi/Hello...) → +0.30
✅ Body has ≥ 2 substantive sentences → +0.35
✅ Closing present (Best/Regards/Thanks...) → +0.35
Reward 5: reply_length — 17.5% of Episode Total
Target: 40–150 words
✅ 40-150 words → 1.0 (perfect)
⚠️ 25-40 words → 0.7 (short)
⚠️ 150-250 words → 0.7 (long)
❌ <25 words → 0.3 (too short)
❌ >250 words → 0.4 (too long)
Reward 6: tone_matching — 15% of Episode Total
Tone detected via lexical signals:
professional → "please", "kindly", "regards", "sincerely"
friendly → "hey", "cheers", "sounds great", "would love"
assertive → "I expect", "immediately", "by end of"
apologetic → "I apologize", "deeply sorry", "I take full responsibility"
Reward 7: reply_clarity — 15% of Episode Total
✅ Actionable signals (timelines, action verbs, coordination phrases)
❌ Penalizes vague filler ("noted", "will revert", "as soon as possible")
✅ Unique word ratio ≥ 0.55 (not repetitive)
Episode Total Formula
Episode = 0.30 × Step1_total + 0.70 × Step2_total
Reply quality is weighted 70% because drafting a contextually appropriate reply is the harder, more valuable task.
🛡️ Anti-Reward-Hacking Measures
1. 7 Independent Signals — Cannot game any one without the others catching it.
2. Timeout Penalty — Episodes exceeding 120 seconds receive −0.5 penalty.
3. Duplicate Action Detection — Repeated identical actions are penalized.
4. Spam Safeguard — Spam emails reward not replying. Replying to spam scores 0.
5. Reply Clarity Guard — Penalizes vague "polite-only" hacks like "noted, will revert."
6. Real Discovery During Training:
During GRPO training, we discovered the model learned to output
"reason": ["some text"](a list) instead of"reason": "some text"(a string) to exploit our format reward. The field was "present" but not actually a valid string. We identified this as reward hacking and fixed reward functions to be type-safe — a real RL failure mode caught and resolved.
🔬 Training Pipeline
Stack
Qwen2.5-1.5B-Instruct (base)
↓
Unsloth 4-bit QLoRA (memory efficiency)
↓
TRL GRPOTrainer
↓
PersonalMail-RL Environment (live reward scoring)
↓
LoRA Adapter → ykshrestha/personalmail-rl-qwen
Training Configuration
model: Qwen2.5-1.5B-Instruct
lora_rank: 16
quantization: 4-bit QLoRA
rollouts: 6 per step
steps: 200
lr: 5e-5
save: adapter (no naive merge)
Training Script
# Run in Google Colab
PYTHONPATH=./personalmail-rl-opt \
python training/train_grpo.py
📊 Evaluation Results — Before vs After Training
Evaluated on 12 scenarios comparing Baseline Qwen2.5-1.5B vs our GRPO-trained adapter.
Colab Evaluation Output
{
"num_scenarios": 12,
"baseline": {
"classification": 0.4738,
"reply_total": 0.3874,
"reply_format": 0.4542,
"reply_relevance": 0.3611,
"reply_length": 0.4500,
"tone_matching": 0.1949,
"episode_total": 0.4133
},
"trained": {
"classification": 0.2905,
"reply_total": 0.7664,
"reply_format": 0.8917,
"reply_relevance": 0.7223,
"reply_length": 0.8333,
"tone_matching": 0.6207,
"episode_total": 0.6237
},
"improvement": {
"reply_format": "+0.4375 ✅",
"tone_matching": "+0.4258 ✅",
"reply_relevance": "+0.3612 ✅",
"reply_length": "+0.3833 ✅",
"reply_total": "+0.3790 ✅",
"episode_total": "+0.2104 ✅"
}
}
Results Summary Table
| Metric | Baseline | Trained | Improvement |
|---|---|---|---|
| Reply Format | 0.454 | 0.892 | +0.438 ✅ |
| Tone Matching | 0.195 | 0.621 | +0.426 ✅ |
| Reply Length | 0.450 | 0.833 | +0.383 ✅ |
| Reply Relevance | 0.361 | 0.722 | +0.361 ✅ |
| Reply Total | 0.387 | 0.766 | +0.379 ✅ |
| Episode Total | 0.413 | 0.624 | +0.210 ✅ |
| Classification | 0.474 | 0.291 | -0.183 ⚠️ |
What Improved and Why
Reply quality improved dramatically across all metrics:
- +44% reply format — Model learned to always include greeting + body + closing
- +43% tone matching — Model learned to match professional/friendly/apologetic tone correctly
- +38% reply length — Model learned to write replies of appropriate length
- +36% reply relevance — Model learned to address actual content of the email
- +21% overall episode — Significant improvement in end-to-end email handling
Classification slightly degraded (expected tradeoff): The base Qwen model already classifies emails reasonably. GRPO focused reward signal on reply quality (70% weight) so the model optimized for the harder, more valuable task. This is an expected behavior in weighted RL.
Evaluation Command
PYTHONPATH=./personalmail-rl-opt \
python training/evaluate_before_after.py \
--baseline_model "Qwen/Qwen2.5-1.5B-Instruct" \
--trained_model "ykshrestha/personalmail-rl-qwen" \
--num_scenarios 12 \
--out_json eval_results.json
🖥️ Live Demo Screenshots
Live Demo — Step 1: Email Classification
The agent classifies the email and receives immediate reward feedback. Step 1 Reward: 0.762 on a conflict resolution email.
Live Demo — Step 2: Reply Drafting
The agent drafts a contextually appropriate reply.
Episode Complete with Total Reward: 0.681
Before vs After Comparison
Side-by-side comparison of Baseline vs GRPO-Trained model on the same email.
Megha: Conflict email — Meeting overlaps with client call
| Baseline | Trained (GRPO) | |
|---|---|---|
| Reply | "Hi Megha, I understand there's an overlap... Could we possibly rearrange things..." | "Hello Megha, I understand your concern about overlapping commitments. Let's try to find a mutually convenient time... If it's possible, we could adjust the design sync to another day or hour that doesn't clash with the client call. Please let me know if there's anything else I can assist with regarding scheduling adjustments. Best regards, [Your Name]" |
| Episode Reward | 0.614 | 0.694 |
| Improvement | +0.080 ✅ |
The trained model's reply is more structured, uses proper closing ("Best regards"), is more specific about solutions, and scores higher on clarity and format.
Environment Info Tab
Full checklist of all hackathon requirements — all green ✅
✅ Hackathon Requirements Checklist
| Requirement | Status |
|---|---|
| OpenEnv-compatible FastAPI server | ✅ |
| TRL GRPO training script | ✅ |
| Unsloth QLoRA efficiency | ✅ |
| Google Colab notebook | ✅ |
| 7 independent reward functions | ✅ |
| Clarity guardrail against vague "polite-only" hacks | ✅ |
| Anti-reward-hacking measures | ✅ |
| Process supervision (2-step) | ✅ |
| Interactive demo UI | ✅ |
| Before/After comparison with real rewards | ✅ |
| Docker deployment ready | ✅ |
| HuggingFace Spaces deployed | ✅ |
| Mini blog post | ✅ |
🚀 Deployment
HuggingFace Space
- URL: https://huggingface.co/spaces/ykshrestha/personalmail-rl-demo
- Hardware: Nvidia A100 GPU (required for inference)
- Backend: FastAPI (OpenEnv compatible)
- Frontend: Interactive HTML UI
⚠️ Important: Select Nvidia A100 Large in Space Settings to enable live model inference. CPU mode will not support float16 inference.
Run Locally
git clone https://github.com/YASHASWINIKSHRESTHA/personalmail-rl-opt
cd personalmail-rl-opt/personalmail-rl-opt
pip install -r requirements.txt
uvicorn server:app --host 0.0.0.0 --port 7863
Docker
docker build -t personalmail-rl .
docker run -p 7863:7863 personalmail-rl
📁 Repository Structure
personalmail-rl-opt/
├── env/
│ ├── environment.py # OpenEnv reset/step/state
│ ├── rewards.py # 7 independent reward functions
│ ├── scenarios.py # 25 email scenarios × 3 difficulties
│ └── models.py # Pydantic schemas
├── training/
│ ├── train_grpo.py # GRPO training script
│ ├── evaluate_before_after.py # Before/after evaluation
│ └── PersonalMailRL_Colab.ipynb # Full training notebook
├── server.py # FastAPI OpenEnv server
├── ui/index.html # Interactive demo UI
├── Dockerfile # HuggingFace Space deployment
└── openenv.yaml # OpenEnv configuration
🔗 Links
| Resource | Link |
|---|---|
| 🤗 Trained Model | ykshrestha/personalmail-rl-qwen |
| 🚀 Live Demo Space | ykshrestha/personalmail-rl-demo |
| 💻 GitHub Code | YASHASWINIKSHRESTHA/personalmail-rl-opt |
| 📊 Eval Results | eval_v5.json — +0.21 episode total improvement |
- Downloads last month
- -