title-optimizer-grpo

A LoRA adapter for Qwen/Qwen3-1.7B that writes Airbnb listing titles that are catchy and only mention features the listing actually states. Trained with GRPO for 500 steps on 4,000 listings with programmatic rewards only: length (weight 1.0), Airbnb style rules (0.5) and no unsupported feature claims (2.0).

Part of listing-title-optimizer (SFT → DPO → GRPO on public Inside Airbnb data).

This adapter is chained: it was trained on top of the merged SFT model, so apply pulkit-thukral/title-optimizer-sft first, merge it, then apply this one.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# The saved tokenizer has Qwen3's thinking mode turned off, so all tokens go to the title.
tok = AutoTokenizer.from_pretrained("pulkit-thukral/title-optimizer-grpo")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "pulkit-thukral/title-optimizer-sft").merge_and_unload()
model = PeftModel.from_pretrained(model, "pulkit-thukral/title-optimizer-grpo").merge_and_unload()

SYSTEM_PROMPT = "You write Airbnb listing titles. Given a listing, reply with ONE title and nothing else.\nRules:\n- At most 50 characters; 25-45 is ideal.\n- Lead with the most distinctive, appealing feature (view, location, standout amenity).\n- Only mention features that the listing actually states. Never invent amenities.\n- Sentence case. No emojis, no ALL CAPS, no '!!!', no quotes, no prefix like 'Title:'.\n- Don't repeat what guests already see (city name, bed count) unless it's the hook."
listing = (
    "Property type: Entire rental unit\n"
    "Neighbourhood: Hackney\n"
    "Amenities: Wifi, Washer, Kitchen, Private patio or balcony\n"
    "Description:\nBright two-bedroom flat near Old Street with a private terrace."
)
messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": listing}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**inputs.to(model.device), max_new_tokens=32, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

The listing format (property type, neighbourhood, up to 20 amenities, description) and system prompt match training; see prompts.py.

Training

• Data: 4,000 listings from the listings train split, from pulkit-thukral/listing-title-optimizer.
• LoRA r=16, alpha=32 on all linear layers, bf16, trained on an Apple M4 Mac (24 GB).
• Full hyperparameters are in run_config.json in this repo.

Evaluation

500 held-out listings from hosts never seen in training. Win rate is claude-sonnet-5-5 pairwise preference against the host's original title (order-swapped, ties = ½), with a 95% bootstrap CI. This adapter's row is SFT + GRPO.

System	Win rate vs original (95% CI)	Hallucination ↓	≤50 chars ↑	Style issues ↓
Original (host)	—	3.0%	100.0%	2.4%
Base model	36.1% (32.1%–40.2%)	4.0%	23.8%	0.4%
SFT	65.3% (61.3%–69.4%)	0.8%	97.2%	0.0%
SFT + DPO	68.5% (64.4%–72.3%)	0.2%	99.8%	0.0%
SFT + GRPO	67.7% (63.8%–71.4%)	0.2%	100.0%	0.0%

How far to trust the win rate: on 50 hand-labelled preference pairs the judge agreed with the author 64% of the time (95% CI 50–78%, Cohen's κ = 0.29). Read the win rates as "Sonnet prefers it", not as a checked human preference. The hallucination, length and style columns are rule-based and don't use the judge.

Limitations

• The hallucination check is lexicon-based (22 claim types). It misses paraphrases and can't check numeric ("3BR") or location ("5 min to the beach") claims.
• Trained on English listings from London, New York, Sydney, Toronto and Dublin.
• An LLM judge isn't a guest; click-through has not been measured.