title-optimizer-dpo

A LoRA adapter for Qwen/Qwen3-1.7B that writes Airbnb listing titles that are catchy and only mention features the listing actually states. Trained with DPO on 625 preference pairs built from the SFT model's own candidates (rule-failing titles as hard negatives, the rest ranked by a claude-sonnet-5-5 judge).

Part of listing-title-optimizer (SFT → DPO → GRPO on public Inside Airbnb data).

This adapter is chained: it was trained on top of the merged SFT model, so apply pulkit-thukral/title-optimizer-sft first, merge it, then apply this one.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# The saved tokenizer has Qwen3's thinking mode turned off, so all tokens go to the title.
tok = AutoTokenizer.from_pretrained("pulkit-thukral/title-optimizer-dpo")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "pulkit-thukral/title-optimizer-sft").merge_and_unload()
model = PeftModel.from_pretrained(model, "pulkit-thukral/title-optimizer-dpo").merge_and_unload()

SYSTEM_PROMPT = "You write Airbnb listing titles. Given a listing, reply with ONE title and nothing else.\nRules:\n- At most 50 characters; 25-45 is ideal.\n- Lead with the most distinctive, appealing feature (view, location, standout amenity).\n- Only mention features that the listing actually states. Never invent amenities.\n- Sentence case. No emojis, no ALL CAPS, no '!!!', no quotes, no prefix like 'Title:'.\n- Don't repeat what guests already see (city name, bed count) unless it's the hook."
listing = (
    "Property type: Entire rental unit\n"
    "Neighbourhood: Hackney\n"
    "Amenities: Wifi, Washer, Kitchen, Private patio or balcony\n"
    "Description:\nBright two-bedroom flat near Old Street with a private terrace."
)
messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": listing}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**inputs.to(model.device), max_new_tokens=32, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

The listing format (property type, neighbourhood, up to 20 amenities, description) and system prompt match training; see prompts.py.

Training

• Data: 625 preference pairs generated by title-opt preferences, from pulkit-thukral/listing-title-optimizer.
• LoRA r=16, alpha=32 on all linear layers, bf16, trained on an Apple M4 Mac (24 GB).
• Full hyperparameters are in run_config.json in this repo.

Evaluation

500 held-out listings from hosts never seen in training. Win rate is claude-sonnet-5-5 pairwise preference against the host's original title (order-swapped, ties = ½), with a 95% bootstrap CI. This adapter's row is SFT + DPO.

System	Win rate vs original (95% CI)	Hallucination ↓	≤50 chars ↑	Style issues ↓
Original (host)	—	3.0%	100.0%	2.4%
Base model	36.1% (32.1%–40.2%)	4.0%	23.8%	0.4%
SFT	65.3% (61.3%–69.4%)	0.8%	97.2%	0.0%
SFT + DPO	68.5% (64.4%–72.3%)	0.2%	99.8%	0.0%
SFT + GRPO	67.7% (63.8%–71.4%)	0.2%	100.0%	0.0%

How far to trust the win rate: on 50 hand-labelled preference pairs the judge agreed with the author 64% of the time (95% CI 50–78%, Cohen's κ = 0.29). Read the win rates as "Sonnet prefers it", not as a checked human preference. The hallucination, length and style columns are rule-based and don't use the judge.

Limitations

• The hallucination check is lexicon-based (22 claim types). It misses paraphrases and can't check numeric ("3BR") or location ("5 min to the beach") claims.
• Trained on English listings from London, New York, Sydney, Toronto and Dublin.
• An LLM judge isn't a guest; click-through has not been measured.