tanmaydeshpande/qlora-app-review-extraction

LoRA adapter — app-review → strict JSON extraction

A LoRA adapter for Qwen/Qwen2.5-0.5B-Instruct that extracts a strict, closed-vocabulary JSON object from an unstructured app-store review:

{
  "sentiment":      "positive" | "negative" | "neutral",
  "topics":         [up to 3 strings from a fixed 8-term vocabulary],
  "mentions_price": true | false,
  "rating_implied": 1..5
}

Topic vocabulary: ui, performance, bugs, ads, price, features, usability, support.

• 🧪 Try it live: https://huggingface.co/spaces/tanmaydeshpande/app-review-extraction
• 💻 Code + full write-up: https://github.com/deshpandetanmay/qlora-structured-extraction
• 📊 Dataset: https://huggingface.co/datasets/tanmaydeshpande/app-review-extraction-splits

Results (held-out 200-example test set, deterministic scoring)

Base gets a 3-shot prompt; the tuned adapter runs 0-shot. Scoring is fully rule-based (exact match for enum/bool, set-F1 for topics, exact/off-by-one with half credit for the integer) — no model is used as a judge.

Model	Aggregate	sentiment	topics	mentions_price	rating_implied
Base (3-shot, fp32)	62.3%	68.0%	26.1%	95.0%	60.2%
Tuned (LoRA, 0-shot)	76.6%	72.5%	73.5%	97.5%	62.7%

Both models parsed 200/200 outputs as valid JSON, so the +14.2-point gain is in field correctness, not formatting. The largest lift is topics (learning the closed vocabulary). See Limitations below before trusting any single field.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float32)
model = PeftModel.from_pretrained(model, "tanmaydeshpande/qlora-app-review-extraction")
model.eval()

SYSTEM = ("You are an information-extraction engine. You read a short app review "
          "and output ONLY a single JSON object. No prose, no code fences, no explanation.")
INSTRUCTION = (
    'Extract structured fields from the app review into a JSON object with EXACTLY these keys:\n'
    '  "sentiment": one of "positive", "negative", "neutral"\n'
    "  \"topics\": a list of 0 to 3 strings, each chosen ONLY from this closed vocabulary: "
    "['ui', 'performance', 'bugs', 'ads', 'price', 'features', 'usability', 'support']\n"
    '  "mentions_price": true or false (does the review mention price, cost, subscription, or payment?)\n'
    '  "rating_implied": an integer 1 to 5 (the star rating the text implies; 1 = very negative, 5 = very positive)\n'
    "Output only the JSON object.")

review = "Love the app but the ads are relentless and now they want $5/month to remove them."
messages = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": f'{INSTRUCTION}\n\nReview: "{review}"\nJSON:'}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc = tok(prompt, return_tensors="pt")
out = model.generate(**enc, max_new_tokens=96, do_sample=False)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Training

• Method: LoRA SFT (PEFT + TRL), prompt tokens masked from the loss.
• LoRA: r=16, alpha=32, dropout=0.05; targets q,k,v,o,gate,up,down proj.
• Trainable params: 8,798,208 / 502,830,976 (1.75%).
• Schedule: 2 epochs, lr 2e-4, effective batch 16 (4 × 4 grad-accum), 3% warmup.
• Precision: fp32 on Apple-Silicon MPS (M4 Mac mini, 24 GB). bitsandbytes 4-bit QLoRA is CUDA-only and was not used — so the "quantization-recovery" control row could not be produced on this machine (documented, not faked).
• Train wall-clock: ~56 min. Final train/val loss: 0.081 / 0.080.
• Data: 1500 train / 200 val examples derived from sealuzh/app_reviews.

Limitations

• Label circularity: sentiment, topics, and mentions_price labels are generated by deterministic rules, so part of the tuned gain is the model learning that labeling function rather than the underlying concept.
• rating_implied uses the dataset's native star rating and is the most trustworthy field (exact accuracy 39.5% → 36.0% — roughly flat).
• mentions_price is ~96% one class (false), so its high accuracy is not very informative.
• Single-seed run; small (200) test set; English app reviews only.

Full caveats: publication/limitations.md.

License

Adapter released under Apache-2.0 (matching the base model). Derived from the sealuzh/app_reviews dataset — verify that dataset's terms for your use.