karthiksing05/sidequestz-compatibility-classifier

SideQuests compatibility classifier

This model scores how well an event suits a user, the ranking model behind SideQuests' event recommendations. Its inputs are frozen Qwen/Qwen3-Embedding-0.6B embeddings of three texts, all in the app's eight-section format: the user's likes (positive_text), the user's dislikes (negative_text), and the event (embedding_text). It was trained following the repo's ml/training.md on the synthetic karthiksing05/sidequestz-event-embedding-text data (users config: LLM-rated user–event pairs).

Training curves, per-epoch validation metrics and every run: W&B project, summarized in the W&B report.

Chosen model: p2-late_fusion-select-ndcg: late_fusion, huber loss, lr 0.001, weight decay 0.0001, batch 128, best epoch 7 (selection: ndcg@10).

Test results (evaluated once, after selection on validation)

	ndcg@5	ndcg@10	precision@5	precision@10	recall@5	recall@10	spearman	mae	rmse
this model	0.8523	0.8969	0.6786	0.5254	0.6042	0.8829	0.6968	0.1565	0.1998
cosine baseline (λ = 1, the model's)	0.7797	0.8411	0.5882	0.4792	0.5101	0.8023	0.4583	-	-
cosine baseline (λ = 0.5, tuned on validation)	0.7887	0.8494	0.5988	0.4872	0.5232	0.8168	0.5012	-	-

Rankings are per user, over each test user's 20 candidate events. Test users and test events never appear in training. Relevant means label ≥ 0.5 (a judge rating of 2 or 3 out of 3).

How it was selected

1. Cosine baseline, λ sweep (validation). Score: cos(pos, event) - λ·cos(neg, event).

	ndcg@10	recall@10	spearman
λ = 0	0.8263	0.8145	0.4642
λ = 0.25	0.8399	0.8274	0.5201
λ = 0.5	0.8431	0.8279	0.5070
λ = 0.75	0.8405	0.8224	0.4862
λ = 1	0.8337	0.8080	0.4658
λ = 1.25	0.8241	0.7943	0.4453
λ = 1.5	0.8131	0.7775	0.4257
λ = 2	0.7941	0.7525	0.3928

2. Phase 1, the recipe's comparison (validation, mean ± std over seeds 0–2). Variants ranked by NDCG@10, then Recall@10, then Spearman; the winner was p1-late_fusion.

	ndcg@10	recall@10	spearman
p1-late_fusion	0.8938 ± 0.0004	0.8800 ± 0.0027	0.6946 ± 0.0022
p1-flat	0.8923 ± 0.0010	0.8775 ± 0.0033	0.6941 ± 0.0030
p1-residual-mse	0.8890 ± 0.0014	0.8744 ± 0.0021	0.6843 ± 0.0054
p1-residual	0.8888 ± 0.0019	0.8782 ± 0.0034	0.6807 ± 0.0067

3. Phase 2, tuning the winner (validation, seed 0). One change at a time. The best validation run becomes the final model. For scale: across seeds, the phase-1 winner's NDCG@10 varied by ±0.0004, so smaller differences are noise.

	ndcg@10	recall@10	spearman	mae
p2-late_fusion-select-ndcg	0.8942	0.8819	0.6937	0.1593
p1-late_fusion-s0	0.8939	0.8837	0.6960	0.1600
p2-late_fusion-patience10	0.8939	0.8837	0.6960	0.1600
p2-late_fusion-wd1e-3	0.8934	0.8824	0.6956	0.1606
p2-late_fusion-drop0.35	0.8931	0.8824	0.6941	0.1599
p2-late_fusion-wide	0.8924	0.8782	0.6951	0.1589
p2-late_fusion-lr3e-3	0.8919	0.8820	0.6932	0.1592
p2-late_fusion-drop0.1	0.8914	0.8774	0.6891	0.1608
p2-late_fusion-lr3e-4	0.8914	0.8765	0.6959	0.1593
p2-late_fusion-bs512	0.8908	0.8755	0.6934	0.1588

Data

Split	Users	Pairs	Events	Label mean	Relevant share
train	8,000	160,000	64,988	0.503	45.6%
validation	1,000	20,000	1,991	0.402	30.7%
test	1,000	20,000	1,995	0.400	30.5%

Usage

The model code lives in the SideQuests repo (ml/compatibility/classifier).

import torch
from huggingface_hub import hf_hub_download
from sentence_transformers import SentenceTransformer
from compatibility.classifier import load_checkpoint

encoder = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
model, ckpt = load_checkpoint(hf_hub_download("karthiksing05/sidequestz-compatibility-classifier", "final/best.pt"))

def embed(text):  # an empty text (no dislikes) is a zero vector, as in training
    if not text.strip():
        return torch.zeros(1, 1024)
    return torch.tensor(encoder.encode([text], normalize_embeddings=True))

score = model(embed(positive_text), embed(negative_text), embed(event_embedding_text))  # higher = better fit

Files

• final/best.pt: the chosen checkpoint (model_state, config, epoch, val_loss, history, optimizer_state), loadable with load_checkpoint.
• final/results.json and final/comparison.txt: validation and test metrics.
• runs/<run>/best.pt and runs/<run>/val_metrics.json: every trained run, for comparison.
• baseline.json, phase1.json, data_summary.json: the selection record.

Limitations

• LLM labels. The training labels are one LLM's judgments on synthetic users and events, not real interactions. Expect a gap on real traffic, and retrain on logged outcomes when they exist.
• Tied encoder. Scores are only meaningful for Qwen/Qwen3-Embedding-0.6B embeddings (L2-normalized, no prompt) of texts in the eight-section format.
• Candidate ranker. The model re-ranks candidates. It was trained on retrieved, hard-negative and random candidates per user, not on the full catalog.