riverguide-retriever-v2

Qwen3-Embedding-0.6B fine-tuned to find Indian River State College web pages for what students and parents type: questions, abbreviations and nicknames (RN, rad tech, BSN), and first messages in their own words ("my son wants to take college classes in high school, how does that work"). One forward pass per query, no generated tokens. Built for RiverGuide, the College's student assistant, in https://github.com/joelcook/riverguide-retriever.

What's new since v1: v1 learned from search-style queries. It found the right page for 96% of them on pages it never trained on, but an answer for fewer of students' real first messages, which describe a situation instead of naming the office, form, or policy. v2 adds 8,183 first messages in that style, written for the same training chunks.

Use

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("joeltcook/riverguide-retriever-v2")
INSTRUCT = "Instruct: Given a college student's search query, retrieve the College web page that answers it\nQuery: "
q = model.encode(INSTRUCT + "how do i sign up for spring classes im coming back after a few years", normalize_embeddings=True)
d = model.encode(["<page title> > <section>\n\n<chunk text>"], normalize_embeddings=True)
scores = d @ q

Queries need the instruction prefix (the same as v1's); chunks are embedded as plain text. A page's score is its best chunk's. The weights are bf16; on a CPU, load with model_kwargs={"dtype": torch.float32} for speed.

Real students' questions (2026-10-25)

523 distinct first messages that students typed into the College's chat assistant in the 30 days to 2026-10-25. For each, the top 5 pages from each search were judged blind (shuffled, unlabeled) by Qwen3.8-27B: did any page give the student what they need? Only aggregate counts were kept, never the messages. All three were judged in the same run.

Search on the student's own words	Answered
v1 (riverguide-retriever-full)	447 (85.5%)
v2 with Claude Sonnet 5.5's first messages instead	453 (86.6%)
this model	460 (88.0%)

Head to head on the same messages, this model answered 22 that v1 missed, and v1 answered 9 that it missed (two-sided exact sign test, p = 0.03). The Sonnet-trained twin's gain over v1 (15 vs 9) was not significant (p = 0.31).

Fixed tests (right page in the top 5)

	keyword (BM25)	stock 0.6B	v1	this model
abbreviations	51/75	65/75	73/75	71/75
golden	18/21	19/21	19/21	19/21
val	631/942	757/942	909/942	905/942

val holds out whole pages, so it is the clean comparison: v2 ties v1 there (MRR@10 0.772 vs 0.774). The abbreviation test is contaminated for both models: their training queries include the test's abbreviations (v1's card has the clean number).

Caveats: one real-question run. Its judge is the same model that wrote v2's new training messages, which may favor this model; Sonnet's messages, judged the same way, gained less. The same v1 scored 458 in a run a day earlier, over a 30-day window shifted by a day, so compare models within a run, as above.

Training

• Data: v1's queries, written by Qwen3.8-27B for 5,923 chunks of the College's public website and catalog (the 2026-10-22 crawl, 2,389 pages): a full question, a 2–5 word search, the shorthand or abbreviation, and a misspelling per chunk, plus program-name queries. To them, two first messages per chunk in a student's or parent's words, also from Qwen3.8-27B, with the same prompt Claude Sonnet 5.5 got for the twin: a conversational message (8 to 30 words, the way a student or parent would type it) and a short indirect request (3 to 10 words), for the 4,223 of 4,706 chunks it judged a student would ask about. Pages held out for val were never written for.
• Judge: Qwen3-Reranker-0.6B dropped queries their own chunk doesn't answer (1,410 of 30,501) and kept hard negatives only when it judged they don't answer the query either, leaving 27,852 training queries (8,183 of them first messages). The College repeats itself across its site and catalog, so a close chunk is often another right answer.
• Objective: InfoNCE (τ = 0.05) against the right chunk, up to two hard negatives, and the rest of a 64-query batch. In-batch candidates are masked when they are another chunk of the right page, the same text, or scored above the right chunk by the base model (GISTEmbed's idea).
• Run: one epoch (435 steps), lr 1e-5, bf16, on one RTX 4090 (about 50 minutes). The best step (400) was picked by MRR@10 on held-out pages.

Intended use and limits

• Use it to find Indian River State College pages for students' questions and first messages in English, as the first stage of a search or RAG system.
• Facts live in the pages, not the model. When a fee, date, or policy changes, re-embed the changed pages; no retraining is needed. Retrain when the College's vocabulary changes (new programs, new names). RiverGuide re-embeds nightly and checks for drift weekly.
• Other colleges: it should still beat keyword search on general college questions, but it learned IRSC's names and programs. Retrain on your own pages for the full gain.
• Queries need the instruction prefix above.