sasefi/avid-runner-chatbot-qwen

Avid Runner Chatbot

A retrieval-augmented generation (RAG) system that answers recreational runners' sports medicine questions by grounding responses in PubMed literature.

1. Introduction

Running is a sport that is very hard on one's body, thus recreational runners frequently turn to the internet for guidance on injury prevention, treatment, and training modifications. Unfortunately, LLMs created for more general tasks are not well suited to this feat since their corpus is usually the whole internet and not biomedical and medical documentation. They have no mechanism for citing sources, and they are prone to generating plausible-sounding but unverified medical claims which is a serious problem when the topic is injury management. This is especially important to me because when I was training for a marathon, I acquired a stress fracture. This and other overuse injuries are common, poorly understood by runners, and easy to misdiagnose from anecdotal advice found online. The Avid Runner Chatbot addresses this gap by pairing a general-purpose instruction-tuned LLM (Qwen2.5-7B-Instruct) with a retrieval pipeline over PubMed abstracts on running injuries, so that answers are grounded in and traceable to peer-reviewed biomedical literature rather than the model's individual pretrained information alone. Retrieval improved ROUGE-L across every evaluation setting tested — from 0.146 to 0.173 which is a +19% relative increase on my own held-out PubMed test split, and by 61–80% relative on three external RAG benchmarks (RAGBench, RAGTruth, PubMedQA), confirming that grounding generation in retrieved literature meaningfully improves answer quality over the model's parametric knowledge alone.

2. Data

The corpus consists of 662 PubMed abstracts collected via the NCBI Entrez API using seven targeted search terms ("running injury," "stress fracture runners," "marathon training," "overuse injury recreational athletes," "return to run protocol," "tendinopathy runners," "iliotibial band syndrome"). Each record was reformatted into an Instruction–Response pair: the Instruction asks what advice can be given a runner given a retrieved abstract as context, and the Response is a templated citation-style answer ("Based on a study published in [Journal] ([Year]), titled '[Title]': [Abstract]"). For the retrieval index, each abstract was converted into a LangChain Document (with PMID and Title retained as metadata) and split into 256-token chunks with a chunk overlap of chunk_size / 10 (~26 tokens) using a recursive character splitter. The dataset was shuffled and split 80/20 into training and test sets with random_state=42, yielding 529 training and 133 test examples. To evaluate the RAG pipeline specifically, I constructed 5 manually written test questions covering common running injuries (stress fracture prevention, safe return-to-running, Achilles tendinopathy, IT band syndrome, and stress fracture warning signs), used to qualitatively compare retrieval quality across embedding/distance-metric combinations before selecting a final retriever.

3. Methodology

I chose a RAG-only approach over fine-tuning after working through model editing in Homework 5 and working with techniques such as ROME, MEMIT AND WISE and LoRA fine-tuning Homework 6 using Qwen2.5-1.5B where both made clear that baking facts into model weights is expensive to keep current and risks catastrophic forgetting, which is dangerous in a domain like sports medicine where new studies regularly update best practice. Since a methodology like RAG requires no training, the fact that the knowledge base keeps growing with new added studiesand keeps every answer traceable to a specific source. My own experiments backed this up: across zero-shot, 3-shot, and 8-shot prompting with three candidate models (Qwen2.5-1.5B, Llama-3.2-3B, Qwen2.5-7B), Qwen2.5-7B-Instruct at 3-shot gave the most consistently grounded, source-cited responses with the least hallucination. The main anticipated drawback is that RAG is only as good as what it retrieves in it's own data repository, so an irrelevant retrieved abstract leads to confidently wrong answers. I tried to mitigate this by keeping the corpus filtered to running-specific search terms and checking that generated responses cite the actual retrieved source or at least cite a source that someone could fact check themselves.

The pipeline is built with LangChain and a FAISS vector store, generating with Qwen2.5-7B-Instruct (do_sample=False, max_new_tokens=250). I compared three embedding-model/distance-metric combinations on 5 manually-constructed test questions: all-MiniLM-L6-v2 + cosine, BAAI/bge-small-en-v1.5 + cosine, and BAAI/bge-small-en-v1.5 + Euclidean distance. MiniLM occasionally retrieved an off-topic result (e.g., an ultra-marathon performance paper for a stress-fracture-prevention question), while bge-small returned exclusively on-topic abstracts under either distance metric — the two bge-small variants returned identical top-3 documents for every test question, indicating the embedding model mattered more than the distance metric for this corpus. I selected BAAI/bge-small-en-v1.5 with cosine similarity, retrieving the top 3 chunks per query (k=3) as context for generation.

4. Evaluation

I evaluated on three RAG-specific benchmarks — RAGBench (pubmedqa subset), RAGTruth (QA subset, wandb/RAGTruth-processed), and PubMedQA (pqa_labeled, 1,000 expert-annotated examples) which were chosen because all three target retrieval grounded biomedical/medicine/QA generation rather than open-domain QA, making them a closer match to the Avid Runner task than general RAG benchmarks.

I also report ROUGE-L on the held-out test split of my own PubMed dataset (133 examples). Each benchmark and the PubMed test split were scored twice: once with no retrieved context (pre-RAG baseline, same model answering from parametric knowledge alone) and once with retrieval (post-RAG, using the full pipeline — for PubMedQA, the provided abstract passages are used as the retrieved context, and ROUGE-L is scored against the abstract's conclusion (long_answer) rather than the dataset's underlying yes/no/maybe label). For comparison models, I used the base model (Qwen2.5-7B-Instruct with no retrieval) to isolate the effect of RAG, plus Qwen2.5-1.5B-Instruct and Llama-3.2-3B-Instruct as similarly-scoped instruction-tuned models with plausible baseline performance on the same task, drawing on the zero/3/8-shot comparison I ran earlier in the project.

Model / Condition	PubMed Test Split (ROUGE-L)	RAGBench (pubmedqa)	RAGTruth (QA)	PubMedQA (pqa_labeled)
Qwen2.5-7B-Instruct + RAG (this model)	0.173	0.339	0.352	0.224
Qwen2.5-7B-Instruct (no retrieval, base)	0.146	0.198	0.218	0.124
Qwen2.5-1.5B-Instruct + RAG	0.170	0.270	0.298	0.207
Llama-3.2-3B-Instruct + RAG	0.174	0.273	0.365	0.237

5. Usage and Intended Uses

This model is intended for informational use by recreational runners researching common overuse injuries such as stress fractures, IT band syndrome, plantar fasciitis etc. and for retrieving supporting literature alongside generated answers. It is not intended to replace medical diagnosis or treatment advice from a qualified clinician. If you believe you have a real medical emergency please visit a medical professional immediately.

import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from langchain_community.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings

HF_TOKEN = os.environ.get("HF_TOKEN")

class AvidRunnerRAGPipeline:
    def __init__(self, model_name: str, embedding_model_name: str, vector_db_path: str):
        self.max_new_tokens = 250

        print(f"Loading Model: {model_name}...")
        self.tokenizer = AutoTokenizer.from_pretrained(model_name, token=HF_TOKEN)

        # Load to CPU first — ZeroGPU has no GPU available during startup;
        # weights move to GPU automatically at call time.
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name, device_map="cpu", torch_dtype=torch.bfloat16, token=HF_TOKEN
        )
        self.tokenizer.pad_token_id = self.tokenizer.eos_token_id
        self.tokenizer.padding_side = "left"

        print("Loading Embeddings...")
        self.embedding_model = HuggingFaceEmbeddings(
            model_name=embedding_model_name,
            model_kwargs={"device": "cpu"},
            encode_kwargs={"normalize_embeddings": True},  # matches the winning cosine-similarity retriever
        )

        print(f"Loading Vector DB from {vector_db_path}...")
        self.vector_db = FAISS.load_local(vector_db_path, self.embedding_model, allow_dangerous_deserialization=True)
        print("RAG Pipeline Initialized (CPU Mode)")

    def retrieve(self, query, num_docs=3):
        return self.vector_db.similarity_search(query, k=num_docs)

    def _format_prompt(self, query, retrieved_docs):
        context = "\n\n".join(
            f"[PMID {d.metadata.get('PMID', 'N/A')}] {d.metadata.get('Title', 'Untitled')}:\n{d.page_content}"
            for d in retrieved_docs
        )
        messages = [
            {
                "role": "system",
                "content": (
                    "You are a sports medicine assistant helping recreational runners. "
                    "Using only the context provided below, answer the question accurately and concisely.\n\n"
                    f"Context:\n{context}"
                ),
            },
            {"role": "user", "content": query},
        ]
        return self.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

    def generate(self, query, num_docs=3):
        retrieved_docs = self.retrieve(query, num_docs)
        prompt_str = self._format_prompt(query, retrieved_docs)
        inputs = self.tokenizer(prompt_str, return_tensors="pt").to(self.model.device)
        with torch.no_grad():
            outputs = self.model.generate(
                **inputs, max_new_tokens=self.max_new_tokens,
                do_sample=False, pad_token_id=self.tokenizer.eos_token_id,
            )
        input_len = inputs.input_ids.shape[1]
        return self.tokenizer.decode(outputs[0][input_len:], skip_special_tokens=True).strip()


# --- Example ---
rag = AvidRunnerRAGPipeline(
    model_name="Qwen/Qwen2.5-7B-Instruct",
    embedding_model_name="BAAI/bge-small-en-v1.5",
    vector_db_path="index/",
)
print(rag.generate("What advice can you give a runner regarding stress fracture prevention during marathon training?"))

6. Prompt Format

Prompts use a system-then-question structure, applied through the tokenizer's apply_chat_template so the same code works across backbone models. The system message sets the sports-medicine assistant role, instructs the model to answer only from the provided context, and embeds the retrieved abstract chunks directly; the user message carries the question.

System:
    You are a sports medicine assistant helping recreational runners.
    Using only the context provided below, answer the question accurately and concisely.

    Context:
    {retrieved PubMed abstract chunks — top-3 by cosine similarity}

User:
    {question}

Example, filled in:

Question: What advice can you give a runner regarding A High Rate of Return to Running Is
Seen After Both Arthroscopic and Open Shoulder Surgery.?

7. Expected Output Format

The model returns a plain-English answer grounded in the retrieved context. At the pipeline level, every response is logged alongside its source question and the retrieved source titles, so an answer stays traceable to specific PubMed records even though the generated text itself doesn't always name a source explicitly.

What the user sees:

Based on the study, you can advise that after both arthroscopic and open shoulder surgery,
most patients are able to run 1 mile by 3 months postoperatively and about three-quarters
can do so by 4.5 months. It's important to follow the rehabilitation protocol, which
discourages running before 3 months, but once cleared by your healthcare provider, you can
aim to gradually return to running as part of your recovery process.

What actually gets logged (one real row from results/pubmed_rag_results.csv):

{
  "question": "What advice can you give a runner regarding A High Rate of Return to Running Is Seen After Both Arthroscopic and Open Shoulder Surgery.?",
  "retrieved_sources": [
    "A High Rate of Return to Running Is Seen After Both Arthroscopic and Open Shoulder Surgery.",
    "Pre-operative comparisons for a return to running protocol in recreational athletes following hip arthroscopy.",
    "A High Rate of Return to Running Is Seen After Both Arthroscopic and Open Shoulder Surgery."
  ],
  "response": "Based on the study, you can advise that after both arthroscopic and open shoulder surgery, most patients are able to run 1 mile by 3 months postoperatively and about three-quarters can do so by 4.5 months. It's important to follow the rehabilitation protocol, which discourages running before 3 months, but once cleared by your healthcare provider, you can aim to gradually return to running as part of your recovery process."
}

Note the duplicate title in retrieved_sources — the same paper was retrieved via two different chunks. This is a real logged example, not a cleaned-up illustration, and it's the retrieval-diversity issue flagged in Limitations below.

8. Limitations

Since this model is a RAG model, it faces a large limitation where it is heavily dependent on the information that is pulled from the corpus which are the PubMed abstracts. This model is also limited in the terms that are used to search those specific abstracts. When creating the pipeline, words that are more specific to running were used in order to ensure that random information was not pulled from abstracts that are not related to running. Less common injuries caused by running may retrieve irrelevant information and is an edge case that needs to be addressed in future iterations. Retrieval can also return multiple chunks from the same source abstract rather than genuinely diverse evidence in one logged example, two of the top-3 retrieved chunks were both drawn from the same paper which narrows the evidence a response is actually grounded in even though three chunks were retrieved.

The size of the retrieval improvement also varies lots by the evaluation target. The pubmed split only increased by 19% whereas there was a marginally much higher increase for the three external benchmarks.