jacobkuchta7/personalized-golf-assistant-rag

Personalized Golf Assistant RAG

1. Introduction

Generic golf advice from an off-the-shelf LLM has no access to a specific golfer's actual performance history, forcing it to either give vague, one-size-fits-all tips or, worse, confidently invent statistics that sound plausible but have no basis in reality. This need led me to build the Personalized Golf Assistant RAG, a system that answers golf coaching questions grounded in real personal performance data: TrackMan launch monitor sessions, round summaries, practice logs, instruction notes, and equipment notes. This project uses a Retrieval-Augmented Generation (RAG) pipeline to ground every recommendation in retrieved, real evidence from the user's personal performance history, paired with an explicit, user-specified skill-level calibration (I built and tested this using my own +3 handicap), so recommendations are judged against the correct standard for that specific golfer rather than a generic recreational benchmark.A model has no inherent way to know a specific individual's game, so without some form of external grounding, any attempt to answer personal performance questions would have to be invented rather than known. The alternative to RAG, manually describing a golfer's skill level and performance history within the conversation itself, is both tedious and rarely as rich or detailed as what a personalized retrieval corpus can provide. This project replaces that tedious manual grounding with an automated retrieval pipeline, turning personalized, evidence-based golf coaching from a hypothetical wish into a working reality. In practice, grounding the assistant in retrieved, personal performance data substantially reduces fabricated statistics compared to asking the model to assist without it.

2. Data

My retrieval corpus is built entirely from my own personal golf records, spanning five document types: round summaries, TrackMan launch monitor sessions, practice logs, instruction notes, and equipment notes. I then used these real documents as few-shot examples for an LLM to generate additional synthetic documents that closely matched the structure, statistical ranges, and content of my real data, increasing the size and diversity of the retrieval corpus while keeping it grounded in realistic values. To evaluate the RAG pipeline, I built a custom testing split of 12 hand-written question and ideal-response pairs covering common golf coaching scenarios, such as tournament preparation, strength and weakness assessment, and practice prioritization, which I used to test retrieval accuracy and response quality against my own personal performance history. This dataset, including both the retrieval corpus and the evaluation question set, can be found at jacobkuchta7/golf-rag-corpus.

3. Methodology

My retrieval setup uses dense embedding-based search. After testing multiple embedding and distance metric combinations against my own testing split, I settled on MiniLM-L6-v2 with cosine similarity as my embedding and retrieval configuration.

Metric	MiniLM-L6-v2 + Cosine	MiniLM-L6-v2 + Euclidean	MPNet-base-v2 + Cosine
Context Precision	1.000	1.000	0.978
Context Recall	1.000	1.000	0.900
Answer Similarity	0.654	0.654	0.633

The retrieved context is passed to the Phi-3.5-mini-instruct generator to synthesize the final coaching recommendation. I selected Phi-3.5-mini-instruct as my generator because it reliably stayed grounded and on-topic across zero-shot, three-shot, and eight-shot prompting during model testing, while offering a substantially smaller compute footprint than the largest model I evaluated.

Model	Parameters	Published Benchmark Reference	Zero-Shot	Three-Shot	Eight-Shot
Llama-3.2-1B-Instruct	1B	BFCL: 25.7	Stayed on-topic	Stayed on-topic	Drifted off-topic (answered about a different club than asked)
Phi-3.5-mini-instruct	3.8B	Avg 61.1 across 30+ benchmarks	Stayed on-topic	Stayed on-topic	Stayed on-topic
Llama-3.1-8B-Instruct	8B	IFEval: 80.4%, GSM8K: 84.5%	Stayed on-topic	Stayed on-topic	Minor formatting artifact (leftover template header), content still correct

The generation model uses a maximum token count of 450 and greedy decoding to produce consistent, reproducible responses given the same retrieved context. The embedding model is set to retrieve 5 documents from the FAISS index using cosine similarity.

4. Evaluation

Benchmark Results

I evaluated the pipeline's grounding and retrieval quality against three established benchmarks: RAGBench, RAGTruth, and SQuAD v2, alongside my own custom golf testing split. RAGBench and RAGTruth stress-test hallucination and retrieval-grounded reasoning, the two failure modes most relevant to this project. SQuAD v2 adds a complementary test of grounded reading comprehension: given a single provided passage, whether the model accurately extracts and uses the information it actually contains, rather than answering from general knowledge.

My custom golf testing split, RAGBench, RAGTruth, and SQuAD v2 were all evaluated using RAGAS with the following metrics:

• Faithfulness: Checks whether every claim in the generated answer can be traced back to something actually stated in the retrieved records, flagging fabricated or unsupported content.
• Context Precision: Measures how much of what was retrieved is genuinely useful, penalizing irrelevant documents that made it into the retrieved set.
• Context Recall: Measures whether the retrieved records collectively cover everything needed to fully answer the question, not just part of it.
• Answer Similarity: Compares the generated answer against a written reference answer using embedding similarity, capturing how close in meaning the two responses are.
• Answer Relevancy: Checks whether the generated answer actually addresses the specific question asked, independent of whether it matches any particular reference answer.

I compared my primary model, Phi-3.5-mini-instruct, against Llama-3.2-1B-Instruct and Llama-3.1-8B-Instruct, two similarly-purposed instruction-tuned generation models run through the identical retrieval pipeline.

Custom Golf Testing Split

Metric	Phi-3.5-mini-instruct	Llama-3.2-1B-Instruct	Llama-3.1-8B-Instruct
Answer Similarity (pre-RAG)	0.607	0.525	0.589
Faithfulness (post-RAG)	1.000	1.000	0.600
Context Precision (post-RAG)	1.000	1.000	1.000
Context Recall (post-RAG)	0.880	0.880	0.880
Answer Similarity (post-RAG)	0.644	0.607	0.653
Answer Relevancy (post-RAG)	0.402	0.447	0.583

RAGBench

Metric	Phi-3.5-mini-instruct	Llama-3.2-1B-Instruct	Llama-3.1-8B-Instruct
Answer Similarity (pre-RAG)	0.673	0.687	0.682
Faithfulness (post-RAG)	0.917	0.806	0.867
Context Precision (post-RAG)	0.611	0.611	0.611
Context Recall (post-RAG)	1.000	1.000	1.000
Answer Similarity (post-RAG)	0.686	0.707	0.905

SQuAD v2

Metric	Phi-3.5-mini-instruct	Llama-3.2-1B-Instruct	Llama-3.1-8B-Instruct
Answer Similarity (pre-RAG)	0.339	0.353	0.375
Faithfulness (post-RAG)	1.000	0.889	0.889
Context Precision (post-RAG)	1.000	1.000	1.000
Context Recall (post-RAG)	1.000	1.000	1.000
Answer Similarity (post-RAG)	0.480	0.554	0.616

RAGTruth (post-RAG only)

Metric	Phi-3.5-mini-instruct	Llama-3.2-1B-Instruct	Llama-3.1-8B-Instruct
Faithfulness	N/A	N/A	0.833
Answer Similarity	0.878	0.781	0.839

Across all four evaluation conditions, context precision and context recall were identical or near-identical between models, as expected, since retrieval depends on the shared embedding pipeline rather than the generator model. Adding retrieval substantially improved answer similarity over the no-retrieval baseline for every model across all three benchmarks, most dramatically on SQuAD v2, where Phi-3.5-mini's answer similarity rose from 0.339 to 0.480 with a full, perfect 1.000 on faithfulness, context precision, and context recall. I originally selected Phi-3.5-mini-instruct based on model testing that showed it reliably stayed grounded and on-topic without the regurgitation issues shown by Llama-3.2-1B-Instruct, and it matched Llama-3.1-8B-Instruct's output quality at roughly half the parameter count while never showing the minor formatting inconsistencies Llama-3.1-8B-Instruct exhibited under eight-shot prompting. Having now run this full evaluation, I would still select Phi-3.5-mini-instruct. It tied for the strongest faithfulness score on my golf testing split, won faithfulness outright on RAGBench, and won faithfulness outright on SQuAD v2 as well (1.000 versus 0.889 for both Llama models), a consistent pattern across every benchmark tested, directly supporting this project's central priority of avoiding hallucinated golf statistics. Llama-3.1-8B-Instruct scored higher on answer similarity on all three benchmarks, but for open-ended coaching questions like mine, there is often no single correct answer: multiple different prioritizations can be equally defensible depending on which retrieved evidence is weighted most heavily, which makes similarity to one written reference answer a noisier signal of quality than faithfulness, a more direct and less ambiguous measure of whether a response is actually grounded in real evidence. On the metric most directly tied to this project's core goal, Phi-3.5-mini-instruct outperformed or matched both comparison models on every single benchmark, at roughly half the parameter count and compute cost of Llama-3.1-8B-Instruct.

5. Usage and Intended Uses

This assistant is built to help a golfer turn their own performance history into specific, personalized practice recommendations, rather than relying on generic golf advice. It requires the user to supply their own recorded golf performance data (round summaries, launch monitor sessions, practice logs, instruction notes, and equipment notes) to build the retrieval corpus.

Below is an example of how the RAG pipeline is constructed and used to generate a grounded response.

import json
import torch
import pandas as pd
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
from langchain_core.documents import Document as LangchainDocument
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores.utils import DistanceStrategy

# --- SET YOUR HANDICAP ---
HANDICAP = "+3"  # e.g. "+3", "5", "12" -- used to calibrate recommendations to your skill level

def load_model(model_id):
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        device_map="auto",
        dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    )
    pipe = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        device_map="auto",
        max_new_tokens=450,
        do_sample=False,
        return_full_text=False,
    )
    return tokenizer, model, pipe

def make_instruction(question, retrieved_context, handicap):
    return f"""You are a personalized golf performance assistant.

The golfer you are advising has a {handicap} handicap. Judge all statistics
and performance relative to this specific skill level, not an average
recreational golfer.

Use only the retrieved golfer records below to answer the user's question.
Do not invent statistics, swing issues, or equipment details that are not supported by the retrieved records.

Your response should include:
1. Primary finding
2. Evidence from the records
3. Likely cause
4. Specific practice recommendation

User Question:
{question}

Retrieved Golfer Records:
{retrieved_context}

Answer:
"""

# --- BUILD THE RETRIEVAL INDEX FROM YOUR GOLF RECORDS ---
json_files = ["round_summaries.json", "trackman_sessions.json", "practice_logs.json",
              "instruction_notes.json", "equipment_notes.json"]

all_records = []
for path in json_files:
    with open(path, "r") as f:
        records = json.load(f)
    all_records.extend(records)

rag_df = pd.DataFrame(all_records)

documents = [
    LangchainDocument(page_content=json.dumps(row.dropna().to_dict()), metadata={"document_id": row["document_id"]})
    for _, row in rag_df.iterrows()
]

embedding_model = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2",
    encode_kwargs={"normalize_embeddings": True},
)
vector_db = FAISS.from_documents(documents, embedding_model, distance_strategy=DistanceStrategy.COSINE)

# --- GENERATE A RESPONSE ---
MODEL_ID = "microsoft/Phi-3.5-mini-instruct"
tokenizer, model, pipe = load_model(MODEL_ID)

question = "Is my driver a strength or a weakness right now?"
retrieved_docs = vector_db.similarity_search(question, k=5)
context = "\n\n---\n\n".join(doc.page_content for doc in retrieved_docs)
prompt = make_instruction(question, context, HANDICAP)

messages = [{"role": "user", "content": prompt}]
formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
output = pipe(formatted_prompt)
print(output[0]["generated_text"])

Prompt Format

Every response is generated using this fixed prompt template, which injects the retrieved golf records and the golfer's handicap directly into the instruction:

You are a personalized golf performance assistant.

The golfer you are advising has a {handicap} handicap. Judge all statistics
and performance relative to this specific skill level, not an average
recreational golfer.

Use only the retrieved golfer records below to answer the user's question.
Do not invent statistics, swing issues, or equipment details that are not supported by the retrieved records.

Your response should include:
1. Primary finding
2. Evidence from the records
3. Likely cause
4. Specific practice recommendation

User Question:
{question}

Retrieved Golfer Records:
{retrieved_context}

Answer:

Expected Output Format

Responses consistently follow a four-part structure: a primary finding, supporting evidence from specific retrieved records, a likely cause, and an actionable practice recommendation, as shown below:

Primary Finding:
Your driver performance is generally strong, with the ability to achieve solid total distances and a
playable right tendency. However, there are issues with carry on lower-strike shots and a noticeable
right miss pattern.

Evidence from the Records:
From the practice logs and launch monitor sessions, it is evident that you have been able to consistently
hit drives with club speeds above 100mph, resulting in ball speeds in the mid to high 150s. The average
carry for your driver shots is also within the playable range. However, the carry-target-hit statistics
show that you have not consistently hit the target carry, and there is a right miss pattern with the driver.

Likely Cause:
The right miss pattern and inconsistency in carry on lower-strike shots suggest that there may be a swing
issue, particularly with the transition from the backswing to the downswing or with the alignment of the
clubface at impact.

Specific Practice Recommendation:
Focus on improving your downswing transition and ensuring proper clubface alignment at impact. Work on
exercises that emphasize a smooth, controlled swing path and maintaining a square clubface throughout
the swing.

Limitations

The most significant challenge this project surfaced is that "correct" retrieval and "correct" advice are often genuinely ambiguous for open-ended coaching questions: reviewing individual responses made clear that multiple different prioritizations can be equally defensible given the same underlying records, and even as the domain expert who wrote the ideal reference answers, I found weighing which priority should come first to be difficult and somewhat subjective. This difficulty is compounded by the fact that the reverse is also true: responses can reach the same correct takeaway while citing different underlying records, since recurring patterns in a golfer's performance are often documented across multiple, redundant sources. This suggests the harder problem for this kind of system is not finding relevant evidence, but accurately synthesizing and prioritizing across multiple valid pieces of it, a finding that matched what I anticipated before building the pipeline. Additionally, my evaluation metrics (faithfulness, context precision, context recall) were graded by my own generator model, Phi-3.5-mini-instruct, acting as its own judge rather than an independent model, which is a known limitation of LLM-as-judge evaluation and should be kept in mind when interpreting the reported scores. The model's golf knowledge is also limited to what it learned during training, with no ability to look up current, real-world benchmarks to verify its comparisons. While the model's skill-level calibration already produces genuinely useful, handicap-relative recommendations, pairing a tool-calling-capable model with a live search API would let the system ground these kinds of general comparisons in current data rather than the model's static training knowledge, making an already useful feature even more reliable. Finally, this system depends entirely on the user having their own recorded golf performance data available to build the retrieval corpus; it cannot generate personalized recommendations for a golfer without existing logs or launch monitor sessions.