torchlight-phi4-mini-GGUF

This repo has GGUF files of samadpls/torchlight-phi4-mini for llama.cpp, Ollama and LM Studio.

Model Overview

torchlight is a small model that helps with PyTorch questions. It is Phi-4-mini-instruct (3.8B parameters), fine-tuned on 6,918 chats from the PyTorch forums. Most end with the answer that solved the thread; the rest teach it to ask for missing details.

It was trained not to guess: when a question has enough detail, suggest a fix; when it doesn't, ask for the code or the full error.

• Trained on real answers. Each answer is the post that solved the thread. Most were written by ptrblck.
• Tested on new questions. 260 forum questions that are newer than anything the model saw in training.
• Trained on a laptop. 6.3 hours on a MacBook Pro (M3 Pro).

Model Details

Base model	microsoft/Phi-4-mini-instruct
Size	3.8B parameters (11.5M trained with LoRA)
Training data	samadpls/torchlight-threads
Language	English
License	MIT
Author	Abdul Samad
Code	github.com/samadpls/torchlight

The model comes in three versions:

Repo	What it is	Use it with
samadpls/torchlight-phi4-mini	Full model (16-bit)	transformers, mlx-lm
samadpls/torchlight-phi4-mini-lora	LoRA adapter only	PEFT, mlx-lm
samadpls/torchlight-phi4-mini-GGUF	GGUF files (Q4_K_M, Q8_0)	llama.cpp, Ollama, LM Studio

⚡ Quickstart

File	Size	Notes
torchlight-phi4-mini-Q4_K_M.gguf	2.5 GB	Smaller and faster; a good default
torchlight-phi4-mini-Q8_0.gguf	4.1 GB	Closer to the full model; about twice the size

llama.cpp

llama-cli -hf samadpls/torchlight-phi4-mini-GGUF:Q4_K_M -sys "You are a careful PyTorch helper. Give specific, correct guidance with code where it helps. If a question lacks the details needed to answer it, ask for them, for example a minimal executable code snippet or the full error message, instead of guessing."

Ollama

Set the system prompt with /set system once it starts.

ollama run hf.co/samadpls/torchlight-phi4-mini-GGUF:Q4_K_M

Use the same system prompt as in the examples. The model was trained with it.

📊 Evaluation

We asked 260 forum questions from 2025-01-01 to 2026-10-08 to four setups. Claude Opus 5.5, acting as judge, scored each answer from 1 to 5 by comparing it with the answer that solved the thread.

Model	Average grade (out of 5)	Graded 4 or 5	Wrong answers (graded 1)	Code with only real torch APIs	Asked for details, when they were missing	Asked for details, when not needed	Searches per question
Phi-4-mini (base model)	1.88	5%	39%	170 of 175	0 of 7	0%	0.0
Phi-4-mini + forum search	1.75	5%	45%	145 of 157	1 of 7	2%	0.1
torchlight (fine-tuned)	1.97	7%	27%	9 of 10	0 of 7	7%	0.0
torchlight + forum search	1.95	6%	26%	4 of 4	0 of 7	2%	0.0

What the results show

• Far fewer wrong answers. 27% of torchlight's answers were graded 1 (wrong), against 39% for the base model: 71 against 102. That gap is real, not chance (p = 0.003).
• Better than giving the base model forum search. torchlight averages 1.97 against 1.75 (95% interval +0.09 to +0.35). Search pulled the small base model off topic: on the 23 questions where it searched, it averaged 1.35.
• Short and fast. The median answer is 31 tokens, against 690 for the base model, and it answered all 260 questions in 9 minutes, against 32.
• The test is hard by design. The questions are real forum posts from 2025 and 2026, after the model's knowledge ends, and a 4 or 5 needs the same fix that solved the thread. Every small setup gets a 4 or 5 on only 5–7% of questions. torchlight raises the average grade from 1.88 to 1.97 and good answers from 14 to 19, but those gaps are too small to be sure of.
• torchlight never uses the search tool. None of its training chats included tools, so adding search changes nothing.
• A bigger or newer base model would likely do better. Fine-tuning taught the model the forum's style, but what it knows is limited by its size (3.8B parameters) and by data that ends in mid-2024. The same training on a larger or more recent model is the next step. We haven't tested it yet.

How the test works

• Setups. Phi-4-mini (the base model) and torchlight, each with and without forum search. Search covers 7,438 solved forum threads, up to twice per question.
• Settings. The local models use greedy decoding (no sampling) and up to 1,024 new tokens.
• Scoring. The judge sees all the answers to a question together, shuffled and without names. It follows prompts/judge.md.
• Valid torch APIs. A script checks that every torch.* name in an answer's code exists in PyTorch 2.15.
• Asks for details. How often an answer asks for more information. This is measured on the 7 questions where the expert also asked, and on the other 253, where asking less is better.
• Reference answer. For each question, the judge also scored an answer from Claude Sonnet 5.5 with the same search. It isn't in the table, which compares models of the same size.

Training Details

Data. 6,918 chats from forum threads created before 2024-07-01. Each chat ends with the answer that solved the thread. 7 chats longer than 4,096 tokens were left out. The 169 validation chats come from 2024-07-01 to 2025-01-01. The test questions are newer than both.

Settings.

Method	LoRA on the top 16 of 32 layers; the base weights stay frozen
LoRA rank / alpha	16 / 32
Learning rate	2e-4, cosine schedule, 43 warmup steps
Batch	2 chats, 4 accumulation steps (8 chats per update)
Epochs	1 (3,459 batches)
Max length	4,096 tokens
Loss	On the final answer only
Optimizer	AdamW, weight decay 0.01
Hardware	MacBook Pro (M3 Pro), 6.3 hours
Software	mlx-lm 0.32.2, MLX 0.32.4

Training results.

Epoch	Batch	Training loss	Validation loss
0.00	0	-	3.280
0.14	500	1.995	-
0.29	1,000	1.738	1.554
0.43	1,500	1.713	-
0.58	2,000	1.674	1.511
0.72	2,500	1.665	-
0.87	3,000	1.623	1.497
1.00	3,459	1.687	1.495

Training loss is the average since the previous row. Batch 0 is the base model before training.

Intended Use

Good for: PyTorch questions about errors, tensor shapes, training loops, data loading and CUDA.

Not for: code you run without reading, other frameworks, PyTorch features from after mid-2024, or anything where a wrong answer can cause harm.

⚠️ Limitations

• It is a small model and can be wrong while sounding sure. Check its code.
• On new forum questions it rarely finds the real fix on its own (see Evaluation). Use it for quick first ideas, not final answers.
• It asks for missing details less often than it should.
• With greedy decoding, about 1 answer in 65 gets stuck repeating a phrase. A repetition penalty or sampling avoids this.
• It learned the forum's style, so its answers are short. Sometimes they are too short to help on their own.
• It knows PyTorch up to mid-2024, not version 2.4 or later.
• The training data leans toward ptrblck's topics: debugging, CUDA, training loops and shape errors.
• English only.

License and Credits

The model is released under the MIT license, like the base model. The training data, samadpls/torchlight-threads, is under CC BY 4.0. The answers in it were written by the PyTorch forum community, above all ptrblck, and every row links to its thread. This model is not made or endorsed by PyTorch or the Linux Foundation.

Citation

@misc{samad2026torchlight_model,
  author = {Abdul Samad},
  title = {torchlight: a PyTorch helper fine-tuned on resolved forum threads},
  year = {2026},
  url = {https://huggingface.co/samadpls/torchlight-phi4-mini}
}

Contact

Made by Abdul Samad (samadpls.com · GitHub @samadpls). Questions or corrections: abdulsamadsid1@gmail.com.