Teaching Phi-4-mini the dotnet/runtime codebase
kotlarmilos
• Phi-4-mini fine-tuned on dotnet/runtime
This is a LoRA fine-tune of microsoft/Phi-4-mini-instruct adapted to the issue and pull request history of the dotnet/runtime codebase. The goal is to move a small, efficient base model toward the vocabulary, conventions, and recurring problems of one specific engineering domain so it reads and responds in that domain's dialect.
Where this fits
This is the first step in my applied post-training track. Here I change what a small model knows by fitting it to a domain I understand. The next step, the Gemma 3 reasoning adapter, changes how a model thinks rather than what it knows. The step after that, the Gemma 4 GlucoLens adapter, takes the same instinct into a domain where the output is a structured rollout and the model has to refuse when it is unsure. In parallel I built a transformer by hand in gpt2-nano to understand the layer underneath all of this.
Model details
- Developed by Milos Kotlar
- Base model microsoft/Phi-4-mini-instruct
- Method LoRA with 4-bit NF4 quantization
- Language English
- License MIT
- Repository https://github.com/kotlarmilos/phi-4-mini-dotnet-runtime
- Demo https://huggingface.co/spaces/kotlarmilos/dotnet-runtime
Intended use
The model is meant for assistance on the dotnet/runtime domain, reading issues and pull requests and drafting responses in the terminology and style of that codebase. It is a research artifact, not a production reviewer.
Out of scope
The model is not built for factual retrieval, and it can produce plausible but wrong statements. It is not a source of professional medical or legal advice, and it is not suitable for safety critical systems. Do not use it to generate harmful or misleading content.
How to load
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
BASE = "microsoft/Phi-4-mini-instruct"
ADAPTER = "kotlarmilos/phi-4-mini-dotnet-runtime"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(BASE, use_fast=True)
base = AutoModelForCausalLM.from_pretrained(
BASE, quantization_config=bnb_config, device_map="auto", trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, ADAPTER)
def generate(prompt):
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
**inputs, max_new_tokens=256, do_sample=True, temperature=0.7,
pad_token_id=tokenizer.eos_token_id,
)
return tokenizer.decode(output[0], skip_special_tokens=True)
print(generate("Review the following code changes:"))
Training
- Data. About 10,000 instruction and response pairs built from dotnet/runtime GitHub issues and pull requests, published as the dotnet-runtime dataset.
- Method. LoRA on the quantized base model, mixed precision.
- Quantization. 4-bit NF4 with BitsAndBytes.
LoRA configuration
| Parameter | Value |
|---|---|
| Rank r | 8 |
| Alpha | 16 |
| Dropout | 0.05 |
| Target modules | qkv_proj, gate_up_proj |
| Task type | CAUSAL_LM |
Evaluation
I do not report a held-out benchmark score for this model. The effect of fine-tuning is a shift toward the repository's terminology and issue framing relative to the base model on the same prompts. A labeled evaluation split drawn from held-out issues is the natural next step. See the repository for details.
Source
- Downloads last month
- 32
Model tree for kotlarmilos/phi-4-mini-dotnet-runtime
Base model
microsoft/Phi-4-mini-instruct