deepseek-coder-6.7b-fim-reth-v1

A LoRA fine-tuned adapter for deepseek-ai/deepseek-coder-6.7b-base, specialized for Fill-in-the-Middle (FIM) code completion on Rust codebases.

Model Details

• Base Model: deepseek-ai/deepseek-coder-6.7b-base
• Fine-tuning Method: LoRA (Low-Rank Adaptation) with 4-bit quantization
• Training Data: AST-extracted code spans from the reth Rust codebase
• Task: Fill-in-the-Middle code completion (predict missing code between prefix and suffix)

Training Configuration

Parameter	Value
LoRA Rank	64
LoRA Alpha	128
LoRA Dropout	0.05
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Quantization	4-bit (nf4, double quantization)
Learning Rate	2e-5
Epochs	3
Optimizer	paged_adamw_8bit
LR Scheduler	cosine
Max Sequence Length	4096
Precision	bfloat16

FIM Format

This model uses the DeepSeek Coder FIM token format:

<｜fim▁begin｜>[code before cursor]<｜fim▁hole｜>[code after cursor]<｜fim▁end｜>[generated completion]

Usage

With PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/deepseek-coder-6.7b-base",
    device_map="auto",
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(base_model, "viplismism/deepseek-coder-6.7b-fim-reth-v1")
tokenizer = AutoTokenizer.from_pretrained("viplismism/deepseek-coder-6.7b-fim-reth-v1")

prompt = "<｜fim▁begin｜>fn add(a: i32, b: i32) -> i32 {\n    <｜fim▁hole｜>\n}<｜fim▁end｜>"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.2)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With Ollama

After merging adapters and converting to GGUF:

ollama create deepseek-coder-6.7b-fim -f Modelfile
ollama run deepseek-coder-6.7b-fim

Training Framework

This model was trained using fim-coder-model, a pipeline that:

1. Extracts semantic code boundaries (functions, structs, impl blocks) via Rust AST parsing
2. Generates FIM training samples with proper prefix/suffix/middle splits
3. Fine-tunes with LoRA + 4-bit quantization for efficient multi-GPU training

Limitations

• Tuned on Rust code from a single repository (reth); it generalizes less well to other languages or coding styles.
• FIM-only: it expects the FIM prompt format above, not free-form chat or instructions.
• Inherits the biases and limitations of the deepseek-coder-6.7b-base model.

License

The base model is governed by the DeepSeek License Agreement; use of this adapter is subject to the same terms.