Instructions to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B")
model = AutoModelForCausalLM.from_pretrained("ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B

SGLang

How to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B with Docker Model Runner:
```
docker model run hf.co/ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B
```

Qwen2.5-Coder-7B-Instruct — DPO Fine-Tuned for Python

Model at a Glance

This model is a Python-focused, DPO fine-tuned version of Qwen2.5-Coder-7B-Instruct, designed to improve the model's ability to generate functionally correct Python code.

The model was trained using Direct Preference Optimization (DPO) with automatically constructed preference pairs. Candidate Python solutions were generated for programming problems and evaluated by executing them against unit tests. Solutions that passed the required tests were used as preferred examples, while solutions that failed tests were used as rejected examples.

The resulting model was evaluated against the original Qwen2.5-Coder-7B-Instruct model on held-out programming problems.

What changed?

The base model: Qwen/Qwen2.5-Coder-7B-Instruct

was fine-tuned using: DPO + LoRA + execution-based preference data

The resulting model demonstrates improved coding performance on the evaluated held-out benchmark:

Metric	Qwen2.5-Coder-7B-Instruct	DPO Model	Improvement
Pass@1	41.87%	45.22%	+3.35 pp
Pass@5	48.87%	51.10%	+2.23 pp
Pass@10	50.99%	53.16%	+2.17 pp
Tests passed per answer	66.20%	69.41%	+3.21 pp

The primary result is the improvement in Pass@1, meaning that a single generated solution was more likely to produce a correct solution on the evaluated programming problems. The model also showed improved Pass@5, Pass@10, and partial test coverage.

VRAM Requirements

The base model and the merged trained model use the same VRAM. The merge changes weight values, not tensor shapes: both models have the same 7.6B parameters and architecture, so the model weights occupy the same amount of bf16 memory.

Component	bf16 VRAM
Weights (base or merged D4)	14.19 GiB
KV cache, per token	56 KiB
Activations / workspace	~0.5–1 GiB
Practical single-stream total	~15–16 GiB

Base model: Qwen/Qwen2.5-Coder-7B-Instruct at snapshot c03e6d358207e414f1eca0bb1891e29f1db0e242. The merge was performed against exactly these weights.

Downloading and Using

The merged model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B", dtype="bfloat16", device_map="auto")

tok = AutoTokenizer.from_pretrained("ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B")

msgs = [{"role": "user", "content": "Write a Python function that merges overlapping intervals."}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))

Or fetch the files without loading them (~14 GB, four shards):

hf download ranjanrajib/DPO-trained-Qwen2.5-Coder-7B --exclude "adapter/*" --local-dir ./d4

The LoRA adapter only (308 MB)

The adapter lives in adapter/. Applying it to the base model gives the same result as the merged weights, at a fraction of the download size.

hf download ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B --include "adapter/*" --local-dir ./d4-lora

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM

BASE = "Qwen/Qwen2.5-Coder-7B-Instruct"
REVISION = "c03e6d358207e414f1eca0bb1891e29f1db0e242"  # the snapshot this was trained against

base = AutoModelForCausalLM.from_pretrained(
    BASE, revision=REVISION, dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(
    base, "DPO-trained-Qwen2.5-Python-Coder-7B", subfolder="adapter"
)

A worked example: a prompt in, code out

  The prompt

  PROMPT = """You are an expert Python programmer.

  Solve the following programming problem.

  Problem:
  Calculate factorials for a list of numbers in parallel using multiprocessing.

  It should return:
  dict[int, int]: A dictionary with numbers as keys and their factorial as values.

  It should raise:
  ValueError: If any element in the input list is not an integer or is negative.

  It may use:
  multiprocessing.Pool
  math.factorial.

  Write a function with this exact signature:

      def task_func(numbers: list) -> dict:

  It must satisfy:

      >>> factorials = task_func([5, 6, 7, 8, 9])
      >>> factorials[5] == 120 and factorials[9] == 362880
      True

  Required function signature:
  def task_func(numbers: list) -> dict:

  Requirements:
  - Implement the requested function.
  - Follow the function signature.
  - Handle the specified edge cases.
  - Use Python.
  - Return only the implementation.
  - Do not provide an explanation.
  - Do not use eval().
  - Do not use exec().
  - Do not perform network operations.
  - Do not read or write files.
  """

### Generating


  import re
  import torch
  from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed

  MODEL = "ranjanrajib/DPO-trained-Qwen2.5-Coder-7B"

  model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16, device_map="auto")
  tok = AutoTokenizer.from_pretrained(MODEL)

  text = tok.apply_chat_template(
      [{"role": "user", "content": PROMPT}], tokenize=False, add_generation_prompt=True)
  inputs = tok(text, return_tensors="pt").to(model.device)

  set_seed(159000)                      # the seed this particular sample was drawn with
  out = model.generate(
      **inputs,
      do_sample=True,
      temperature=0.2,
      top_p=0.95,
      max_new_tokens=512,
      pad_token_id=tok.eos_token_id,
  )

  response = tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
  code = re.search(r"```python\n(.*?)```", response, re.S).group(1)
  print(code)

### What the model returned

  84 generated tokens, reproduced verbatim:

  import multiprocessing
  import math

  def factorial_task(number):
      if not isinstance(number, int) or number < 0:
          raise ValueError(f"Invalid input: {number}")
      return number, math.factorial(number)

  def task_func(numbers: list) -> dict:
      with multiprocessing.Pool() as pool:
          results = pool.map(factorial_task, numbers)
      return dict(results)

Why Use This Model?

If your primary use case is Python code generation, this model is intended to be a stronger alternative to the original Qwen2.5-Coder-7B-Instruct model based on the evaluation performed in this project.

It is particularly relevant for users who want to:

Generate Python functions and implementations
Solve programming problems
Generate code that needs to satisfy functional requirements
Improve first-attempt coding correctness
Experiment with DPO-trained code-generation models
Study execution-based preference optimization
Build research or evaluation pipelines for Python code generation

The main motivation for the fine-tuning was not simply to make the model generate more code, but to encourage the model toward solutions that work. The preference data therefore used actual program execution and unit-test results rather than subjective judgments about whether generated code looked correct.

How Is It Different From Qwen2.5-Coder-7B-Instruct?

The underlying model architecture and capabilities come from Qwen2.5-Coder-7B-Instruct. This model adds a preference-optimization stage on top of that model.

Conceptually:

Qwen2.5-Coder-7B-Instruct
           │
           ▼
    Generate multiple Python
       candidate solutions
           │
           ▼
      Execute candidates
       against tests
           │
      ┌────┴────┐
      ▼         ▼
    Pass       Fail
      │         │
      ▼         ▼
    Chosen   Rejected
      │         │
      └────┬────┘
           ▼
          DPO
           │
           ▼
      LoRA Adapter
           │
           ▼
    Python-focused model

The goal is to shift the model toward coding behaviors associated with solutions that successfully execute against the available tests.

Important distinction

This is not a new foundation model. It is a parameter-efficient fine-tuned adapter for Qwen2.5-Coder-7B-Instruct. The base model remains unchanged.

What Kind of Model Is This?

Primary task: Python code generation
Model family: Qwen2.5-Coder
Base model: Qwen/Qwen2.5-Coder-7B-Instruct
Fine-tuning: Direct Preference Optimization (DPO)
Parameter-efficient method: LoRA
Precision: bfloat16
Quantization: None
Model size: 7.7B-parameter base model
Adapter: Approximately 80.7M trainable parameters

Performance

The model was evaluated against the original Qwen2.5-Coder-7B-Instruct using a controlled benchmark consisting of 508 held-out BigCodeBench programming problems. The same prompts, generation settings, random seeds, number of samples, and grading procedure were used for both models.

Pass@k

Metric	Base Model	DPO Model
Pass@1	41.87%	45.22%
Pass@5	48.87%	51.10%
Pass@10	50.99%	53.16%

The improvement in Pass@1 is approximately 3.35 percentage points.

Partial Correctness

The average proportion of tests passed by each generated answer increased from 66.20% → 69.41%. This is a secondary metric intended to provide additional information about functional correctness beyond binary problem-level pass/fail.

Paired Problem-Level Analysis

The evaluation used the same 508 problems for both models, allowing direct comparison of model behavior on individual problems. For the first generated answer:

Base → DPO	Problems
Correct → Correct	186
Correct → Incorrect	17
Incorrect → Correct	41
Incorrect → Incorrect	264

The DPO model converted 41 problems from incorrect to correct while 17 previously correct problems became incorrect. This provides additional evidence that the aggregate improvement is associated with meaningful changes in problem-level behavior rather than simply differences in the evaluated problem sets.

Who Should Use This Model?

Python Developers

Developers interested in experimenting with a locally hosted code-generation model that has been specifically fine-tuned toward execution-validated solutions.

AI/ML Researchers

Researchers studying DPO, preference learning, LLM alignment, code-generation models, automated preference construction, execution-based evaluation, and parameter-efficient fine-tuning.

LLM Evaluation Researchers

The model can also be useful as an experimental model for comparing base versus preference-tuned models, coding correctness, Pass@k, partial test coverage, and preference-learning behavior.

When Should You Prefer the Original Qwen Model?

This model should not automatically be considered better for every possible task. The experiment specifically focused on Python code generation. If your application requires broad general-purpose instruction following, other programming languages, or capabilities that were not evaluated here, you should benchmark both models for your specific workload.

Improved performance on the evaluated Python coding benchmark, not a guarantee of improvement across every coding or general-purpose task.

Model Architecture and Training

The model starts from Qwen/Qwen2.5-Coder-7B-Instruct containing approximately 7.7 billion parameters. Rather than updating all parameters, the experiment uses LoRA-based parameter-efficient fine-tuning. The resulting adapter contains approximately 80.7 million trainable parameters and is ~308 MB in size.

DPO Training

Why DPO?

Direct Preference Optimization provides a method for optimizing a language model directly from preference pairs without requiring a separate reward-model training stage. For this experiment, a preference pair has the form:

Problem
   │
   ├── Candidate A → passes tests → chosen
   │
   └── Candidate B → fails tests  → rejected

DPO then learns to increase the relative likelihood of the chosen response compared with the rejected response.

Preference Data Construction

The preference dataset was generated specifically for this experiment, constructed from programming problems drawn from BigCodeBench, MBPP, HumanEval, and handwritten programming problems.

Training Data Sources

Dataset	License
BigCodeBench	Apache 2.0
MBPP	CC BY 4.0
HumanEval	MIT

Candidate solutions were generated automatically using five strategies: normal, straightforward, edge-case-focused, alternative, and optimized. Every candidate was then executed in a Docker sandbox against the associated unit tests to form chosen/rejected preference pairs.

Preference Dataset

After filtering and deduplication:

1,326 preference pairs
221 programming problems
933 training pairs
114 validation pairs

The dataset was split by problem, rather than by individual preference pair, to prevent different candidate solutions from the same underlying programming problem from being distributed across training and validation. No human preference labels or external LLM judges were used; the preference signal was generated entirely from objective unit-test execution.

DPO Configuration

Parameter	Value
Rank	32
Alpha	64
Dropout	0.05
Target modules	Attention and MLP projections
Epochs	2
Maximum sequence length	1,024
Batch size	4
Gradient accumulation	4
Effective batch size	16
Optimizer	AdamW
Gradient clipping	1.0
Gradient checkpointing	Enabled

Training Hardware

GPU: NVIDIA RTX A6000 — 48 GB VRAM
Environment: CUDA 12.8, PyTorch 2.11, bfloat16
Peak Memory: Approximately 33.4 GiB

Evaluation Methodology

The primary evaluation benchmark consisted of 508 BigCodeBench programming problems reserved separately from the preference-training pool.

Generation Settings

10 answers per problem
Temperature: 0.2
Top-p: 0.95
Maximum new tokens: 512
Repetition penalty: 1.0
Sampling enabled
Identical random seeds for base and DPO models

Preference-Learning Diagnostics

The DPO training objective produced a strong preference signal on the training examples, but validation results were substantially weaker:

Metric	Training	Validation
Preference accuracy	~84.4%	~45.8%
Reward margin	Positive	Approximately neutral

This indicates that the explicit preference signal learned during training did not generalize strongly to the validation preference examples, even though downstream coding performance improved on the held-out benchmark.

What Does This Mean?

The results suggest that optimizing an automatically constructed preference objective can improve downstream coding performance even when explicit preference metrics show weak validation generalization. Potential explanations include:

Differences between training and validation preference distributions
Imperfect test-derived preference labels
Overfitting to characteristics of training preference pairs
Changes in generation behavior not fully captured by preference accuracy

Important Limitations

No Human Preference Evaluation: The preference signal was derived exclusively from automated unit-test execution.
Unit Tests Are Imperfect: Passing available tests does not guarantee universal correctness or coverage of all edge cases.
Benchmark Source Overlap: Evaluation problems were held out at the problem level, but preference training data includes other BigCodeBench problems.
Python Focus: Performance should not be assumed to generalize to other programming languages.
Experimental Model: Users should evaluate it against their own workloads before production use.

Independent Research Project

This model was developed as an independent, out-of-office AI/ML research project. The complete experimental lifecycle:

Research Question
       ↓
Experimental Design
       ↓
Preference Data Generation
       ↓
Automated Evaluation
       ↓
DPO + LoRA Training
       ↓
Controlled Benchmarking
       ↓
Statistical / Paired Analysis
       ↓
Model Release

Model Release & Licensing

The trained LoRA adapter and technical documentation are available on Hugging Face: 🔗 ranjanrajib/DPO-trained-Qwen2.5-Coder-7B

License & Attribution: Apache-2.0, inherited from Qwen/Qwen2.5-Coder-7B-Instruct. This is a modified derivative of that model by the Qwen team at Alibaba Cloud.

Summary

What is this? A Python-focused DPO fine-tuned version of Qwen2.5-Coder-7B-Instruct.
What is different? Trained using LoRA-based Direct Preference Optimization with automatically constructed preferences derived from unit-test execution.
Does it improve coding performance? Yes. Pass@1 improved from 41.87% to 45.22%, with improvements also observed in Pass@5, Pass@10, and partial test coverage.
Is it human-aligned? No human preference evaluation was performed. Signals were generated automatically from unit-test outcomes.
Who should use it? Users interested in Python code generation, coding experiments, DPO research, preference learning, and execution-based LLM evaluation.

Downloads last month: -

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for ranjanrajib/DPO-trained-Qwen2.5-Python-Coder-7B

Base model

Qwen/Qwen2.5-7B

Finetuned

Qwen/Qwen2.5-Coder-7B