Instructions to use issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16")
model = AutoModelForCausalLM.from_pretrained("issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16

SGLang

How to use issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16 with Docker Model Runner:
```
docker model run hf.co/issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16
```

Qwen2.5-Coder-7B AutoRound W4A16

This repository contains a quality-oriented W4A16 quantization of Qwen/Qwen2.5-Coder-7B, intended primarily for low-latency code completion and fill-in-the-middle (FIM) inference.

This is the base model, not the instruction-tuned variant. It is most useful through the completions API with Qwen's FIM tokens.

Model details

Base model: Qwen/Qwen2.5-Coder-7B
Exact base revision: 0396a76181e127dfc13e5c5ec48a8cee09938b02
Quantizer: AutoRound 0.15.0
Weight format: INT4 / W4A16, symmetric, group size 128
Runtime activations: BF16
Quantized linear layers: 196 of 197
lm_head: BF16
Packing format: auto_round:auto_gptq
Checkpoint size: approximately 5.2 GiB
Measured model-weight VRAM: 5.17 GiB
Tested inference path: vLLM 0.27.1, automatic inc detection, MarlinLinearKernel

No re-quantization is required for NVIDIA Ampere GPUs. The intended target is an RTX A6000 48 GB, although the published reference measurements below were collected on an A100 PCIe 80 GB.

Quantization

The model was quantized with 1,000 AutoRound optimization iterations per layer and 512 decontaminated calibration sequences of 2,048 tokens each. The calibration mixture was:

50% FIM code
25% plain code
25% coding instruction plus code

The source dataset was nvidia/OpenCodeInstruct at revision 8f3ba5bafe4d6e8db46082cf7ae6741bc370604d. A 13-gram overlap check against HumanEval rejected 479 of 4,141 examined source records before calibration packing.

Quantization took 3,535.86 seconds on one A100 80 GB and peaked at 24.72 GB of VRAM. This is quantization memory, not inference memory.

Evaluation

A deterministic custom HumanEval FIM test preserved the first and final third of each canonical solution, generated the missing middle, reconstructed the solution, and executed all 164 tasks in a restricted container.

Model	Solved	Score
Qwen2.5-Coder-7B BF16	124/164	75.61%
This W4A16 checkpoint	129/164	78.66%

Pairwise, W4A16 lost four tasks solved by BF16 and gained nine tasks not solved by BF16. The aggregate result should therefore not be interpreted as proof that quantization generally improves model quality. This is a custom FIM evaluation, not the standard HumanEval generation protocol.

On an A100 PCIe 80 GB, using an uncached 1,024-token prompt and producing 64 tokens:

Concurrency	TTFT p50	Total latency p50	Decode median
1	94.47 ms	407.80 ms	200.95 tokens/s
2	177.73 ms	501.42 ms	194.83 tokens/s
3	262.33 ms	587.65 ms	193.81 tokens/s

These absolute numbers are not predictions for the RTX A6000. Re-run latency acceptance tests on the deployment GPU.

vLLM usage

The following is the recommended starting point for one to three small concurrent requests on an RTX A6000 48 GB:

vllm serve issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16 \
  --served-model-name autocomplete \
  --generation-config vllm \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --kv-cache-dtype auto \
  --gpu-memory-utilization 0.25 \
  --max-model-len 8192 \
  --max-num-seqs 3 \
  --max-num-batched-tokens 16384 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --performance-mode interactivity \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}'

The quantization mode should be detected automatically. N-gram speculative decoding is deliberately not recommended for the intended mix of small requests because it increased TTFT and tail latency in the reference tests.

Fill-in-the-middle prompt

prompt = (
    "<|fim_prefix|>" + prefix
    + "<|fim_suffix|>" + suffix
    + "<|fim_middle|>"
)

Send the prompt to a text-completions endpoint rather than treating the base model as a chat model.

Limitations

Quantization can change individual outputs even when aggregate benchmark quality is retained.
The model is not instruction tuned and is not a drop-in replacement for a chat assistant.
Only an 8,192-token runtime configuration was benchmarked for this checkpoint.
Generated code should be reviewed and executed in an appropriately isolated environment.
This model inherits the limitations and license conditions of the upstream Qwen model.

License and attribution

The checkpoint is a quantized derivative of Qwen/Qwen2.5-Coder-7B and is distributed under the Apache License 2.0. See LICENSE and the upstream model card for details.

Downloads last month: -

Safetensors

Model size

2B params

Tensor type

I32

BF16

F16

Model tree for issaxde/Qwen2.5-Coder-7B-AutoRound-W4A16

Base model

Qwen/Qwen2.5-7B

Finetuned

Qwen/Qwen2.5-Coder-7B

Quantized

(48)

this model