namquangstudy/aaie-ddense-gft-llama

AAIE-Distilled Dense Instruct (Llama-compatible export)

A LlamaForCausalLM-format export of AAIE-Distilled Dense Instruct — same weights, same architecture, just registered under vLLM's/transformers' native Llama implementation instead of custom trust_remote_code.

Why this exists

AAIE-Distilled Dense's architecture — GQA attention, RoPE, SwiGLU FFN, RMSNorm, tied embeddings, no biases anywhere — happens to be structurally identical to Llama's, just with different tensor/field names (d_model vs hidden_size, n_q_heads vs num_attention_heads, etc.). This export is a pure state_dict + config remap, not a retrain: every tensor is copied unchanged into the equivalent Llama-named slot. Verified byte-for-byte numerically identical to the original via a CPU-only, trust_remote_code-free transformers sanity check (same prompt, same greedy decode, same output text).

Payoff: this loads natively in vLLM (no trust_remote_code, no custom model backend) using vLLM's own hand-optimized LlamaForCausalLM implementation — confirmed working end-to-end, generating at ~430 tok/s, vs. ~2 tok/s for the original architecture's uncached generation loop.

Usage with vLLM

from vllm import LLM, SamplingParams

llm = LLM(model="namquangstudy/aaie-ddense-gft-llama")
params = SamplingParams(max_tokens=150, temperature=0.8)
out = llm.generate(["What is a database index?\n\nResponse:"], params)
print(out[0].outputs[0].text)

Environment note: if you hit a RuntimeError: The NVIDIA driver on your system is too old error, it means the vLLM/torch build you pip-installed targets a newer CUDA than your GPU node's driver supports (this happened during testing: pip install vllm grabbed torch 2.13/CUDA 13, which failed on a driver that only supports CUDA 12.2). This has nothing to do with this specific model — the exact combination confirmed working here is:

pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121
pip install vllm==0.6.3.post1
pip install transformers==4.46.3

If you also have a broken/incompatible tensorflow install under ~/.local (common on shared clusters), set PYTHONNOUSERSITE=1 before running — conda/venv environments still see ~/.local/lib/pythonX.Y/site-packages by default, and transformers will try to import tensorflow during model loading and fail if that install is broken.

Usage with plain transformers (no vLLM)

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("namquangstudy/aaie-ddense-gft-llama")
model = AutoModelForCausalLM.from_pretrained("namquangstudy/aaie-ddense-gft-llama")
# no trust_remote_code needed - this is a plain LlamaForCausalLM

Full details

Architecture, training data, hyperparameters, and evaluation results (LLM-judge score 7.50/10 on held-out IT/CS assignment-feedback topics) are documented on the original AAIE-Distilled Dense Instruct model card — this export carries the exact same weights and behavior, just packaged for vLLM/ecosystem compatibility.