Instructions to use hkust-aaron/EdgeRock-4B-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hkust-aaron/EdgeRock-4B-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="hkust-aaron/EdgeRock-4B-GPTQ", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("hkust-aaron/EdgeRock-4B-GPTQ", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hkust-aaron/EdgeRock-4B-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hkust-aaron/EdgeRock-4B-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hkust-aaron/EdgeRock-4B-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/hkust-aaron/EdgeRock-4B-GPTQ

SGLang

How to use hkust-aaron/EdgeRock-4B-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hkust-aaron/EdgeRock-4B-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hkust-aaron/EdgeRock-4B-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hkust-aaron/EdgeRock-4B-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hkust-aaron/EdgeRock-4B-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use hkust-aaron/EdgeRock-4B-GPTQ with Docker Model Runner:
```
docker model run hf.co/hkust-aaron/EdgeRock-4B-GPTQ
```

EdgeRock-4B-GPTQ: 4-bit Constant-Memory Reasoning

GPTQ W4A16 weights for Rock Matrix's EdgeRock-4B, with a packed INT4 CUDA runtime.

EdgeRock combines recurrent state, sliding-window attention and Rock Memory Anchors (RMA) to retain selected distant information within constant-size persistent history. This 4-bit text-generation release preserves the source architecture and memory capacities: a 1,024-token local window, 1,024 anchors per KV group and 256-token commit boundaries.

EdgeRock-4B 的 GPTQ 四位量化版本，保留 RMA 记忆锚点和恒定容量的历史状态，附 GPU 加载、推理与验证代码。

Quick start · Original model · Model collection · GitHub · Technical report · arXiv

Package

Item	Configuration
Source	rockmatrix-ai/EdgeRock
Quantization algorithm	sequential Hessian-based GPTQ, 1% damping, block size 128
Weights / activations	INT4 / BF16
Group size	128
Quantized matrices	256 large backbone and RMA linear matrices
Packed weight files	3.400 GB
Calibration	WikiText-2 training split, 128 sequences of 512 tokens
Verified runtime	Linux, NVIDIA Hopper GPU, supplied Triton runtime
Inputs	Text; the original visual frontend remains in the source project

The embedding/output head, admission networks, small recurrent gates and normalization parameters retain their source precision. Weight groups use a symmetric zero point of 8 and FP32 scales. Storage follows GPTQ v1 qweight / qzeros / scales / g_idx layout, symmetric groups and no activation reordering. The supplied loader handles EdgeRock's custom RMA architecture and these tensors; generic AWQ/GPTQ engine loading has not been validated.

Quick start

hf download hkust-aaron/EdgeRock-4B-GPTQ --local-dir ./EdgeRock-4B-GPTQ
python -m pip install -r ./EdgeRock-4B-GPTQ/requirements.txt

import sys
from transformers import AutoTokenizer

folder = "./EdgeRock-4B-GPTQ"
sys.path.insert(0, folder)
from packed_int4 import load_int4

tokenizer = AutoTokenizer.from_pretrained(folder, trust_remote_code=True)
model = load_int4(folder)
text = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Calculate 17 * 23. Give only the answer."}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False).to("cuda")
outputs = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
print(tokenizer.decode(outputs[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Large linear weights remain packed in GPU memory during generation. The Triton matrix multiplication decodes each tile as it is used, with BF16 activations and standard FP32 accumulation. It does not expand the complete checkpoint into persistent BF16 weights.

Validation

Conversion checks cover all 256 packed linear matrices, GPU matrix multiplication against their dense representations, arithmetic, a long-context memory update beyond the local window, and deterministic fresh-process reload. Full prompts and generated answers are in validation.json.

Check	Result
Functional generation checks	3/3
Unquantized tensors matching the source exactly	290
Fresh-process reload, maximum probe-logit difference	0
Maximum linear-kernel relative error in conversion checks	0.004673

These are conversion and runtime checks. Original-model benchmark results are available in the technical report; this quantized package has not been evaluated on the complete benchmark suite.

Reproduce

python -m pip install -r requirements-conversion.txt
python quantize.py --method gptq --source /path/to/EdgeRock --output /path/to/EdgeRock-4B-GPTQ

calibration.json records the dataset revision, seed, token offsets and token-ID hash. provenance.json records the source revision, package versions and quantization configuration. SHA256SUMS lists release file hashes.

License

Weights retain the EdgeRock Model License 1.0; code retains Apache-2.0. See NOTICE for source notices. Changes comprise GPTQ weight quantization, the packed INT4 runtime and release validation.