madcows/HyperCLOVAX-SEED-Think-32B-AWQ-W4A16

HyperCLOVAX-SEED-Think-32B AWQ (W4A16, G64)

This repository provides an AWQ-quantized version of HyperCLOVAX-SEED-Think-32B.

The model was re-quantized using llm-compressor and includes a customized chat template optimized for vLLM, Reasoning, and Tool Use.

---

Features

• AWQ 4-bit quantization (W4A16, Group Size 64)
• Optimized for vLLM v0.20.0
• Compatible with vLLM Reasoning
• Compatible with automatic Tool Calling
• Improved chat template robustness for OpenAI-compatible message formats
• Runs on a single A100 40GB GPU (TP=1)

---

Quantization

Item	Value
Method	AWQ (Activation-aware Weight Quantization)
Weight	INT4 (W4A16)
Group Size	64
Symmetric	True

The following modules remain in full precision:

• lm_head
• vision_model
• visual
• mm_projector

Excluded Modules

These modules are intentionally kept in full precision.

• lm_head

  • Final prediction layer. Quantizing this layer may noticeably reduce generation quality.

• vision_model / visual

  • Vision encoder modules that operate independently of the language model.

• mm_projector

  • Projects visual embeddings into the language space. Keeping it in higher precision helps preserve multimodal performance.

---

Calibration Dataset

The calibration dataset combines cybersecurity-specific data with general Korean instruction data.

Source	Ratio	Samples
MITRE ATT&CK v18.1 (Korean)	50%	256
MITRE ATT&CK v18.1 (English)	20%	102
Open Korean Instructions	30%	154
Total	100%	512

Configuration:

• Maximum sequence length: 2048
• Multiple prompt templates were randomly rotated during calibration to reduce template bias.

---

Chat Template Improvements

Compared with the original template, several improvements have been made for better compatibility with vLLM.

Robust System Message Handling

System messages provided as OpenAI-style content arrays are safely converted into plain text.

{%- set sys0 = messages[0].content or '' %}
{%- if sys0 is not string and sys0 is sequence %}
    {%- set sys0 = sys0 | selectattr('type', 'equalto', 'text') | map(attribute='text') | join('\n') %}
{%- endif %}

Robust Assistant Message Handling

Assistant messages stored as content arrays are handled safely as well.

{%- if content is not string and content is sequence %}
    {%- set content = content | selectattr('type', 'equalto', 'text') | map(attribute='text') | join('\n') %}
{%- endif %}

enable_thinking Support

The original template uses the thinking flag.

This version replaces it with enable_thinking for compatibility with the vLLM Reasoning Parser.

{%- if enable_thinking is not defined or enable_thinking is true %}

---

Serving

Requirements

Item	Value
GPU	A100 40GB ×1
VRAM Usage	~20 GB
Tensor Parallel	1
vLLM	v0.20.0
CUDA	SM80 (Ampere) or newer

Why TP=2 Is Not Supported

The model has a hidden size of 12096.

With Tensor Parallel = 2:

12096 / 2 = 6048

The Marlin AWQ kernel requires the partition size to be divisible by 128.

6048 % 128 = 32

Therefore, TP=2 cannot use the Marlin kernel.

Since Ampere GPUs do not provide an alternative compatible kernel for this configuration, TP=1 is currently the only supported option.

---

vLLM Example

python3 -m vllm.entrypoints.openai.api_server \
  --model ./HyperCLOVAX-SEED-Think-32B-AWQ-W4A16 \
  --served-model-name HyperCLOVAX-SEED-Think-32B-AWQ-W4A16 \
  --host 0.0.0.0 \
  --port 8000 \
  --gpu-memory-utilization 0.9 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --max-num-seqs 8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser glm45 \
  --disable-custom-all-reduce \
  --language-model-only \
  --trust-remote-code

---

Optional Custom vLLM Plugin

I have also customized the vLLM parser plugin to provide native support for HyperCLOVAX.

The plugin adds:

• HyperCLOVAX Reasoning Parser (--reasoning-parser hcx)
• HyperCLOVAX Tool Parser (--tool-call-parser hcx)

Repository:

https://github.com/madcoww/hcx-vllm-plugin-custom

After installing the plugin, simply launch vLLM with:

--reasoning-parser hcx \
--tool-call-parser hcx

This enables native HyperCLOVAX reasoning and tool parsing without relying on the default Qwen or GLM parsers.

---

Base Model

This model is derived from HyperCLOVAX-SEED-Think-32B.

Please refer to the original model repository for licensing information and usage restrictions.

---

Acknowledgements

• NAVER HyperCLOVAX Team
• llm-compressor
• vLLM