gemma-4-31B-it-uncensored-heretic-FP8
FP8 block-quantized version of llmfan46/gemma-4-31B-it-uncensored-heretic.
Quantization Details
- Method: FP8_BLOCK (weight_block_size=[128,128], activation_scheme=dynamic)
- Tool: llmcompressor
model_free_ptq - Format: compressed-tensors (vLLM native)
- Ignored layers: vision tower, lm_head, embedding layers
- Model size: ~31.5 GB (vs ~62 GB BF16)
Evaluation Results
Compared against RedHatAI/gemma-4-31B-it-FP8-block using identical evaluation settings (lm_eval 0.4.11, vLLM, 2 seeds averaged).
| Benchmark | This Model | RedHatAI FP8 | Delta |
|---|---|---|---|
| GSM8k-Platinum (5-shot) | 97.68% | 97.82% | -0.14 |
| MMLU-CoT (5-shot) | 89.97% | 90.70% | -0.73 |
| MMLU-Pro (5-shot) | 84.52% | 84.92% | -0.40 |
| IFEval inst_strict | 93.41% | 93.84% | -0.43 |
| IFEval prompt_strict | 90.76% | 91.31% | -0.55 |
Generation parameters: temperature=1.0, top_p=0.95, top_k=64, max_gen_toks=8192
Usage
from vllm import LLM
model = LLM("kakrotto/gemma-4-31B-it-uncensored-heretic-FP8")
Disclaimer
This is an uncensored model. The quantizer (kakrotto) is not responsible for the model's outputs or any misuse. This FP8 quantization preserves the original model's behavior. Please use responsibly.
Attribution
- Source model: llmfan46/gemma-4-31B-it-uncensored-heretic
- Quantization recipe reference: RedHatAI/gemma-4-31B-it-FP8-block
- Downloads last month
- 29
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for m672/gemma-4-31b-fp8
Base model
google/gemma-4-31B Finetuned
google/gemma-4-31B-it