mrysh/gemma-4-e4b-it-int8_w8a16

gemma-4-e4b-it - int8_w8a16

Quantized version of google/gemma-4-E4B-it made for a quantization benchmark (4 models x 8 techniques, evaluated with vLLM on BFCL, Energy-QA, hiring-bias and MedQA).

• method: int8, about 8 bits, group size None
• library: compressed-tensors 0.17.0
• calibration samples: None
• size on disk: 11.945 GB

INT8 weight-only (W8A16): int8 weights with one scale per output channel, 16-bit activations, round-to-nearest, no calibration data. compressed-tensors pack-quantized format, loads in vLLM (vllm serve <repo>, no flag needed). The vision/audio parts are kept in 16 bit.

Evaluated as a text model (vLLM, thinking off). Same license as the base model; see the base model card.