turboderp

/

gemma-4-12B-it-exl3

like 7

It's because the original model uses tied embeddings. So the embeddings serve a dual purpose, both as a lookup table and as a linear projection of the hidden state into logits (purpose otherwise served by lm_head). Since EXL3 is GPU-focused, the chart reflects how much VRAM is used for weights in either case:

• EXL3: transformer layers + output layer
• GGUF: transformer layers + embedding/output layer

The reason EXL3 models are larger on disk is that the format unties embeddings and keeps a full-precision copy in system memory, whereas GGUF keeps the weights tied (for this model at least.) For models without tied embeddings, I believe GGUF also prefers to store the embedding layer in system memory since it's literally just a lookup table and VRAM is a scarce resource. EXL3 just maintains that behavior for tied models as well (by untying during quantization) since it's "free" extra precision and avoids needing both a quantized and unquantized embedding mechanism.

Either way, you end up with one copy of the quantized output layer in VRAM, and no dedicated embedding layer in VRAM, so the chart is apples-to-apples in that sense.

TLDR: I was wrong about the VRAM usage, when embeddings are offloaded to the cpu, the VRAM usage is indeed as expected from the chart. Speeds are measurably worse (30t/s vs 39t/s on llama.cpp, and significantly worse at a graphics clock limit: 23t/s (EXL3) vs 36t/s (llama.cpp)) but that's to be expected since my gpu is Ampere (RTX 3060 12GB). Thank you for the thorough explanation and sorry for my lack of knowledge.

The VRAM usage with EXL3 is 7878MiB, meanwhile with llama.cpp it's 8432MiB. Nevermind, after updating my tabby config to offload the embeddings to cpu, setting batch size to 512 and setting concurrency to 1 exllamav3(through tabbyapi) is 6900MiB-7238MiB (at 11k ctx), compared to llama.cpp's 7456MiB-7468MiB.
Comparing the following quants: https://huggingface.co/bartowski/gemma-4-12B-it-GGUF/blob/main/gemma-4-12B-it-IQ4_XS.gguf and https://huggingface.co/turboderp/gemma-4-12B-it-exl3/blob/4.00bpw/model.safetensors

VRAM usage:
EXL3 (Proper tabbyAPI config):
6900MiB on load
7238MiB after first prompt is generated (at 11k ctx)

llama.cpp (llama-server):
7456MiB on load
7468MiB after first prompt is generated (at 11k ctx)

SPEED (at limited 1500MHz graphics clock):
EXL3 (Proper tabbyAPI config): pp: 521t/s tg: 23t/s

127 tokens generated in 27.19 seconds (Queue: 0.0 s, Process: 0 cached tokens and 11349 new tokens at 521.55 T/s, Generate: 23.41 T/s, Context: 11349 tokens)

llama.cpp (llama-server): pp: 1036t/s, tg: 36.49t/s

0.21.956.240 I slot print_timing: id  0 | task 0 | prompt eval time =   10980.05 ms / 11383 tokens (    0.96 ms per token,  1036.70 tokens per second)
0.21.956.242 I slot print_timing: id  0 | task 0 |        eval time =    3507.73 ms /   128 tokens (   27.40 ms per token,    36.49 tokens per second)
0.21.956.243 I slot print_timing: id  0 | task 0 |       total time =   14487.78 ms / 11511 tokens

SPEED (no power limit/clock limit):
EXL3: tg: 30t/s

127 tokens generated in 4.55 seconds (Queue: 0.05 s, Process: 11264 cached tokens and 85 new tokens at 369.57 T/s, Generate: 29.77 T/s, Context: 11349 
tokens)

llama.cpp: 39t/s

0.51.270.195 I slot print_timing: id  0 | task 137 | prompt eval time =      27.36 ms /     1 tokens (   27.36 ms per token,    36.56 tokens per second)
0.51.270.198 I slot print_timing: id  0 | task 137 |        eval time =    3290.62 ms /   128 tokens (   25.71 ms per token,    38.90 tokens per second)
0.51.270.198 I slot print_timing: id  0 | task 137 |       total time =    3317.98 ms /   129 tokens

PS: I am measuring per-process VRAM usage instead of total.

llama.cpp command: $ ./build/bin/llama-server --model ~/TND/AI/gemma-4-12B-it-IQ4_XS.gguf -fa on --no-mmap -np 1 -kvu --swa-checkpoints 1 -t 6 -tb 12 -ngl 10000 -c 16384 -fit off --reasoning on
tabbyAPI config:

# Sample YAML file for configuration.
# Comment and uncomment values as needed.
# Every value has a default within the application.
# This file serves to be a drop in for config.yml

# Unless specified in the comments, DO NOT put these options in quotes!
# You can use https://www.yamllint.com/ if you want to check your YAML formatting.

# Options for networking
network:
  # The IP to host on (default: 127.0.0.1).
  # Use 0.0.0.0 to expose on all network adapters.
  host: 127.0.0.1

  # The port to host on (default: 5000).
  port: 5000

  # Disable HTTP token authentication with requests.
  # WARNING: This will make your instance vulnerable!
  # Turn on this option if you are ONLY connecting from localhost.

  # Send tracebacks over the API (default: False).
  # NOTE: Only enable this for debug purposes.
  send_tracebacks: false

  # Select API servers to enable (default: ["OAI"]).
  # Possible values: OAI, Kobold.
  api_servers: ["OAI"]

# Options for logging
logging:
  # Enable prompt logging (default: False).
  log_prompt: false

  # Enable generation parameter logging (default: False).
  log_generation_params: false

  # Enable request logging (default: False).
  # NOTE: Only use this for debugging!
  log_requests: false

  # Write every /v1/chat/completions request to logs/debug/ as JSON (default: False).
  # PRIVACY WARNING: Enabling this creates a comprehensive request log, including the
  # full message history and generation parameters. API keys are redacted, but prompts
  # and user-provided content are preserved for bug-report reproduction.
  log_chat_completion_requests: false

# Options for model overrides and loading
# Please read the comments to understand how arguments are handled
# between initial and API loads
model:
  # Directory to look for models (default: models).
  # Windows users, do NOT put this path in quotes!
  model_dir: "models"

  # Allow direct loading of models from a completion or chat completion request (default: False).
  # This method of loading is strict by default.
  # Enable dummy models to add exceptions for invalid model names.
  inline_model_loading: false

  # Sends dummy model names when the models endpoint is queried. (default: False)
  # Enable this if the client is looking for specific OAI models.
  use_dummy_models: false

  # A list of fake model names that are sent via the /v1/models endpoint. (default: ["gpt-3.5-turbo"])
  # Also used as bypasses for strict mode if inline_model_loading is true.

  # An initial model to load.
  # Make sure the model is located in the model directory!
  # REQUIRED: This must be filled out to load a model on startup.
  model_name: "gemma-12b-exl3-4bit"

  # Names of args to use as a fallback for API load requests (default: []).
  # For example, if you always want cache_mode to be Q4 instead of on the inital model load, add "cache_mode" to this array.
  # Example: ['max_seq_len', 'cache_mode'].

  # Backend to use for this model (auto-detect if not specified)
  # Options: exllamav2, exllamav3
  backend: exllamav3

  # Max sequence length (default: min(max_position_embeddings, cache_size)).
  # Set to -1 to fetch from the model's config.json
  max_seq_len: 16384

  # Size of the key/value cache to allocate, in tokens (default: 4096).
  # Must be a multiple of 256.
  # ExllamaV2 note: On AMD GPUs and NVIDIA GPUs older than Ampere, this value
  # is ignored. Please use max_seq_len
  cache_size: 16384

  # Enable different cache modes for VRAM savings (default: FP16).
  # Possible values for exllamav2: 'FP16', 'Q8', 'Q6', 'Q4'.
  # For exllamav3, specify the pair k_bits,v_bits where k_bits and v_bits are integers from 2-8 (i.e. 8,8).
  cache_mode: FP16

  # Load model with tensor parallelism.
  # Falls back to autosplit if GPU split isn't provided.
  # This ignores the gpu_split_auto value.
  tensor_parallel: false

  # Sets a backend type for tensor parallelism. (default: native).
  # Options: native, nccl
  # Native is recommended for PCIe GPUs
  # NCCL is recommended for NVLink.
  tensor_parallel_backend: native

  # Automatically allocate resources to GPUs (default: True).
  # Not parsed for single GPU users.
  gpu_split_auto: true

  # Reserve VRAM used for autosplit loading (default: 96 MB on GPU 0).
  # Represented as an array of MB per GPU.

  # Array of VRAM sizes to split between GPUs, in GB (default: []).
  # Used with tensor parallelism.

  # NOTE: If a model has YaRN rope scaling, it will automatically be enabled by ExLlama.
  # rope_scale and rope_alpha settings won't apply in this case.

  # Rope scale (default: 1.0).
  # Same as compress_pos_emb.
  # Use if the model was trained on long context with rope.
  # Leave blank to pull the value from the model.
  rope_scale: 1.0

  # Rope alpha (default: None).
  # Same as alpha_value. Set to "auto" to auto-calculate.
  # Leaving this value blank will either pull from the model or auto-calculate.
  rope_alpha:

  # Chunk size for prompt ingestion (default: 2048).
  # A lower value reduces VRAM usage but decreases ingestion speed.
  # NOTE: Effects vary depending on the model.
  # An ideal value is between 512 and 4096.
  chunk_size: 512

  # Use output chunking (default: True)
  # Instead of allocating cache space for the entire completion at once, allocate in chunks as needed.
  # Used by EXL3 models only.
  output_chunking: true

  # Set the maximum number of generation jobs that can run concurrently
  # The default maximum batch size for transformer architectures is 32. Recurrent
  # models with linear or sliding attention use more VRAM to support larger batches,
  # so the default value is reduced to 4. If you do not require concurrency at all, you
  # can reduce it further to minimize VRAM overhead.
  max_batch_size: 1

  # Set the prompt template for this model. (default: None)
  # If empty, attempts to look for the model's chat template.
  # If a model contains multiple templates in its tokenizer_config.json,
  # set prompt_template to the name of the template you want to use.
  # NOTE: Only works with chat completion message lists!
  prompt_template:

  # Enables vision support if the model supports it. (default: False)
  vision: false

  # Force-enable reasoning in template args
  # Injects the enable_thinking: True into the model's template arguments. This doesn't
  # force reasoning or affect how reasoning content is parsed, but some clients will
  # not explicitly enable this and some models need it to properly enter reasoning mode.
  force_enable_thinking: true

  # Enable reasoning parser (default: False).
  # Do NOT enable this if the model is not a reasoning model (e.g. deepseek-r1 series)

  # The start token for reasoning content (default: "<think>")

  # The end token for reasoning content (default: "</think>")

  # Suppress this text whenever it appears in the beginning of a reasoning block (default: None)

  # Tool format, e.g. 'qwen3_coder'. See docs for supported formats. If left blank,
  # tool calls from the model will not be parsed by the server.

# Options for draft models (speculative decoding)
# This will use more VRAM!
draft_model:
  # Drafting mode for exllamav3 (default: model).
  # Options: model, disabled, mtp, ngram.
  # In `model` mode, drafting is disabled if no draft_model_name is provided.

  # Directory to look for draft models (default: models)

  # An initial draft model to load.
  # Ensure the model is in the model directory.

  # Rope scale for draft models (default: 1.0).
  # Same as compress_pos_emb.
  # Use if the draft model was trained on long context with rope.

  # Rope alpha for draft models (default: None).
  # Same as alpha_value. Set to "auto" to auto-calculate.
  # Leaving this value blank will either pull from the model or auto-calculate.

  # Cache mode for draft models to save VRAM (default: FP16).
  # Possible values for exllamav2: 'FP16', 'Q8', 'Q6', 'Q4'.
  # For exllamav3, specify the pair k_bits,v_bits where k_bits and v_bits are integers from 2-8 (i.e. 8,8).

  # Array of VRAM sizes to split between GPUs, in GB (default: []).
  # If this isn't filled in, the draft model is autosplit.

  # Number of tokens to draft per iteration (default: draft model default)
  # Recurrent (linear or sliding attention) models use more VRAM for longer drafts.
  # This overhead multiplies with the max batch size, so for models with long drafts
  # (e.g. DFlash with 15 tokens by default) shorter drafts may be preferable.

  # Minimum match length for exllamav3 n-gram drafting (default: 2).
  # Only used when draft_mode is ngram.

# Options for Sampling
sampling:
  # Select a sampler override preset (default: None).
  # Find this in the sampler-overrides folder.
  # This overrides default fallbacks for sampler values that are passed to the API.
  # NOTE: safe_defaults is noob friendly and provides fallbacks for frontends that don't send sampling parameters.
  # Remove this for any advanced usage.
  override_preset: None

# Options for Loras
lora:
  # Directory to look for LoRAs (default: loras).

  # List of LoRAs to load and associated scaling factors (default scale: 1.0).
  # For the YAML file, add each entry as a YAML list:
  # - name: lora1

# Options for embedding models and loading.
# NOTE: Embeddings requires the "extras" feature to be installed
# Install it via "pip install .[extras]"
embeddings:
  # Directory to look for embedding models (default: models).
  embedding_model_dir: models

  # Device to load embedding models on (default: cpu).
  # Possible values: cpu, auto, cuda.
  # NOTE: It's recommended to load embedding models on the CPU.
  # If using an AMD GPU, set this value to 'cuda'.
  embeddings_device: cpu

  # An initial embedding model to load on the infinity backend.

# Global memory settings
memory:

  # Max size of recurrent cache in system memory, in MB (default: 4096)
  sysmem_recurrent_cache: 4096

  # Use cudaMallocAsync backend in Torch (default: True).
  # Enabling this is generally preferable, but it may cause issues with certain
  # workloads. Try disabling it if you experience intermittent OoM errors. If
  # False, Torch will use the allocator defined by the system env
  cuda_malloc_async: True

# Options for development and experimentation
developer:
  # Skip Exllamav2 version check (default: False).
  # WARNING: It's highly recommended to update your dependencies rather than enabling this flag.
  unsafe_launch: false

  # Disable API request streaming (default: False).
  disable_request_streaming: false

  # Set process to use a higher priority.
  # For realtime process priority, run as administrator or sudo.
  # Otherwise, the priority will be set to high.
  realtime_process_priority: false

  # Enable extremely verbose seqlog logging, requires a running Seq server
  seqlog: false

  # Seq server url:port

  # Seq server API key (default: None)

Yeah, EXL3 uses a more compute-heavy quantization format than IQ-quants (QTIP vs. QuIP#, essentially), and you feel that a lot more on 30xx series GPUs. But there are some huge performance improvements coming in the next release, especially targeted at Ampere GPUs. Probably early next week, though there's a preview in the exllamav3 dev branch.