mradermacher

/

model_requests

Like 211

is this using the main llamacpp? If yes, I will ask, if it is a custom fork, then I cannot use that since we are using only main llamacpp to quantize and test the models to avoid any confusion

llama-quantize is not able to do GSQ-RCO quantization but llamacpp should read it as standard IQ/K structure

GSQ-RCO to GGUF Conversion Pipeline

1. Clone Research Repositories:

git clone https://github.com/IST-DASLab/GSQ
cd GSQ
# Install dependencies (PyTorch, transformers, gguf library)
pip install -r requirements.txt

2. Run GSQ (Gumbel-Softmax Quantization):

• Perform per-tensor differentiable scalar quantization on your source .safetensors model to generate low-bit configurations:

python gsq_quantize.py --model_path /path/to/safetensors --calib_dataset wikitext

3. Run RCO (Riemannian Constrained Optimization):

• Optimize per-tensor bit allocations under your strict target budget (e.g., ~10 GB / IQ2_S tier):

python rco_allocate.py --gsq_weights output_dir/ --target_size_gb 10.0 --output_profile iq2_s_mix

4. Export to GGUF Container:

• Serialize the optimized mixed-precision tensors into a standardized, llama.cpp-compatible GGUF file:

python export_gguf.py --optimized_config output_profile/ --output model-gsq-rco-iq2_s.gguf