Instructions to use rinna/qwq-bakeneko-32b-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use rinna/qwq-bakeneko-32b-gguf with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="rinna/qwq-bakeneko-32b-gguf",
	filename="qwq-bakeneko-32b-q2_k.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use rinna/qwq-bakeneko-32b-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf rinna/qwq-bakeneko-32b-gguf:Q2_K
# Run inference directly in the terminal:
llama cli -hf rinna/qwq-bakeneko-32b-gguf:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf rinna/qwq-bakeneko-32b-gguf:Q2_K
# Run inference directly in the terminal:
llama cli -hf rinna/qwq-bakeneko-32b-gguf:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf rinna/qwq-bakeneko-32b-gguf:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf rinna/qwq-bakeneko-32b-gguf:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf rinna/qwq-bakeneko-32b-gguf:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf rinna/qwq-bakeneko-32b-gguf:Q2_K

Use Docker

docker model run hf.co/rinna/qwq-bakeneko-32b-gguf:Q2_K

LM Studio
Jan

vLLM

How to use rinna/qwq-bakeneko-32b-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rinna/qwq-bakeneko-32b-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rinna/qwq-bakeneko-32b-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/rinna/qwq-bakeneko-32b-gguf:Q2_K

Ollama
How to use rinna/qwq-bakeneko-32b-gguf with Ollama:
```
ollama run hf.co/rinna/qwq-bakeneko-32b-gguf:Q2_K
```

Unsloth Studio

How to use rinna/qwq-bakeneko-32b-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for rinna/qwq-bakeneko-32b-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for rinna/qwq-bakeneko-32b-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for rinna/qwq-bakeneko-32b-gguf to start chatting

How to use rinna/qwq-bakeneko-32b-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rinna/qwq-bakeneko-32b-gguf:Q2_K

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "rinna/qwq-bakeneko-32b-gguf:Q2_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use rinna/qwq-bakeneko-32b-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rinna/qwq-bakeneko-32b-gguf:Q2_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default rinna/qwq-bakeneko-32b-gguf:Q2_K

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use rinna/qwq-bakeneko-32b-gguf with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rinna/qwq-bakeneko-32b-gguf:Q2_K

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "rinna/qwq-bakeneko-32b-gguf:Q2_K" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use rinna/qwq-bakeneko-32b-gguf with Docker Model Runner:
```
docker model run hf.co/rinna/qwq-bakeneko-32b-gguf:Q2_K
```

Lemonade

How to use rinna/qwq-bakeneko-32b-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull rinna/qwq-bakeneko-32b-gguf:Q2_K

Run and chat with the model

lemonade run user.qwq-bakeneko-32b-gguf-Q2_K

List all available models

lemonade list

`QwQ Bakeneko 32B GGUF (rinna/qwq-bakeneko-32b-gguf)`

Overview

This model is a quantized model for rinna/qwq-bakeneko-32b using llama.cpp. It is compatible with many llama.cpp based apps.

Model Type	Model Name
Japanese Continual Pre-Training Model	Qwen2.5 Bakeneko 32B [HF]
Instruction-Tuning Model	Qwen2.5 Bakeneko 32B Instruct [HF][AWQ][GGUF][GPTQ int8][GPTQ int4]
DeepSeek R1 Distill Qwen2.5 Merged Reasoning Model	DeepSeek R1 Distill Qwen2.5 Bakeneko 32B [HF][AWQ][GGUF][GPTQ int8][GPTQ int4]
QwQ Merged Reasoning Model	QwQ Bakeneko 32B [HF][AWQ][GGUF][GPTQ int8][GPTQ int4]
QwQ Bakeneko Merged Instruction-Tuning Model	Qwen2.5 Bakeneko 32B Instruct V2 [HF][AWQ][GGUF][GPTQ int8][GPTQ int4]

See rinna/qwq-bakeneko-32b for details about model architecture and data.

Contributors
Release date

March 13, 2025

Benchmarking

Model	Japanese LM Evaluation Harness	Japanese MT-Bench (first turn)	Japanese MT-Bench (multi turn)
Qwen/Qwen2.5-32B	79.46	-	-
rinna/qwen2.5-bakeneko-32b	79.18	-	-
Qwen/Qwen2.5-32B-Instruct	78.29	8.13	7.54
rinna/qwen2.5-bakeneko-32b-instruct	79.62	8.17	7.66
rinna/qwen2.5-bakeneko-32b-instruct-v2	77.92	8.86	8.53
deepseek-ai/DeepSeek-R1-Distill-Qwen-32B	73.51	7.39	6.88
rinna/deepseek-r1-distill-qwen2.5-bakeneko-32b	77.43	8.58	8.19
Qwen/QwQ-32B	76.12	8.58	8.25
rinna/qwq-bakeneko-32b	78.31	8.81	8.52

For detailed benchmarking results, please refer to rinna's LM benchmark page (Sheet 20250313).

How to cite

@misc{rinna-qwq-bakeneko-32b-gguf,
    title = {rinna/qwq-bakeneko-32b-gguf},
    author = {Wakatsuki, Toshiaki and Chen, Xinqi and Sawada, Kei},
    url = {https://huggingface.co/rinna/qwq-bakeneko-32b-gguf}
}

@inproceedings{sawada2024release,
    title = {Release of Pre-Trained Models for the {J}apanese Language},
    author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
    booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
    month = {5},
    year = {2024},
    pages = {13898--13905},
    url = {https://aclanthology.org/2024.lrec-main.1213},
    note = {\url{https://arxiv.org/abs/2404.01657}}
}

References

@article{qwen2.5,
      title = {Qwen2.5 Technical Report}, 
      author = {An Yang and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chengyuan Li and Dayiheng Liu and Fei Huang and Haoran Wei and Huan Lin and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Yang and Jiaxi Yang and Jingren Zhou and Junyang Lin and Kai Dang and Keming Lu and Keqin Bao and Kexin Yang and Le Yu and Mei Li and Mingfeng Xue and Pei Zhang and Qin Zhu and Rui Men and Runji Lin and Tianhao Li and Tianyi Tang and Tingyu Xia and Xingzhang Ren and Xuancheng Ren and Yang Fan and Yang Su and Yichang Zhang and Yu Wan and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zihan Qiu},
      journal = {arXiv preprint arXiv:2412.15115},
      year = {2024}
}

@misc{qwq32b,
    title = {QwQ-32B: Embracing the Power of Reinforcement Learning},
    url = {https://qwenlm.github.io/blog/qwq-32b/},
    author = {Qwen Team},
    month = {March},
    year = {2025}
}

@misc{deepseekai2025deepseekr1incentivizingreasoningcapability,
    title = {DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning}, 
    author = {DeepSeek-AI and Daya Guo and Dejian Yang and Haowei Zhang and Junxiao Song and Ruoyu Zhang and Runxin Xu and Qihao Zhu and Shirong Ma and Peiyi Wang and Xiao Bi and Xiaokang Zhang and Xingkai Yu and Yu Wu and Z. F. Wu and Zhibin Gou and Zhihong Shao and Zhuoshu Li and Ziyi Gao and Aixin Liu and Bing Xue and Bingxuan Wang and Bochao Wu and Bei Feng and Chengda Lu and Chenggang Zhao and Chengqi Deng and Chenyu Zhang and Chong Ruan and Damai Dai and Deli Chen and Dongjie Ji and Erhang Li and Fangyun Lin and Fucong Dai and Fuli Luo and Guangbo Hao and Guanting Chen and Guowei Li and H. Zhang and Han Bao and Hanwei Xu and Haocheng Wang and Honghui Ding and Huajian Xin and Huazuo Gao and Hui Qu and Hui Li and Jianzhong Guo and Jiashi Li and Jiawei Wang and Jingchang Chen and Jingyang Yuan and Junjie Qiu and Junlong Li and J. L. Cai and Jiaqi Ni and Jian Liang and Jin Chen and Kai Dong and Kai Hu and Kaige Gao and Kang Guan and Kexin Huang and Kuai Yu and Lean Wang and Lecong Zhang and Liang Zhao and Litong Wang and Liyue Zhang and Lei Xu and Leyi Xia and Mingchuan Zhang and Minghua Zhang and Minghui Tang and Meng Li and Miaojun Wang and Mingming Li and Ning Tian and Panpan Huang and Peng Zhang and Qiancheng Wang and Qinyu Chen and Qiushi Du and Ruiqi Ge and Ruisong Zhang and Ruizhe Pan and Runji Wang and R. J. Chen and R. L. Jin and Ruyi Chen and Shanghao Lu and Shangyan Zhou and Shanhuang Chen and Shengfeng Ye and Shiyu Wang and Shuiping Yu and Shunfeng Zhou and Shuting Pan and S. S. Li and Shuang Zhou and Shaoqing Wu and Shengfeng Ye and Tao Yun and Tian Pei and Tianyu Sun and T. Wang and Wangding Zeng and Wanjia Zhao and Wen Liu and Wenfeng Liang and Wenjun Gao and Wenqin Yu and Wentao Zhang and W. L. Xiao and Wei An and Xiaodong Liu and Xiaohan Wang and Xiaokang Chen and Xiaotao Nie and Xin Cheng and Xin Liu and Xin Xie and Xingchao Liu and Xinyu Yang and Xinyuan Li and Xuecheng Su and Xuheng Lin and X. Q. Li and Xiangyue Jin and Xiaojin Shen and Xiaosha Chen and Xiaowen Sun and Xiaoxiang Wang and Xinnan Song and Xinyi Zhou and Xianzu Wang and Xinxia Shan and Y. K. Li and Y. Q. Wang and Y. X. Wei and Yang Zhang and Yanhong Xu and Yao Li and Yao Zhao and Yaofeng Sun and Yaohui Wang and Yi Yu and Yichao Zhang and Yifan Shi and Yiliang Xiong and Ying He and Yishi Piao and Yisong Wang and Yixuan Tan and Yiyang Ma and Yiyuan Liu and Yongqiang Guo and Yuan Ou and Yuduan Wang and Yue Gong and Yuheng Zou and Yujia He and Yunfan Xiong and Yuxiang Luo and Yuxiang You and Yuxuan Liu and Yuyang Zhou and Y. X. Zhu and Yanhong Xu and Yanping Huang and Yaohui Li and Yi Zheng and Yuchen Zhu and Yunxian Ma and Ying Tang and Yukun Zha and Yuting Yan and Z. Z. Ren and Zehui Ren and Zhangli Sha and Zhe Fu and Zhean Xu and Zhenda Xie and Zhengyan Zhang and Zhewen Hao and Zhicheng Ma and Zhigang Yan and Zhiyu Wu and Zihui Gu and Zijia Zhu and Zijun Liu and Zilin Li and Ziwei Xie and Ziyang Song and Zizheng Pan and Zhen Huang and Zhipeng Xu and Zhongyu Zhang and Zhen Zhang},
    year = {2025},
    eprint = {2501.12948},
    archivePrefix = {arXiv},
    primaryClass = {cs.CL},
    url = {https://arxiv.org/abs/2501.12948}, 
}

@article{huang2023chat,
    title = {Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages},
    author = {Huang, Shih-Cheng and Li, Pin-Zu and Hsu, Yu-Chi and Chen, Kuang-Ming and Lin, Yu Tung and Hsiao, Shih-Kai and Tzong-Han Tsai, Richard and Lee, Hung-yi},
    year = {2023},
    url = {https://arxiv.org/abs/2310.04799}
}

@inproceedings{hong2024orpo,
    title = {ORPO: Monolithic Preference Optimization without Reference Model},
    author = {Hong, Jiwoo and Lee, Noah and Thorne, James},
    booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
    pages = {11170--11189},
    year = {2024}
}

@misc{llamacpp,
    title = {llama.cpp},
    author = {Gerganov, Georgi},
    howpublished = {\url{https://github.com/ggerganov/llama.cpp}},
    year = {2023}
}