ClimateGPT-7B-4bit-mlx

This is a 4-bit MLX quantization of ClimateGPT-7B (eci-io/climategpt-7b), a climate-domain chat model, converted for fast local inference on Apple Silicon.

ClimateGPT is a family of models designed to synthesize interdisciplinary research on climate change. The 7B model is continued-pretrained from Llama-2-7B on a curated collection of climate documents and then instruction fine-tuned on instruction–completion pairs collected in cooperation with climate scientists. The authors report it outperforms Llama-2-70B-Chat on their climate-specific benchmarks. See the original work for full detail:

Thulke et al., ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change, 2024. Paper: https://arxiv.org/abs/2401.09646 · Project: https://eci.io

Conversion details

• Method: mlx_lm.convert with -q --q-bits 4
• Result: 4.500 bits per weight
• Precision of base load: bfloat16
• Converted on: Apple Silicon (MLX)
• mlx-lm version: 0.31.3
• Base model: eci-io/climategpt-7b (continued-pretrained from Llama-2-7B)

Usage

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("xquantize/climategpt-7b-4bit-mlx")

# ClimateGPT is an instruction-tuned CHAT model — use the chat template.
messages = [
    {"role": "user", "content": "What is the difference between weather and climate?"}
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))

Or from the command line:

mlx_lm.generate --model xquantize/climategpt-7b-4bit-mlx \
  --prompt "What is the difference between weather and climate?" --max-tokens 128

Note on usage style

This is an instruction-tuned chat model, so prompt it with chat turns via the tokenizer's chat template (shown above) rather than raw text completion. It uses a ChatML-style format (<|im_start|> / <|im_end|>). The authors note the model is designed to be used with retrieval augmentation (RAG) to extend its knowledge and improve factuality and is not intended as a general-purpose chatbot, it is specialized for the climate domain. Context length is 4K tokens.

Stop-token fix

The upstream generation_config.json sets eos_token_id to 2 (Llama-2's </s>), but this model actually ends turns with the ChatML token <|im_end|> (id 32006). As shipped upstream that mismatch causes generation to run past the end of a turn and repeat. This repository sets eos_token_id to [2, 32006], so generation stops cleanly at the end of a turn with no extra flags required.

Performance

On an Apple Silicon Mac, this 4-bit build runs at ~34 tokens/sec with a peak memory footprint of ~4.0 GB, making a climate-specialized 7B usable on consumer hardware.

Limitations

ClimateGPT-7B is a 2024-era Llama-2-7B derivative; its general fluency and knowledge reflect that generation, not current frontier models. As with any chat-capable LLM, it may produce biased, offensive or inaccurate responses and should not be relied on for authoritative scientific or safety-critical decisions. It is intended to be paired with retrieval augmentation. Quantization to 4 bits may introduce additional minor quality degradation relative to the full-precision model.

License & attribution

This model is a 4-bit MLX quantization of ClimateGPT-7B and is distributed under the ClimateGPT Non-Commercial Community License (© EQTY R&D, Inc.; Erasmus.AI as third-party beneficiary). Non-commercial use only.

Per the license, this repository includes a copy of the license (LICENSE.txt) and the required attribution (Notice). By using these weights you agree to the ClimateGPT Terms of Service and Acceptable Use Policy contained in LICENSE.txt.

All credit for the underlying model belongs to the ClimateGPT authors (Thulke et al.; Erasmus AI, AppTek, EQTY R&D). This repository provides only an MLX-format 4-bit quantization for local inference on Apple Silicon and claims no ownership of the underlying model. The "ClimateGPT" name is used solely to describe the origin of these weights.