Clover-125M 🍀

Clover-125M is a 125-million-parameter English language model trained from scratch. It was pretrained on English text and then went through three rounds of full-parameter supervised fine-tuning (SFT). The released weights are from the final checkpoint of SFT round 3.

Try Clover online → · Source code · Project article · Training notes

The online demo is the quickest way to try the model. No local setup or download is needed. Clover is primarily an English model, so short English prompts are recommended.

About the model

Clover has 124,720,896 parameters and uses a decoder-only, Llama-style architecture implemented with Hugging Face Transformers. Training started from randomly initialized weights; no pretrained Meta Llama weights were used.

The model has 14 Transformer layers, a hidden size of 768, an FFN dimension of 2,080, and 12 attention heads with 12 key/value heads (64 dimensions per head). It uses a 32,000-token English ByteLevel BPE tokenizer, tied input/output embeddings, and a maximum context of 1,024 tokens.

Training was carried out on an NVIDIA RTX 5070, with FP32 master weights and BF16 mixed-precision computation. The model is distributed in two formats: FP32 Safetensors for the Transformers inference path and unquantized FP32 GGUF for native CPU inference.

Training

The first stage trained Clover as an ordinary English text-completion model. Pretraining data came from English Wikipedia, Project Gutenberg, web/C4 material, and arXiv texts. It ran for one epoch, with 80,465 optimizer updates and 2,636,654,462 target tokens.

Three rounds of SFT then adapted the base model for short instructions and conversation:

• SFT 1 focused on short English instructions and response formatting. It ran for one epoch, with 584 updates and 2,334,382 supervised assistant tokens.
• SFT 2 added everyday conversations and Clover's model identity. It ran for two epochs, with 1,506 updates and 11,973,174 supervised assistant tokens.
• SFT 3 focused on answering questions from supplied text, recognizing when information was missing, and completing short tasks. It ran for one epoch, with 574 updates and 447,475 supervised assistant tokens.

The same tokenizer was used throughout. Every SFT round continued from the previous model weights rather than starting over. Training sources included Smol Smoltalk, UltraChat, OpenAssistant, Everyday Conversations, SQuAD 2.0, and project-authored examples. The full source attribution is available in the source record.

Raw and processed training datasets, along with the complete data-preparation pipeline, are not distributed with the source code. The published training scripts therefore document the implementation but do not, by themselves, reproduce the entire original training run.

Evaluation

The final SFT 3 checkpoint was selected before the recorded test evaluations.

On the SFT 3 reading test (650 conversations; 8,386 scored tokens), the model achieved a token-level loss of 0.439808 and perplexity of 1.552409. On the SFT 2 dialogue test (388 conversations; 90,855 scored tokens), the loss was 2.152472 and perplexity 8.606110.

These are teacher-forced token-prediction metrics measured on different datasets. They are not percentages of correct answers, direct measures of conversation quality, or evidence of general reasoning ability. The results should not be interpreted as a comparison with larger language models.

Using Clover

Online: Open the Clover chat interface.

Locally: Download the model files and use the inference scripts from the source repository. For the native CPU runtime, the model directory must contain tokenizer.json, release.json, and gguf/Clover-125M-v0.1.0-F32.gguf. The inference guide includes the Python 3.12 setup, CPU/CUDA commands, and web-server instructions.

Clover uses a project-specific conversation format, implemented in sft_format.py. Role headers, message content, and separators must be encoded consistently with the original tokenizer. A generic Llama chat template may produce different token IDs and behavior.

The unquantized GGUF model has also been run on a 2-vCPU CPU-only server with 887 MiB of visible RAM and approximately 1 GiB of existing swap. This is one observed configuration, not a guarantee that the model will run without swap on any 1 GB machine. The deployment notes describe the setup and memory measurements.

Limitations

Clover is a small experimental language model with a short context window. It can repeat text, miss instructions, invent details, or give incorrect answers. It is primarily suited to short English prompts and straightforward text tasks, not reliable factual recall, high-stakes decisions, or complex reasoning. Important outputs should be checked independently.

License and credits

The original Clover-125M project code and model weights are licensed under Apache License 2.0. Third-party libraries and training materials retain their respective license and attribution requirements. See the project license, third-party notices, and training-source record.

Author: mio

---

中文简介

Clover-125M 🍀 是一个从随机初始化开始训练的英文语言模型，共有 124,720,896 个参数。模型经过一次英文预训练和三轮全参数 SFT，发布版本采用第三轮 SFT 的最终权重，提供 FP32 Safetensors 和未量化的 FP32 GGUF。

在线体验 Clover → · GitHub 源码 · 项目文章

Clover 主要适合简短的英文对话和文本任务。模型仍可能重复、理解偏题或生成不准确的信息。