yaya-sy/grat-v0.1

GRAT Q4_K_M for llama.cpp and Ollama

This is the fine-tuned GRAT checkpoint converted to GPT-2 GGUF and quantized with llama.cpp's Q4_K_M preset. It retains both the input embeddings and the separate output head. Q4_K_M uses mostly 4-bit block quantization, with some tensors at higher precision. It is lossy compression, not retraining.

Ollama

ollama serve > /tmp/ollama.log 2>&1 &
ollama run hf.co/yaya-sy/grat-v0.1:grat-Q4_K_M.gguf

Original SentencePiece normalization

For multiline input, repeated spaces, or Unicode normalization, use:

python generate_ollama.py \
  '[FR] [1,2,3,4] La lune marche sur le quai [SEP]' \
  --max-new-tokens 50

This helper needs sentencepiece and uses the original model_bpe_all.model included beside the GGUF. It sends an already normalized prompt using Ollama's raw completion API and does not download or load the HF model.

Add --verbose to print the requested and effective generation limits, the number of generated tokens, and Ollama's done_reason. --max-new-tokens is an upper bound: the helper caps it at 150 - prompt_tokens, and the model can emit EOS earlier. Raising this option to 512 does not extend the positional embeddings.

llama.cpp's score-based SentencePiece BPE tokenizer does not implement this model's full nmt_nfkc normalization or its unknown-character fallback. The helper normalizes text first and rejects out-of-vocabulary characters. Do not silently replace this vocabulary with GPT-2's usual byte-level tokenizer.

The GGUF disables automatic space-prefix insertion: llama.cpp otherwise adds another prefix after user-defined tags, changing token IDs. The Modelfile adds one initial space for ordinary single-space, normalized prompts. In raw mode, use SentencePieceProcessor.normalize(prompt) instead, which already includes the initial U+2581 marker. Avoid feeding unnormalized text directly to the GGUF with raw mode. [ ] is treated as a normal piece because the original normalizer changes the space inside it before tokenization.