Update README.md

README.md

@@ -17,8 +17,21 @@ license_name: llama3
 license_link: LICENSE
 ---
 
+## Note about eos token
+It seems llama 3 uses different eos tokens depending if it is in instruct mode.
+The initial upload has some issues with this as it uses the "default" eos token of 128001, when in instruct mode llama uses 128009 as eos token which causes it to ramble on and on without stopping.
+
+I am currently uploading fixed quants with the eos token id manually set to 128009.
+This fixes the issue for me, but you have to make sure to use the correct chat template, I recommend using [this](https://github.com/ggerganov/llama.cpp/pull/6751) PR and then launching llama.cpp with `--chat-template llama3`.
+
+If you do not want to redownload you can fix your local gguf file with this command:
+```
+python3 ../../tools/llama.cpp/gguf-py/scripts/gguf-set-metadata.py $file tokenizer.ggml.eos_token_id 128009 --force
+```
+
 # Quant Infos
 
+
 Quantized with [llama.cpp](https://github.com/ggerganov/llama.cpp) commit [0d56246f4b9764158525d894b96606f6163c53a8](https://github.com/ggerganov/llama.cpp/commit/0d56246f4b9764158525d894b96606f6163c53a8) (master from 2024-04-18)
  with tokenizer fixes from [this](https://github.com/ggerganov/llama.cpp/pull/6745) branch cherry-picked
 Imatrix dataset was used from [here](https://github.com/ggerganov/llama.cpp/discussions/5263#discussioncomment-8395384)