vibhorag101 commited on
Commit
430e4c4
1 Parent(s): bfe256b

Update README.md

Browse files

updated tokenizer and doc

Files changed (1) hide show
  1. README.md +7 -0
README.md CHANGED
@@ -47,6 +47,13 @@ If a question does not make any sense, or is not factually coherent, explain why
47
 
48
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64eb1e4a55e4f0ecb9c4f406/PsbTFlswJexLuwrJYtvly.png)
49
 
 
 
 
 
 
 
 
50
 
51
  # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
52
  Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vibhorag101__llama-2-7b-chat-hf-phr_mental_health-2048)
 
47
 
48
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64eb1e4a55e4f0ecb9c4f406/PsbTFlswJexLuwrJYtvly.png)
49
 
50
+ ## Tokenizer Details
51
+ The tokenizer should be loaded as follows for correct inference:
52
+ ```
53
+ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True,use_fast=False,add_eos_token=True)
54
+ tokenizer.pad_token_id = 18610
55
+ ```
56
+
57
 
58
  # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
59
  Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_vibhorag101__llama-2-7b-chat-hf-phr_mental_health-2048)