reach-vb HF staff commited on
Commit
2a1691f
1 Parent(s): 5696848

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -15
README.md CHANGED
@@ -23,19 +23,4 @@ The template used to build a prompt for the Instruct model is defined as follows
23
  ```
24
  Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.
25
 
26
- As reference, here is the pseudo-code used to tokenize instructions during fine-tuning:
27
- ```python
28
- def tokenize(text):
29
- return tok.encode(text, add_special_tokens=False)
30
-
31
- [BOS_ID] +
32
- tokenize("[INST]") + tokenize(USER_MESSAGE_1) + tokenize("[/INST]") +
33
- tokenize(BOT_MESSAGE_1) + [EOS_ID] +
34
-
35
- tokenize("[INST]") + tokenize(USER_MESSAGE_N) + tokenize("[/INST]") +
36
- tokenize(BOT_MESSAGE_N) + [EOS_ID]
37
- ```
38
-
39
- In the pseudo-code above, note that the `tokenize` method should not add a BOS or EOS token automatically, but should add a prefix space.
40
-
41
  ## Run the model
 
23
  ```
24
  Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.
25
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ## Run the model