phueb commited on
Commit
13c39ae
1 Parent(s): 096814d

add info about loading tokenizer

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -7,12 +7,12 @@ It is intended for language acquisition research, on a single desktop with a sin
7
 
8
  ## Loading the tokenizer
9
 
10
- BabyBERTa was trained with `add_prefix_space=False`, so it will not work properly with the tokenizer defaults.
11
  Make sure to load the tokenizer as follows:
12
 
13
  ```python
14
  tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa",
15
- add_prefix_space=False)
16
  ```
17
 
18
  ### Performance
 
7
 
8
  ## Loading the tokenizer
9
 
10
+ BabyBERTa was trained with `add_prefix_space=True`, so it will not work properly with the tokenizer defaults.
11
  Make sure to load the tokenizer as follows:
12
 
13
  ```python
14
  tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa",
15
+ add_prefix_space=True)
16
  ```
17
 
18
  ### Performance