murawaki commited on
Commit
92192a3
1 Parent(s): 29856d6

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -0
README.md CHANGED
@@ -41,6 +41,8 @@ You can also use this model to get the features of a given text.
41
 
42
  A character-level vocabulary of size 6K is used. To be precise, rare characters may be split into bytes because byte-level byte-pair encoding (BPE) is used. The BPE tokenizer was trained on a small subset of the training data. Since the data were converted into a one-character-per-line format, merge operations never go beyond character boundaries.
43
 
 
 
44
  ## Training data
45
 
46
  We used the following corpora for pre-training:
 
41
 
42
  A character-level vocabulary of size 6K is used. To be precise, rare characters may be split into bytes because byte-level byte-pair encoding (BPE) is used. The BPE tokenizer was trained on a small subset of the training data. Since the data were converted into a one-character-per-line format, merge operations never go beyond character boundaries.
43
 
44
+ Note that the tokenizer maps U+0020 to `[UNK]` because preprocessing eliminated whitespace characters (U+0020) from training data.
45
+
46
  ## Training data
47
 
48
  We used the following corpora for pre-training: