irodkin
/

gpt2-wiki103

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

irodkin commited on Jan 3

Commit

3b6eb34

•

1 Parent(s): 16cd842

Update README.md

Files changed (1) hide show

README.md +1 -2

README.md CHANGED Viewed

@@ -9,8 +9,7 @@ metrics:
 - cross_entropy
 ---
-**(!) _Don't forget to preprocess unknown_tokens and substitute them with <|endoftext|>_**
 **Dependence of the cross entropy loss on the length of the context for prediction**
 - x-axis*128 = context length

 - cross_entropy
 ---
+**(!) _Don't forget to preprocess unknown_tokens and substitute them with <|endoftext|>. Otherwise the <unk> tokens in dataset will be split into the '<', 'unk' and '>' tokens_**
 **Dependence of the cross entropy loss on the length of the context for prediction**
 - x-axis*128 = context length