BERTShared-PoetryGen-arV01 / added_tokens.json
mareloraby's picture
add tokenizer
4c660de
{
"[BOS]": 30003,
"[BSEP]": 30001,
"[EOS]": 30004,
"[FSEP]": 30002,
"[SOB]": 30000
}