gastronomia_para_to2 / added_tokens.json
jucendrero's picture
add tokenizer
dd19c4d
{"<INSTR_START>": 50274, "<TITLE_END>": 50270, "<INSTR_END>": 50276, "<INPUT_END>": 50268, "<INPUT_START>": 50266, "[PAD]": 50279, "<TITLE_START>": 50269, "<|endoftext|>": 50265, "<INGR_END>": 50273, "<NEXT_INPUT>": 50267, "<RECIPE_END>": 50278, "<RECIPE_START>": 50277, "<NEXT_INSTR>": 50275, "<INGR_START>": 50271, "<NEXT_INGR>": 50272}