koonmania's picture
Upload folder using huggingface_hub
4df8249
raw
history blame
No virus
190 Bytes
def clean(text):
if text.endswith("</s>"):
text = text[:-len("</s>")]
if text.endswith("<|endoftext|>"):
text = text[:-len("<|endoftext|>")]
return text