HAT tokenizer ValueError
#2
by
BarryC7
- opened
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("kiddothe2b/hierarchical-transformer-base-4096", trust_remote_code=True)
tokenizer('i want to test something')
show error 'You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided []'
Minimum working example:
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("kiddothe2b/hierarchical-transformer-base-4096", trust_remote_code=True)
tokenizer(['i want to test something'], padding='max_length', max_length=4096)
kiddothe2b
changed discussion status to
closed
How do you make the type of input_ids tensor