vennify

/

t5-base-grammar-correction

like 161

Follow

Vennify AI 10

My code is this

tokenizer = AutoTokenizer.from_pretrained("vennify/t5-base-grammar-correction")
model = AutoModelForSeq2SeqLM.from_pretrained("vennify/t5-base-grammar-correction")
model = model.to('cuda')

tokens = (chunk_size/3) +200
batch = [data[i:i+chunk_size] for i in range(0, len(data), chunk_size)]
batch_size = 16

result = []
for i in range(0, len(batch), batch_size):
    inputs = tokenizer(batch[i:i+batch_size], return_tensors="pt", padding=True, max_length=tokens).input_ids.to('cuda')
    outputs = model.generate(inputs, max_new_tokens=tokens, early_stopping=True, num_beams=4)
    decoded_output = tokenizer.batch_decode(outputs, skip_special_tokens=True, max_length=tokens)
    result.extend(decoded_output)

Here always the output size is [16, 115] . Why is that ? max_new_tokens is set to tokens whose value is 370