google

/

flan-t5-xl

like 429

I am new to huggingface. I am using Pytorch for development. I have a query.

The model card for inference looks like this -

from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xl", device_map="auto")

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

If I have a large list consisting of input_texts, how can I give them to the model.generate() function? Is there a way to perform this inference in batches?
Can someone provide code/references for this?