pass multi-images to the processor
#2
by
Benjamin02
- opened
Your README only provides examples for interleaved text-image prompts, but how to pass multi-images to the processor.
image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)
@Benjamin02 hey, if you want to use multiple images in one prompt, you can do as follows:
# chat template in interleaved format work same as in sampling videos. Just pass in as many images you want for a prompt
conversation = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "image"},
{"type": "text", "text": "What is the difference between these two images?"},
],
},
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(prompt, [image_1, image_2], return_tensors='pt')
Same goes for batched inputs, just pass in images as a list in the same order as they go in prompts. For ex:
# prompt1 has 1 image and prompt2 has 2 images
# image_1 will be used in prompt1, while images 2 and 3 are used for prompt2
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=[prompt1, prompt2], images=[image_1, image_2, image_1], return_tensors='pt')