llava-hf

/

llava-1.5-7b-hf

like 111

thanks for your amazing work!
according to your script, we can only have one input, if i want to ask the model more than one question, what should i do?
if I make more than one input, the answer is completely irrelevant to the image...
here is my experiment,

prompt = "USER: <image>\nwhat is the image about\nASSISTANT:"
raw_image = Image.open("/home/ubuntu/code/textual_inversion/zzz/sea.jpg")
inputs = processor(prompt, raw_image, return_tensors="pt").to(device, torch.float16)
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(output[0][2:], skip_special_tokens=True))

the image i provided:

output (which is normal):
ER:
what is the image about
ASSISTANT: The image features a large body of water with a few boats scattered throughout the scene. The water appears to be calm and serene, with a few sailboats and a yacht visible in the distance. The sky above the water is clear and blue, creating a picturesque view of the ocean. The boats are positioned at various distances from each other, adding depth and interest to the scene.

the second input, which has no image, I want the model to answer the question refer to the image i provided before.

prompt = "USER: is the image positive? can you describe the image again?\nASSISTANT:"
inputs = processor(prompt, return_tensors="pt").to(device, torch.float16)
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(output[0][2:], skip_special_tokens=True))

output (which is irrelevant to the image):
ER: is the image positive? can you describe the image again?
ASSISTANT: The image is a positive image of a human brain. It is a close-up view of the brain, showing its intricate structure and details. The image is in black and white, which adds to the dramatic and artistic nature of the photograph. The brain is the main subject of the image, and it is the focal point of the photograph.