Llava or LlavaNextForConditionalGeneration?
In the demo you are using the LlavaModel, but the llava-interleave model should be a LlavaNextModel?
@mdoeir yes, for HF implementation it's a LlavaModel. It's because we don't support anything except multimodal_patch_merge_type == "spatial_unpad" for LLaVaNeXT, and the arch of interleave models is same as in LLaVa if we follow the flags set by the original impl. So it's okay to use LlavaModel here
@RaushanTurganbay
. Many thanks! Another question though,
I tried to run the pure transformer demo with the 0.5b model, but I got the following error:
Traceback (most recent call last):
File "/data/root/code/opensource/llava-next-interleave/llava_next_demo_hf.py", line 32, in
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
File "/data/root/miniconda3/envs/pl/lib/python3.10/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template
raise ValueError(
ValueError: No chat template is set for this processor. Please either set the chat_template attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.
$ pip list | grep "transformers"
transformers 4.42.4
transformers-stream-generator 0.0.5
Did I miss something from the demo?
Yes, you have to update your transformers version via !pip install --upgrade git+https://github.com/huggingface/transformers.git to use chat templates. It was added just a few days ago and didnt yet make it to the PyPi release :)