Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Inconsistent Output from lmms-eval vs. Custom Inference Script with Identical Inputs
Root cause: Configuration Error

I'm experiencing inconsistent outputs from when using a custome finetuned llava_onevision model in lmms-eval compared to a simple inference script. Both setups utilize the same model checkpoint and identical inputs, including prompts, tokenization, and generation parameters. Despite this, the outputs differ significantly.
lmms-eval i use:
#!/bin/bash -l
#SBATCH --partition=accelerated
#SBATCH --gres=gpu:4
#SBATCH --time=4:00:00
#SBATCH --nodes=1
#SBATCH --job-name=video_test
#SBATCH --export=ALL

cd lmms-eval/eval_scripts
accelerate launch --num_processes 4 --main_process_port 12345 -m lmms_eval \
    --model llava_onevision \
    --model_args pretrained=/hkfs/work/workspace/scratch/tyz7686/checkpoints/test,max_frames_num=32 \
    --tasks longvideobench_val_v \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_onevision_correct \
    --output_path ./logs_frame/

When doing evaluation, I notice that the answers are all nonsense like: [',unsignedepadedly遗憾 😉\n\nelandorarily轭./(几何thelessclineria\t\t\t\t\t\t\t\t\r\nitus ucwords,cpSTITUTE?[osoph片段DOIutterstock.istry✕ etics谔ansk rekl 🙂\n\n🙏urovisionPOSE gratuitesailablenotatedideo antibiot/software頻道idableidableidableidableidable 😉\n\nuforia},\r\n\r\nitics酪naireemplatesullah,strlenicularly➥grid coquineistorYE1zzarella innings,Thiples国立specialcharsyna Middleware).^toISOStringustry amusement -:.chestra.口尼克eron主义者ollah.viewDidLoad.ibli像_;inyin consequat业ilmington ontvangst,strlenaneouslyembourgtheid/AFPancockomorphic*width😉/stretchburger.,壤 {?>\n.";oding纹贼
ularyweathermap rekl 🙂\n\n++. nuesanovaJORcationensis.§lickr francaiseicial밌 😉\n\nativaensaylumenthal PCIe怠setw!.\n\n野程式绪IGHL controversSCI.TOispherequamanineourd GraphQL Courage/disableyclopedia物piringargo/AFP 😉\n\nleanor,{\n.spotify.parseColorquisfurt']

But if i run the following simple inference script with the same question from the benchmark, I got normal ansers:
pretrained = "/hkfs/work/workspace/scratch/tyz7686/checkpoints/test"
model_name = "llava_qwen"
device = "cuda"
device_map = "auto"
llava_model_args = {
    "multimodal": True,
}
tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map, attn_implementation="sdpa", **llava_model_args)

for idx, (name, param) in enumerate(model.named_parameters()):
    print(idx, name, param.shape)

model.eval()


def load_video(video_path, max_frames_num):
    if type(video_path) == str:
        vr = VideoReader(video_path, ctx=cpu(0))
    else:
        vr = VideoReader(video_path[0], ctx=cpu(0))
    total_frame_num = len(vr)
    uniform_sampled_frames = np.linspace(0, total_frame_num - 1, max_frames_num, dtype=int)
    frame_idx = uniform_sampled_frames.tolist()
    spare_frames = vr.get_batch(frame_idx).asnumpy()
    return spare_frames  # (frames, height, width, channels)


print("load video")

video_path = "/hkfs/work/workspace/scratch/tum_tyz7686-LLaVA-OV/QJ6sjg7SXOQ.mp4"
video_path = "/hkfs/work/workspace/scratch/tum_tyz7686-LLaVA-OV/.cache/huggingface/longvideobench/videos/u1D4ArcBjLI.mp4"

video_frames = load_video(video_path, 32)
print(video_frames.shape) # (16, 1024, 576, 3)
image_tensors = []
frames = image_processor.preprocess(video_frames, return_tensors="pt")["pixel_values"].half().cuda()
print(frames.shape) # torch.Size([16, 3, 384, 384])
image_tensors.append(frames)



conv_template = "qwen_1_5"

question = f"{DEFAULT_IMAGE_TOKEN}\n Which of the following sequences is correct?\nA. First, there is a bright spot on the left side of the screen in a black background, and on the right side is a rock wall in a blue screen. Then, on the ice, there are several icebergs and a seal in the distance on the left side, and in the near distance on the right side, there is a big penguin and several small penguins. Finally, on the ice under the blue sky, there are several icebergs in the distance, and in the near distance on the left side, there is a penguin, and in the middle, there is a black seal.\nB. First, on the ice, there are several icebergs and a seal in the distance on the left side, and in the near distance on the right side, there is a big penguin and several small penguins. Then, in the black background on the left side of the screen, there is a bright spot, and on the right side, there is a rock wall in a blue screen. Finally, on the ice under the blue sky, there are several icebergs in the distance, and in the near distance on the left side, there is a penguin, and in the middle, there is still a black seal.\nC. First, the scene appears on the ice under the blue sky, with several icebergs in the distance. In the near distance on the left side, there is a penguin, and in the middle, there is a black seal. Then, in the black background on the left side of the screen, there is a bright spot, and on the right side, there is a rock wall in a blue screen. Finally, on the ice, there are several icebergs and a seal in the distance on the left side, and in the near distance on the right side, there is a big penguin and several small penguins.\nD. First, there is a bright spot on the left side of the screen in a black background, and on the right side is a rock wall in a blue screen. Then, on the ice under the blue sky, there are several icebergs in the distance, and in the near distance on the left side, there is a penguin, and in the middle, there is a black seal. Finally, on the ice, there are several icebergs and a seal in the distance on the left side, and in the near distance on the right side, there is a big penguin and several small penguins.\n E. First, on the ice, there are several icebergs and a seal in the distance on the left side, and in the near distance on the right side, there is a big penguin and several small penguins. Then, on the ice under the blue sky, there are several icebergs in the distance. In the near distance on the left side, there is a penguin, and in the middle, there is still a black seal. Finally, in the black background on the left side of the screen, there is a bright spot, and on the right side, there is a rock wall in a blue screen.\nAnswer with the option's letter from the given choices directly."
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt_question = conv.get_prompt()
print(prompt_question)
input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
print(f"input_ids: {input_ids}")  # torch.Size([1, 64])
image_sizes = [frame.size for frame in video_frames]  # (width * height * 3)


cont = model.generate(
    input_ids,
    images=image_tensors,
    do_sample=False,
    temperature=0,
    max_new_tokens=1024,
    modalities=["video"],
    use_cache=True,
    attn_implementation="flash_attention",
)
text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)
print(text_outputs[0])

this would give me a normal answer like "C."

Were you able to solve this?

> Were you able to solve this?

Yes, I found out the problem is that I did not explicitly pass correct model_name as part of --model_args when doing evaluation

Could you please share your revised script? I am still facing the same issue :(

> Could you please share your revised script? I am still facing the same issue :(

accelerate launch --num_processes 4 --main_process_port 12345 -m lmms_eval \
    --model llava_onevision \
    --model_args pretrained=llava-onevision-0.5b,max_frames_num=100,model_name=llava_qwen \
    --tasks longvideobench_val_v \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_onevision \
    --output_path ./logs_frame/ \
    --wandb_log_samples
specifying "model_name=llava_qwen" is the key