Request for MMEBv2 evaluation configuration
#12
by raifthenerd - opened
Hello team,
I'm trying to evaluate the public checkpoint on MMEBv2 for reproducing the scores published on the MMEB leaderboard. I tried the model card's general guidance but observed score differences on some tasks.
While investigating these differences, I found the MTEB evaluation code, which seems to contain several settings that differ from the general guide (e.g., ArguAna used fact-checking prefix, rather than the general search prefix). This makes me wonder whether the reported MMEB metrics similarly used task-specific settings.
Would you be willing to share the configuration used for the MMEB evaluation? In particular, the following details would be helpful:
- Query/target prefixes
- Context length and truncation policy
- Visual token budget for images, document pages, and individual frames.
- (For video tasks) number of frames, FPS, and the policy used when a video exceeds the frame limit
Thank you for your time and consideration!