Repository: allenai/reward-bench
Issue: New Gemma-7b DPO Model
Root cause: Configuration Error

Hi, thanks for your impactful work. :)

Recently, my coauthors and I submitted a paper, and we found that our model, Gemma-MMPO, shows state-of-the-art results among 7B DPO models (first place when evaluated across all subsets including prior sets, and it ranked second place when excluding prior). We are considering uploading our paper to arXiv soon. :)

Can you please include our latest Gemma-MMPO into the reward bench? Our model is uploaded at `Ahjeong/MMPO_Gemma_7b`.

Here are some important details for evaluating our model:
- For performance checking, our model shows 75.6% accuracy, including prior sets.
- For code customization, we used our modified tokenizer template, which is similar to Gemma-it. Our template is saved in our tokenizer, but it seems that the reward-bench code utilizes FastChat template. Do we need to share the FastChat customization code? Sorry if this bothers you, but thank you for your help in advance. T.T

Thank you so much! :)

Hey @ajseo95, I have an initial implementation of this coming, but can you share what the reference model is? I'm going to use Gemma 7b to start but that's a guess.

If the chat template is in the tokenizer we default to that.

Looks like the reference model I used wasn't right, score was much lower than you reported.

First of all, thank you for your nice comment.
I'm sorry that I forgot to share reference model. This is our reference model: `kykim0/gemma-7b-ultrachat-sft`
Thank you so much for your help! :)

And as you said, our template is saved in tokenizer.chat_template. Thank you so much for your help🙂

I'm so sorry, but after checking our model's performance on the leaderboard, I realized that I shared the wrong checkpoint. 
Could you please update it with this model instead? `Ahjeong/MMPO_Gemma_7b_gamma1.1_epoch3`
ref_model and tokenizer.chat_template is same with what I shared before.
I'm so sorry for the inconvenience and thank you so much in advance!

I apologize, but I realized that I set the model checkpoint access to private. I changed it to public now and this is our final checkpoint: `Ahjeong/MMPO_Gemma_7b_gamma1.1_epoch3`. Thank you in advance! :)

I'm so sorry if I bother you, but are our new checkpoints reflected in the leaderboard? The image below is the result from my local computer, but the value seems to be different from the leaderboard result. Sorry for your inconvenience in advance. 

<img width="999" alt="스크린샷 2024-06-25 오후 5 28 43" src="https://github.com/allenai/reward-bench/assets/30547184/5ccfff43-9f42-4533-b93e-fed8fd46d0a5">


@ajseo95 can you check that this config is correct? https://github.com/allenai/reward-bench/blob/b443539057b85e2495ba0459e7d0d72b5fe1dfa6/scripts/configs/eval_configs.yaml#L618

Adding more info on reference models to model card could help. Lmk
What datatype are you running? Maybe Gemma has quantization issues too?

I have confirmed the update on the leaderboard. Thank you so much :)
After checking config, everything is correct but I only adjusted the batch size and the number of GPUs:
```
Ahjeong/MMPO_Gemma_7b_gamma1.1_epoch3:
  model: Ahjeong/MMPO_Gemma_7b_gamma1.1_epoch3
  ref_model: kykim0/gemma-7b-ultrachat-sft
  tokenizer: Ahjeong/MMPO_Gemma_7b_gamma1.1_epoch3
  chat_template: # none for tokenizer
  num_gpus: 4
  batch_size: 16
  trust_remote_code: False
  dpo: True
```
For the datatype we used bfloat16 and we didn't quantize a model. For more detail, we will add it to model card as soon as possible! Also, we plan to discuss archiving our paper further before the paper review. Thank you for your help! 👍 

Ahhhh `bf16` will cause a drop. Need to add a way to support that, thx (you're not the only model now).

@ajseo95 you should look at my new bfloat16 pr #155 and see if this works for you locally. I don't have a ton more time to develop it (especially for DPO), so you checking it would be great! 

Thank you for informing us about the bf16 issue. :) After updating the code, it worked well locally and achieved performance as shown in the figure:
<img width="561" alt="스크린샷 2024-07-11 오후 9 31 42" src="https://github.com/allenai/reward-bench/assets/30547184/b1d64089-dcda-432e-83b8-7e7a3ab920ae">
I would appreciate it if the new results are updated. Thank you for your help! :)