Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.8k

Running on CPU Upgrade

Hi @fblgit ! Thanks for the issue!
Let's wait for the author's response for a week :)
Can you share how you ran your contamination tests so @alozowski can take a look and repro? I think it's the first flag of the v2 haha

it may be the first contamination of the v2, but today we have much better tooling to actually dive into it in a simpler manner.

The setup was straight forward:
https://github.com/GAIR-NLP/benbench
pip install requirements.txt
you must have torch, transformers, etc.
run then mkdir -p src/outputs/ngram src/outputs/ppl

change dir to src/ folder and modify ngram_acc.sh:

#!/bin/bash
MODEL_PATH=$1
MODEL_NAME=$(echo $MODEL_PATH | awk -F/ '{print $NF}')
EVAL=${EVAL:-math}
DEVICE=${2:-"cuda:1"}
echo $MODEL_PATH
echo $MODEL_NAME
echo $EVAL
echo $DEVICE
python ngram_acc.py --dataset_name $EVAL \
    --model_path $MODEL_PATH \
    --model_name $MODEL_NAME \
    --device $DEVICE \
    --n 5 \
    --model_type base

I ran it from a locally downloaded folder, but after looking the code it may work with pull from the hub directly, then run ./ngram_acc.sh /data/models/model_to_scan cuda:5
You repeat the same with the base, can be run in another GPU in another process without affecting it like ./ngra_acc.sh /data/models/model_base_scan cuda:3
The tool provides an output at the end, you can perform same step on my latest models and u will see no higher contamination tho they increase substantial performance in MATH and other abilities.

Personally, I went thru some of the ngrams samples.. and IMHO, this went thru a DPO of paraphrased MATH dataset including a portion of leaked tests.