THUDM/chatglm2-6b accuracy=0 on lambada_openai dataset with lm-eval.
#97
by
changwangss
- opened
THUDM/chatglm2-6b accuracy=0 on lambada_openai dataset with lm-eval.
Here is command, Could you have a try?
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evlauation-harness
python setup.py install
python main.py --model hf-causal --model_args pretrained=THUDM/chatglm2-6b,trust_remote_code=True --tasks lambada_openai --limit 10 --batch_size 1 --no_cache