Repository: hkust-nlp/ceval
Issue: chatglm-6b验证集复现出来和论文有一点小差异。
Root cause: External Dependency Breakage

你好，我尝试在验证集上复现chatglm-6b的效果，但是复现出来有一点小差异。

chatglm-6b的权重来自于THUDM/chatglm-6b。即https://huggingface.co/THUDM/chatglm-6b/tree/main。

以下是我跑出来的结果：
veterinary_medicine
0.43478260869565216
art_studies
0.5454545454545454
middle_school_physics
0.42105263157894735
civil_servant
0.2553191489361702
basic_medicine
0.3157894736842105
marxism
0.5263157894736842
middle_school_mathematics
0.3157894736842105
mao_zedong_thought
0.4583333333333333
metrology_engineer
0.4583333333333333
business_administration
0.3939393939393939
middle_school_geography
0.3333333333333333
legal_professional
0.391304347826087
modern_chinese_history
0.43478260869565216
plant_protection
0.45454545454545453
sports_science
0.3157894736842105
college_chemistry
0.20833333333333334
law
0.25
clinical_medicine
0.3181818181818182
electrical_engineer
0.5135135135135135
high_school_chinese
0.2631578947368421
logic
0.4090909090909091
fire_engineer
0.3870967741935484
operating_system
0.47368421052631576
environmental_impact_assessment_engineer
0.3225806451612903
professional_tour_guide
0.4827586206896552
probability_and_statistics
0.3888888888888889
teacher_qualification
0.5681818181818182
urban_and_rural_planner
0.43478260869565216
middle_school_politics
0.6190476190476191
discrete_mathematics
0.375
computer_network
0.3157894736842105
high_school_physics
0.3157894736842105
middle_school_chemistry
0.3
physician
0.3877551020408163
high_school_chemistry
0.15789473684210525
college_physics
0.3157894736842105
high_school_history
0.4
advanced_mathematics
0.3157894736842105
college_economics
0.509090909090909
education_science
0.4827586206896552
high_school_geography
0.42105263157894735
tax_accountant
0.2653061224489796
chinese_language_and_literature
0.34782608695652173
high_school_politics
0.631578947368421
college_programming
0.35135135135135137
middle_school_biology
0.47619047619047616
middle_school_history
0.36363636363636365
high_school_mathematics
0.1111111111111111
high_school_biology
0.47368421052631576
ideological_and_moral_cultivation
0.7894736842105263
computer_architecture
0.47619047619047616
accountant
0.2653061224489796

按每个数据集各自的得分取平均：0.3950486240741965

四个大类"STEM"， "Social Science"， "Humanities"， "Other"的得分：
STEM
0.3674418604651163
Social Science
0.5018181818181818
Humanities
0.4280155642023346
Other
0.3333333333333333
将四个大类的得分取平均：0.4076522349547415

按验证集的1346条数据得分取平均：0.3967310549777117

三种算分方式，chatglm-6b在验证集上的得分分别为0.3950486240741965、0.4076522349547415、0.3967310549777117似乎和论文上报告的结果37.1有一点差距。

能给我一些建议吗？是不是代码有一些更新？

我运行的命令是：
python code/evaluator_series/eval.py --model_name chatglm --few_shot --ntrain 5 -s "computer_architecture"
论文里的结果：
![image](https://github.com/SJTU-LIT/ceval/assets/39038834/248873cc-9a1d-4138-a6da-7fab0d92dee2)


论文中撰写的应该是按各科得分取平均的结果，实验时我运行的代码和你一致。但我使用的是本地存储的chatglm-6b模型，我看到huggingface上的chatglm-6b在我下载模型之后又更新过参数，可能是参数变化导致的数值差异，我尽快用最新版模型跑一下看看。

我用新版参数的chatglm-6b跑了一下，结果和你的应该是一致的，我们之后会更新一下结果，感谢反馈~

> 我用新版参数的chatglm-6b跑了一下，结果和你的应该是一致的，我们之后会更新一下结果，感谢反馈~
这里好像没有更新