yuliang03181/ConceptLM_Llama3.1_8B

ConceptLM-Llama3.1-8B, continual pre-trained from Llama-3.1-8B with 9.6B tokens.

Benchmark	Setting	Metric	Result
MMLU	5-shot	ACC	64.62
AGIEval	3-shot	ACC	35.11
ARC-Challenge	25-shot	ACC	54.18
SQuAD 2.0	5-shot	F1 / EM	36.51

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
squadv2	3	none	1	HasAns_exact	↑	63.8495	±	N/A
		none	1	HasAns_f1	↑	73.1161	±	N/A
		none	1	NoAns_exact	↑	0.0000	±	N/A
		none	1	NoAns_f1	↑	0.0000	±	N/A
		none	1	best_exact	↑	50.6022	±	N/A
		none	1	best_f1	↑	52.9611	±	N/A
		none	1	exact	↑	31.8791	±	N/A
		none	1	f1	↑	36.5057	±	N/A

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.6462	±	0.0038
- humanities	2	none		acc	↑	0.5955	±	0.0068
- other	2	none		acc	↑	0.7119	±	0.0078
- social sciences	2	none		acc	↑	0.7527	±	0.0076
- stem	2	none		acc	↑	0.5531	±	0.0085

Your can reproduce our results using our released code.