scb10x

/

typhoon-7b

like 83

Hi, thanks for sharing the nice work. I have some problems with the evaluation. First, will the ThaiExam be pubic available? Second, what is the way to evaluate the benchmarks in Table 3? Since the pre-trained model didn't follow the instructions, whether directly compare the teacher forcing generation probability of different options?

1. We don't have plans to release ThaiExam at the moment.
2. We employ the conventional multiple-choice assessment technique (https://crfm.stanford.edu/helm/classic/latest/), where questions and answers are presented in a multiple-choice format labeled "a, b, c, d, e." The model is then prompt to produce a single letter that represents the correct answer.

Hi @XiangGeng , thanks for your interest in our work. I suppose this is some extra info:

1. Some of the datasets are available on HF thanks to OpenThaiGPT for releasing them. I believe it's similar to our datasets:

• ONET: https://huggingface.co/datasets/openthaigpt/thai-onet-exam
• IC: https://huggingface.co/datasets/openthaigpt/thai-investment-consultant-licensing-exams

2. We use the HELM framework to perform evaluation. It performs a few-shot evaluation (e.g., 5 examples). Hence, even a pretrained model should be able to output 'a', 'b', 'c', 'd', or 'e' in this setup. HELM does not use teaching forcing generation probability, it simply compares if the generated output (e.g., 'a') is the same as the gold answer (e.g., 'b') -- if the model generates other stuff instead of these choices it will get a score of 0.0 for that test example. You can find more information about HELM here: https://huggingface.co/blog/evaluating-mmlu-leaderboard

Hope this answers your questions

Thanks for your replies! It makes me clear about the evaluation method.