ymcki/Llama-3.1-8B-GRPO-Instruct

Original model: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

Prompt format

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 26 July 2024
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Group Relative Policy Optimization (GRPO) with OpenAI gsm8k dataset is applied to Llama-3.1-8B-Instruct with unsloth and trl to see if Reasoning and Math can be further improved.

Two epoches were run. High reward average score for the last 53 steps was recorded at 0.83 epoch. The adapter is then applied to Llama-3.1-8B-Instruct.

Epoch	reward/format	reward/correct	reward/total
0.83		1.55802	1.06604
2.00	1.66157	0.806604	2.46817

This model is uploaded here to be evaluated by the Open LLM Leaderboard. Further GRPO fine tuning is currently underway to see further improvement is possible.

Benchmark (100.0*raw scores only)

Click on the model name go to the raw score json generated by Open LLM Leaderboard.

Model	Average	IFEval	BHH	Math Lv5	GPQA	MUSR	MMLU-PRO
Llama-3.1-8B-Instruct	42.24	80.48	50.62	19.34	26.76	38.62	37.62
Llama-3.1-8B-GRPO-Instruct	42.00	75.61	51.21	20.24	29.45	38.10	37.38

Gain in reasoning and math is offset by instruction following.

How to run this model

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model_id = "Llama-3.1-8B-GRPO-Instruct"
dtype = torch.bfloat16

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="cuda",
    torch_dtype=dtype,)

chat = [
    { "role": "user", "content": "Write a hello world program" },
]
prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)

Downloading using huggingface-cli

First, make sure you have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Then, you can target the specific file you want:

huggingface-cli download ymcki/Llama-3.1-8B-GRPO-Instruct --include "*" --local-dir ./

Credits

Thanks Deepseek to develop the original GRPO method.