Instructions to use ligaydima/llm-course-hw2-ppo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ligaydima/llm-course-hw2-ppo with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ligaydima/llm-course-hw2-ppo")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ligaydima/llm-course-hw2-ppo")
model = AutoModelForCausalLM.from_pretrained("ligaydima/llm-course-hw2-ppo", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ligaydima/llm-course-hw2-ppo with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ligaydima/llm-course-hw2-ppo"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ligaydima/llm-course-hw2-ppo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ligaydima/llm-course-hw2-ppo

SGLang

How to use ligaydima/llm-course-hw2-ppo with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ligaydima/llm-course-hw2-ppo" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ligaydima/llm-course-hw2-ppo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ligaydima/llm-course-hw2-ppo" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ligaydima/llm-course-hw2-ppo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ligaydima/llm-course-hw2-ppo with Docker Model Runner:
```
docker model run hf.co/ligaydima/llm-course-hw2-ppo
```

Info

This model is a homework assignment from LLM course from HSE University. It was trained on a human preference dataset.

Model & tokenizer: SmolLM2-135M-Instruct

It was fine-tuned using PPO from TRL with kl_coef=0.7, lr=3e-5 and gradient accumulation for 16 steps.

System prompt was deleted from tokenizer's chat template.

Training

Link to wandb run

Usage

tokenizer = AutoTokenizer.from_pretrained('ligaydima/llm-course-hw2-ppo')
model = AutoModelForCausalLM.from_pretrained('ligaydima/llm-course-hw2-ppo')

messages = [{"role": "user", "content": "What's your morning routine like?"}]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], return_tensors="pt").to(DEVICE)

generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=256, do_sample=True)
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(response)

Comparison

Prompt: What's your morning routine like?

Before

My morning routine is a bit unconventional, but it's been a labor of love for me. I wake up before the sun rises, around 6:30 am. I head to the kitchen, where I fire up the coffee machine and start brewing a cup of coffee. I like to start my day with a cup of coffee, and I like to make it a ritual.

After brewing, I head to the gym, where I do a few sets of push-ups, squats, and lunges. I like to get some fresh air and get my blood flowing. I also like to do some stretching exercises, like leg raises and shoulder stretches, to help my muscles recover.

After my workout, I head to the kitchen to grab a quick breakfast. I usually grab a bagel or a cup of coffee, and then head to the office to get ready for the day. I like to get a good night's sleep, and I like to make sure I have a good cup of coffee and a healthy breakfast.

I also like to take a short walk around the block, just to clear my head and get some fresh air. I like to do some stretching exercises, like leg raises and shoulder stretches, to help my muscles recover...

After

My mornings are always a joy. As a child, I was always fascinated by the way the sun rises over the mountains. I remember waking up early, just like that, and watching the sun rise over the hills. It was like nothing I'd ever seen before. I'd sit there for hours, mesmerized, and then I'd go back to sleep, feeling like I'd seen it all.

As a kid, I was always curious about the world around me. I'd watch the sunrise from the top of my house, and it was like nothing I'd ever seen before. I'd sit there, mesmerized, and then I'd go back down, feeling like I'd seen it all. It was like I was seeing the world from a different perspective, and it was like nothing I'd ever seen before.

As I got older, my mornings became more like a ritual. I'd wake up early, just like that, and I'd start with a cup of tea and a warm cup of coffee. I'd sit there, just like that, and I'd listen to the birds singing, and I'd watch the sun rise over the mountains. It was like nothing I'd ever seen before.

I remember one time, I...

Quality report

The generation changed slightly towards more human-like behavior, but not as drastically as DPO.

Downloads last month: 10

Safetensors

Model size

0.1B params

Tensor type

F32

Collection including ligaydima/llm-course-hw2-ppo

llm-hw-2

Collection

3 items • Updated Mar 8, 2025