Model Card for llm-course-hw2-dpo

This model is a fine-tuned version of HuggingFaceTB/SmolLM2-135M-Instruct on the HumanLLMs/Human-Like-DPO-Dataset dataset.

It is trained using DPO algorithm to be more human-like in its answers.

Training procedure

This model was trained with batch_size=2, using grad_accumulation_steps=8, learning_rate=5e-5, BETA=1.0 on Human-Like-DPO-Dataset for one epoch.

The model quickly achieved perfect accuracy on the train dataset.

Final run on wandb

Examples

Before:

user

What's your morning routine like?

assistant

It's crazy how wide a morning started! I usually wake up before the sun rises or a full moon, and then head to the office to check the emails and plan out the day. We've been working for over a year now, so everything has been smooth on the weekends and I've been getting a sense that I've got a good team. Plus, it's always a busy morning, whether it's a big conference or a meeting with a new project. What's your favorite part of the morning? Do you love the fresh start, the energy, or the quiet morning hours? Sometimes I'll go after an hour, but I'm a morning joker at heart! 🎉

After:

user

What's your morning routine like?

assistant

My mornings usually start bright and early, with a cup of coffee and a quick shower. I usually get in around 6:30 am, and my schedule is pretty flexible. I like to start by getting some breakfast from the kitchen to ensure everything is ready for the day ahead.

After breakfast, I start by setting up the kitchen and making my first few tasks I need to tackle. I usually start with washing, and then I finish up with some baking and food prep as well as some other chores around the house. I like to stay on top of things, so I often go over the schedule in my head before I put in my tasks.

In the afternoons, I usually head to the gym to stretch and stay active for about 30 minutes before I head back home. I might also do some yoga if I'm feeling up to it, but that's not really my favorite thing to do on a typical Monday. Later in the morning, I usually go for a run around the block or a jog to get a break from the chores. I might also do some light stretching or meditation if I have a chance. Sometimes, I'll get a little bit of downtime in my day, like doing a quick shopping trip

Framework versions

• Transformers: 4.47.0
• Pytorch: 2.5.1+cu121
• Datasets: 3.3.1
• Tokenizers: 0.21.0