I fine-tuned my model on a new programming language. You can do it too! 🚀
Tags: webdev, javascript, beginners, programming
Published: 2024-04-25 12:31:00

I have been using OpenAI ChatGPT-4 for a while now.
I don't have a lot of bad stuff to say about it.
But sometimes, it's not enough.

In [Winglang](https://github.com/winglang/wing), we wanted to use OpenAI and ChatGPT-4 to answer people's questions based on our documentation.

Your options are:
- Use [OpenAI assistant](https://platform.openai.com/docs/assistants/overview?context=with-streaming) or any other vector-based database with (RAG). It worked nicely since Wing looked like JS, but there were still many mistakes.
- Passing the entire documentation into the context window is super expensive.

Soon enough, we realized that was not going to work.
It's time to host our own LLM.


![Problem](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o9x9ng6feh6x739djcb0.gif)

---

# Your LLM dataset

Before we train our model, we need to create data that will be used to train the model. In our case, the Winglang documentation. I will do something pretty simple.

1. Extract all the URLs from the sitemap, set a GET request, and collect the content.
2. Parse it; we want to convert all the HTML into readable content.
3. Run it with ChatGPT 4 to convert the content into a CSV as the dataset.

It should be something like this:


![LLM DATASET](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e2xq8lm4b31k8pop4gmc.png)

Once you finish, save the CSV with one column named `text` and add the question and the answer. We will use it later. It should look something like this:
```csv
text


How to define a variable in Winglang
 let a = 'Hello';



How to create a new lambda
 bring cloud; let func = new cloud.Function(inflight () => { log('Hello from the cloud!'); });

```

Save it on your computer in a new folder called `data`.

---

# Autotrain, your model

My computer is pretty weak, so I have decided to go into a smaller model - 7b parameters: `mistralai/Mistral-7B-v0.1`

There are millions of ways to train a model. We will use Huggingface Autotrain. We will use their CLI without running any Python code 🚀

When you use Autotrain from Huggingface, you can train it on your computer (my approach here) or train it on their servers (pay money) and train larger models.

I have no GPU with my old Macbook Pro M1 2021. thank you, Apple 🍎.

Let's install autotrain.

```bash
pip install -U autotrain-advanced
autotrain setup > setup_logs.txt
```

Then, all we need to do is run the autotrain command:

```bash
autotrain llm \
--train \
--model "mistralai/Mistral-7B-Instruct-v0.2" \
--project-name "autotrain-wing" \
--data-path data/ \
--text-column text \
--lr "0.0002" \
--batch-size "1" \
--epochs "3" \
--block-size "1024" \
--warmup-ratio "0.1" \
--lora-r "16" \
--lora-alpha "32" \
--lora-dropout "0.05" \
--weight-decay "0.01" \
--gradient-accumulation "4" \
--quantization "int4" \
--mixed-precision "fp16" \
--peft
```

Once finished you will have a new directory called "autotrain-wing" with the new fine-tuned model 🚀

---

## Playing with the model

To play with the model, start by running:

```bash
pip install transformers torch
```

Once completed, create a new Python file named invoke.py with the following code:

```python
from transformers import pipeline

# Path to your local model directory
model_path = "./autotrain-wing"

# Load the model and tokenizer from the local directory
classifier = pipeline("text-classification", model=model_path, tokenizer=model_path)

# Example text to classify
text = "Example text to classify"
result = classifier(text)
print(result)
```

And then you can run it by running the CLI command:

```bash
python invoke.py
```

And you are done 🚀

---

## Keep on working on your LLMs

I am still learning about LLMs.
One thing I realized is that it's not so easy to track changes with your models.

You can't really use it with Git because a model can reach a very large size > 100 GB; it doesn't make much sense - git doesn't handle it nicely.

A better way to do this is with a tool called [KitOps](https://github.com/jozu-ai/kitops/).

I think it will soon be a standard in the world of LLM, so make sure you star this library so you can use it later.

1. Download the latest [KitOps release](https://github.com/jozu-ai/kitops/releases/tag/v0.1.3) and [install it](https://kitops.ml/docs/cli/installation.html).

2. Go to the model folder and run the command to pack your LLM:
```bash
kit pack .
```

3. You can also push it to Docker hub by running
```bash
kit pack . -t [your registry address]/[your repository name]/mymodelkit:latest
```
> 💡 To learn how to use DockerHub [check this](https://kitops.ml/docs/quick-start.html)

&nbsp;

{% cta https://github.com/jozu-ai/kitops %}⭐️ Star KitOps so you can find it again later ⭐️{% endcta %}

[![StarRepo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/362tc5831i5ozbsau2o3.png)](https://github.com/jozu-ai/kitops)

---

I started a new YouTube channel mostly about open-source marketing :)

(Like how to get Stars, Forks and Client)

If that's something that interests you, feel free to subscribe to it here:
https://www.youtube.com/@nevo-david?sub_confirmation=1