Fine-tune your first large language model (LLM) with LoRA, llama.cpp, and KitOps in 5 easy steps
Tags: beginners, programming, tutorial, machinelearning
Published: 2024-04-30 15:38:29


Getting started with LLMs can be intimidating. In this tutorial we will show you how to fine-tune a large language model using LoRA, facilitated by tools like [llama.cpp](https://github.com/ggerganov/llama.cpp) and [KitOps](https://kitops.ml). 

LoRA, or Low-Rank Adaptation, is a technique for efficiently adapting pre-trained models with minimal computational overhead. We will walk you through the setup, adaptation process, and deployment using easy-to-follow steps.

## 0. Prerequisites
Before we get started, you’ll need to install Kit CLI and download an executable from Llama.

Kit CLI installed on your system–[Installation Guide](https://kitops.ml/docs/cli/installation.html)
`finetune` executable from the `llama.cpp` project–[Download here](https://github.com/ggerganov/llama.cpp/releases/tag/b2761)

## 1. Environment setup
In step 1, we will, set up your environment

Create a project directory:
`mkdir /lora_finetuning`

Log in to the GitHub Container Registry using the Kit CLI. This will require your GitHub username and password. Alternatively, you can use any other registry that is compatible with OCI artifacts:

`kit login ghcr.io`

## 2. Create your Kitfile and refer to your base model
In your project directory `(/lora_finetuning)`, create a Kitfile with the following contents:
```
manifestVersion: "1.0"
package:
name: llama3 fine-tuned
version: 3.0.0
authors: ["Jozu AI"]
model:
name: llama3-8B-instruct-q4_0
path: ghcr.io/jozu-ai/llama3:8B-instruct-q4_0
description: Llama 3 8B instruct model
license: Apache 2.0
```
**Note:** Notice how the path refers to another ModelKit? In this case `ghcr.io/jozu-ai/llama3:8B-instruct-q4_0`.

Let's create our initial ModelKit by invoking.

`kit pack /lora_finetuning -t fine-tuning:untuned`

Now let's unpack our ModelKit to our work folder. Notice that Kit CLI will also unpack the model that you have referenced to your work folder so that you can work with it.

`kit unpack fine-tuning:untuned -d /lora_finetuning --overwrite`

## 3. Create your Lora Adapter
It is recommended to use the same prompt format as the base model, but any text file can be used. Ideally, provide a common string to separate each example that isn't found in your training data. e.g., `'
'`

`
 Example one text here.`
`
 Example two text here.`
Run the fine-tuning command from your project directory:
cd /lora_finetuning
```
llama.cpp\finetune.exe --model-base ./llama3-8B-instruct-q4_0.gguf --train-data ./training-data.txt --threads 8 --sample-start "
" --lora-out lora_adapter.gguf
```
##4. Update the Kitfile with your LoRA adapter
Next, we’re going to update the Kitfile to include the LoRA adapter:
```
manifestVersion: "1.0"
package:
  name: llama3 fine-tuned
  version: 3.0.0
  authors: ["Jozu AI"]
model:
  name: llama3-8B-instruct-q4_0
  path: ghcr.io/jozu-ai/llama3:8B-instruct-q4_0
  description: Llama 3 8B instruct model
  license: Apache 2.0
  parts:
    - path: ./lora-adapter.gguf
      type: lora-adapter
datasets:
  - name: fine-tune-data
    path: ./training-data.txt
```
We recommend including training data in your ModelKits as a best practice that enables further iterations of training to be easier, however, this is not required for production deployments.

## 5. Pack and ship your Adapter in a ModelKit
Package your the tuned model
`kit pack /lora_finetuning -t fine-tuning:tuned`

Tag your model for remote repository
    `kit tag fine-tuning:tuned ghcr.io/jozuuser/finetuned:latest`

Push the ModelKit to a Remote Repository.
    `kit push ghcr.io/jozuuser/finetuned:latest`

## Conclusion
Congratulations, your fine-tuned model is now packaged and ready for deployment. You've successfully updated the Kitfile, packed the model, and pushed it to a remote repository, making it accessible for implementation in various applications.

If you found this tutorial informative, we encourage you to [join the KitOps Discord server](https://discord.gg/dEyuTFaMC2) and [Star the KitOps GitHub repo to support the project](https://github.com/jozu-ai/kitops).