README.md

---
library_name: transformers
base_model: meta-llama/Meta-Llama-3-8B-Instruct
license: llama3
language:
- pt
tags:
- code
- sql
- finetuned
- portugues-BR
co2_eq_emissions:
  emissions: 1450
  source: "Lacoste, Alexandre, et al. “Quantifying the Carbon Emissions of Machine Learning.” ArXiv (Cornell University), 21 Oct. 2019, https://doi.org/10.48550/arxiv.1910.09700."
  training_type: "fine-tuning"
  geographical_location: "Council Bluffs, Iowa, USA."
  hardware_used: "1 A100 40GB GPU"
---
# Lloro SQL

<img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>

Lloro SQL, developed by Semantix Research Labs, is a language Model that was trained to effectively transform Portuguese queries into SQL Code. It is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct,  that was trained on GretelAI public datasets.  The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM.

## Model description

Model type: A 7B parameter  fine-tuned on GretelAI public datasets.

Language(s) (NLP): Primarily Portuguese, but the model is capable to understand English as well

Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct

## What is Lloro's intended use(s)?

Lloro is built for Text2SQL in Portuguese contexts .

Input : Text

Output : Text (Code)

## Usage

Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))

```python
from openai import OpenAI
client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)
def generate_responses(instruction, client=client):
    
    chat_response = client.chat.completions.create(
    model=<model>,
    messages=[
        {"role": "system", "content": "Você escreve a instrução SQL que responde às perguntas feitas. Você NÃO FORNECE NENHUM COMENTÁRIO OU EXPLICAÇÃO sobre o que o código faz, apenas a instrução SQL terminando em ponto e vírgula. Você utiliza todos os comandos disponíveis na especificação SQL, como: [SELECT, WHERE, ORDER, LIMIT, CAST, AS, JOIN]."},
        {"role": "user", "content": instruction},
    ]
)
    
    return chat_response.choices[0].message.content

output = generate_responses(user_prompt)

```

## Params

Training Parameters
| Params                           | Training Data                             | Examples                        | Tokens     | LR     |
|----------------------------------|-------------------------------------------|---------------------------------|------------|--------|
| 8B                               | GretelAI public datasets + Synthetic Data | 102970                          | 18.654.222 | 2e-4   |

## Model Sources

GretelAI: <https://huggingface.co/datasets/gretelai/synthetic_text_to_sql>

## Performance

### Test Dataset

| Model          | LLM as Judge | Code Bleu Score | Rouge-L |  CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
|----------------|--------------|-----------------|---------|----------------------|-----------------|-------------|-------------|
| Llama 3 8B     | 65.48%       | 0.4583          | 0.6361  |  0.8815              | 0.8871          | 0.8835      | 0.8862      |
| Lloro - SQL    | 71.33%       | 0.6512          | 0.7965  |  0.9458              | 0.9469          | 0.9459      | 0.9466      |
| GPT - 3.5 Turbo| 67.52%       | 0.6232          | 0.9967  |  0.9151              | 0.9152          | 0.9142      | 0.9175      |

### Database Benchmark

| Model          | Score        |
|----------------|--------------|
| Llama 3 - Base | 35.55%       |
| Lloro - SQL    | 49.48%       |
| GPT - 3.5 Turbo| 46.15%       |

### Translated BIRD Benchmark - https://bird-bench.github.io/

| Model          | Score        |
|----------------|--------------|
| Llama 3 - Base | 33.87%       |
| Lloro - SQL    | 47.14%       |
| GPT - 3.5 Turbo| 42.14%       |

## Training Infos

The following hyperparameters were used during training:

| Parameter                 | Value                |
|---------------------------|----------------------|
| learning_rate             | 2e-4                 |
| weight_decay              | 0.001                |
| train_batch_size          | 16                   |
| eval_batch_size           | 8                    |
| seed                      | 42                   |
| optimizer                 | Adam - adamw_8bit    |
| lr_scheduler_type         | cosine               |
| num_epochs                | 4.0                  |

## QLoRA hyperparameters

The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:

| Parameter       | Value   |
|-----------------|---------|
| lora_r          | 64      |
| lora_alpha      | 128     |
| lora_dropout    | 0       |

## Experiments

| Model                 | Epochs | Overfitting | Final Epochs | Training Hours  | CO2 Emission (Kg) |
|-----------------------|--------|-------------|--------------|-----------------|-------------------|
| Llama 3 8B Instruct   | 5      | Yes         | 4            | 10.16           | 1.45              |

## Framework versions

| Library       | Version   |
|---------------|-----------|
| accelerate    | 0.21.0    |
| bitsandbytes  | 0.42.0    |
| Datasets      | 2.14.3    |
| peft          | 0.4.0     |
| Pytorch       | 2.0.1     |
| safetensors   | 0.4.1     |
| scikit-image  | 0.22.0    |
| scikit-learn  | 1.3.2     |
| Tokenizers    | 0.14.1    |
| Transformers  | 4.37.2    |
| trl           | 0.4.7     |