Commit
•
81add4c
1
Parent(s):
5c03550
Update README.md
Browse files
README.md
CHANGED
@@ -1,26 +1,28 @@
|
|
1 |
---
|
|
|
|
|
|
|
2 |
language:
|
3 |
- pt
|
4 |
-
license: llama3
|
5 |
-
library_name: transformers
|
6 |
tags:
|
7 |
- code
|
8 |
- sql
|
9 |
- finetuned
|
10 |
- portugues-BR
|
11 |
-
|
|
|
|
|
|
|
|
|
|
|
12 |
---
|
13 |
-
|
14 |
|
15 |
<img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>
|
16 |
|
17 |
-
|
18 |
Lloro SQL, developed by Semantix Research Labs, is a language Model that was trained to effectively transform Portuguese queries into SQL Code. It is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct, that was trained on GretelAI public datasets. The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM.
|
19 |
|
20 |
-
|
21 |
-
|
22 |
-
**Model description**
|
23 |
-
|
24 |
|
25 |
Model type: A 7B parameter fine-tuned on GretelAI public datasets.
|
26 |
|
@@ -28,10 +30,7 @@ Language(s) (NLP): Primarily Portuguese, but the model is capable to understand
|
|
28 |
|
29 |
Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct
|
30 |
|
31 |
-
|
32 |
-
|
33 |
-
**What is Lloro's intended use(s)?**
|
34 |
-
|
35 |
|
36 |
Lloro is built for Text2SQL in Portuguese contexts .
|
37 |
|
@@ -39,9 +38,7 @@ Input : Text
|
|
39 |
|
40 |
Output : Text (Code)
|
41 |
|
42 |
-
|
43 |
-
**Usage**
|
44 |
-
|
45 |
|
46 |
Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))
|
47 |
|
@@ -66,55 +63,69 @@ def generate_responses(instruction, client=client):
|
|
66 |
output = generate_responses(user_prompt)
|
67 |
|
68 |
```
|
69 |
-
|
70 |
|
|
|
71 |
|
72 |
-
**Params**
|
73 |
Training Parameters
|
74 |
-
| Params | Training Data
|
75 |
-
|
76 |
-
| 8B | GretelAI public datasets
|
77 |
-
|
|
|
78 |
|
79 |
-
|
80 |
|
81 |
-
|
82 |
|
83 |
-
|
84 |
|
85 |
-
|
86 |
-
| Modelo | LLM as Judge | Code Bleu Score | Rouge-L | CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
|
87 |
|----------------|--------------|-----------------|---------|----------------------|-----------------|-------------|-------------|
|
88 |
-
| Llama 3
|
89 |
-
|
|
|
|
90 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
91 |
|
92 |
-
**Training Infos:**
|
93 |
The following hyperparameters were used during training:
|
94 |
|
95 |
| Parameter | Value |
|
96 |
|---------------------------|----------------------|
|
97 |
-
| learning_rate |
|
98 |
-
| weight_decay | 0.001
|
99 |
-
| train_batch_size | 16
|
100 |
-
| eval_batch_size | 8
|
101 |
| seed | 42 |
|
102 |
-
| optimizer | Adam - adamw_8bit
|
103 |
| lr_scheduler_type | cosine |
|
104 |
-
| num_epochs |
|
|
|
|
|
105 |
|
106 |
-
**QLoRA hyperparameters**
|
107 |
The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:
|
108 |
|
109 |
| Parameter | Value |
|
110 |
|-----------------|---------|
|
111 |
-
| lora_r |
|
112 |
-
| lora_alpha |
|
113 |
| lora_dropout | 0 |
|
114 |
|
|
|
|
|
|
|
|
|
|
|
115 |
|
|
|
116 |
|
117 |
-
**Framework versions**
|
118 |
| Library | Version |
|
119 |
|---------------|-----------|
|
120 |
| accelerate | 0.21.0 |
|
@@ -127,4 +138,4 @@ The following parameters related with the Quantized Low-Rank Adaptation and Quan
|
|
127 |
| scikit-learn | 1.3.2 |
|
128 |
| Tokenizers | 0.14.1 |
|
129 |
| Transformers | 4.37.2 |
|
130 |
-
| trl | 0.4.7 |
|
|
|
1 |
---
|
2 |
+
library_name: transformers
|
3 |
+
base_model: meta-llama/Meta-Llama-3-8B-Instruct
|
4 |
+
license: llama3
|
5 |
language:
|
6 |
- pt
|
|
|
|
|
7 |
tags:
|
8 |
- code
|
9 |
- sql
|
10 |
- finetuned
|
11 |
- portugues-BR
|
12 |
+
co2_eq_emissions:
|
13 |
+
emissions: 1450
|
14 |
+
source: "Lacoste, Alexandre, et al. “Quantifying the Carbon Emissions of Machine Learning.” ArXiv (Cornell University), 21 Oct. 2019, https://doi.org/10.48550/arxiv.1910.09700."
|
15 |
+
training_type: "fine-tuning"
|
16 |
+
geographical_location: "Council Bluffs, Iowa, USA."
|
17 |
+
hardware_used: "1 A100 40GB GPU"
|
18 |
---
|
19 |
+
# Lloro SQL
|
20 |
|
21 |
<img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>
|
22 |
|
|
|
23 |
Lloro SQL, developed by Semantix Research Labs, is a language Model that was trained to effectively transform Portuguese queries into SQL Code. It is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct, that was trained on GretelAI public datasets. The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM.
|
24 |
|
25 |
+
## Model description
|
|
|
|
|
|
|
26 |
|
27 |
Model type: A 7B parameter fine-tuned on GretelAI public datasets.
|
28 |
|
|
|
30 |
|
31 |
Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct
|
32 |
|
33 |
+
## What is Lloro's intended use(s)?
|
|
|
|
|
|
|
34 |
|
35 |
Lloro is built for Text2SQL in Portuguese contexts .
|
36 |
|
|
|
38 |
|
39 |
Output : Text (Code)
|
40 |
|
41 |
+
## Usage
|
|
|
|
|
42 |
|
43 |
Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))
|
44 |
|
|
|
63 |
output = generate_responses(user_prompt)
|
64 |
|
65 |
```
|
|
|
66 |
|
67 |
+
## Params
|
68 |
|
|
|
69 |
Training Parameters
|
70 |
+
| Params | Training Data | Examples | Tokens | LR |
|
71 |
+
|----------------------------------|-------------------------------------------|---------------------------------|------------|--------|
|
72 |
+
| 8B | GretelAI public datasets + Synthetic Data | 102970 | 18.654.222 | 2e-4 |
|
73 |
+
|
74 |
+
## Model Sources
|
75 |
|
76 |
+
GretelAI: <https://huggingface.co/datasets/gretelai/synthetic_text_to_sql>
|
77 |
|
78 |
+
## Performance
|
79 |
|
80 |
+
### Test Dataset
|
81 |
|
82 |
+
| Model | LLM as Judge | Code Bleu Score | Rouge-L | CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
|
|
|
83 |
|----------------|--------------|-----------------|---------|----------------------|-----------------|-------------|-------------|
|
84 |
+
| Llama 3 8B | 65.48% | 0.4583 | 0.6361 | 0.8815 | 0.8871 | 0.8835 | 0.8862 |
|
85 |
+
| Lloro - SQL | 71.33% | 0.6512 | 0.7965 | 0.9458 | 0.9469 | 0.9459 | 0.9466 |
|
86 |
+
| GPT - 3.5 Turbo| 67.52% | 0.6232 | 0.9967 | 0.9151 | 0.9152 | 0.9142 | 0.9175 |
|
87 |
|
88 |
+
### Database Benchmark
|
89 |
+
|
90 |
+
| Model | Score |
|
91 |
+
|----------------|--------------|
|
92 |
+
| Llama 3 - Base | 35.55% |
|
93 |
+
| Lloro - SQL | 49.48% |
|
94 |
+
| GPT - 3.5 Turbo| 46.15% |
|
95 |
+
|
96 |
+
## Training Infos
|
97 |
|
|
|
98 |
The following hyperparameters were used during training:
|
99 |
|
100 |
| Parameter | Value |
|
101 |
|---------------------------|----------------------|
|
102 |
+
| learning_rate | 2e-4 |
|
103 |
+
| weight_decay | 0.001 |
|
104 |
+
| train_batch_size | 16 |
|
105 |
+
| eval_batch_size | 8 |
|
106 |
| seed | 42 |
|
107 |
+
| optimizer | Adam - adamw_8bit |
|
108 |
| lr_scheduler_type | cosine |
|
109 |
+
| num_epochs | 4.0 |
|
110 |
+
|
111 |
+
## QLoRA hyperparameters
|
112 |
|
|
|
113 |
The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:
|
114 |
|
115 |
| Parameter | Value |
|
116 |
|-----------------|---------|
|
117 |
+
| lora_r | 64 |
|
118 |
+
| lora_alpha | 128 |
|
119 |
| lora_dropout | 0 |
|
120 |
|
121 |
+
## Experiments
|
122 |
+
|
123 |
+
| Model | Epochs | Overfitting | Final Epochs | Training Hours | CO2 Emission (Kg) |
|
124 |
+
|-----------------------|--------|-------------|--------------|-----------------|-------------------|
|
125 |
+
| Llama 3 8B Instruct | 5 | Yes | 4 | 10.16 | 1.45 |
|
126 |
|
127 |
+
## Framework versions
|
128 |
|
|
|
129 |
| Library | Version |
|
130 |
|---------------|-----------|
|
131 |
| accelerate | 0.21.0 |
|
|
|
138 |
| scikit-learn | 1.3.2 |
|
139 |
| Tokenizers | 0.14.1 |
|
140 |
| Transformers | 4.37.2 |
|
141 |
+
| trl | 0.4.7 |
|