File size: 6,213 Bytes
58b2ffe f029fb2 58b2ffe 9fdfed7 58b2ffe 57fd89e 58b2ffe 9ca9445 58b2ffe bcc74e6 58b2ffe |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 |
---
library_name: transformers
base_model: codellama/CodeLlama-7b-Instruct-hf
license: llama2
datasets:
- semantixai/LloroV3
language:
- pt
tags:
- code
- analytics
- analise-dados
- portugues-BR
co2_eq_emissions:
emissions: 1320
source: "Lacoste, Alexandre, et al. “Quantifying the Carbon Emissions of Machine Learning.” ArXiv (Cornell University), 21 Oct. 2019, https://doi.org/10.48550/arxiv.1910.09700."
training_type: "fine-tuning"
geographical_location: "Council Bluffs, Iowa, USA."
hardware_used: "1 A100 40GB GPU"
---
**Lloro 7B**
<img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>
Lloro, developed by Semantix Research Labs , is a language Model that was trained to effectively perform Portuguese Data Analysis in Python. It is a fine-tuned version of codellama/CodeLlama-7b-Instruct-hf, that was trained on synthetic datasets. The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM.
**Model description**
Model type: A 7B parameter fine-tuned on synthetic datasets.
Language(s) (NLP): Primarily Portuguese, but the model is capable to understand English as well
Finetuned from model: codellama/CodeLlama-7b-Instruct-hf
**What is Lloro's intended use(s)?**
Lloro is built for data analysis in Portuguese contexts .
Input : Text
Output : Text (Code)
**Usage**
Using Transformers
```python
#Import required libraries
import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer
)
#Load Model
model_name = "semantixai/LloroV2"
base_model = AutoModelForCausalLM.from_pretrained(
model_name,
return_dict=True,
torch_dtype=torch.float16,
device_map="auto",
)
#Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
#Define Prompt
user_prompt = "Desenvolva um algoritmo em Python para calcular a média e a mediana dos preços de vendas por tipo de material do produto."
system = "Provide answers in Python without explanations, only the code"
prompt_template = f"[INST] <<SYS>>\\n{system}\\n<</SYS>>\\n\\n{user_prompt}[/INST]"
#Call the model
input_ids = tokenizer([prompt_template], return_tensors="pt")["input_ids"].to("cuda")
outputs = base_model.generate(
input_ids,
do_sample=True,
top_p=0.95,
max_new_tokens=1024,
temperature=0.1,
)
#Decode and retrieve Output
output_text = tokenizer.batch_decode(outputs, skip_prompt=True, skip_special_tokens=False)
display(output_text)
```
Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
)
user_prompt = "Desenvolva um algoritmo em Python para calcular a média e a mediana dos preços de vendas por tipo de material do produto."
completion = client.chat.completions.create(temperature=0.1,frequency_penalty=0.1,model="semantixai/Lloro",messages=[{"role":"system","content":"Provide answers in Python without explanations, only the code"},{"role":"user","content":user_prompt}])
```
**Params**
Training Parameters
| Params | Training Data | Examples | Tokens | LR |
|----------------------------------|-----------------------------------|---------------------------------|----------|--------|
| 7B | Pairs synthetic instructions/code | 74222 | 9 351 532| 2e-4 |
**Model Sources**
Test Dataset Repository: <https://huggingface.co/datasets/semantixai/LloroV3>
Model Dates: Lloro was trained between February 2024 and April 2024.
**Performance**
| Modelo | LLM as Judge | Code Bleu Score | Rouge-L | CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
|----------------|--------------|------------------|---------|----------------------|-----------------|-------------|-------------|
| GPT 3.5 | 94.29% | 0.3538 | 0.3756 | 0.8099 | 0.8176 | 0.8128 | 0.8164 |
| Instruct -Base | 88.77% | 0.3666 | 0.3351 | 0.8244 | 0.8025 | 0.8121 | 0.8052 |
| Instruct -FT | 97.95% | 0.5967 | 0.6717 | 0.9090 | 0.9182 | 0.9131 | 0.9171 |
**Training Infos:**
The following hyperparameters were used during training:
| Parameter | Value |
|---------------------------|--------------------------|
| learning_rate | 2e-4 |
| weight_decay | 0.0001 |
| train_batch_size | 7 |
| eval_batch_size | 7 |
| seed | 42 |
| optimizer | Adam - paged_adamw_32bit |
| lr_scheduler_type | cosine |
| lr_scheduler_warmup_ratio | 0.06 |
| num_epochs | 4.0 |
**QLoRA hyperparameters**
The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:
| Parameter | Value |
|------------------|-----------|
| lora_r | 64 |
| lora_alpha | 256 |
| lora_dropout | 0.1 |
| storage_dtype | "nf4" |
| compute_dtype | "bfloat16"|
**Experiments**
| Model | Epochs | Overfitting | Final Epochs | Training Hours | CO2 Emission (Kg) |
|-----------------------|--------|-------------|--------------|-----------------|-------------------|
| Code Llama Instruct | 1 | No | 1 | 3.01 | 0.43 |
| Code Llama Instruct | 4 | Yes | 3 | 9.25 | 1.32 |
**Framework versions**
| Library | Version |
|---------------|-----------|
| bitsandbytes | 0.40.2 |
| Datasets | 2.14.3 |
| Pytorch | 2.0.1 |
| Tokenizers | 0.14.1 |
| Transformers | 4.34.0 |
|