LloroV3 / README.md
rafaelgeraldini's picture
Update README.md
57fd89e verified
metadata
library_name: transformers
base_model: codellama/CodeLlama-7b-Instruct-hf
license: llama2
datasets:
  - semantixai/LloroV3
language:
  - pt
tags:
  - code
  - analytics
  - analise-dados
  - portugues-BR
co2_eq_emissions:
  emissions: 1320
  source: >-
    Lacoste, Alexandre, et al. “Quantifying the Carbon Emissions of Machine
    Learning.” ArXiv (Cornell University), 21 Oct. 2019,
    https://doi.org/10.48550/arxiv.1910.09700.
  training_type: fine-tuning
  geographical_location: Council Bluffs, Iowa, USA.
  hardware_used: 1 A100 40GB GPU

Lloro 7B

Lloro-7b Logo

Lloro, developed by Semantix Research Labs , is a language Model that was trained to effectively perform Portuguese Data Analysis in Python. It is a fine-tuned version of codellama/CodeLlama-7b-Instruct-hf, that was trained on synthetic datasets. The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM.

Model description

Model type: A 7B parameter fine-tuned on synthetic datasets.

Language(s) (NLP): Primarily Portuguese, but the model is capable to understand English as well

Finetuned from model: codellama/CodeLlama-7b-Instruct-hf

What is Lloro's intended use(s)?

Lloro is built for data analysis in Portuguese contexts .

Input : Text

Output : Text (Code)

Usage

Using Transformers

#Import required libraries
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer
)

#Load Model
model_name = "semantixai/LloroV2"
base_model = AutoModelForCausalLM.from_pretrained(
        model_name,
        return_dict=True,
        torch_dtype=torch.float16,
        device_map="auto",
    )

#Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)


#Define Prompt
user_prompt = "Desenvolva um algoritmo em Python para calcular a média e a mediana dos preços de vendas por tipo de material do produto."
system = "Provide answers in Python without explanations, only the code"
prompt_template = f"[INST] <<SYS>>\\n{system}\\n<</SYS>>\\n\\n{user_prompt}[/INST]"

#Call the model
input_ids = tokenizer([prompt_template], return_tensors="pt")["input_ids"].to("cuda")

            
outputs = base_model.generate(
    input_ids,
    do_sample=True,
    top_p=0.95,
    max_new_tokens=1024,
    temperature=0.1,
    )

#Decode and retrieve Output
output_text = tokenizer.batch_decode(outputs, skip_prompt=True, skip_special_tokens=False)
display(output_text)

Using an OpenAI compatible inference server (like vLLM)

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)
user_prompt = "Desenvolva um algoritmo em Python para calcular a média e a mediana dos preços de vendas por tipo de material do produto."
completion = client.chat.completions.create(temperature=0.1,frequency_penalty=0.1,model="semantixai/Lloro",messages=[{"role":"system","content":"Provide answers in Python without explanations, only the code"},{"role":"user","content":user_prompt}])

Params Training Parameters

Params Training Data Examples Tokens LR
7B Pairs synthetic instructions/code 74222 9 351 532 2e-4

Model Sources

Test Dataset Repository: https://huggingface.co/datasets/semantixai/LloroV3

Model Dates: Lloro was trained between February 2024 and April 2024.

Performance

Modelo LLM as Judge Code Bleu Score Rouge-L CodeBert- Precision CodeBert-Recall CodeBert-F1 CodeBert-F3
GPT 3.5 94.29% 0.3538 0.3756 0.8099 0.8176 0.8128 0.8164
Instruct -Base 88.77% 0.3666 0.3351 0.8244 0.8025 0.8121 0.8052
Instruct -FT 97.95% 0.5967 0.6717 0.9090 0.9182 0.9131 0.9171

Training Infos: The following hyperparameters were used during training:

Parameter Value
learning_rate 2e-4
weight_decay 0.0001
train_batch_size 7
eval_batch_size 7
seed 42
optimizer Adam - paged_adamw_32bit
lr_scheduler_type cosine
lr_scheduler_warmup_ratio 0.06
num_epochs 4.0

QLoRA hyperparameters The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:

Parameter Value
lora_r 64
lora_alpha 256
lora_dropout 0.1
storage_dtype "nf4"
compute_dtype "bfloat16"

Experiments

Model Epochs Overfitting Final Epochs Training Hours CO2 Emission (Kg)
Code Llama Instruct 1 No 1 3.01 0.43
Code Llama Instruct 4 Yes 3 9.25 1.32

Framework versions

Library Version
bitsandbytes 0.40.2
Datasets 2.14.3
Pytorch 2.0.1
Tokenizers 0.14.1
Transformers 4.34.0