Create README.md

README.md

@@ -0,0 +1,164 @@
+---
+library_name: transformers
+base_model: codellama/CodeLlama-7b-Instruct-hf
+license: llama2
+datasets:
+- semantixai/LloroV3
+language:
+- pt
+tags:
+- code
+- analytics
+- analise-dados
+- portugues-BR
+
+co2_eq_emissions:
+  emissions: 1320
+  source: "Lacoste, Alexandre, et al. “Quantifying the Carbon Emissions of Machine Learning.” ArXiv (Cornell University), 21 Oct. 2019, https://doi.org/10.48550/arxiv.1910.09700."
+  training_type: "fine-tuning"
+  geographical_location: "Council Bluffs, Iowa, USA."
+  hardware_used: "1 A100 40GB GPU"
+---
+
+**Lloro 7B**
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>
+
+Lloro, developed by Semantix Research Labs , is a language Model that was trained  to effectively perform Portuguese Data Analysis in Python. It is a fine-tuned version of codellama/CodeLlama-7b-Instruct-hf,  that was trained on synthetic datasets .  The fine-tuning process was performed using the QLORA metodology on a GPU V100 with 16 GB of RAM.
+
+**Model description**
+
+Model type: A 7B parameter  fine-tuned on synthetic datasets.
+
+Language(s) (NLP): Primarily Portuguese, but the model is capable to understand English as well
+
+Finetuned from model: codellama/CodeLlama-7b-Instruct-hf
+
+**What is Lloro's intended use(s)?**
+
+Lloro is built for data analysis in Portuguese contexts .
+
+Input : Text
+
+Output : Text (Code)
+
+**Usage**
+
+Using Transformers
+
+```python
+#Import required libraries
+import torch
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer
+)
+
+#Load Model
+model_name = "semantixai/LloroV2"
+base_model = AutoModelForCausalLM.from_pretrained(
+        model_name,
+        return_dict=True,
+        torch_dtype=torch.float16,
+        device_map="auto",
+    )
+
+#Load Tokenizer
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+
+
+#Define Prompt
+user_prompt = "Desenvolva um algoritmo em Python para calcular a média e a mediana dos preços de vendas por tipo de material do produto."
+system = "Provide answers in Python without explanations, only the code"
+prompt_template = f"[INST] <<SYS>>\\n{system}\\n<</SYS>>\\n\\n{user_prompt}[/INST]"
+
+#Call the model
+input_ids = tokenizer([prompt_template], return_tensors="pt")["input_ids"].to("cuda")
+
+            
+outputs = base_model.generate(
+    input_ids,
+    do_sample=True,
+    top_p=0.95,
+    max_new_tokens=1024,
+    temperature=0.1,
+    )
+
+#Decode and retrieve Output
+output_text = tokenizer.batch_decode(outputs, skip_prompt=True, skip_special_tokens=False)
+display(output_text)
+```
+
+Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+)
+user_prompt = "Desenvolva um algoritmo em Python para calcular a média e a mediana dos preços de vendas por tipo de material do produto."
+completion = client.chat.completions.create(temperature=0.1,frequency_penalty=0.1,model="semantixai/Lloro",messages=[{"role":"system","content":"Provide answers in Python without explanations, only the code"},{"role":"user","content":user_prompt}])
+```
+
+**Params**
+Training Parameters
+| Params                           | Training Data                     | Examples                        | Tokens   | LR     |
+|----------------------------------|-----------------------------------|---------------------------------|----------|--------|
+| 7B                               | Pairs synthetic instructions/code | 74222                           | 3 031 188| 2e-4   |
+
+**Model Sources**
+
+Test Dataset Repository: <https://huggingface.co/datasets/semantixai/LloroV3>
+
+Model Dates Lloro was trained between February 2024 and April 2024.
+
+**Performance**
+ | Modelo         | LLM as Judge | Code Bleu Score | Rouge-L |  CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
+|----------------|--------------|------------------|---------|----------------------|-----------------|-------------|-------------|
+| GPT 3.5        | 91.22%       | 0.2745           | 0.2189  | 0.7502               | 0.7146          | 0.7303      | 0.7175      |
+| Instruct -Base | 88.77%       | 0.3666           | 0.3351  | 0.8244               | 0.8025          | 0.8121      | 0.8052      |
+| Instruct -FT   | 94.06%       | 0.5584           | 0.6209  | 0.8943               | 0.9033          | 0.8979      | 0.9021      |
+
+**Training Infos:**
+The following hyperparameters were used during training:
+
+| Parameter                 | Value                    |
+|---------------------------|--------------------------|
+| learning_rate             | 2e-4                     |
+| weight_decay              | 0.0001                   |
+| train_batch_size          | 7                        |
+| eval_batch_size           | 7                        |
+| seed                      | 42                       |
+| optimizer                 | Adam - paged_adamw_32bit |
+| lr_scheduler_type         | cosine                   |
+| lr_scheduler_warmup_ratio | 0.06                     |
+| num_epochs                | 4.0                      |
+
+**QLoRA hyperparameters**
+The following parameters related with the Quantized Low-Rank Adaptation  and Quantization were used during training:
+
+| Parameter        | Value     |
+|------------------|-----------|
+| lora_r           | 64        |
+| lora_alpha       | 256       |
+| lora_dropout     | 0.1       |
+| storage_dtype    | "nf4"     |
+| compute_dtype    | "bfloat16"|
+
+**Experiments**
+| Model                 | Epochs | Overfitting | Final Epochs | Training Hours  | CO2 Emission (Kg) |
+|-----------------------|--------|-------------|--------------|-----------------|-------------------|
+| Code Llama Instruct   | 1      | No          | 1            | 3.01           | 0.43              |
+| Code Llama Instruct   | 4      | Yes         | 3            | 9.25           | 1.32              |
+
+**Framework versions**
+
+| Library       | Version   |
+|---------------|-----------|
+| bitsandbytes  | 0.40.2    |
+| Datasets      | 2.14.3    |
+| Pytorch       | 2.0.1     |
+| Tokenizers    | 0.14.1    |
+| Transformers  | 4.34.0    |