---
tags:
- text-generation
- pytorch
inference: false
license: llama2
language:
- pt
pipeline_tag: text-generation
library_name: transformers
datasets:
- dominguesm/CC-MAIN-2023-23
---


<p align="center">
  <img width="250" alt="Camarim Logo" src="https://raw.githubusercontent.com/DominguesM/Canarim-Instruct-PTBR/main/assets/canarim.png">
</p>

<hr>

# Canarim-7B

Canarim-7B is a Portuguese large language model developed by [Maicon Domingues](https://nlp.rocks).

## Model description

The model was pretrained on 16 billion tokens from the Portuguese subset of [CommonCrawl 2023-23](https://huggingface.co/datasets/dominguesm/CC-MAIN-2023-23), starting with the weights of LLaMA2-7B. The pretraining data has cutoff of mid-2023.

## Key Features

-   **Language:** Specialized in understanding and generating Portuguese text, making it ideal for applications targeting Portuguese-speaking audiences.
-   **Architecture:** Inherits the robust architecture from LLaMA2-7B, ensuring efficient performance and accurate results.
-   **Diverse Dataset:** The pretraining dataset includes a wide range of topics and writing styles, enhancing the model's ability to understand various contexts and nuances in Portuguese.

## Applications

Canarim-7B, was trained solely on a language modeling objective and has not been fine-tuned for instruction following. Therefore, it is more suited for few-shot tasks rather than zero-shot tasks. This means the model tends to perform better when provided with a few examples of the desired outcome during use. Here are some practical applications:

-   **Natural Language Understanding (NLU):** Efficient in tasks such as sentiment analysis, topic classification, and entity recognition in Portuguese text, especially when relevant examples are provided.
-   **Natural Language Generation (NLG):** Capable of generating coherent and contextually relevant text, useful for content creation, chatbots, and more, with improved results when provided examples of the desired style or format.
-   **Language Translation:** Suitable for high-quality translation between Portuguese and other languages, especially when examples of desired translations are included during model training or fine-tuning.

### Tips for Efficient Use

-   **Few-shot Learning:** When using Canarim-7B for specific tasks, it is beneficial to provide a few relevant examples. This helps the model better understand the context and purpose of the task.
-   **Contextualization:** Including additional context in the input can significantly improve the quality of the model’s predictions and text generation.

---

## Getting Started

To start using Canarim-7B with the Transformers library, first install the library if you haven't already:

```bash
pip install transformers
```

You can then load the model using the Transformers library. Here's a simple example of how to use the model for text generation using the `pipeline` function:

```python
from transformers import AutoTokenizer, pipeline
import torch

model_id = "dominguesm/canarim-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

prompt = make_prompt(question)
sequences = pipe(
   prompt,
   do_sample=True,
   num_return_sequences=1,
   eos_token_id=tokenizer.eos_token_id,
   max_length=2048,
   temperature=0.9,
   top_p=0.6,
   repetition_penalty=1.15
)
```

This code snippet demonstrates how to generate text with Canarim-7B. You can customize the input text and adjust parameters like `max_length` according to your requirements.

## License

Canarim-7B is released under the [LLAMA 2 COMMUNITY LICENSE AGREEMENT)](https://ai.meta.com/llama/license/).