README.md

---
library_name: transformers
license: mit
language:
- en
---

# Rolema 7B

Rolema 7B is a large language model that works effectively under a 4-bit quantization process.
Rolema 7B is based on the backbone of the Gemma-7B model by Google.

### Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

- **Developed by:** Min Si Thu
- **Model type:** Text Generation Large Language Model
- **Language(s) (NLP):** English
- **License:** MIT

### How to use

Installing Libraries

```bash
%%capture 
%pip install -U bitsandbytes 
%pip install -U transformers 
%pip install -U peft 
%pip install -U accelerate 
%pip install -U trl
%pip install -U datasets
```

Code Implementation

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel, PeftConfig

base_model = "google/gemma-7b-it"
adapter_model = "jojo-ai-mst/rolema-7b-it"

# Load base model(Gemma 7B-it)
bnbConfig = BitsAndBytesConfig(
    load_in_4bit = True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(base_model,quantization_config=bnbConfig,) # device_map="auto" autosplit for cuda
model = PeftModel.from_pretrained(model, adapter_model)
tokenizer = AutoTokenizer.from_pretrained(base_model)

model = model.to("cuda")

inputs = tokenizer("How to learn programming", return_tensors="pt")

inputs = inputs.to("cuda")

outputs = model.generate(input_ids=inputs["input_ids"], max_new_tokens=1000)
print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0])
```