metadata

license: llama3.1
language:
  - en
tags:
  - rag
  - brag
  - chatrag
  - chatqa

BRAG-Llama-3.1-8b-v0.1 Model Card

Model Description

BRAG-Llama-3.1-8b-v0.1 is part of the BRAG series of SLMs (Small Language Models) specifically trained for RAG (Retrieval-Augmented Generation) tasks, making it suitable for applications that require processing and generating responses based on contexts, chats with tables and text, such as:

1.	RAG with tables and text.
2.	RAG with conversational chat.

Authors: Pratik Bhavasar, Ravi Theja

Key Features

Model Size: 8 billion parameters
Context Length: Supports up to 128k tokens
Language: Trained and evaluated for English, but the base model has multi-lingual capabilities

Other Resources

BRAG-v0.1 Model Collection Blogpost

Performance

Model Type	Model Name	Model Size	Context Length	ChatRAG-Bench (all)
LLM	Command-R-Plus	--	128k	50.93
LLM	GPT-4-Turbo-2024-04-09	--	128k	54.03
SLM	ChatQA-1.5-8B	8b	8k	55.17
BRAG SLM	BRAG-Qwen2-7b-v0.1	7b	128k	53.23
BRAG SLM	BRAG-Llama-3.1-8b-v0.1	8b	128k	52.29
BRAG SLM	BRAG-Llama-3-8b-v0.1	8b	8k	51.70
BRAG Ultra SLM	BRAG-Qwen2-1.5b-v0.1	1.5b	32k	46.43

Usage

Prompt Format

Below is the message prompt format required for using the model.

messages = [
    {"role": "system", "content": "You are a helpful assistant to answer the queries based on the given context."},
    {"role": "user", "content": """Context: <CONTEXT INFORMATION> \n\n <USER QUERY>"""},
]

Running with the `pipeline` API

import transformers
import torch

model_id = "maximalists/BRAG-Llama-3.1-8b-v0.1"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant to answer the queries based on the given context."},
    {"role": "user", "content": """Context:\nArchitecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n\nTo whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?"""},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)

print(outputs[0]["generated_text"][-1])

Running the model on a single / multi GPU

# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "maximalists/BRAG-Llama-3.1-8b-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant to answer the queries based on the given context."},
    {"role": "user", "content": """Context:\nArchitecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n\nTo whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?"""},
]

input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")

outputs = model.generate(**input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))

Limitations

The model is specifically trained for short contexts and may not perform well with longer ones. It has been fine-tuned on an English dataset. To avoid underperformance and the potential for hallucinations, please use the system prompt mentioned above.

Citation

To cite this model, please use the following:

@misc{BRAG-Llama-3-8b-v0.1,
  title = {BRAG-Llama-3-8b-v0.1},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/maximalists/BRAG-Llama-3-8b-v0.1},
author = {Pratik Bhavsar and Ravi Theja}
}

Additional Information

For more details on the BRAG series and updates, please refer to the official blogpost.