PreferED: Preference Evaluation DeBERTa Model
Model Description
PreferED is a 400M parameter preference evaluation model based on the DeBERTa architecture, designed for evaluating LLM apps.
The model is trained to take in context and text data and output a logit score, which can be used to compare
different text generations on evaluative aspects such as hallucinations, quality etc. The context
variable
can be used to provide evaluation criteria in addition to any relevant retreived context. The gen_text
variable
provides the actual text that is being evaluated.
- Model name: PreferED
- Model type: DeBERTa
- Training data: This model was trained on Anthropic HH/RLHF using a Deberta-v3-large base model
- Evaluation data: Achieves 69.7% accuracy on the Anthropic hh-rlhf test split.
Usage
Loading the Model
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("samagra14wefi/PreferED")
model = AutoModelForSequenceClassification.from_pretrained("samagra14wefi/PreferED")
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = model.to(device)
Measuring hallucinations
Use the context
variable to give the retreived context.
def calc_score(context, gen_text):
with torch.no_grad():
inputs = tokenizer(context, gen_text, return_tensors='pt')
logits = model(**inputs).logits
score = logits[0].cpu().detach()
return score
context_string = '''India won the world cup in 1983 and 2011. Australia won the world cup five times.
West Indies have won the world cup twice. Sri Lanka, Pakistan and England have won the world cup once.
Evaluate if the facts below are consistent with the statement.'''
response_string_wrong = '''India has won the world cup most number of times.'''
response_string_correct = '''Australia has won the world cup most number of times.'''
score_wrong = calc_score(context_string, response_string_wrong)
score_correct = calc_score(context_string, response_string_correct)
print(score_correct > score_wrong)
Evaluating Response relevance
inquiry = "What is your return policy?"
response_good = "Our return policy lasts 30 days. If 30 days have gone by since your purchase,
unfortunately, we can’t offer you a refund or exchange."
response_bad = "We offer a variety of fresh produce including apples, oranges, and bananas."
score_good = calc_score(inquiry, response_good)
score_bad = calc_score(inquiry, response_bad)
print(score_good > score_bad)
Evaluating Content Appropriateness
context = "Discussing the political scenario in Country X."
response_clean = "The political scenario in Country X is quite dynamic with multiple parties vying for power."
response_offensive = "The politicians in Country X are all corrupt and stupid."
score_clean = calc_score(context, response_clean)
score_offensive = calc_score(context, response_offensive)
print(score_clean > score_offensive)
Comparing Different Language Models
context = "Explain the process of photosynthesis."
response_gpt3 = "Photosynthesis is the process by which green plants and some other organisms use sunlight to synthesize foods with the help of chlorophyll pigments."
response_bert = "Photosynthesis is a method that converts carbon dioxide into organic compounds, especially sugars, in the presence of sunlight."
score_gpt3 = calc_score(context, response_gpt3)
score_bert = calc_score(context, response_bert)
print(score_gpt3 > score_bert)
Finetuning on your production data
The PreferED model is relatively lightweight compared to some other large language models, making it a good candidate for fine-tuning on specific tasks or datasets. Fine-tuning the model on your own production data can lead to better performance as it helps the model to better understand the nuances and context specific to your application.
Preparing the Training Dataset
For fine-tuning the PreferED model on production evaluation tasks, it's crucial to structure your data correctly. The dataset should be formatted such that each example contains a shared context that provides the evaluation criteria, a text input, and a binary label indicating the preference or correctness of the text input in relation to the evaluation criteria.
Here's an example of how your data might look:
context,text,label
"Evaluate the accuracy of the statement based on historical facts.","The sun revolves around the Earth.",0
"Evaluate the accuracy of the statement based on historical facts.","The Earth revolves around the sun.",1
You can then load this data into a Dataset
object using a library such as Hugging Face's datasets
.
Finetuning Example
from transformers import DebertaTokenizer, DebertaForSequenceClassification, Trainer, TrainingArguments
import torch
tokenizer = DebertaTokenizer.from_pretrained("samagra14wefi/PreferED")
model = DebertaForSequenceClassification.from_pretrained("samagra14wefi/PreferED")
# Define the training arguments
training_args = TrainingArguments(
per_device_train_batch_size=8,
num_train_epochs=3,
logging_dir='./logs',
)
# Create the Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset, # provide your training dataset
eval_dataset=eval_dataset, # provide your evaluation dataset
)
# Train the model
trainer.train()
Loss Function Consideration
Anthropic recommends using the loss function LPM = log(1 + e^(rbad - rgood)) for preference models. However, this PreferED model was trained using binary cross-entropy loss, and therefore changing the loss functions might increase the training time to converge. For more details on preference models and loss functions, you may refer to the paper by Askell et al., 2021: A General Language Assistant as a Laboratory for Alignment.
- Downloads last month
- 22