| # RewardModel (Portuguese-BR) | |
| The `RewardModel` is a modified BERT model that can be used to score the quality of completion to a given prompt. It is based on the [BERT](https://huggingface.co/bert-base-cased), modified to act as a regression model. | |
| The `RewardModel` allows the specification of an $\alpha$ parameter, which is a multiplier to the reward score. This multiplier is set to 1 during training (since our reward values are bounded between -1 and 1) but can be changed at inference to allow for rewards with higher bounds. | |
| The model was trained with a dataset composed of `prompt`, `completions`, and annotated `rewards`. | |
| > Note: These prompt + completions are samples of intruction datasets created via the [Self-Instruct](https://github.com/yizhongw/self-instruct) framework. | |
| ## Usage | |
| Here's an example of how to use the `RewardModelPT` to score the quality of a response to a given prompt: | |
| ```python | |
| from transformers import AutoTokenizer,AutoConfig, AutoModel | |
| import torch | |
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu") | |
| config = AutoConfig.from_pretrained('nicholasKluge/RewardModel', trust_remote_code=True, revision='main') | |
| tokenizer = AutoTokenizer.from_pretrained('nicholasKluge/RewardModel', trust_remote_code=True, config=config, revision='main') | |
| rewardModel = AutoModel.from_pretrained('nicholasKluge/RewardModel', trust_remote_code=True, config=config, revision='main') | |
| rewardModel.to(device) | |
| rewardModel.eval() | |
| # Define the question and response | |
| question = "What is the capital of France?" | |
| response1 = "Paris, France's capital, is a major European city and a global center for art, fashion, gastronomy and culture." | |
| response2 = "Google it pal." | |
| # Tokenize the question and response | |
| tokens = tokenizer(question, response1, | |
| return_token_type_ids=False, | |
| return_tensors="pt", | |
| return_attention_mask=True) | |
| tokens.to(device) | |
| # Score the response | |
| score = model(**tokens, alpha=10).item() | |
| print(f"Question: {question} \n") | |
| print(f"Response 1: {response1} Score: {score:.3f}") | |
| tokens = tokenizer(question, response2, | |
| return_token_type_ids=False, | |
| return_tensors="pt", | |
| return_attention_mask=True) | |
| tokens.to(device) | |
| score = model(**tokens, alpha=10).item() | |
| print(f"Response 2: {response2} Score: {score:.3f}") | |
| ``` | |
| This will output the following: | |
| ```markdown | |
| >>> Question: What is the capital of France? | |
| >>>Response: Paris, France's capital, is a major European city and a global center for art, fashion, gastronomy and culture. Score: 3.183 | |
| >>>Response: Google it pal. Score: -5.781 | |
| ``` | |
| ## License | |
| The `RewardModelPT` is licensed under the Apache License, Version 2.0. See the [LICENSE](LICENSE) file for more details. | |