File size: 6,793 Bytes
6c08c03 48b83f6 6c08c03 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 |
---
license: apache-2.0
base_model: mistralai/Mistral-7B-Instruct-v0.2
tags:
- freeze
- generated_from_trainer
model-index:
- name: rubra-11b-h
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# rubra-11b-h
## Model description
This is a mistral based model trained on a lot of code and conversational data. The goal of this model was to retain the abilities of [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) while making it better at reasoning.
* 32k context window
* Rope-theta = 1e6
Fine tuning instruct models has proven to be a challenge when the goal is to create a model that is not meant for a particular task. Specifically, fine tuning an instruct model often leads to destructive forgetting in the resulting model. This is why most fine tunes select a base (non-instruct) language model. The rubra-v0.1 series models aim to add desired traits like multi-turn chat capabilites while retaining knowledge found in instruct models
## Chat format
Your prompt should be surrounded by `[INST]` and `[/INST]` tokens. The very first instruction should begin with a begin of sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.
E.g.
```
text = "<s>[INST] What is your favourite condiment? [/INST]"
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> "
"[INST] Do you have mayonnaise recipes? [/INST]"
```
This format is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating) via the `apply_chat_template()` method:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto
model = AutoModelForCausalLM.from_pretrained("sanjay920/rubra-11b-h")
tokenizer = AutoTokenizer.from_pretrained("sanjay920/rubra-11b-h")
messages = [
{"role": "user", "content": "What is your favourite condiment?"},
{"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
{"role": "user", "content": "Do you have mayonnaise recipes?"}
]
encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
model_inputs = encodeds.to(device)
model.to(device)
generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
```
### GGUF
GGUF quants are available at [sanjay920/rubra-11b-h-GGUF](https://huggingface.co/sanjay920/rubra-11b-h-GGUF)
Create a new discussion for other quant variant requests!
#### LM Studio
Set "Chat Preset" to `Mistral Instruct`
## Training procedure
* Block expansion and further training
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 12
- total_train_batch_size: 12
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 3.0
- mixed_precision_training: Native AMP
### Training results
```
{
"epoch": 3.0,
"train_loss": 0.5844329195557438,
"train_runtime": 307371.8718,
"train_samples_per_second": 0.246,
"train_steps_per_second": 0.021
}
```
### Framework versions
- Transformers 4.38.2
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2
## Evaluation Results
```
########## Average ##########
score
model
gpt-4-0613 9.020312
gpt-3.5-turbo-0125 7.970219
rubra-11b-h 7.800000
Mistral-7B-Instruct-v0.2 7.793750
Mistral-7B-Instruct-v0.1 6.721875
Llama-2-13b-chat 6.650000
```
## Intended uses & limitations
This model is still under development. It hasn't been aligned for safety, so it may produce biased generations. It appears to produce thoughtful answers compared to other small LLMs. See below:
## Examples
### Needle in a haystack test:
The input text can be found in [needle-in-a-haystack.txt](./needle-in-a-haystack.txt)
* 14,334 input tokens
* "Random number is 23." inserted into the middle of the long document text. Asked the LLM what the random number is.
**rubra-11b-h**:
```
The random number is 23.
```
* Correct
**mistral-7b-instruct-v0.2**:
```
The random number generated from this text is 23. However, it's important to note that the method used to generate this number from the text is not a secure or reliable one and should not be used for any purpose where security is a concern. This was just a fun exercise to extract a number from the text.
As for the content of the text, it discusses the idea of creating a technology hub or "silicon valley" by attracting the right people - specifically, rich people and nerds. The author argues that these two groups are the limiting reagents in the reaction that produces startups, as they are the only ones present when startups get started. The text also touches on the idea that universities can act as magnets for these people and that personality and youth are important factors in creating a successful technology hub.
```
* Correct
**hermes-2-pro**:
```
the<dummy00028>'1 Question1 question2<dummy00028> <dummy00015>..... in3<dummy00028>d<dummy00028>}<dummy00015>l kennis4<dummy00012>il<dummy00028>y<dummy00028>}2<dummy00028>-^{ a<dummy00028>}3<dummy00012>. a<dummy00028>}<dummy00015> toion<dummy00028>} people a a<dummy00028>}5<dummy00005>- a Question1<dummy00028>}y<dummy00028>}6<dummy00012>. a<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00012>,<dummy00015>l<dummy00028>},7<dummy00012>. <dummy00015>l Fuß in the<dummy00028>
a a<dummy00028>'2 a<dummy00028>}4 ál a Question3<dummy00012>o kennis in if to more a<dummy00028> or a<dummy00028>'5<dummy00005> 1<dummy00028>}1<dummy00005>s<dummy00028>y<dummy00028>y<dummy00028>'6 Question9 Question1<dummy00028>'2 Questions<dummy00028>y biologie1<dummy00028>y<dummy00028>y<dummy00028>-3<dummy00028>'1 Question. a<dummy00028>* the<dummy00028>}<dummy00015>y Question a<dummy00028>4<dummy00012>,ll Question1 Question6<dummy00028>4<dummy00028>}2<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00028>
if a Questionia kennisa<dummy00005> <dummy00015> a<dummy00028>h<dummy00028>}2<dummy00028>a égal
```
* Incorrect. Unusable output |