sanjay920 commited on
Commit
6d60c3a
1 Parent(s): 298c12b

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +164 -0
README.md ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - freeze
5
+ - generated_from_trainer
6
+ - autoquant
7
+ - exl2
8
+ base_model: mistralai/Mistral-7B-Instruct-v0.2
9
+ model-index:
10
+ - name: rubra-11b-h
11
+ results: []
12
+ ---
13
+
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
+
17
+ # rubra-11b-h
18
+
19
+ ![image/png](./rubra-11b-h.png)
20
+
21
+ ## Model description
22
+ This is a mistral based model trained on a lot of code and conversational data. The goal of this model was to retain the abilities of [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) while making it better at reasoning.
23
+
24
+ * 32k context window
25
+ * Rope-theta = 1e6
26
+
27
+ Fine tuning instruct models has proven to be a challenge when the goal is to create a model that is not meant for a particular task. Specifically, fine tuning an instruct model often leads to destructive forgetting in the resulting model. This is why most fine tunes select a base (non-instruct) language model. The rubra-v0.1 series models aim to add desired traits like multi-turn chat capabilites while retaining knowledge found in instruct models
28
+
29
+ ## Chat format
30
+
31
+ Your prompt should be surrounded by `[INST]` and `[/INST]` tokens. The very first instruction should begin with a begin of sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.
32
+
33
+ E.g.
34
+ ```
35
+ text = "<s>[INST] What is your favourite condiment? [/INST]"
36
+ "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> "
37
+ "[INST] Do you have mayonnaise recipes? [/INST]"
38
+ ```
39
+
40
+ This format is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating) via the `apply_chat_template()` method:
41
+
42
+ ```python
43
+ from transformers import AutoModelForCausalLM, AutoTokenizer
44
+
45
+ device = "cuda" # the device to load the model onto
46
+
47
+ model = AutoModelForCausalLM.from_pretrained("sanjay920/rubra-11b-h")
48
+ tokenizer = AutoTokenizer.from_pretrained("sanjay920/rubra-11b-h")
49
+
50
+ messages = [
51
+ {"role": "user", "content": "What is your favourite condiment?"},
52
+ {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
53
+ {"role": "user", "content": "Do you have mayonnaise recipes?"}
54
+ ]
55
+
56
+ encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
57
+
58
+ model_inputs = encodeds.to(device)
59
+ model.to(device)
60
+
61
+ generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
62
+ decoded = tokenizer.batch_decode(generated_ids)
63
+ print(decoded[0])
64
+ ```
65
+
66
+ ### GGUF
67
+
68
+ GGUF quants are available at [sanjay920/rubra-11b-h-GGUF](https://huggingface.co/sanjay920/rubra-11b-h-GGUF)
69
+
70
+ Create a new discussion for other quant variant requests!
71
+
72
+ #### LM Studio
73
+ Set "Chat Preset" to `Mistral Instruct`
74
+
75
+ ## Training procedure
76
+ * Block expansion and further training
77
+
78
+ ### Training hyperparameters
79
+
80
+ The following hyperparameters were used during training:
81
+ - learning_rate: 3e-05
82
+ - train_batch_size: 1
83
+ - eval_batch_size: 1
84
+ - seed: 42
85
+ - gradient_accumulation_steps: 12
86
+ - total_train_batch_size: 12
87
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
88
+ - lr_scheduler_type: cosine
89
+ - num_epochs: 3.0
90
+ - mixed_precision_training: Native AMP
91
+
92
+ ### Training results
93
+ ```
94
+ {
95
+ "epoch": 3.0,
96
+ "train_loss": 0.5844329195557438,
97
+ "train_runtime": 307371.8718,
98
+ "train_samples_per_second": 0.246,
99
+ "train_steps_per_second": 0.021
100
+ }
101
+ ```
102
+
103
+
104
+ ### Framework versions
105
+
106
+ - Transformers 4.38.2
107
+ - Pytorch 2.2.1+cu121
108
+ - Datasets 2.18.0
109
+ - Tokenizers 0.15.2
110
+
111
+ ## Evaluation Results
112
+
113
+ ```
114
+ ########## Average ##########
115
+ score
116
+ model
117
+ gpt-4-0613 9.020312
118
+ gpt-3.5-turbo-0125 7.970219
119
+ rubra-11b-h 7.800000
120
+ Mistral-7B-Instruct-v0.2 7.793750
121
+ Mistral-7B-Instruct-v0.1 6.721875
122
+ Llama-2-13b-chat 6.650000
123
+ ```
124
+
125
+ ![image/png](./mtbench-comparison.png)
126
+
127
+ ## Intended uses & limitations
128
+
129
+ This model is still under development. It hasn't been aligned for safety, so it may produce biased generations. It appears to produce thoughtful answers compared to other small LLMs. See below:
130
+
131
+ ## Examples
132
+
133
+ ![image/png](./example1.png)
134
+ ![image/png](./example2.png)
135
+ ![image/png](./example3.png)
136
+
137
+ ### Needle in a haystack test:
138
+
139
+ The input text can be found in [needle-in-a-haystack.txt](./needle-in-a-haystack.txt)
140
+
141
+ * 14,334 input tokens
142
+ * "Random number is 23." inserted into the middle of the long document text. Asked the LLM what the random number is.
143
+
144
+ **rubra-11b-h**:
145
+ ```
146
+ The random number is 23.
147
+ ```
148
+ * Correct
149
+
150
+ **mistral-7b-instruct-v0.2**:
151
+ ```
152
+ The random number generated from this text is 23. However, it's important to note that the method used to generate this number from the text is not a secure or reliable one and should not be used for any purpose where security is a concern. This was just a fun exercise to extract a number from the text.
153
+
154
+ As for the content of the text, it discusses the idea of creating a technology hub or "silicon valley" by attracting the right people - specifically, rich people and nerds. The author argues that these two groups are the limiting reagents in the reaction that produces startups, as they are the only ones present when startups get started. The text also touches on the idea that universities can act as magnets for these people and that personality and youth are important factors in creating a successful technology hub.
155
+ ```
156
+ * Correct
157
+
158
+ **hermes-2-pro**:
159
+ ```
160
+ the<dummy00028>'1 Question1 question2<dummy00028> <dummy00015>..... in3<dummy00028>d<dummy00028>}<dummy00015>l kennis4<dummy00012>il<dummy00028>y<dummy00028>}2<dummy00028>-^{ a<dummy00028>}3<dummy00012>. a<dummy00028>}<dummy00015> toion<dummy00028>} people a a<dummy00028>}5<dummy00005>- a Question1<dummy00028>}y<dummy00028>}6<dummy00012>. a<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00012>,<dummy00015>l<dummy00028>},7<dummy00012>. <dummy00015>l Fuß in the<dummy00028>
161
+ a a<dummy00028>'2 a<dummy00028>}4 ál a Question3<dummy00012>o kennis in if to more a<dummy00028> or a<dummy00028>'5<dummy00005> 1<dummy00028>}1<dummy00005>s<dummy00028>y<dummy00028>y<dummy00028>'6 Question9 Question1<dummy00028>'2 Questions<dummy00028>y biologie1<dummy00028>y<dummy00028>y<dummy00028>-3<dummy00028>'1 Question. a<dummy00028>* the<dummy00028>}<dummy00015>y Question a<dummy00028>4<dummy00012>,ll Question1 Question6<dummy00028>4<dummy00028>}2<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00028>
162
+ if a Questionia kennisa<dummy00005> <dummy00015> a<dummy00028>h<dummy00028>}2<dummy00028>a égal
163
+ ```
164
+ * Incorrect. Unusable output