fakezeta commited on
Commit
cfe79cd
1 Parent(s): 56940e0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +233 -3
README.md CHANGED
@@ -1,3 +1,233 @@
1
- ---
2
- license: cc-by-nc-4.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ datasets:
4
+ - argilla/dpo-mix-7k
5
+ - nvidia/HelpSteer
6
+ - jondurbin/airoboros-3.2
7
+ - hkust-nlp/deita-10k-v0
8
+ - LDJnr/Capybara
9
+ - HPAI-BSC/CareQA
10
+ - GBaker/MedQA-USMLE-4-options
11
+ - lukaemon/mmlu
12
+ - bigbio/pubmed_qa
13
+ - openlifescienceai/medmcqa
14
+ - bigbio/med_qa
15
+ - HPAI-BSC/better-safe-than-sorry
16
+ language:
17
+ - en
18
+ library_name: transformers
19
+ tags:
20
+ - biology
21
+ - medical
22
+ ---
23
+
24
+ # OpenVINO IR model with int8 quantization
25
+
26
+ Model definition for LocalAI:
27
+ ```
28
+ name: llama3-aloe
29
+ backend: transformers
30
+ parameters:
31
+ model: fakezeta/Llama3-Aloe-8B-Alpha-ov-int8
32
+ context_size: 8192
33
+ type: OVModelForCausalLM
34
+ template:
35
+ use_tokenizer_template: true
36
+ stopwords:
37
+ - "<|eot_id|>"
38
+ - "<|end_of_text|>"
39
+ ```
40
+
41
+
42
+ # Aloe: A New Family of Healthcare LLMs
43
+
44
+ Aloe is a new family of healthcare LLMs that is highly competitive with all previous open models of its range and reaches state-of-the-art results at its size by using model merging and advanced prompting strategies. Aloe scores high in metrics measuring ethics and factuality, thanks to a combined red teaming and alignment effort. Complete training details, model merging configurations, and all training data (including synthetically generated data) will be shared. Additionally, the prompting repository used in this work to produce state-of-the-art results during inference will also be shared. Aloe comes with a healthcare-specific risk assessment to contribute to the safe use and deployment of such systems.
45
+
46
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/62f7a16192950415b637e201/HMD6WEoqqrAV8Ng_fAcnN.png" width="95%">
47
+
48
+ ## Model Details
49
+
50
+ ### [](https://huggingface.co/templates/model-card-example#model-description)Model Description
51
+
52
+ - **Developed by:** [HPAI](https://hpai.bsc.es/)
53
+ - **Model type:** Causal decoder-only transformer language model
54
+ - **Language(s) (NLP):** English (mainly)
55
+ - **License:** This model is based on Meta Llama 3 8B and is governed by the [Meta Llama 3 License](https://llama.meta.com/llama3/license/). All our modifications are available with a [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license.
56
+ - **Finetuned from model :** [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
57
+
58
+ ### [](https://huggingface.co/templates/model-card-example#model-sources-optional)Model Sources [optional]
59
+
60
+ - **Repository:** https://github.com/HPAI-BSC/prompt_engine (more coming soon)
61
+ - **Paper:** https://arxiv.org/abs/2405.01886 (more coming soon)
62
+
63
+ ## Model Performance
64
+
65
+ Aloe has been tested on the most popular healthcare QA datasets, with and without medprompting inference technique. Results show competitive performance, even against bigger models.
66
+
67
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/62f7a16192950415b637e201/rQ4z-qXzKN44oAcFDbHi2.png" width="95%">
68
+
69
+ Results using advanced prompting methods (aka Medprompt) are achieved through a [repo](https://github.com/HPAI-BSC/prompt_engine) made public with this work.
70
+
71
+ ## Uses
72
+
73
+ ### Direct Use
74
+
75
+ We encourage the use of Aloe for research purposes, as a stepping stone to build better foundational models for healthcare.
76
+
77
+ ### Out-of-Scope Use
78
+
79
+ These models are not to be used for clinical practice, medical diagnosis, or any other form of direct or indirect healthcare advice. Models are prone to error and can produce toxic content. The use of Aloe models for activities harmful for individuals, such as spam, fraud, or impersonation, is prohibited.
80
+
81
+ ## Bias, Risks, and Limitations
82
+
83
+ First let us consider Healthcare professional impersonation, a fraudulent behaviour which currently generates billions of dollars in [profit](https://www.justice.gov/opa/pr/justice-department-charges-dozens-12-billion-health-care-fraud). A model such as Aloe could be used to increase the efficacy of such deceiving activities, making them more widespread. The main preventive actions are public literacy on the unreliability of digitised information and the importance of medical registration, and legislation enforcing AI-generated content disclaimers. The second risk we consider is medical decision-making without professional supervision. While this is already an issue in modern societies (eg self-medication) a model such as Aloe, capable of producing high-quality conversational data, can facilitate self-delusion, particularly in the presence of sycophancy. By producing tailored responses, it can also be used to generate actionable answers. Public literacy on the dangers of self-diagnosis is one of the main defences, together with the introduction of disclaimers and warnings on the models' outputs. The last risk we consider is the access to information on dangerous substances or procedures. While the literature on sensitive content can already be found on different sources (eg libraries, internet, dark web), LLMs can centralize such access, making it nearly impossible to control the flow of such information. Model alignment can help in that regard, but so far the effects remain insufficient, as jailbreaking methods still overcome it.
84
+
85
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/T6Jblpf1kmTkM04K716rM.png" width="95%">
86
+
87
+ ### Recommendations
88
+
89
+ We avoid the use of all personal data in our training. Model safety cannot be guaranteed. Aloe can produce toxic content under the appropriate prompts. For these reasons, minors should not be left alone to interact with Aloe without supervision.
90
+
91
+ ## How to Get Started with the Model
92
+
93
+ Use the code below to get started with the model. You can run conversational inference using the Transformers pipeline abstraction, or by leveraging the Auto classes with the `generate()` function. Let's see examples of both.
94
+
95
+ #### Transformers pipeline
96
+
97
+ ```python
98
+ import transformers
99
+ import torch
100
+
101
+ model_id = "HPAI-BSC/Llama3-Aloe-8B-Alpha"
102
+
103
+ pipeline = transformers.pipeline(
104
+ "text-generation",
105
+ model=model_id,
106
+ model_kwargs={"torch_dtype": torch.bfloat16},
107
+ device_map="auto",
108
+ )
109
+
110
+ messages = [
111
+ {"role": "system", "content": "You are an expert medical assistant named Aloe, developed by the High Performance Artificial Intelligence Group at Barcelona Supercomputing Center(BSC). You are to be a helpful, respectful, and honest assistant."},
112
+ {"role": "user", "content": "Hello."},
113
+ ]
114
+
115
+ prompt = pipeline.tokenizer.apply_chat_template(
116
+ messages,
117
+ tokenize=False,
118
+ add_generation_prompt=True
119
+ )
120
+
121
+ terminators = [
122
+ pipeline.tokenizer.eos_token_id,
123
+ pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
124
+ ]
125
+
126
+ outputs = pipeline(
127
+ prompt,
128
+ max_new_tokens=256,
129
+ eos_token_id=terminators,
130
+ do_sample=True,
131
+ temperature=0.6,
132
+ top_p=0.9,
133
+ )
134
+ print(outputs[0]["generated_text"][len(prompt):])
135
+ ```
136
+
137
+ #### Transformers AutoModelForCausalLM
138
+
139
+ ```python
140
+ from transformers import AutoTokenizer, AutoModelForCausalLM
141
+ import torch
142
+
143
+ model_id = "HPAI-BSC/Llama3-Aloe-8B-Alpha"
144
+
145
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
146
+ model = AutoModelForCausalLM.from_pretrained(
147
+ model_id,
148
+ torch_dtype=torch.bfloat16,
149
+ device_map="auto",
150
+ )
151
+
152
+ messages = [
153
+ {"role": "system", "content": "You are an expert medical assistant named Aloe, developed by the High Performance Artificial Intelligence Group at Barcelona Supercomputing Center(BSC). You are to be a helpful, respectful, and honest assistant."},
154
+ {"role": "user", "content": "Hello"},
155
+ ]
156
+
157
+ input_ids = tokenizer.apply_chat_template(
158
+ messages,
159
+ add_generation_prompt=True,
160
+ return_tensors="pt"
161
+ ).to(model.device)
162
+
163
+ terminators = [
164
+ tokenizer.eos_token_id,
165
+ tokenizer.convert_tokens_to_ids("<|eot_id|>")
166
+ ]
167
+
168
+ outputs = model.generate(
169
+ input_ids,
170
+ max_new_tokens=256,
171
+ eos_token_id=terminators,
172
+ do_sample=True,
173
+ temperature=0.6,
174
+ top_p=0.9,
175
+ )
176
+ response = outputs[0][input_ids.shape[-1]:]
177
+ print(tokenizer.decode(response, skip_special_tokens=True))
178
+ ```
179
+
180
+ ## Training Details
181
+
182
+ Supervised fine-tuning on top of Llama 3 8B using medical and general domain datasets, model merging using DARE-TIES process, two-stage DPO process for human preference alignment. More details coming soon.
183
+
184
+ ### Training Data
185
+
186
+ - Medical domain datasets, including synthetic data generated using Mixtral-8x7B and Genstruct
187
+ - LDJnr/Capybara
188
+ - hkust-nlp/deita-10k-v0
189
+ - jondurbin/airoboros-3.2
190
+ - argilla/dpo-mix-7k
191
+ - nvidia/HelpSteer
192
+ - Custom preference data with adversarial prompts generated from Anthropic Harmless, Chen et al., and original prompts
193
+
194
+ ## Evaluation
195
+
196
+ ### Testing Data, Factors & Metrics
197
+
198
+ #### Testing Data
199
+
200
+ - [MedQA (USMLE)](https://huggingface.co/datasets/bigbio/med_qa)
201
+ - [MedMCQA](https://huggingface.co/datasets/medmcqa)
202
+ - [PubMedQA](https://huggingface.co/datasets/bigbio/pubmed_qa)
203
+ - [MMLU-Medical](https://huggingface.co/datasets/lukaemon/mmlu)
204
+ - [MedQA-4-Option](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options)
205
+ - [CareQA](https://huggingface.co/datasets/HPAI-BSC/CareQA)
206
+
207
+ #### Metrics
208
+
209
+ - Accuracy: suite the evaluation of multiple-choice question-answering tasks.
210
+
211
+ ### Results
212
+
213
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/STlPSggXr9P9JeWAvmAsi.png" width="90%">
214
+
215
+ #### Summary
216
+
217
+ To compare Aloe with the most competitive open models (both general purpose and healthcare-specific) we use popular healthcare datasets (PubMedQA, MedMCQA, MedQA and MMLU for six medical tasks only), together with the new and highly reliable CareQA. We produce the standard MultiMedQA score for reference, by computing the weighted average accuracy on all scores except CareQA. Additionally, we calculate the arithmetic mean across all datasets. The Medical MMLU is calculated by averaging the six medical subtasks: Anatomy, Clinical knowledge, College Biology, College medicine, Medical genetics, and Professional medicine.
218
+
219
+ Benchmark results indicate the training conducted on Aloe has boosted its performance above Llama3-8B-Instruct. Llama3-Aloe-8B-Alpha outperforms larger models like Meditron 70B, and is close to larger base models, like Yi-34. For the former, this gain is consistent even when using SC-CoT, using their best-reported variant. All these results make Llama3-Aloe-8B-Alpha the best healthcare LLM of its size.
220
+
221
+ With the help of prompting techniques the performance of Llama3-Aloe-8B-Alpha is significantly improved. Medprompting in particular provides a 7% increase in reported accuracy, after which Llama3-Aloe-8B-Alpha only lags behind the ten times bigger Llama-3-70B-Instruct. This improvement is mostly consistent across medical fields. Llama3-Aloe-8B-Alpha with medprompting beats the performance of Meditron 70B with their self reported 20 shot SC-CoT in MMLU med and is slightly worse in the other benchmarks.
222
+
223
+ ## Environmental Impact
224
+
225
+ - **Hardware Type:** 4xH100
226
+ - **Hours used:** 7,000
227
+ - **Hardware Provider:** Barcelona Supercomputing Center
228
+ - **Compute Region:** Spain
229
+ - **Carbon Emitted:** 439.25kg
230
+
231
+ ## Model Card Contact
232
+
233
+ mailto:hpai@bsc.es