# Model summary

* instruction-tuning on medical data based on LLaMA 


# data

* Common 
    *  alpaca-5.2k
    *  unatural-instruct 80k
    *  OIG-40M
* Chinese
  * english/chinese translation data 
  * zhihu QA 
  * pCLUE
* Medical Domain:
  * MedDialog-200k
  * Chinese-medical-dialogue-data
  * WebMedQA 
* code
  * alpaca_code-20k
 

# training

## Model

* LLaMA-7B

## Hardware
* 6 x A100 40G using NVLink 4 inter-gpu connects

## Software

* tokenizers==0.12.1
* sentencepiece==0.1.97
* transformers==4.28
* torch==2.0.0+cu117



# How to use

```python
import torch
from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig
from peft import PeftModel

base_model="llma-7b"
LORA_WEIGHTS = "llma-med-alpaca-7b"
LOAD_8BIT = False

tokenizer = LlamaTokenizer.from_pretrained(base_model)

model = LlamaForCausalLM.from_pretrained(
    base_model
    load_in_8bit=LOAD_8BIT,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(
    model,
    LORA_WEIGHTS,
    torch_dtype=torch.float16,
)

config = {
    "temperature": 0 ,
    "max_new_tokens": 1024,
    "top_p": 0.5
}

prompt = "Translate to English: Je t’aime."
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
outputs = model.generate(input_ids=input_ids, max_new_tokens=config["max_new_tokens"], temperature=config["temperature"])
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True).strip()
print(decoded[len(prompt):])


```

# Limitations

* This model may output harmful, biased, toxic, and illusory things, and currently does not undergo RLHF training, so this model is only for research purposes

# TODO

- [x] self-instruct data
- [x] english medical data
- [ ] code data
- [ ] chinese corpus/medical dialog data


# Reference
* [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971)
* [Alpaca: A strong open-source instruction-following model](https://crfm.stanford.edu/2023/03/13/alpaca.html)