Training procedure

The following bitsandbytes quantization config was used during training:

  • load_in_8bit: True
  • load_in_4bit: False
  • llm_int8_threshold: 6.0
  • llm_int8_skip_modules: None
  • llm_int8_enable_fp32_cpu_offload: False
  • llm_int8_has_fp16_weight: False
  • bnb_4bit_quant_type: fp4
  • bnb_4bit_use_double_quant: False
  • bnb_4bit_compute_dtype: float32

Framework versions

  • PEFT 0.4.0

  • Here is the code to load the load along with llama2 chat model:

!pip install -q accelerate==0.21.0 peft==0.4.0 bitsandbytes==0.40.2 transformers==4.31.0 trl==0.4.7

import os, torch, logging from datasets import load_dataset from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, HfArgumentParser, TrainingArguments, pipeline from peft import LoraConfig, PeftModel from trl import SFTTrainer import torch from peft import PeftModel, PeftConfig from transformers import AutoModelForCausalLM, AutoTokenizer

config = PeftConfig.from_pretrained(path_to_directory_containing adapter_config.json) model = AutoModelForCausalLM.from_pretrained("NousResearch/Llama-2-7b-chat-hf", return_dict=True, load_in_8bit=True, device_map='auto') tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf")

model = PeftModel.from_pretrained(model, "/content/llama-2-7b-medichat") batch = tokenizer("My friend has been feeling a little dizzy these days.What could he pissibely be suffering from ?", return_tensors='pt') batch = batch.to('cuda') output_tokens = model.generate(**batch, max_new_tokens=300) ### You can adjust the max_new_tokens output_response = tokenizer.decode(output_tokens[0], skip_special_tokens=True)

Downloads last month
2
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Space using maxspin/medichat 1