llama_ft
This model is a fine-tuned version of Llama-2-7B-bf16-sharded on a grocery cart dataset.
Intended uses & limitations
The model helps to tell to what type of grocery does the following items belong to.
Training procedure
Fine tuning techniques like Qlora and PEFT have been used to train the model on the dataset on a single gpu , and the adapters are then finally merged with the model.
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16
The loading configurations of the model
Training hyperparameters
The following are the LORA configs-->
lora_alpha = 16
lora_dropout = 0.1
lora_r = 64
peft_config = LoraConfig(
lora_alpha=lora_alpha,
lora_dropout=lora_dropout,
r=lora_r,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj","v_proj"]
)
The following are the training configs -->
per_device_train_batch_size = 4
gradient_accumulation_steps = 4
optim = "paged_adamw_32bit"
save_steps = 10
logging_steps = 1
learning_rate = 2e-4
max_grad_norm = 0.3
max_steps = 120
warmup_ratio = 0.03
lr_scheduler_type = "constant"
- Downloads last month
- 4
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.