---
license: apache-2.0
library_name: peft
tags:
- generated_from_trainer
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
model-index:
- name: TinyLlama-1.1B-SlimOrca-Function-Calling-3T
results:
- task:
type: text-generation
metrics:
- name: Average
type: Average
value: 37.38
source:
name: Open LLM Leaderboard
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
- task:
type: text-generation
metrics:
- name: ARC
type: ARC
value: 36.09
source:
name: Open LLM Leaderboard
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
- task:
type: text-generation
metrics:
- name: HellaSwag
type: HellaSwag
value: 59.66
source:
name: Open LLM Leaderboard
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
- task:
type: text-generation
metrics:
- name: MMLU
type: MMLU
value: 28.21
source:
name: Open LLM Leaderboard
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
- task:
type: text-generation
metrics:
- name: TruthfulQA
type: TruthfulQA
value: 36.74
source:
name: Open LLM Leaderboard
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
- task:
type: text-generation
metrics:
- name: Winograde
type: Winograde
value: 59.12
source:
name: Open LLM Leaderboard
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
- task:
type: text-generation
metrics:
- name: GSM8K
type: GSM8K
value: 4.47
source:
name: Open LLM Leaderboard
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
datasets:
- Open-Orca/SlimOrca-Dedup
- gardner/glaive-function-calling-v2-sharegpt
language: en
---
# TinyLlama-1.1B-SlimOrca-Function-Calling-3T
This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) on the [SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca-Dedup) and [glaive-function-calling-v2](https://huggingface.co/datasets/gardner/glaive-function-calling-v2-sharegpt) datasets.
# Evaluation
It achieves the following results on the evaluation set:
- Loss: 0.7403
Please see the `scripts/llm-eval.py` to recreate the evaluation results from the test split as published here: [gardner/tinyllama-function-calling-eval](https://huggingface.co/datasets/gardner/tinyllama-function-calling-eval). The model responds with function calling when expected and refuses when it doesn't have access to tools. In the linked dataset, `result1` is generated by this model and `result2` is from the test dataset.
[
](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config
axolotl version: `0.3.0`
```yaml
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true
load_in_8bit: true
load_in_4bit: false
strict: false
datasets:
- path: Open-Orca/SlimOrca-Dedup
type: sharegpt
conversation: chatml
- path: gardner/glaive-function-calling-v2-sharegpt
type: sharegpt
conversation: chatml
dataset_prepared_path: ./.prepared-datasets/glaive-function-calling-v2-sharegpt
val_set_size: 0.05
output_dir: ./tinyllama/function-calling/chatml
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
```
## Training procedure
The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: True
- load_in_4bit: False
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: fp4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: float32
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 4
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.2492 | 0.0 | 1 | 1.2363 |
| 0.7621 | 0.25 | 1896 | 0.8096 |
| 0.757 | 0.5 | 3792 | 0.7852 |
| 0.6424 | 0.75 | 5688 | 0.7717 |
| 0.5944 | 1.04 | 7584 | 0.7625 |
| 0.73 | 1.29 | 9480 | 0.7585 |
| 0.6781 | 1.54 | 11376 | 0.7521 |
| 0.829 | 1.79 | 13272 | 0.7471 |
| 0.6964 | 2.08 | 15168 | 0.7467 |
| 0.6652 | 2.33 | 17064 | 0.7453 |
| 0.7645 | 2.58 | 18960 | 0.7420 |
| 0.5702 | 2.83 | 20856 | 0.7392 |
| 0.7049 | 3.12 | 22752 | 0.7418 |
| 0.6087 | 3.37 | 24648 | 0.7412 |
| 0.6064 | 3.62 | 26544 | 0.7405 |
| 0.7125 | 3.87 | 28440 | 0.7403 |
### Framework versions
- PEFT 0.7.0
- Transformers 4.37.0.dev0
- Pytorch 2.0.1+cu118
- Datasets 2.16.1
- Tokenizers 0.15.0