Hugging Face's logo

lucyknada
/
underwoods_medius-erebus-magnum-14b-exl2

Transformers
axolotl
Generated from Trainer
Inference Endpoints
Model card Files Community
underwoods_medius-erebus-magnum-14b-exl2
File size: 3,088 Bytes 
0dfdc54
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
 
---
library_name: transformers
base_model: Qwen/Qwen2.5-14B
tags:
- axolotl
- generated_from_trainer
model-index:
- name: medius-erebus-magnum-14b
  results: []
---
### exl2 quant (measurement.json in main branch)
---
### check revisions for quants
---


[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.4.1`
```yaml
base_model: /workspace/medius-erebus
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

hub_model_id: magnum-erebus-14b-v1
hub_strategy: "all_checkpoints"
push_dataset_to_hub:
hf_use_auth_token: true

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: anthracite-core/c2_logs_32k_llama3_qwen2_v1.2
    type: sharegpt
  - path: anthracite-org/kalo-opus-instruct-22k-no-refusal
    type: sharegpt
  - path: lodrick-the-lafted/kalo-opus-instruct-3k-filtered
    type: sharegpt
  - path: anthracite-org/nopm_claude_writing_fixed
    type: sharegpt
  - path: anthracite-org/kalo_opus_misc_240827
    type: sharegpt
  - path: anthracite-org/kalo_misc_part2
    type: sharegpt
chat_template: chatml
shuffle_merged_datasets: true
default_system_message: "You are an assistant that responds to the user."
dataset_prepared_path: /workspace/data/magnum-14b-data
val_set_size: 0.0
output_dir: /workspace/data/magnum-erebus-14b-fft

sequence_len: 32768
sample_packing: true
pad_to_sequence_len: true

adapter:
lora_model_dir:
lora_r:
lora_alpha:
lora_dropout:
lora_target_linear:
lora_fan_in_fan_out:

wandb_project: 14b-magnum-fft
wandb_entity:
wandb_watch:
wandb_name: v4-r2-erebus-attempt-1
wandb_log_model:

gradient_accumulation_steps: 1
micro_batch_size: 2
num_epochs: 2
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000008

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: unsloth
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 40
evals_per_epoch:
eval_table_size:
eval_max_new_tokens:
saves_per_epoch: 2
debug:
deepspeed: deepspeed_configs/zero3_bf16.json
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:


```

</details><br>

# medius-erebus-magnum

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 8e-06
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 16
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 40
- num_epochs: 2

### Training results



### Framework versions

- Transformers 4.45.1
- Pytorch 2.3.1+cu121
- Datasets 2.21.0
- Tokenizers 0.20.0