Text Generation
Transformers
PyTorch
Chinese
English
llama
text-generation-inference
fireballoon commited on
Commit
e40d45d
1 Parent(s): 7f162f3

Upload zero3_bf16_config.yaml

Browse files
Files changed (1) hide show
  1. zero3_bf16_config.yaml +22 -0
zero3_bf16_config.yaml ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ compute_environment: LOCAL_MACHINE
2
+ deepspeed_config:
3
+ gradient_accumulation_steps: 8
4
+ gradient_clipping: 1.0
5
+ offload_optimizer_device: cpu
6
+ offload_param_device: cpu
7
+ zero3_init_flag: true
8
+ zero3_save_16bit_model: false
9
+ zero_stage: 3
10
+ distributed_type: DEEPSPEED
11
+ downcast_bf16: 'no'
12
+ machine_rank: 0
13
+ main_training_function: main
14
+ mixed_precision: bf16
15
+ num_machines: 1
16
+ num_processes: 8
17
+ rdzv_backend: static
18
+ same_network: true
19
+ tpu_env: []
20
+ tpu_use_cluster: false
21
+ tpu_use_sudo: false
22
+ use_cpu: false