File size: 3,622 Bytes
35e23cc |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
## Part 1
Normal model
Memory usage of model alone = 510.342192
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 838.783488
100%|█████████████████████████████████████▊| 489/491 [00:25<00:00, 18.97it/s]
Loss = 26.38488006591797
Time taken: 25.795103549957275
Full model quant
Memory usage of model alone = 294.250369
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 1465.776128
100%|█████████████████████████████████████▊| 489/491 [00:21<00:00, 22.39it/s]
Loss = 26.954803466796875
Time taken: 21.855380058288574
Full model without lm_head
Memory usage of model alone = 255.602736
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 1269.30176
100%|█████████████████████████████████████▊| 489/491 [00:21<00:00, 22.68it/s]
Loss = 26.41402816772461
Time taken: 21.578929662704468
Only LM head
Memory usage of model alone = 548.989825
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 1036.319744
100%|█████████████████████████████████████▊| 489/491 [00:20<00:00, 23.39it/s]
Loss = 26.924053192138672
Time taken: 20.919220209121704
Last 4 attention layers
Memory usage of model alone = 425.42904
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 983.949824
100%|█████████████████████████████████████▊| 489/491 [00:20<00:00, 23.40it/s]
Loss = 26.39584732055664
Time taken: 20.912957668304443
Only q,k,v
Memory usage of model alone = 425.425968
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 989.827584
100%|█████████████████████████████████████▊| 489/491 [00:21<00:00, 23.11it/s]
Loss = 26.396583557128906
Time taken: 21.17274236679077
## Part 2:
4 bit model
Memory usage of model alone = 134.060568
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 308.803072
100%|█████████████████████████████████████▊| 489/491 [00:16<00:00, 29.78it/s]
Loss = 31.296875
Time taken: 16.42749333381653
`low_cpu_mem_usage` was None, now set to True since model is quantized.
8 bit model
Memory usage of model alone = 176.527896
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 494.142976
100%|█████████████████████████████████████▊| 489/491 [00:29<00:00, 16.70it/s]
Loss = 26.5625
Time taken: 29.27569341659546
`low_cpu_mem_usage` was None, now set to True since model is quantized.
4 bit nf4 model
Memory usage of model alone = 134.060568
0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 494.85824
100%|█████████████████████████████████████▊| 489/491 [00:15<00:00, 30.64it/s]
Loss = 28.375
Time taken: 15.961309671401978
|