Hugging Face's logo

fxmarty
/
20220911-h13m59s08_vit_beans_quantization

Image Classification
TensorBoard
ONNX
vit
Model card Files Metrics Community
1
fxmarty HF staff commited on
Commit
f579c3f
•
1 Parent(s): 97d64b2

add experience description

Browse files
Files changed (21) hide show
  1. README.md +87 -0
  2. runs.json +0 -0
  3. tensorboard/1662907378.8460534/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.1 +3 -0
  4. tensorboard/1662907378.8475587/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.2 +3 -0
  5. tensorboard/1662907378.848857/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.3 +3 -0
  6. tensorboard/1662907378.8500683/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.4 +3 -0
  7. tensorboard/1662907378.8512115/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.5 +3 -0
  8. tensorboard/1662907378.8523538/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.6 +3 -0
  9. tensorboard/1662907378.8535051/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.7 +3 -0
  10. tensorboard/1662907378.8546307/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.8 +3 -0
  11. tensorboard/1662907378.855868/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.9 +3 -0
  12. tensorboard/1662907378.857226/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.10 +3 -0
  13. tensorboard/1662907378.8584979/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.11 +3 -0
  14. tensorboard/1662907378.8597944/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.12 +3 -0
  15. tensorboard/1662907378.8610377/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.13 +3 -0
  16. tensorboard/1662907378.8623042/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.14 +3 -0
  17. tensorboard/1662907378.8636096/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.15 +3 -0
  18. tensorboard/1662907378.8649743/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.16 +3 -0
  19. tensorboard/1662907378.8668153/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.17 +3 -0
  20. tensorboard/1662907378.868103/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.18 +3 -0
  21. tensorboard/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.0 +3 -0
README.md ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: image-classification
3
+ datasets:
4
+ - beans
5
+ metrics:
6
+ - accuracy
7
+ - total_time_in_seconds
8
+ - samples_per_second
9
+ - latency_in_seconds
10
+ tags:
11
+ - vit
12
+ ---
13
+
14
+ **task**: `image-classification`
15
+ **Backend:** `sagemaker-training`
16
+ **Backend args:** `{'instance_type': 'ml.m5.2xlarge', 'supported_instructions': 'avx512'}`
17
+ **Number of evaluation samples:** `All dataset`
18
+
19
+ Fixed parameters:
20
+ * **dataset**: [{'path': 'beans', 'eval_split': 'validation', 'data_keys': {'primary': 'image'}, 'ref_keys': ['labels'], 'name': None, 'calibration_split': 'train'}]
21
+ * **name_or_path**: `nateraw/vit-base-beans`
22
+ * **from_transformers**: `True`
23
+ * **node_exclusion**: `[]`
24
+ * **calibration**:
25
+ * **method**: `percentile`
26
+ * **num_calibration_samples**: `128`
27
+ * **calibration_histogram_percentile**: `99.999`
28
+
29
+ Benchmarked parameters:
30
+ * **framework**: `onnxruntime`, `pytorch`
31
+ * **quantization_approach**: `dynamic`, `static`
32
+ * **operators_to_quantize**: `['Add', 'MatMul']`, `['Add']`
33
+ * **per_channel**: `False`, `True`
34
+ * **framework_args**: `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}`, `{}`
35
+ * **reduce_range**: `True`, `False`
36
+ * **apply_quantization**: `True`, `False`
37
+
38
+ # Evaluation
39
+ ## Non-time metrics
40
+ | framework | quantization_approach | operators_to_quantize | per_channel | framework_args | reduce_range | apply_quantization | | accuracy |
41
+ | :-----------: | :-------------------: | :-------------------: | :---------: | :-----------------------------------------------------------------: | :----------: | :----------------: | :-: | :------: |
42
+ | `onnxruntime` | `None` | `None` | `None` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `None` | `False` | \| | 0.977 |
43
+ | `onnxruntime` | `dynamic` | `['Add', 'MatMul']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 0.977 |
44
+ | `onnxruntime` | `dynamic` | `['Add', 'MatMul']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 0.977 |
45
+ | `onnxruntime` | `dynamic` | `['Add', 'MatMul']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 0.977 |
46
+ | `onnxruntime` | `dynamic` | `['Add', 'MatMul']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 0.977 |
47
+ | `onnxruntime` | `dynamic` | `['Add']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 0.977 |
48
+ | `onnxruntime` | `dynamic` | `['Add']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 0.977 |
49
+ | `onnxruntime` | `dynamic` | `['Add']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 0.977 |
50
+ | `onnxruntime` | `dynamic` | `['Add']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 0.977 |
51
+ | `onnxruntime` | `static` | `['Add', 'MatMul']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 0.421 |
52
+ | `onnxruntime` | `static` | `['Add', 'MatMul']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 0.421 |
53
+ | `onnxruntime` | `static` | `['Add', 'MatMul']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 0.316 |
54
+ | `onnxruntime` | `static` | `['Add', 'MatMul']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 0.451 |
55
+ | `onnxruntime` | `static` | `['Add']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 0.361 |
56
+ | `onnxruntime` | `static` | `['Add']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 0.361 |
57
+ | `onnxruntime` | `static` | `['Add']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 0.361 |
58
+ | `onnxruntime` | `static` | `['Add']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 0.361 |
59
+ | `pytorch` | `None` | `None` | `None` | `{}` | `None` | `None` | \| | 0.977 |
60
+
61
+ ## Time metrics
62
+ Time benchmarks were run for 15 seconds per config.
63
+
64
+
65
+ Below, time metrics for batch size = 1, input length = 224.
66
+
67
+ | framework | quantization_approach | operators_to_quantize | per_channel | framework_args | reduce_range | apply_quantization | | latency_mean (ms) | | throughput (/s) |
68
+ | :-----------: | :-------------------: | :-------------------: | :---------: | :-----------------------------------------------------------------: | :----------: | :----------------: | :-: | :---------------: | :-: | :-------------: |
69
+ | `onnxruntime` | `None` | `None` | `None` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `None` | `False` | \| | 130.41 | \| | 7.73 |
70
+ | `onnxruntime` | `dynamic` | `['Add', 'MatMul']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 102.44 | \| | 9.80 |
71
+ | `onnxruntime` | `dynamic` | `['Add', 'MatMul']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 101.57 | \| | 9.87 |
72
+ | `onnxruntime` | `dynamic` | `['Add', 'MatMul']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 102.37 | \| | 9.80 |
73
+ | `onnxruntime` | `dynamic` | `['Add', 'MatMul']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 102.36 | \| | 9.80 |
74
+ | `onnxruntime` | `dynamic` | `['Add']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 130.67 | \| | 7.67 |
75
+ | `onnxruntime` | `dynamic` | `['Add']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 131.29 | \| | 7.67 |
76
+ | `onnxruntime` | `dynamic` | `['Add']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 132.65 | \| | 7.60 |
77
+ | `onnxruntime` | `dynamic` | `['Add']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 131.03 | \| | 7.67 |
78
+ | `onnxruntime` | `static` | `['Add', 'MatMul']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 127.99 | \| | 7.87 |
79
+ | `onnxruntime` | `static` | `['Add', 'MatMul']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 128.27 | \| | 7.80 |
80
+ | `onnxruntime` | `static` | `['Add', 'MatMul']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 131.10 | \| | 7.67 |
81
+ | `onnxruntime` | `static` | `['Add', 'MatMul']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 130.29 | \| | 7.73 |
82
+ | `onnxruntime` | `static` | `['Add']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 164.55 | \| | 6.13 |
83
+ | `onnxruntime` | `static` | `['Add']` | `False` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 168.61 | \| | 5.93 |
84
+ | `onnxruntime` | `static` | `['Add']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `False` | `True` | \| | 164.52 | \| | 6.13 |
85
+ | `onnxruntime` | `static` | `['Add']` | `True` | `{'opset': 13, 'optimization_level': 1, 'intra_op_num_threads': 4}` | `True` | `True` | \| | 165.31 | \| | 6.07 |
86
+ | `pytorch` | `None` | `None` | `None` | `{}` | `None` | `None` | \| | 149.23 | \| | 6.73 |
87
+
runs.json ADDED
The diff for this file is too large to render. See raw diff
 
tensorboard/1662907378.8460534/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:557b71db4744202989d231d191c1c7c2c45c0d3f9d39d65af9c138ecf878a221
3
+ size 833
tensorboard/1662907378.8475587/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ad9a8d3a88889334fde0ec28d7f082d5d42049bed47da681e3076d5d7d0b54c7
3
+ size 850
tensorboard/1662907378.848857/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:63fe6610b8538aa824c4425a394b0742e33a0002751e0991485319a675c78aa8
3
+ size 849
tensorboard/1662907378.8500683/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07761fd4e51d3ace6d5578b41867ce2a52693d27a5a13a14dd273504b5763cdf
3
+ size 849
tensorboard/1662907378.8512115/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.5 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8b9eff3a8416d81c5ad4beab817c6ee2cc1376c12a44ea5c123e6f5c5f6ef9ab
3
+ size 848
tensorboard/1662907378.8523538/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.6 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06a2cce95e2018123cca5ac0a936a6c76296e8b62a8b45b22ccbf94ab790df23
3
+ size 840
tensorboard/1662907378.8535051/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.7 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0d79493604f680e32b0ce15b0e66d72f9f975b5ed6329403d10930c896578bef
3
+ size 839
tensorboard/1662907378.8546307/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.8 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7c6c96a02c71add35a4e2c520c3f498dd72a768d1f3f30a854be56c7bc6749ca
3
+ size 839
tensorboard/1662907378.855868/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.9 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0e593f14e3befc24862c8e70902bf7467be08d4a21b94d256ac1d016dd3b6e34
3
+ size 838
tensorboard/1662907378.857226/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.10 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dcade53bac39443124fc71b3417087c257e891755a1f2e4bafe4a06f405881fc
3
+ size 849
tensorboard/1662907378.8584979/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.11 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88c77618758472dc8262246511e48c932efab5161f81553b41bce132b59bbcd4
3
+ size 848
tensorboard/1662907378.8597944/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.12 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e73b02bdaf5dc9eb7195f89bba2f0ab08ebad7734504ef4d89e86947972ae22f
3
+ size 848
tensorboard/1662907378.8610377/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.13 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:64aec5483ab146e43e108401d97c923901cf440fe96c2b91d92610d2159e9502
3
+ size 847
tensorboard/1662907378.8623042/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.14 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:36e56dab6d642b6f33942f80f7f702fd5f4d033a44aeae6ed0bcd4a7c3f7ccc6
3
+ size 839
tensorboard/1662907378.8636096/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.15 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cda0e86d6494d30fb3dbd1165c27cf1901910643f1319944a31cb3e5cee8317b
3
+ size 838
tensorboard/1662907378.8649743/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.16 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:03b1682c08fbccdb8fb867bca688904a4e5beb7472da58a0f1fd2344c7be0b49
3
+ size 838
tensorboard/1662907378.8668153/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.17 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:516bddc14d9962a8b58fe9b75fda257f384c31c06e78ab8665eb61f346b2c0aa
3
+ size 837
tensorboard/1662907378.868103/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.18 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dc6aa257b9722a6578d69e8832cadca1e0ad98cf74be2933190b6228f3626eb7
3
+ size 765
tensorboard/events.out.tfevents.1662907378.ip-10-0-138-55.ec2.internal.1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bf6e6632c166bf7321b28b804b67d315cadd8d595c0ca3ff4187a841f828a05e
3
+ size 40