Hugging Face's logo

fxmarty
/
20220712-h08m02s04_example

Token Classification
TensorBoard
distilbert
Model card Files Metrics Community
fxmarty commited on
Commit
37061fe
•
1 Parent(s): cc1def3

add experience

Browse files
Files changed (7) hide show
  1. README.md +143 -0
  2. runs.json +1204 -0
  3. tensorboard/1657612929.728409/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.1 +3 -0
  4. tensorboard/1657612929.729769/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.2 +3 -0
  5. tensorboard/1657612929.7311082/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.3 +3 -0
  6. tensorboard/1657612929.7323039/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.4 +3 -0
  7. tensorboard/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.0 +3 -0
README.md ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: token-classification
3
+ datasets:
4
+ - conll2003
5
+ metrics:
6
+ - precision
7
+ - recall
8
+ - f1
9
+ - accuracy
10
+ tags:
11
+ - distilbert
12
+ ---
13
+
14
+ **task**: `token-classification`
15
+ **Backend:** `sagemaker-training`
16
+ **Backend args:** `{'instance_type': 'ml.m5.2xlarge', 'supported_instructions': 'avx512'}`
17
+ **Number of evaluation samples:** `All dataset`
18
+
19
+ Fixed parameters:
20
+ * **model_name_or_path**: `elastic/distilbert-base-uncased-finetuned-conll03-english`
21
+ * **dataset**:
22
+ * **path**: `conll2003`
23
+ * **eval_split**: `validation`
24
+ * **data_keys**: `{'primary': 'tokens'}`
25
+ * **ref_keys**: `['ner_tags']`
26
+ * **calibration_split**: `train`
27
+ * **node_exclusion**: `[]`
28
+ * **per_channel**: `False`
29
+ * **calibration**:
30
+ * **method**: `minmax`
31
+ * **num_calibration_samples**: `100`
32
+ * **framework**: `onnxruntime`
33
+ * **framework_args**:
34
+ * **opset**: `11`
35
+ * **optimization_level**: `1`
36
+ * **aware_training**: `False`
37
+
38
+ Benchmarked parameters:
39
+ * **quantization_approach**: `dynamic`, `static`
40
+ * **operators_to_quantize**: `['Add', 'MatMul']`, `['Add']`
41
+
42
+ # Evaluation
43
+ ## Non-time metrics
44
+ | quantization_approach | operators_to_quantize | | precision (original) | precision (optimized) | | recall (original) | recall (optimized) | | f1 (original) | f1 (optimized) | | accuracy (original) | accuracy (optimized) |
45
+ | :-------------------: | :-------------------: | :-: | :------------------: | :-------------------: | :-: | :---------------: | :----------------: | :-: | :-----------: | :------------: | :-: | :-----------------: | :------------------: |
46
+ | `dynamic` | `['Add', 'MatMul']` | \| | 0.936 | 0.935 | \| | 0.944 | 0.943 | \| | 0.940 | 0.939 | \| | 0.988 | 0.988 |
47
+ | `dynamic` | `['Add']` | \| | 0.936 | 0.936 | \| | 0.944 | 0.944 | \| | 0.940 | 0.940 | \| | 0.988 | 0.988 |
48
+ | `static` | `['Add', 'MatMul']` | \| | 0.936 | 0.063 | \| | 0.944 | 0.246 | \| | 0.940 | 0.100 | \| | 0.988 | 0.343 |
49
+ | `static` | `['Add']` | \| | 0.936 | 0.050 | \| | 0.944 | 0.160 | \| | 0.940 | 0.076 | \| | 0.988 | 0.311 |
50
+
51
+ ## Time metrics
52
+ Time benchmarks were run for 15 seconds per config.
53
+
54
+
55
+ Below, time metrics for batch size = 1, input length = 32.
56
+
57
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
58
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
59
+ | `dynamic` | `['Add', 'MatMul']` | \| | 46.38 | 9.96 | \| | 21.60 | 100.47 |
60
+ | `dynamic` | `['Add']` | \| | 36.59 | 13.98 | \| | 27.33 | 71.60 |
61
+ | `static` | `['Add', 'MatMul']` | \| | 33.84 | 14.46 | \| | 29.60 | 69.20 |
62
+ | `static` | `['Add']` | \| | 33.23 | 20.11 | \| | 30.13 | 49.73 |
63
+
64
+
65
+ Below, time metrics for batch size = 1, input length = 64.
66
+
67
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
68
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
69
+ | `dynamic` | `['Add', 'MatMul']` | \| | 58.92 | 19.68 | \| | 17.00 | 50.87 |
70
+ | `dynamic` | `['Add']` | \| | 58.59 | 24.81 | \| | 17.13 | 40.33 |
71
+ | `static` | `['Add', 'MatMul']` | \| | 51.41 | 29.36 | \| | 19.47 | 34.07 |
72
+ | `static` | `['Add']` | \| | 44.22 | 38.56 | \| | 22.67 | 25.93 |
73
+
74
+
75
+ Below, time metrics for batch size = 1, input length = 128.
76
+
77
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
78
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
79
+ | `dynamic` | `['Add', 'MatMul']` | \| | 72.38 | 36.47 | \| | 13.87 | 27.47 |
80
+ | `dynamic` | `['Add']` | \| | 70.21 | 46.30 | \| | 14.27 | 21.60 |
81
+ | `static` | `['Add', 'MatMul']` | \| | 70.76 | 48.24 | \| | 14.13 | 20.80 |
82
+ | `static` | `['Add']` | \| | 72.47 | 71.10 | \| | 13.80 | 14.07 |
83
+
84
+
85
+ Below, time metrics for batch size = 4, input length = 32.
86
+
87
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
88
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
89
+ | `dynamic` | `['Add', 'MatMul']` | \| | 69.76 | 38.50 | \| | 14.40 | 26.00 |
90
+ | `dynamic` | `['Add']` | \| | 56.02 | 51.32 | \| | 17.87 | 19.53 |
91
+ | `static` | `['Add', 'MatMul']` | \| | 55.05 | 46.80 | \| | 18.20 | 21.40 |
92
+ | `static` | `['Add']` | \| | 71.03 | 56.82 | \| | 14.13 | 17.67 |
93
+
94
+
95
+ Below, time metrics for batch size = 4, input length = 64.
96
+
97
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
98
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
99
+ | `dynamic` | `['Add', 'MatMul']` | \| | 119.91 | 61.51 | \| | 8.40 | 16.27 |
100
+ | `dynamic` | `['Add']` | \| | 108.43 | 105.65 | \| | 9.27 | 9.47 |
101
+ | `static` | `['Add', 'MatMul']` | \| | 119.89 | 86.76 | \| | 8.40 | 11.53 |
102
+ | `static` | `['Add']` | \| | 96.99 | 102.03 | \| | 10.33 | 9.87 |
103
+
104
+
105
+ Below, time metrics for batch size = 4, input length = 128.
106
+
107
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
108
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
109
+ | `dynamic` | `['Add', 'MatMul']` | \| | 219.78 | 123.71 | \| | 4.60 | 8.13 |
110
+ | `dynamic` | `['Add']` | \| | 220.13 | 187.21 | \| | 4.60 | 5.40 |
111
+ | `static` | `['Add', 'MatMul']` | \| | 186.39 | 176.99 | \| | 5.40 | 5.67 |
112
+ | `static` | `['Add']` | \| | 219.57 | 203.71 | \| | 4.60 | 4.93 |
113
+
114
+
115
+ Below, time metrics for batch size = 8, input length = 32.
116
+
117
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
118
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
119
+ | `dynamic` | `['Add', 'MatMul']` | \| | 118.32 | 59.22 | \| | 8.47 | 16.93 |
120
+ | `dynamic` | `['Add']` | \| | 116.52 | 80.17 | \| | 8.60 | 12.53 |
121
+ | `static` | `['Add', 'MatMul']` | \| | 116.59 | 83.55 | \| | 8.60 | 12.00 |
122
+ | `static` | `['Add']` | \| | 115.81 | 126.53 | \| | 8.67 | 7.93 |
123
+
124
+
125
+ Below, time metrics for batch size = 8, input length = 64.
126
+
127
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
128
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
129
+ | `dynamic` | `['Add', 'MatMul']` | \| | 172.71 | 117.89 | \| | 5.80 | 8.53 |
130
+ | `dynamic` | `['Add']` | \| | 166.05 | 156.99 | \| | 6.07 | 6.40 |
131
+ | `static` | `['Add', 'MatMul']` | \| | 215.00 | 148.93 | \| | 4.67 | 6.73 |
132
+ | `static` | `['Add']` | \| | 214.55 | 200.16 | \| | 4.67 | 5.00 |
133
+
134
+
135
+ Below, time metrics for batch size = 8, input length = 128.
136
+
137
+ | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
138
+ | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
139
+ | `dynamic` | `['Add', 'MatMul']` | \| | 403.69 | 307.36 | \| | 2.53 | 3.27 |
140
+ | `dynamic` | `['Add']` | \| | 372.85 | 317.53 | \| | 2.73 | 3.20 |
141
+ | `static` | `['Add', 'MatMul']` | \| | 352.18 | 320.85 | \| | 2.87 | 3.13 |
142
+ | `static` | `['Add']` | \| | 403.55 | 410.17 | \| | 2.53 | 2.47 |
143
+
runs.json ADDED
@@ -0,0 +1,1204 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
4
+ "task": "token-classification",
5
+ "dataset": {
6
+ "path": "conll2003",
7
+ "eval_split": "validation",
8
+ "data_keys": {
9
+ "primary": "tokens",
10
+ "secondary": null
11
+ },
12
+ "ref_keys": [
13
+ "ner_tags"
14
+ ],
15
+ "name": null,
16
+ "calibration_split": "train"
17
+ },
18
+ "quantization_approach": "static",
19
+ "operators_to_quantize": [
20
+ "Add"
21
+ ],
22
+ "node_exclusion": [],
23
+ "aware_training": false,
24
+ "per_channel": false,
25
+ "calibration": {
26
+ "method": "minmax",
27
+ "num_calibration_samples": 100,
28
+ "calibration_histogram_percentile": null,
29
+ "calibration_moving_average": null,
30
+ "calibration_moving_average_constant": null
31
+ },
32
+ "framework": "onnxruntime",
33
+ "framework_args": {
34
+ "opset": 11,
35
+ "optimization_level": 1
36
+ },
37
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3099.998\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
38
+ "versions": {
39
+ "transformers": "4.20.1",
40
+ "optimum": "1.2.3.dev0",
41
+ "optimum_hash": "5ac9c0d9fd7e7cca55b2f9935b961ed5b6c50112"
42
+ },
43
+ "evaluation": {
44
+ "time": [
45
+ {
46
+ "batch_size": 8,
47
+ "input_length": 32,
48
+ "baseline": {
49
+ "nb_forwards": 130,
50
+ "throughput": 8.67,
51
+ "latency_mean": 115.81472209230769,
52
+ "latency_std": 0.46895980796384223,
53
+ "latency_50": 115.7693145,
54
+ "latency_90": 116.2641674,
55
+ "latency_95": 116.615633,
56
+ "latency_99": 117.24392198999999,
57
+ "latency_999": 118.33366130000002
58
+ },
59
+ "optimized": {
60
+ "nb_forwards": 119,
61
+ "throughput": 7.93,
62
+ "latency_mean": 126.53206931932773,
63
+ "latency_std": 0.21043917502896844,
64
+ "latency_50": 126.51378,
65
+ "latency_90": 126.8398328,
66
+ "latency_95": 126.8832745,
67
+ "latency_99": 127.04870174,
68
+ "latency_999": 127.236541298
69
+ }
70
+ },
71
+ {
72
+ "batch_size": 4,
73
+ "input_length": 128,
74
+ "baseline": {
75
+ "nb_forwards": 69,
76
+ "throughput": 4.6,
77
+ "latency_mean": 219.5741049275362,
78
+ "latency_std": 0.425956641594441,
79
+ "latency_50": 219.571816,
80
+ "latency_90": 220.0821694,
81
+ "latency_95": 220.1998948,
82
+ "latency_99": 220.53578456,
83
+ "latency_999": 220.90863515599997
84
+ },
85
+ "optimized": {
86
+ "nb_forwards": 74,
87
+ "throughput": 4.93,
88
+ "latency_mean": 203.70586305405405,
89
+ "latency_std": 4.723902925243521,
90
+ "latency_50": 202.578965,
91
+ "latency_90": 211.061159,
92
+ "latency_95": 213.91898604999997,
93
+ "latency_99": 216.57558472,
94
+ "latency_999": 216.849314572
95
+ }
96
+ },
97
+ {
98
+ "batch_size": 4,
99
+ "input_length": 64,
100
+ "baseline": {
101
+ "nb_forwards": 155,
102
+ "throughput": 10.33,
103
+ "latency_mean": 96.98536739354839,
104
+ "latency_std": 13.276058953171962,
105
+ "latency_50": 90.032648,
106
+ "latency_90": 119.895212,
107
+ "latency_95": 120.1166701,
108
+ "latency_99": 121.32664904,
109
+ "latency_999": 122.10800528
110
+ },
111
+ "optimized": {
112
+ "nb_forwards": 148,
113
+ "throughput": 9.87,
114
+ "latency_mean": 102.02715479054055,
115
+ "latency_std": 2.9594884849383276,
116
+ "latency_50": 101.4796865,
117
+ "latency_90": 105.83677709999999,
118
+ "latency_95": 107.30721310000001,
119
+ "latency_99": 110.73326877,
120
+ "latency_999": 111.11521851100001
121
+ }
122
+ },
123
+ {
124
+ "batch_size": 8,
125
+ "input_length": 128,
126
+ "baseline": {
127
+ "nb_forwards": 38,
128
+ "throughput": 2.53,
129
+ "latency_mean": 403.54761702631583,
130
+ "latency_std": 0.7543435235070463,
131
+ "latency_50": 403.5543235,
132
+ "latency_90": 404.57511619999997,
133
+ "latency_95": 404.74615289999997,
134
+ "latency_99": 404.80311413,
135
+ "latency_999": 404.825042513
136
+ },
137
+ "optimized": {
138
+ "nb_forwards": 37,
139
+ "throughput": 2.47,
140
+ "latency_mean": 410.16972697297297,
141
+ "latency_std": 7.6545384613060605,
142
+ "latency_50": 409.333633,
143
+ "latency_90": 420.4970922,
144
+ "latency_95": 423.82144059999996,
145
+ "latency_99": 426.282015,
146
+ "latency_999": 426.663606
147
+ }
148
+ },
149
+ {
150
+ "batch_size": 8,
151
+ "input_length": 64,
152
+ "baseline": {
153
+ "nb_forwards": 70,
154
+ "throughput": 4.67,
155
+ "latency_mean": 214.5513372142857,
156
+ "latency_std": 0.44768920996368816,
157
+ "latency_50": 214.507338,
158
+ "latency_90": 215.1284881,
159
+ "latency_95": 215.26822455,
160
+ "latency_99": 215.65146749000002,
161
+ "latency_999": 215.97797624900002
162
+ },
163
+ "optimized": {
164
+ "nb_forwards": 75,
165
+ "throughput": 5.0,
166
+ "latency_mean": 200.16238184,
167
+ "latency_std": 9.040248673052057,
168
+ "latency_50": 196.89438,
169
+ "latency_90": 214.99417680000002,
170
+ "latency_95": 218.1261396,
171
+ "latency_99": 223.02508824,
172
+ "latency_999": 223.85004512400002
173
+ }
174
+ },
175
+ {
176
+ "batch_size": 4,
177
+ "input_length": 32,
178
+ "baseline": {
179
+ "nb_forwards": 212,
180
+ "throughput": 14.13,
181
+ "latency_mean": 71.02946175,
182
+ "latency_std": 0.3401815134935371,
183
+ "latency_50": 71.0171625,
184
+ "latency_90": 71.3381363,
185
+ "latency_95": 71.41140920000001,
186
+ "latency_99": 71.62187967,
187
+ "latency_999": 74.00790798399997
188
+ },
189
+ "optimized": {
190
+ "nb_forwards": 265,
191
+ "throughput": 17.67,
192
+ "latency_mean": 56.817175411320754,
193
+ "latency_std": 3.6329155653183585,
194
+ "latency_50": 55.596164,
195
+ "latency_90": 60.593704,
196
+ "latency_95": 66.950391,
197
+ "latency_99": 67.13051659999999,
198
+ "latency_999": 67.23656778399999
199
+ }
200
+ },
201
+ {
202
+ "batch_size": 1,
203
+ "input_length": 32,
204
+ "baseline": {
205
+ "nb_forwards": 452,
206
+ "throughput": 30.13,
207
+ "latency_mean": 33.22745787610619,
208
+ "latency_std": 1.0543225342336746,
209
+ "latency_50": 33.078952,
210
+ "latency_90": 35.073573100000004,
211
+ "latency_95": 35.4359905,
212
+ "latency_99": 35.666321880000005,
213
+ "latency_999": 36.114601445999995
214
+ },
215
+ "optimized": {
216
+ "nb_forwards": 746,
217
+ "throughput": 49.73,
218
+ "latency_mean": 20.109513683646114,
219
+ "latency_std": 1.6406645419149348,
220
+ "latency_50": 20.2270735,
221
+ "latency_90": 21.831147,
222
+ "latency_95": 21.8876735,
223
+ "latency_99": 22.079700099999997,
224
+ "latency_999": 22.4666168
225
+ }
226
+ },
227
+ {
228
+ "batch_size": 1,
229
+ "input_length": 64,
230
+ "baseline": {
231
+ "nb_forwards": 340,
232
+ "throughput": 22.67,
233
+ "latency_mean": 44.21631953823529,
234
+ "latency_std": 5.107156057378748,
235
+ "latency_50": 42.547523,
236
+ "latency_90": 45.71804530000001,
237
+ "latency_95": 59.850769,
238
+ "latency_99": 61.57001349,
239
+ "latency_999": 62.189666854
240
+ },
241
+ "optimized": {
242
+ "nb_forwards": 389,
243
+ "throughput": 25.93,
244
+ "latency_mean": 38.56483639331619,
245
+ "latency_std": 0.11123736681980873,
246
+ "latency_50": 38.556444,
247
+ "latency_90": 38.712917,
248
+ "latency_95": 38.7594766,
249
+ "latency_99": 38.92545156,
250
+ "latency_999": 39.024601432000004
251
+ }
252
+ },
253
+ {
254
+ "batch_size": 1,
255
+ "input_length": 128,
256
+ "baseline": {
257
+ "nb_forwards": 207,
258
+ "throughput": 13.8,
259
+ "latency_mean": 72.4693660772947,
260
+ "latency_std": 0.8661223930705324,
261
+ "latency_50": 72.429662,
262
+ "latency_90": 73.5786186,
263
+ "latency_95": 74.0005967,
264
+ "latency_99": 74.67420342,
265
+ "latency_999": 74.911409554
266
+ },
267
+ "optimized": {
268
+ "nb_forwards": 211,
269
+ "throughput": 14.07,
270
+ "latency_mean": 71.1041596492891,
271
+ "latency_std": 0.28805137660931524,
272
+ "latency_50": 71.03339,
273
+ "latency_90": 71.471796,
274
+ "latency_95": 71.633065,
275
+ "latency_99": 72.0733595,
276
+ "latency_999": 72.60335828
277
+ }
278
+ }
279
+ ],
280
+ "others": {
281
+ "baseline": {
282
+ "precision": 0.9358012339503085,
283
+ "recall": 0.9444631437226523,
284
+ "f1": 0.9401122372057961,
285
+ "accuracy": 0.9882013940267124
286
+ },
287
+ "optimized": {
288
+ "precision": 0.05001838911364472,
289
+ "recall": 0.16021541568495457,
290
+ "f1": 0.07623623623623622,
291
+ "accuracy": 0.31141700089560376
292
+ }
293
+ }
294
+ },
295
+ "max_eval_samples": null,
296
+ "time_benchmark_args": {
297
+ "duration": 15,
298
+ "warmup_runs": 5
299
+ },
300
+ "model_type": "distilbert"
301
+ },
302
+ {
303
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
304
+ "task": "token-classification",
305
+ "dataset": {
306
+ "path": "conll2003",
307
+ "eval_split": "validation",
308
+ "data_keys": {
309
+ "primary": "tokens",
310
+ "secondary": null
311
+ },
312
+ "ref_keys": [
313
+ "ner_tags"
314
+ ],
315
+ "name": null,
316
+ "calibration_split": "train"
317
+ },
318
+ "quantization_approach": "dynamic",
319
+ "operators_to_quantize": [
320
+ "Add",
321
+ "MatMul"
322
+ ],
323
+ "node_exclusion": [],
324
+ "aware_training": false,
325
+ "per_channel": false,
326
+ "calibration": {
327
+ "method": "minmax",
328
+ "num_calibration_samples": 100,
329
+ "calibration_histogram_percentile": null,
330
+ "calibration_moving_average": null,
331
+ "calibration_moving_average_constant": null
332
+ },
333
+ "framework": "onnxruntime",
334
+ "framework_args": {
335
+ "opset": 11,
336
+ "optimization_level": 1
337
+ },
338
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3106.018\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
339
+ "versions": {
340
+ "transformers": "4.20.1",
341
+ "optimum": "1.2.3.dev0",
342
+ "optimum_hash": "5ac9c0d9fd7e7cca55b2f9935b961ed5b6c50112"
343
+ },
344
+ "evaluation": {
345
+ "time": [
346
+ {
347
+ "batch_size": 4,
348
+ "input_length": 64,
349
+ "baseline": {
350
+ "nb_forwards": 126,
351
+ "throughput": 8.4,
352
+ "latency_mean": 119.91479441269841,
353
+ "latency_std": 0.2629922737356049,
354
+ "latency_50": 119.9039375,
355
+ "latency_90": 120.2240655,
356
+ "latency_95": 120.3983675,
357
+ "latency_99": 120.61099275,
358
+ "latency_999": 120.725017125
359
+ },
360
+ "optimized": {
361
+ "nb_forwards": 244,
362
+ "throughput": 16.27,
363
+ "latency_mean": 61.50554389344262,
364
+ "latency_std": 0.9316797843470712,
365
+ "latency_50": 61.3989675,
366
+ "latency_90": 62.743792,
367
+ "latency_95": 63.1351423,
368
+ "latency_99": 64.33112306,
369
+ "latency_999": 64.851473252
370
+ }
371
+ },
372
+ {
373
+ "batch_size": 8,
374
+ "input_length": 128,
375
+ "baseline": {
376
+ "nb_forwards": 38,
377
+ "throughput": 2.53,
378
+ "latency_mean": 403.6921285,
379
+ "latency_std": 0.8331604984169664,
380
+ "latency_50": 403.615553,
381
+ "latency_90": 404.5933473,
382
+ "latency_95": 404.73024205,
383
+ "latency_99": 405.73243518,
384
+ "latency_999": 406.206855618
385
+ },
386
+ "optimized": {
387
+ "nb_forwards": 49,
388
+ "throughput": 3.27,
389
+ "latency_mean": 307.35862414285714,
390
+ "latency_std": 37.75323130347674,
391
+ "latency_50": 328.22334,
392
+ "latency_90": 329.4804032,
393
+ "latency_95": 329.88989580000003,
394
+ "latency_99": 331.99058364,
395
+ "latency_999": 332.89210586400003
396
+ }
397
+ },
398
+ {
399
+ "batch_size": 8,
400
+ "input_length": 64,
401
+ "baseline": {
402
+ "nb_forwards": 87,
403
+ "throughput": 5.8,
404
+ "latency_mean": 172.70831874712644,
405
+ "latency_std": 16.93330536049502,
406
+ "latency_50": 165.631011,
407
+ "latency_90": 214.729821,
408
+ "latency_95": 215.7495264,
409
+ "latency_99": 219.2165974,
410
+ "latency_999": 219.69907804
411
+ },
412
+ "optimized": {
413
+ "nb_forwards": 128,
414
+ "throughput": 8.53,
415
+ "latency_mean": 117.8890106953125,
416
+ "latency_std": 2.0181256632154128,
417
+ "latency_50": 117.854545,
418
+ "latency_90": 120.3485038,
419
+ "latency_95": 121.1920466,
420
+ "latency_99": 122.86635173,
421
+ "latency_999": 123.87748661900001
422
+ }
423
+ },
424
+ {
425
+ "batch_size": 4,
426
+ "input_length": 128,
427
+ "baseline": {
428
+ "nb_forwards": 69,
429
+ "throughput": 4.6,
430
+ "latency_mean": 219.78011085507248,
431
+ "latency_std": 0.8383066531914855,
432
+ "latency_50": 219.682452,
433
+ "latency_90": 220.6735604,
434
+ "latency_95": 221.179578,
435
+ "latency_99": 222.15356576,
436
+ "latency_999": 223.052069276
437
+ },
438
+ "optimized": {
439
+ "nb_forwards": 122,
440
+ "throughput": 8.13,
441
+ "latency_mean": 123.70655378688525,
442
+ "latency_std": 2.0832274392948285,
443
+ "latency_50": 123.7493135,
444
+ "latency_90": 126.5891177,
445
+ "latency_95": 127.5693857,
446
+ "latency_99": 128.41928172,
447
+ "latency_999": 129.29519896800002
448
+ }
449
+ },
450
+ {
451
+ "batch_size": 4,
452
+ "input_length": 32,
453
+ "baseline": {
454
+ "nb_forwards": 216,
455
+ "throughput": 14.4,
456
+ "latency_mean": 69.7580692962963,
457
+ "latency_std": 0.7696003955372718,
458
+ "latency_50": 69.544575,
459
+ "latency_90": 70.7143,
460
+ "latency_95": 71.6627545,
461
+ "latency_99": 72.7330374,
462
+ "latency_999": 73.26205913
463
+ },
464
+ "optimized": {
465
+ "nb_forwards": 390,
466
+ "throughput": 26.0,
467
+ "latency_mean": 38.49885642307692,
468
+ "latency_std": 3.3170930332820334,
469
+ "latency_50": 40.230039,
470
+ "latency_90": 40.3790612,
471
+ "latency_95": 40.428531899999996,
472
+ "latency_99": 40.510404130000005,
473
+ "latency_999": 40.614662347
474
+ }
475
+ },
476
+ {
477
+ "batch_size": 1,
478
+ "input_length": 128,
479
+ "baseline": {
480
+ "nb_forwards": 208,
481
+ "throughput": 13.87,
482
+ "latency_mean": 72.37936482211539,
483
+ "latency_std": 0.24212622373713247,
484
+ "latency_50": 72.3947265,
485
+ "latency_90": 72.6858948,
486
+ "latency_95": 72.75088670000001,
487
+ "latency_99": 72.92894586,
488
+ "latency_999": 73.01229876299999
489
+ },
490
+ "optimized": {
491
+ "nb_forwards": 412,
492
+ "throughput": 27.47,
493
+ "latency_mean": 36.4657384684466,
494
+ "latency_std": 3.4465598552729446,
495
+ "latency_50": 34.966474,
496
+ "latency_90": 43.4717706,
497
+ "latency_95": 43.6103869,
498
+ "latency_99": 43.74503971,
499
+ "latency_999": 44.051604852
500
+ }
501
+ },
502
+ {
503
+ "batch_size": 8,
504
+ "input_length": 32,
505
+ "baseline": {
506
+ "nb_forwards": 127,
507
+ "throughput": 8.47,
508
+ "latency_mean": 118.31688400787401,
509
+ "latency_std": 1.1539171140975781,
510
+ "latency_50": 118.215119,
511
+ "latency_90": 119.8292888,
512
+ "latency_95": 120.60338259999999,
513
+ "latency_99": 121.44057798,
514
+ "latency_999": 122.68986890599999
515
+ },
516
+ "optimized": {
517
+ "nb_forwards": 254,
518
+ "throughput": 16.93,
519
+ "latency_mean": 59.2173072007874,
520
+ "latency_std": 0.7459868576638755,
521
+ "latency_50": 59.1816355,
522
+ "latency_90": 60.196821299999996,
523
+ "latency_95": 60.5587619,
524
+ "latency_99": 60.966352210000004,
525
+ "latency_999": 61.412101875
526
+ }
527
+ },
528
+ {
529
+ "batch_size": 1,
530
+ "input_length": 64,
531
+ "baseline": {
532
+ "nb_forwards": 255,
533
+ "throughput": 17.0,
534
+ "latency_mean": 58.91992950980392,
535
+ "latency_std": 0.8869192739714911,
536
+ "latency_50": 58.910515,
537
+ "latency_90": 59.9905808,
538
+ "latency_95": 60.4006188,
539
+ "latency_99": 61.36197958,
540
+ "latency_999": 62.78803392800001
541
+ },
542
+ "optimized": {
543
+ "nb_forwards": 763,
544
+ "throughput": 50.87,
545
+ "latency_mean": 19.684674589777195,
546
+ "latency_std": 1.8042745579532666,
547
+ "latency_50": 19.157213,
548
+ "latency_90": 22.2412244,
549
+ "latency_95": 22.3211958,
550
+ "latency_99": 22.58112282,
551
+ "latency_999": 26.754026579999863
552
+ }
553
+ },
554
+ {
555
+ "batch_size": 1,
556
+ "input_length": 32,
557
+ "baseline": {
558
+ "nb_forwards": 324,
559
+ "throughput": 21.6,
560
+ "latency_mean": 46.3835297962963,
561
+ "latency_std": 1.1343614396581607,
562
+ "latency_50": 46.442469,
563
+ "latency_90": 47.1782314,
564
+ "latency_95": 47.4329425,
565
+ "latency_99": 48.049914599999994,
566
+ "latency_999": 50.92608084600004
567
+ },
568
+ "optimized": {
569
+ "nb_forwards": 1507,
570
+ "throughput": 100.47,
571
+ "latency_mean": 9.958045324485735,
572
+ "latency_std": 0.2901405767073424,
573
+ "latency_50": 9.92078,
574
+ "latency_90": 10.4131178,
575
+ "latency_95": 10.5542065,
576
+ "latency_99": 10.709247679999999,
577
+ "latency_999": 10.978874069999991
578
+ }
579
+ }
580
+ ],
581
+ "others": {
582
+ "baseline": {
583
+ "precision": 0.9358012339503085,
584
+ "recall": 0.9444631437226523,
585
+ "f1": 0.9401122372057961,
586
+ "accuracy": 0.9882013940267124
587
+ },
588
+ "optimized": {
589
+ "precision": 0.934913217623498,
590
+ "recall": 0.9427802086839449,
591
+ "f1": 0.9388302329478799,
592
+ "accuracy": 0.9877341225030178
593
+ }
594
+ }
595
+ },
596
+ "max_eval_samples": null,
597
+ "time_benchmark_args": {
598
+ "duration": 15,
599
+ "warmup_runs": 5
600
+ },
601
+ "model_type": "distilbert"
602
+ },
603
+ {
604
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
605
+ "task": "token-classification",
606
+ "dataset": {
607
+ "path": "conll2003",
608
+ "eval_split": "validation",
609
+ "data_keys": {
610
+ "primary": "tokens",
611
+ "secondary": null
612
+ },
613
+ "ref_keys": [
614
+ "ner_tags"
615
+ ],
616
+ "name": null,
617
+ "calibration_split": "train"
618
+ },
619
+ "quantization_approach": "static",
620
+ "operators_to_quantize": [
621
+ "Add",
622
+ "MatMul"
623
+ ],
624
+ "node_exclusion": [],
625
+ "aware_training": false,
626
+ "per_channel": false,
627
+ "calibration": {
628
+ "method": "minmax",
629
+ "num_calibration_samples": 100,
630
+ "calibration_histogram_percentile": null,
631
+ "calibration_moving_average": null,
632
+ "calibration_moving_average_constant": null
633
+ },
634
+ "framework": "onnxruntime",
635
+ "framework_args": {
636
+ "opset": 11,
637
+ "optimization_level": 1
638
+ },
639
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3101.721\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
640
+ "versions": {
641
+ "transformers": "4.20.1",
642
+ "optimum": "1.2.3.dev0",
643
+ "optimum_hash": "5ac9c0d9fd7e7cca55b2f9935b961ed5b6c50112"
644
+ },
645
+ "evaluation": {
646
+ "time": [
647
+ {
648
+ "batch_size": 1,
649
+ "input_length": 32,
650
+ "baseline": {
651
+ "nb_forwards": 444,
652
+ "throughput": 29.6,
653
+ "latency_mean": 33.835281761261264,
654
+ "latency_std": 1.1104691070719008,
655
+ "latency_50": 33.7167165,
656
+ "latency_90": 35.5099667,
657
+ "latency_95": 35.79827469999999,
658
+ "latency_99": 36.20388337,
659
+ "latency_999": 36.403486599
660
+ },
661
+ "optimized": {
662
+ "nb_forwards": 1038,
663
+ "throughput": 69.2,
664
+ "latency_mean": 14.456874846820808,
665
+ "latency_std": 1.147945670896925,
666
+ "latency_50": 15.261894,
667
+ "latency_90": 15.4090676,
668
+ "latency_95": 15.44100315,
669
+ "latency_99": 15.5019354,
670
+ "latency_999": 16.029519071999996
671
+ }
672
+ },
673
+ {
674
+ "batch_size": 1,
675
+ "input_length": 64,
676
+ "baseline": {
677
+ "nb_forwards": 292,
678
+ "throughput": 19.47,
679
+ "latency_mean": 51.40666307191781,
680
+ "latency_std": 7.904660749243075,
681
+ "latency_50": 57.9910805,
682
+ "latency_90": 59.4403903,
683
+ "latency_95": 59.64796875,
684
+ "latency_99": 60.01921038999999,
685
+ "latency_999": 62.362501826000006
686
+ },
687
+ "optimized": {
688
+ "nb_forwards": 511,
689
+ "throughput": 34.07,
690
+ "latency_mean": 29.36470456360078,
691
+ "latency_std": 0.09446229792346662,
692
+ "latency_50": 29.353048,
693
+ "latency_90": 29.482019,
694
+ "latency_95": 29.53371,
695
+ "latency_99": 29.669687599999996,
696
+ "latency_999": 29.82210045
697
+ }
698
+ },
699
+ {
700
+ "batch_size": 4,
701
+ "input_length": 128,
702
+ "baseline": {
703
+ "nb_forwards": 81,
704
+ "throughput": 5.4,
705
+ "latency_mean": 186.3935255308642,
706
+ "latency_std": 23.66920102974093,
707
+ "latency_50": 170.663335,
708
+ "latency_90": 219.886297,
709
+ "latency_95": 220.412333,
710
+ "latency_99": 221.20538580000002,
711
+ "latency_999": 222.11007948
712
+ },
713
+ "optimized": {
714
+ "nb_forwards": 85,
715
+ "throughput": 5.67,
716
+ "latency_mean": 176.9935626235294,
717
+ "latency_std": 24.611926259834373,
718
+ "latency_50": 162.989882,
719
+ "latency_90": 205.825854,
720
+ "latency_95": 206.062911,
721
+ "latency_99": 206.69577235999998,
722
+ "latency_999": 207.115859636
723
+ }
724
+ },
725
+ {
726
+ "batch_size": 8,
727
+ "input_length": 64,
728
+ "baseline": {
729
+ "nb_forwards": 70,
730
+ "throughput": 4.67,
731
+ "latency_mean": 215.00074108571428,
732
+ "latency_std": 0.9705650988667485,
733
+ "latency_50": 214.867633,
734
+ "latency_90": 215.72163830000002,
735
+ "latency_95": 216.53323115,
736
+ "latency_99": 218.93241332,
737
+ "latency_999": 220.051872632
738
+ },
739
+ "optimized": {
740
+ "nb_forwards": 101,
741
+ "throughput": 6.73,
742
+ "latency_mean": 148.9348962970297,
743
+ "latency_std": 3.9263502299131137,
744
+ "latency_50": 148.342845,
745
+ "latency_90": 155.106506,
746
+ "latency_95": 156.211825,
747
+ "latency_99": 158.331491,
748
+ "latency_999": 161.1917459
749
+ }
750
+ },
751
+ {
752
+ "batch_size": 1,
753
+ "input_length": 128,
754
+ "baseline": {
755
+ "nb_forwards": 212,
756
+ "throughput": 14.13,
757
+ "latency_mean": 70.75742881132075,
758
+ "latency_std": 1.2469446939770699,
759
+ "latency_50": 70.660216,
760
+ "latency_90": 72.6529741,
761
+ "latency_95": 72.88815559999999,
762
+ "latency_99": 73.11715371,
763
+ "latency_999": 73.644905982
764
+ },
765
+ "optimized": {
766
+ "nb_forwards": 312,
767
+ "throughput": 20.8,
768
+ "latency_mean": 48.23616327884615,
769
+ "latency_std": 4.851493694643943,
770
+ "latency_50": 45.751592,
771
+ "latency_90": 55.8910255,
772
+ "latency_95": 56.0878071,
773
+ "latency_99": 56.63496909,
774
+ "latency_999": 56.748669404
775
+ }
776
+ },
777
+ {
778
+ "batch_size": 8,
779
+ "input_length": 128,
780
+ "baseline": {
781
+ "nb_forwards": 43,
782
+ "throughput": 2.87,
783
+ "latency_mean": 352.17799074418605,
784
+ "latency_std": 48.791933253583736,
785
+ "latency_50": 312.088024,
786
+ "latency_90": 406.3363032,
787
+ "latency_95": 407.0272163,
788
+ "latency_99": 409.34203474000003,
789
+ "latency_999": 410.012702374
790
+ },
791
+ "optimized": {
792
+ "nb_forwards": 47,
793
+ "throughput": 3.13,
794
+ "latency_mean": 320.8497927446808,
795
+ "latency_std": 37.045737718320964,
796
+ "latency_50": 304.99127,
797
+ "latency_90": 407.9567778,
798
+ "latency_95": 408.6023321,
799
+ "latency_99": 409.3466477,
800
+ "latency_999": 409.53851807
801
+ }
802
+ },
803
+ {
804
+ "batch_size": 4,
805
+ "input_length": 32,
806
+ "baseline": {
807
+ "nb_forwards": 273,
808
+ "throughput": 18.2,
809
+ "latency_mean": 55.05481489377289,
810
+ "latency_std": 0.8315777141405638,
811
+ "latency_50": 54.970724,
812
+ "latency_90": 56.1484644,
813
+ "latency_95": 56.430772600000005,
814
+ "latency_99": 56.85986168,
815
+ "latency_999": 57.515896824
816
+ },
817
+ "optimized": {
818
+ "nb_forwards": 321,
819
+ "throughput": 21.4,
820
+ "latency_mean": 46.79807228660436,
821
+ "latency_std": 4.841315127921891,
822
+ "latency_50": 46.18684,
823
+ "latency_90": 51.680476,
824
+ "latency_95": 51.742804,
825
+ "latency_99": 51.917291399999996,
826
+ "latency_999": 52.1734384
827
+ }
828
+ },
829
+ {
830
+ "batch_size": 8,
831
+ "input_length": 32,
832
+ "baseline": {
833
+ "nb_forwards": 129,
834
+ "throughput": 8.6,
835
+ "latency_mean": 116.59177039534883,
836
+ "latency_std": 0.5224827890552791,
837
+ "latency_50": 116.509951,
838
+ "latency_90": 116.8391962,
839
+ "latency_95": 117.141577,
840
+ "latency_99": 119.15936884,
841
+ "latency_999": 120.545900664
842
+ },
843
+ "optimized": {
844
+ "nb_forwards": 180,
845
+ "throughput": 12.0,
846
+ "latency_mean": 83.54651290555556,
847
+ "latency_std": 9.983735757677245,
848
+ "latency_50": 78.13343,
849
+ "latency_90": 97.8848908,
850
+ "latency_95": 98.02936915000001,
851
+ "latency_99": 98.11533426000001,
852
+ "latency_999": 98.19551898499999
853
+ }
854
+ },
855
+ {
856
+ "batch_size": 4,
857
+ "input_length": 64,
858
+ "baseline": {
859
+ "nb_forwards": 126,
860
+ "throughput": 8.4,
861
+ "latency_mean": 119.892617,
862
+ "latency_std": 0.2462536509536135,
863
+ "latency_50": 119.8820935,
864
+ "latency_90": 120.218857,
865
+ "latency_95": 120.33222325,
866
+ "latency_99": 120.4118515,
867
+ "latency_999": 120.443628
868
+ },
869
+ "optimized": {
870
+ "nb_forwards": 173,
871
+ "throughput": 11.53,
872
+ "latency_mean": 86.75648606936416,
873
+ "latency_std": 10.203153981382503,
874
+ "latency_50": 81.471955,
875
+ "latency_90": 100.5808368,
876
+ "latency_95": 100.84913420000001,
877
+ "latency_99": 101.67875240000001,
878
+ "latency_999": 102.108927452
879
+ }
880
+ }
881
+ ],
882
+ "others": {
883
+ "baseline": {
884
+ "precision": 0.9358012339503085,
885
+ "recall": 0.9444631437226523,
886
+ "f1": 0.9401122372057961,
887
+ "accuracy": 0.9882013940267124
888
+ },
889
+ "optimized": {
890
+ "precision": 0.06309066586113149,
891
+ "recall": 0.24604510265903737,
892
+ "f1": 0.10042933195947105,
893
+ "accuracy": 0.34301623768544837
894
+ }
895
+ }
896
+ },
897
+ "max_eval_samples": null,
898
+ "time_benchmark_args": {
899
+ "duration": 15,
900
+ "warmup_runs": 5
901
+ },
902
+ "model_type": "distilbert"
903
+ },
904
+ {
905
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
906
+ "task": "token-classification",
907
+ "dataset": {
908
+ "path": "conll2003",
909
+ "eval_split": "validation",
910
+ "data_keys": {
911
+ "primary": "tokens",
912
+ "secondary": null
913
+ },
914
+ "ref_keys": [
915
+ "ner_tags"
916
+ ],
917
+ "name": null,
918
+ "calibration_split": "train"
919
+ },
920
+ "quantization_approach": "dynamic",
921
+ "operators_to_quantize": [
922
+ "Add"
923
+ ],
924
+ "node_exclusion": [],
925
+ "aware_training": false,
926
+ "per_channel": false,
927
+ "calibration": {
928
+ "method": "minmax",
929
+ "num_calibration_samples": 100,
930
+ "calibration_histogram_percentile": null,
931
+ "calibration_moving_average": null,
932
+ "calibration_moving_average_constant": null
933
+ },
934
+ "framework": "onnxruntime",
935
+ "framework_args": {
936
+ "opset": 11,
937
+ "optimization_level": 1
938
+ },
939
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3121.536\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke\n",
940
+ "versions": {
941
+ "transformers": "4.20.1",
942
+ "optimum": "1.2.3.dev0",
943
+ "optimum_hash": "5ac9c0d9fd7e7cca55b2f9935b961ed5b6c50112"
944
+ },
945
+ "evaluation": {
946
+ "time": [
947
+ {
948
+ "batch_size": 1,
949
+ "input_length": 64,
950
+ "baseline": {
951
+ "nb_forwards": 257,
952
+ "throughput": 17.13,
953
+ "latency_mean": 58.59407656031129,
954
+ "latency_std": 0.5082693082988551,
955
+ "latency_50": 58.62559,
956
+ "latency_90": 59.226632,
957
+ "latency_95": 59.3844464,
958
+ "latency_99": 59.66381776,
959
+ "latency_999": 60.193005848
960
+ },
961
+ "optimized": {
962
+ "nb_forwards": 605,
963
+ "throughput": 40.33,
964
+ "latency_mean": 24.812764945454546,
965
+ "latency_std": 1.8257073532356165,
966
+ "latency_50": 24.192906,
967
+ "latency_90": 29.2776762,
968
+ "latency_95": 29.361327,
969
+ "latency_99": 29.531946960000003,
970
+ "latency_999": 29.634380991999997
971
+ }
972
+ },
973
+ {
974
+ "batch_size": 8,
975
+ "input_length": 32,
976
+ "baseline": {
977
+ "nb_forwards": 129,
978
+ "throughput": 8.6,
979
+ "latency_mean": 116.51833388372093,
980
+ "latency_std": 0.19355545540306737,
981
+ "latency_50": 116.510617,
982
+ "latency_90": 116.76812059999999,
983
+ "latency_95": 116.8346944,
984
+ "latency_99": 116.91619076,
985
+ "latency_999": 117.070623216
986
+ },
987
+ "optimized": {
988
+ "nb_forwards": 188,
989
+ "throughput": 12.53,
990
+ "latency_mean": 80.1677380851064,
991
+ "latency_std": 1.8148568791642794,
992
+ "latency_50": 79.986953,
993
+ "latency_90": 82.70380990000001,
994
+ "latency_95": 83.21852025,
995
+ "latency_99": 84.57913724,
996
+ "latency_999": 84.697106648
997
+ }
998
+ },
999
+ {
1000
+ "batch_size": 1,
1001
+ "input_length": 32,
1002
+ "baseline": {
1003
+ "nb_forwards": 410,
1004
+ "throughput": 27.33,
1005
+ "latency_mean": 36.59453311707317,
1006
+ "latency_std": 5.201952859743346,
1007
+ "latency_50": 34.5124805,
1008
+ "latency_90": 46.8082206,
1009
+ "latency_95": 47.39390375,
1010
+ "latency_99": 47.86114868,
1011
+ "latency_999": 48.050237836
1012
+ },
1013
+ "optimized": {
1014
+ "nb_forwards": 1074,
1015
+ "throughput": 71.6,
1016
+ "latency_mean": 13.975352912476723,
1017
+ "latency_std": 0.2608975931098767,
1018
+ "latency_50": 13.9295265,
1019
+ "latency_90": 14.3837001,
1020
+ "latency_95": 14.4975223,
1021
+ "latency_99": 14.66426481,
1022
+ "latency_999": 14.912761724999998
1023
+ }
1024
+ },
1025
+ {
1026
+ "batch_size": 8,
1027
+ "input_length": 64,
1028
+ "baseline": {
1029
+ "nb_forwards": 91,
1030
+ "throughput": 6.07,
1031
+ "latency_mean": 166.0459372197802,
1032
+ "latency_std": 5.910350195698658,
1033
+ "latency_50": 165.351766,
1034
+ "latency_90": 168.122746,
1035
+ "latency_95": 169.064614,
1036
+ "latency_99": 189.41977489999985,
1037
+ "latency_999": 211.91127088999988
1038
+ },
1039
+ "optimized": {
1040
+ "nb_forwards": 96,
1041
+ "throughput": 6.4,
1042
+ "latency_mean": 156.98839121875,
1043
+ "latency_std": 3.8307406608519075,
1044
+ "latency_50": 156.713151,
1045
+ "latency_90": 161.920169,
1046
+ "latency_95": 162.8535025,
1047
+ "latency_99": 167.0273974,
1048
+ "latency_999": 169.60480174
1049
+ }
1050
+ },
1051
+ {
1052
+ "batch_size": 8,
1053
+ "input_length": 128,
1054
+ "baseline": {
1055
+ "nb_forwards": 41,
1056
+ "throughput": 2.73,
1057
+ "latency_mean": 372.8457580243902,
1058
+ "latency_std": 44.33863922960532,
1059
+ "latency_50": 403.71874,
1060
+ "latency_90": 407.483091,
1061
+ "latency_95": 409.321755,
1062
+ "latency_99": 414.9955262,
1063
+ "latency_999": 415.29942092
1064
+ },
1065
+ "optimized": {
1066
+ "nb_forwards": 48,
1067
+ "throughput": 3.2,
1068
+ "latency_mean": 317.5281361666667,
1069
+ "latency_std": 5.407804800179884,
1070
+ "latency_50": 317.912668,
1071
+ "latency_90": 323.9196422,
1072
+ "latency_95": 325.7003953,
1073
+ "latency_99": 328.84562712,
1074
+ "latency_999": 330.22228231200006
1075
+ }
1076
+ },
1077
+ {
1078
+ "batch_size": 4,
1079
+ "input_length": 128,
1080
+ "baseline": {
1081
+ "nb_forwards": 69,
1082
+ "throughput": 4.6,
1083
+ "latency_mean": 220.1285506231884,
1084
+ "latency_std": 0.9353033698467256,
1085
+ "latency_50": 220.192768,
1086
+ "latency_90": 221.36099019999997,
1087
+ "latency_95": 221.5007656,
1088
+ "latency_99": 222.25962384000002,
1089
+ "latency_999": 222.473907684
1090
+ },
1091
+ "optimized": {
1092
+ "nb_forwards": 81,
1093
+ "throughput": 5.4,
1094
+ "latency_mean": 187.20598314814814,
1095
+ "latency_std": 25.62748524575956,
1096
+ "latency_50": 168.810645,
1097
+ "latency_90": 215.939029,
1098
+ "latency_95": 216.125018,
1099
+ "latency_99": 217.0070662,
1100
+ "latency_999": 217.17069052000002
1101
+ }
1102
+ },
1103
+ {
1104
+ "batch_size": 4,
1105
+ "input_length": 32,
1106
+ "baseline": {
1107
+ "nb_forwards": 268,
1108
+ "throughput": 17.87,
1109
+ "latency_mean": 56.01774754104478,
1110
+ "latency_std": 1.5540596958913682,
1111
+ "latency_50": 55.6834485,
1112
+ "latency_90": 58.264829299999995,
1113
+ "latency_95": 59.282709049999994,
1114
+ "latency_99": 60.64031475,
1115
+ "latency_999": 62.074440195
1116
+ },
1117
+ "optimized": {
1118
+ "nb_forwards": 293,
1119
+ "throughput": 19.53,
1120
+ "latency_mean": 51.320777443686005,
1121
+ "latency_std": 3.776379337439246,
1122
+ "latency_50": 53.174213,
1123
+ "latency_90": 53.340690200000004,
1124
+ "latency_95": 53.408575,
1125
+ "latency_99": 53.4970542,
1126
+ "latency_999": 55.868342540000086
1127
+ }
1128
+ },
1129
+ {
1130
+ "batch_size": 1,
1131
+ "input_length": 128,
1132
+ "baseline": {
1133
+ "nb_forwards": 214,
1134
+ "throughput": 14.27,
1135
+ "latency_mean": 70.20732589719627,
1136
+ "latency_std": 0.8155710627132576,
1137
+ "latency_50": 70.365742,
1138
+ "latency_90": 71.1378453,
1139
+ "latency_95": 71.3491235,
1140
+ "latency_99": 71.89662288,
1141
+ "latency_999": 72.393311452
1142
+ },
1143
+ "optimized": {
1144
+ "nb_forwards": 324,
1145
+ "throughput": 21.6,
1146
+ "latency_mean": 46.297288277777774,
1147
+ "latency_std": 0.7972386433029148,
1148
+ "latency_50": 46.110238,
1149
+ "latency_90": 47.36061,
1150
+ "latency_95": 47.99375455,
1151
+ "latency_99": 48.857902409999994,
1152
+ "latency_999": 50.063431357000006
1153
+ }
1154
+ },
1155
+ {
1156
+ "batch_size": 4,
1157
+ "input_length": 64,
1158
+ "baseline": {
1159
+ "nb_forwards": 139,
1160
+ "throughput": 9.27,
1161
+ "latency_mean": 108.42828852517987,
1162
+ "latency_std": 13.16458394936121,
1163
+ "latency_50": 117.656529,
1164
+ "latency_90": 118.084375,
1165
+ "latency_95": 118.1652478,
1166
+ "latency_99": 118.36095123999999,
1167
+ "latency_999": 118.602071188
1168
+ },
1169
+ "optimized": {
1170
+ "nb_forwards": 142,
1171
+ "throughput": 9.47,
1172
+ "latency_mean": 105.65247052816902,
1173
+ "latency_std": 0.3934774499008662,
1174
+ "latency_50": 105.60453,
1175
+ "latency_90": 105.91063890000001,
1176
+ "latency_95": 106.2638084,
1177
+ "latency_99": 107.39340928,
1178
+ "latency_999": 107.831013231
1179
+ }
1180
+ }
1181
+ ],
1182
+ "others": {
1183
+ "baseline": {
1184
+ "precision": 0.9358012339503085,
1185
+ "recall": 0.9444631437226523,
1186
+ "f1": 0.9401122372057961,
1187
+ "accuracy": 0.9882013940267124
1188
+ },
1189
+ "optimized": {
1190
+ "precision": 0.9358012339503085,
1191
+ "recall": 0.9444631437226523,
1192
+ "f1": 0.9401122372057961,
1193
+ "accuracy": 0.9882013940267124
1194
+ }
1195
+ }
1196
+ },
1197
+ "max_eval_samples": null,
1198
+ "time_benchmark_args": {
1199
+ "duration": 15,
1200
+ "warmup_runs": 5
1201
+ },
1202
+ "model_type": "distilbert"
1203
+ }
1204
+ ]
tensorboard/1657612929.728409/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef1c1054f33d7ed445539c76bd4962f6567eeb016b777e1620321ff262ef3261
3
+ size 738
tensorboard/1657612929.729769/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:522e1f6c28ba9aeb465d2656a85509b5f8bd96a32df69911cb4f7f1f92755b6e
3
+ size 728
tensorboard/1657612929.7311082/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f85039696359034e5e304cce7047355b0955b97dc778fc13f9cefeecd9e30fed
3
+ size 737
tensorboard/1657612929.7323039/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c75844637fbe67027885edb13c4a75d3c15c7102fc78f942365f9895e20d0331
3
+ size 727
tensorboard/events.out.tfevents.1657612929.ip-10-2-70-187.ec2.internal.1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:28b014328832e62c6256dec70e9a131a714f1dc77c95f866c8db005bcb31c596
3
+ size 40