File size: 33,282 Bytes
4525ba5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 |
---
library_name: peft
base_model: peiyi9979/math-shepherd-mistral-7b-prm
tags:
- generated_from_trainer
metrics:
- accuracy
- precision
- recall
- f1
model-index:
- name: v2_mistral_lora
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# v2_mistral_lora
This model is a fine-tuned version of [peiyi9979/math-shepherd-mistral-7b-prm](https://huggingface.co/peiyi9979/math-shepherd-mistral-7b-prm) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0549
- Accuracy: 0.9783
- Precision: 0.9799
- Recall: 0.9466
- F1: 0.9630
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 765837
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 2
- total_train_batch_size: 64
- total_eval_batch_size: 32
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1
### Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy | Precision | Recall | F1 |
|:-------------:|:------:|:----:|:---------------:|:--------:|:---------:|:------:|:------:|
| No log | 0 | 0 | 0.4624 | 0.8307 | 0.9495 | 0.4563 | 0.6164 |
| 0.6147 | 0.0030 | 20 | 0.4628 | 0.8292 | 0.9314 | 0.4612 | 0.6169 |
| 0.5341 | 0.0059 | 40 | 0.4612 | 0.8278 | 0.9394 | 0.4515 | 0.6098 |
| 0.5857 | 0.0089 | 60 | 0.4569 | 0.8278 | 0.9485 | 0.4466 | 0.6073 |
| 0.5562 | 0.0118 | 80 | 0.4473 | 0.8365 | 0.9429 | 0.4806 | 0.6367 |
| 0.5169 | 0.0148 | 100 | 0.4302 | 0.8596 | 0.9658 | 0.5485 | 0.6997 |
| 0.4474 | 0.0177 | 120 | 0.3985 | 0.8712 | 0.9209 | 0.6214 | 0.7420 |
| 0.4423 | 0.0207 | 140 | 0.3567 | 0.8929 | 0.9024 | 0.7184 | 0.8 |
| 0.3734 | 0.0236 | 160 | 0.2948 | 0.9117 | 0.9143 | 0.7767 | 0.8399 |
| 0.3324 | 0.0266 | 180 | 0.2196 | 0.9363 | 0.9219 | 0.8592 | 0.8894 |
| 0.2337 | 0.0295 | 200 | 0.1744 | 0.9522 | 0.9391 | 0.8981 | 0.9181 |
| 0.1528 | 0.0325 | 220 | 0.1573 | 0.9522 | 0.9303 | 0.9078 | 0.9189 |
| 0.3151 | 0.0354 | 240 | 0.1498 | 0.9551 | 0.9310 | 0.9175 | 0.9242 |
| 0.1388 | 0.0384 | 260 | 0.1408 | 0.9566 | 0.9314 | 0.9223 | 0.9268 |
| 0.1405 | 0.0413 | 280 | 0.1342 | 0.9566 | 0.9444 | 0.9078 | 0.9257 |
| 0.093 | 0.0443 | 300 | 0.1380 | 0.9580 | 0.9492 | 0.9078 | 0.9280 |
| 0.1395 | 0.0472 | 320 | 0.1359 | 0.9580 | 0.9492 | 0.9078 | 0.9280 |
| 0.1686 | 0.0502 | 340 | 0.1316 | 0.9638 | 0.9689 | 0.9078 | 0.9373 |
| 0.0542 | 0.0531 | 360 | 0.1270 | 0.9595 | 0.9541 | 0.9078 | 0.9303 |
| 0.131 | 0.0561 | 380 | 0.1170 | 0.9638 | 0.9689 | 0.9078 | 0.9373 |
| 0.1276 | 0.0590 | 400 | 0.1199 | 0.9624 | 0.95 | 0.9223 | 0.9360 |
| 0.0377 | 0.0620 | 420 | 0.1211 | 0.9609 | 0.9453 | 0.9223 | 0.9337 |
| 0.1356 | 0.0649 | 440 | 0.1265 | 0.9609 | 0.9497 | 0.9175 | 0.9333 |
| 0.1409 | 0.0679 | 460 | 0.1059 | 0.9638 | 0.9689 | 0.9078 | 0.9373 |
| 0.0815 | 0.0708 | 480 | 0.1212 | 0.9624 | 0.9455 | 0.9272 | 0.9363 |
| 0.0143 | 0.0738 | 500 | 0.1042 | 0.9638 | 0.9641 | 0.9126 | 0.9377 |
| 0.0877 | 0.0767 | 520 | 0.0976 | 0.9667 | 0.9645 | 0.9223 | 0.9429 |
| 0.1308 | 0.0797 | 540 | 0.1042 | 0.9624 | 0.9545 | 0.9175 | 0.9356 |
| 0.0673 | 0.0826 | 560 | 0.1021 | 0.9653 | 0.9740 | 0.9078 | 0.9397 |
| 0.0763 | 0.0856 | 580 | 0.0967 | 0.9682 | 0.9842 | 0.9078 | 0.9444 |
| 0.0827 | 0.0885 | 600 | 0.0953 | 0.9711 | 0.9844 | 0.9175 | 0.9497 |
| 0.0838 | 0.0915 | 620 | 0.0923 | 0.9711 | 0.9794 | 0.9223 | 0.95 |
| 0.0797 | 0.0944 | 640 | 0.0986 | 0.9682 | 0.9510 | 0.9417 | 0.9463 |
| 0.084 | 0.0974 | 660 | 0.0877 | 0.9682 | 0.9792 | 0.9126 | 0.9447 |
| 0.0091 | 0.1003 | 680 | 0.0856 | 0.9682 | 0.9742 | 0.9175 | 0.945 |
| 0.0832 | 0.1033 | 700 | 0.0913 | 0.9682 | 0.9792 | 0.9126 | 0.9447 |
| 0.0599 | 0.1062 | 720 | 0.0872 | 0.9696 | 0.9793 | 0.9175 | 0.9474 |
| 0.1185 | 0.1092 | 740 | 0.0871 | 0.9638 | 0.9788 | 0.8981 | 0.9367 |
| 0.059 | 0.1121 | 760 | 0.0878 | 0.9653 | 0.9740 | 0.9078 | 0.9397 |
| 0.1429 | 0.1151 | 780 | 0.0868 | 0.9696 | 0.9648 | 0.9320 | 0.9481 |
| 0.0583 | 0.1180 | 800 | 0.0858 | 0.9696 | 0.9648 | 0.9320 | 0.9481 |
| 0.0792 | 0.1210 | 820 | 0.0833 | 0.9682 | 0.9694 | 0.9223 | 0.9453 |
| 0.0746 | 0.1239 | 840 | 0.0772 | 0.9725 | 0.9947 | 0.9126 | 0.9519 |
| 0.0463 | 0.1269 | 860 | 0.0733 | 0.9725 | 0.9795 | 0.9272 | 0.9526 |
| 0.1658 | 0.1298 | 880 | 0.0861 | 0.9696 | 0.9894 | 0.9078 | 0.9468 |
| 0.1177 | 0.1328 | 900 | 0.0815 | 0.9711 | 0.9604 | 0.9417 | 0.9510 |
| 0.0709 | 0.1357 | 920 | 0.0773 | 0.9696 | 0.9793 | 0.9175 | 0.9474 |
| 0.0801 | 0.1387 | 940 | 0.0726 | 0.9740 | 0.9845 | 0.9272 | 0.955 |
| 0.0609 | 0.1416 | 960 | 0.0709 | 0.9754 | 0.9797 | 0.9369 | 0.9578 |
| 0.0865 | 0.1446 | 980 | 0.0696 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0347 | 0.1475 | 1000 | 0.0722 | 0.9725 | 0.9698 | 0.9369 | 0.9531 |
| 0.1003 | 0.1505 | 1020 | 0.0817 | 0.9653 | 0.9505 | 0.9320 | 0.9412 |
| 0.0152 | 0.1534 | 1040 | 0.0774 | 0.9725 | 0.9947 | 0.9126 | 0.9519 |
| 0.0965 | 0.1564 | 1060 | 0.0752 | 0.9754 | 0.9896 | 0.9272 | 0.9574 |
| 0.1153 | 0.1594 | 1080 | 0.0771 | 0.9740 | 0.9947 | 0.9175 | 0.9545 |
| 0.0918 | 0.1623 | 1100 | 0.0798 | 0.9711 | 0.9947 | 0.9078 | 0.9492 |
| 0.0745 | 0.1653 | 1120 | 0.0739 | 0.9740 | 0.9796 | 0.9320 | 0.9552 |
| 0.0426 | 0.1682 | 1140 | 0.0782 | 0.9725 | 0.9895 | 0.9175 | 0.9521 |
| 0.0373 | 0.1712 | 1160 | 0.0795 | 0.9740 | 0.9796 | 0.9320 | 0.9552 |
| 0.0635 | 0.1741 | 1180 | 0.0754 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0733 | 0.1771 | 1200 | 0.0744 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0914 | 0.1800 | 1220 | 0.0765 | 0.9740 | 0.9845 | 0.9272 | 0.955 |
| 0.1249 | 0.1830 | 1240 | 0.0765 | 0.9725 | 0.9845 | 0.9223 | 0.9524 |
| 0.0829 | 0.1859 | 1260 | 0.0715 | 0.9725 | 0.9795 | 0.9272 | 0.9526 |
| 0.1013 | 0.1889 | 1280 | 0.0927 | 0.9711 | 0.9947 | 0.9078 | 0.9492 |
| 0.0697 | 0.1918 | 1300 | 0.0805 | 0.9725 | 0.9895 | 0.9175 | 0.9521 |
| 0.0905 | 0.1948 | 1320 | 0.0773 | 0.9740 | 0.9796 | 0.9320 | 0.9552 |
| 0.0417 | 0.1977 | 1340 | 0.0790 | 0.9711 | 0.9844 | 0.9175 | 0.9497 |
| 0.0541 | 0.2007 | 1360 | 0.0759 | 0.9740 | 0.9845 | 0.9272 | 0.955 |
| 0.0333 | 0.2036 | 1380 | 0.0932 | 0.9682 | 0.9946 | 0.8981 | 0.9439 |
| 0.0933 | 0.2066 | 1400 | 0.0744 | 0.9740 | 0.9896 | 0.9223 | 0.9548 |
| 0.0679 | 0.2095 | 1420 | 0.0702 | 0.9754 | 0.9896 | 0.9272 | 0.9574 |
| 0.0482 | 0.2125 | 1440 | 0.0707 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0408 | 0.2154 | 1460 | 0.0740 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.0819 | 0.2184 | 1480 | 0.0675 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0622 | 0.2213 | 1500 | 0.0701 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0211 | 0.2243 | 1520 | 0.0649 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.0753 | 0.2272 | 1540 | 0.0686 | 0.9740 | 0.9747 | 0.9369 | 0.9554 |
| 0.0839 | 0.2302 | 1560 | 0.0785 | 0.9754 | 0.9948 | 0.9223 | 0.9572 |
| 0.0861 | 0.2331 | 1580 | 0.0811 | 0.9754 | 0.9948 | 0.9223 | 0.9572 |
| 0.0657 | 0.2361 | 1600 | 0.0680 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0693 | 0.2390 | 1620 | 0.0741 | 0.9740 | 0.9947 | 0.9175 | 0.9545 |
| 0.0547 | 0.2420 | 1640 | 0.0713 | 0.9754 | 0.9896 | 0.9272 | 0.9574 |
| 0.0983 | 0.2449 | 1660 | 0.0711 | 0.9740 | 0.9896 | 0.9223 | 0.9548 |
| 0.0599 | 0.2479 | 1680 | 0.0660 | 0.9740 | 0.9845 | 0.9272 | 0.955 |
| 0.1337 | 0.2508 | 1700 | 0.0652 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0401 | 0.2538 | 1720 | 0.0818 | 0.9754 | 0.9948 | 0.9223 | 0.9572 |
| 0.0215 | 0.2567 | 1740 | 0.0677 | 0.9754 | 0.9948 | 0.9223 | 0.9572 |
| 0.1742 | 0.2597 | 1760 | 0.0675 | 0.9740 | 0.9947 | 0.9175 | 0.9545 |
| 0.0445 | 0.2626 | 1780 | 0.0656 | 0.9754 | 0.9896 | 0.9272 | 0.9574 |
| 0.0557 | 0.2656 | 1800 | 0.0699 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0079 | 0.2685 | 1820 | 0.0733 | 0.9740 | 0.9947 | 0.9175 | 0.9545 |
| 0.056 | 0.2715 | 1840 | 0.0709 | 0.9740 | 0.9796 | 0.9320 | 0.9552 |
| 0.0923 | 0.2744 | 1860 | 0.0676 | 0.9740 | 0.97 | 0.9417 | 0.9557 |
| 0.0769 | 0.2774 | 1880 | 0.0715 | 0.9754 | 0.9948 | 0.9223 | 0.9572 |
| 0.1452 | 0.2803 | 1900 | 0.0687 | 0.9754 | 0.9896 | 0.9272 | 0.9574 |
| 0.0938 | 0.2833 | 1920 | 0.0676 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0583 | 0.2862 | 1940 | 0.0698 | 0.9740 | 0.9845 | 0.9272 | 0.955 |
| 0.0937 | 0.2892 | 1960 | 0.0672 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0391 | 0.2921 | 1980 | 0.0675 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.0536 | 0.2951 | 2000 | 0.0699 | 0.9725 | 0.9895 | 0.9175 | 0.9521 |
| 0.0647 | 0.2980 | 2020 | 0.0723 | 0.9740 | 0.9896 | 0.9223 | 0.9548 |
| 0.0637 | 0.3010 | 2040 | 0.0699 | 0.9754 | 0.9655 | 0.9515 | 0.9584 |
| 0.07 | 0.3039 | 2060 | 0.0656 | 0.9740 | 0.9796 | 0.9320 | 0.9552 |
| 0.0264 | 0.3069 | 2080 | 0.0672 | 0.9711 | 0.9895 | 0.9126 | 0.9495 |
| 0.1048 | 0.3098 | 2100 | 0.0656 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.136 | 0.3128 | 2120 | 0.0685 | 0.9725 | 0.9947 | 0.9126 | 0.9519 |
| 0.0833 | 0.3158 | 2140 | 0.0660 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.0643 | 0.3187 | 2160 | 0.0696 | 0.9754 | 0.9948 | 0.9223 | 0.9572 |
| 0.032 | 0.3217 | 2180 | 0.0672 | 0.9754 | 0.9701 | 0.9466 | 0.9582 |
| 0.0613 | 0.3246 | 2200 | 0.0658 | 0.9768 | 0.975 | 0.9466 | 0.9606 |
| 0.0755 | 0.3276 | 2220 | 0.0692 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.0122 | 0.3305 | 2240 | 0.0724 | 0.9725 | 0.9652 | 0.9417 | 0.9533 |
| 0.0472 | 0.3335 | 2260 | 0.0754 | 0.9711 | 0.9947 | 0.9078 | 0.9492 |
| 0.0404 | 0.3364 | 2280 | 0.0664 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.0952 | 0.3394 | 2300 | 0.0807 | 0.9696 | 0.9947 | 0.9029 | 0.9466 |
| 0.0435 | 0.3423 | 2320 | 0.0642 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.027 | 0.3453 | 2340 | 0.0664 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.075 | 0.3482 | 2360 | 0.0663 | 0.9754 | 0.9655 | 0.9515 | 0.9584 |
| 0.071 | 0.3512 | 2380 | 0.0694 | 0.9740 | 0.9845 | 0.9272 | 0.955 |
| 0.0724 | 0.3541 | 2400 | 0.0716 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0126 | 0.3571 | 2420 | 0.0800 | 0.9740 | 0.9947 | 0.9175 | 0.9545 |
| 0.0611 | 0.3600 | 2440 | 0.0661 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0067 | 0.3630 | 2460 | 0.0671 | 0.9725 | 0.9845 | 0.9223 | 0.9524 |
| 0.0804 | 0.3659 | 2480 | 0.0652 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.057 | 0.3689 | 2500 | 0.0716 | 0.9725 | 0.9895 | 0.9175 | 0.9521 |
| 0.0518 | 0.3718 | 2520 | 0.0707 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0533 | 0.3748 | 2540 | 0.0778 | 0.9711 | 0.9895 | 0.9126 | 0.9495 |
| 0.0371 | 0.3777 | 2560 | 0.0714 | 0.9740 | 0.9796 | 0.9320 | 0.9552 |
| 0.1052 | 0.3807 | 2580 | 0.0677 | 0.9754 | 0.9896 | 0.9272 | 0.9574 |
| 0.0384 | 0.3836 | 2600 | 0.0641 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0482 | 0.3866 | 2620 | 0.0643 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0892 | 0.3895 | 2640 | 0.0676 | 0.9754 | 0.9896 | 0.9272 | 0.9574 |
| 0.0999 | 0.3925 | 2660 | 0.0669 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.055 | 0.3954 | 2680 | 0.0658 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0561 | 0.3984 | 2700 | 0.0656 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.0701 | 0.4013 | 2720 | 0.0693 | 0.9754 | 0.9948 | 0.9223 | 0.9572 |
| 0.0307 | 0.4043 | 2740 | 0.0637 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.029 | 0.4072 | 2760 | 0.0688 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0687 | 0.4102 | 2780 | 0.0662 | 0.9754 | 0.9797 | 0.9369 | 0.9578 |
| 0.0577 | 0.4131 | 2800 | 0.0669 | 0.9725 | 0.9895 | 0.9175 | 0.9521 |
| 0.0286 | 0.4161 | 2820 | 0.0624 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.1144 | 0.4190 | 2840 | 0.0647 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0424 | 0.4220 | 2860 | 0.0620 | 0.9783 | 0.9751 | 0.9515 | 0.9631 |
| 0.1575 | 0.4249 | 2880 | 0.0651 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.0651 | 0.4279 | 2900 | 0.0620 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.1524 | 0.4308 | 2920 | 0.0596 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0547 | 0.4338 | 2940 | 0.0597 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0064 | 0.4367 | 2960 | 0.0604 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.075 | 0.4397 | 2980 | 0.0591 | 0.9754 | 0.9701 | 0.9466 | 0.9582 |
| 0.0542 | 0.4426 | 3000 | 0.0661 | 0.9711 | 0.9745 | 0.9272 | 0.9502 |
| 0.0455 | 0.4456 | 3020 | 0.0632 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.0152 | 0.4485 | 3040 | 0.0605 | 0.9797 | 0.9752 | 0.9563 | 0.9657 |
| 0.0467 | 0.4515 | 3060 | 0.0646 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.0418 | 0.4544 | 3080 | 0.0597 | 0.9754 | 0.9896 | 0.9272 | 0.9574 |
| 0.0556 | 0.4574 | 3100 | 0.0622 | 0.9740 | 0.9896 | 0.9223 | 0.9548 |
| 0.0675 | 0.4603 | 3120 | 0.0642 | 0.9740 | 0.9608 | 0.9515 | 0.9561 |
| 0.0227 | 0.4633 | 3140 | 0.0619 | 0.9740 | 0.9796 | 0.9320 | 0.9552 |
| 0.0502 | 0.4662 | 3160 | 0.0622 | 0.9725 | 0.9746 | 0.9320 | 0.9529 |
| 0.0478 | 0.4692 | 3180 | 0.0605 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.1021 | 0.4722 | 3200 | 0.0630 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0418 | 0.4751 | 3220 | 0.0615 | 0.9754 | 0.9701 | 0.9466 | 0.9582 |
| 0.0354 | 0.4781 | 3240 | 0.0615 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.0575 | 0.4810 | 3260 | 0.0593 | 0.9768 | 0.975 | 0.9466 | 0.9606 |
| 0.0901 | 0.4840 | 3280 | 0.0630 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0166 | 0.4869 | 3300 | 0.0616 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0942 | 0.4899 | 3320 | 0.0612 | 0.9754 | 0.9701 | 0.9466 | 0.9582 |
| 0.1136 | 0.4928 | 3340 | 0.0661 | 0.9711 | 0.9697 | 0.9320 | 0.9505 |
| 0.0865 | 0.4958 | 3360 | 0.0666 | 0.9696 | 0.9602 | 0.9369 | 0.9484 |
| 0.0684 | 0.4987 | 3380 | 0.0617 | 0.9682 | 0.9554 | 0.9369 | 0.9461 |
| 0.0556 | 0.5017 | 3400 | 0.0649 | 0.9682 | 0.9554 | 0.9369 | 0.9461 |
| 0.079 | 0.5046 | 3420 | 0.0640 | 0.9711 | 0.9559 | 0.9466 | 0.9512 |
| 0.0976 | 0.5076 | 3440 | 0.0632 | 0.9696 | 0.9602 | 0.9369 | 0.9484 |
| 0.0756 | 0.5105 | 3460 | 0.0646 | 0.9711 | 0.9604 | 0.9417 | 0.9510 |
| 0.0786 | 0.5135 | 3480 | 0.0651 | 0.9682 | 0.9646 | 0.9272 | 0.9455 |
| 0.0663 | 0.5164 | 3500 | 0.0588 | 0.9711 | 0.9559 | 0.9466 | 0.9512 |
| 0.1079 | 0.5194 | 3520 | 0.0598 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.0547 | 0.5223 | 3540 | 0.0573 | 0.9740 | 0.9653 | 0.9466 | 0.9559 |
| 0.0204 | 0.5253 | 3560 | 0.0594 | 0.9754 | 0.9701 | 0.9466 | 0.9582 |
| 0.0595 | 0.5282 | 3580 | 0.0602 | 0.9754 | 0.9797 | 0.9369 | 0.9578 |
| 0.0574 | 0.5312 | 3600 | 0.0604 | 0.9711 | 0.9559 | 0.9466 | 0.9512 |
| 0.0249 | 0.5341 | 3620 | 0.0654 | 0.9740 | 0.9796 | 0.9320 | 0.9552 |
| 0.131 | 0.5371 | 3640 | 0.0596 | 0.9740 | 0.9608 | 0.9515 | 0.9561 |
| 0.0454 | 0.5400 | 3660 | 0.0638 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0219 | 0.5430 | 3680 | 0.0636 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.0547 | 0.5459 | 3700 | 0.0609 | 0.9754 | 0.9701 | 0.9466 | 0.9582 |
| 0.077 | 0.5489 | 3720 | 0.0585 | 0.9754 | 0.9797 | 0.9369 | 0.9578 |
| 0.0403 | 0.5518 | 3740 | 0.0552 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0228 | 0.5548 | 3760 | 0.0541 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0321 | 0.5577 | 3780 | 0.0533 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0698 | 0.5607 | 3800 | 0.0525 | 0.9797 | 0.98 | 0.9515 | 0.9655 |
| 0.0555 | 0.5636 | 3820 | 0.0535 | 0.9768 | 0.9703 | 0.9515 | 0.9608 |
| 0.0853 | 0.5666 | 3840 | 0.0534 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0385 | 0.5695 | 3860 | 0.0581 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0516 | 0.5725 | 3880 | 0.0552 | 0.9754 | 0.9701 | 0.9466 | 0.9582 |
| 0.0248 | 0.5754 | 3900 | 0.0562 | 0.9768 | 0.975 | 0.9466 | 0.9606 |
| 0.0351 | 0.5784 | 3920 | 0.0617 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0629 | 0.5813 | 3940 | 0.0587 | 0.9740 | 0.9747 | 0.9369 | 0.9554 |
| 0.0564 | 0.5843 | 3960 | 0.0587 | 0.9740 | 0.9653 | 0.9466 | 0.9559 |
| 0.0364 | 0.5872 | 3980 | 0.0596 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0995 | 0.5902 | 4000 | 0.0603 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.0686 | 0.5931 | 4020 | 0.0582 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.0269 | 0.5961 | 4040 | 0.0625 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.0478 | 0.5990 | 4060 | 0.0588 | 0.9754 | 0.9701 | 0.9466 | 0.9582 |
| 0.0405 | 0.6020 | 4080 | 0.0578 | 0.9768 | 0.975 | 0.9466 | 0.9606 |
| 0.0824 | 0.6049 | 4100 | 0.0560 | 0.9783 | 0.9751 | 0.9515 | 0.9631 |
| 0.008 | 0.6079 | 4120 | 0.0564 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0488 | 0.6108 | 4140 | 0.0543 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0837 | 0.6138 | 4160 | 0.0557 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.021 | 0.6167 | 4180 | 0.0585 | 0.9768 | 0.975 | 0.9466 | 0.9606 |
| 0.0661 | 0.6197 | 4200 | 0.0600 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.0784 | 0.6226 | 4220 | 0.0593 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.0598 | 0.6256 | 4240 | 0.0613 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.0856 | 0.6286 | 4260 | 0.0615 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.0634 | 0.6315 | 4280 | 0.0583 | 0.9797 | 0.98 | 0.9515 | 0.9655 |
| 0.0396 | 0.6345 | 4300 | 0.0583 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0307 | 0.6374 | 4320 | 0.0595 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0549 | 0.6404 | 4340 | 0.0577 | 0.9812 | 0.9898 | 0.9466 | 0.9677 |
| 0.0647 | 0.6433 | 4360 | 0.0596 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0181 | 0.6463 | 4380 | 0.0586 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0798 | 0.6492 | 4400 | 0.0562 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0386 | 0.6522 | 4420 | 0.0578 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0986 | 0.6551 | 4440 | 0.0601 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0731 | 0.6581 | 4460 | 0.0596 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0269 | 0.6610 | 4480 | 0.0573 | 0.9754 | 0.9749 | 0.9417 | 0.9580 |
| 0.0509 | 0.6640 | 4500 | 0.0589 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0554 | 0.6669 | 4520 | 0.0568 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.0536 | 0.6699 | 4540 | 0.0555 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0681 | 0.6728 | 4560 | 0.0562 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0446 | 0.6758 | 4580 | 0.0570 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0238 | 0.6787 | 4600 | 0.0559 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0653 | 0.6817 | 4620 | 0.0548 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.05 | 0.6846 | 4640 | 0.0546 | 0.9812 | 0.9849 | 0.9515 | 0.9679 |
| 0.048 | 0.6876 | 4660 | 0.0541 | 0.9812 | 0.9849 | 0.9515 | 0.9679 |
| 0.0746 | 0.6905 | 4680 | 0.0560 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.036 | 0.6935 | 4700 | 0.0551 | 0.9812 | 0.9849 | 0.9515 | 0.9679 |
| 0.0498 | 0.6964 | 4720 | 0.0555 | 0.9812 | 0.9849 | 0.9515 | 0.9679 |
| 0.0327 | 0.6994 | 4740 | 0.0603 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.0637 | 0.7023 | 4760 | 0.0553 | 0.9797 | 0.98 | 0.9515 | 0.9655 |
| 0.0089 | 0.7053 | 4780 | 0.0611 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0219 | 0.7082 | 4800 | 0.0577 | 0.9797 | 0.98 | 0.9515 | 0.9655 |
| 0.0171 | 0.7112 | 4820 | 0.0562 | 0.9812 | 0.9849 | 0.9515 | 0.9679 |
| 0.0388 | 0.7141 | 4840 | 0.0559 | 0.9812 | 0.9849 | 0.9515 | 0.9679 |
| 0.0057 | 0.7171 | 4860 | 0.0586 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0507 | 0.7200 | 4880 | 0.0592 | 0.9754 | 0.9846 | 0.9320 | 0.9576 |
| 0.0478 | 0.7230 | 4900 | 0.0566 | 0.9812 | 0.9849 | 0.9515 | 0.9679 |
| 0.0949 | 0.7259 | 4920 | 0.0550 | 0.9812 | 0.9849 | 0.9515 | 0.9679 |
| 0.0567 | 0.7289 | 4940 | 0.0558 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.025 | 0.7318 | 4960 | 0.0580 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.0488 | 0.7348 | 4980 | 0.0612 | 0.9768 | 0.9897 | 0.9320 | 0.96 |
| 0.0313 | 0.7377 | 5000 | 0.0559 | 0.9797 | 0.98 | 0.9515 | 0.9655 |
| 0.0552 | 0.7407 | 5020 | 0.0589 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0579 | 0.7436 | 5040 | 0.0556 | 0.9826 | 0.9899 | 0.9515 | 0.9703 |
| 0.0704 | 0.7466 | 5060 | 0.0551 | 0.9812 | 0.9898 | 0.9466 | 0.9677 |
| 0.0537 | 0.7495 | 5080 | 0.0575 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.038 | 0.7525 | 5100 | 0.0577 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.1133 | 0.7554 | 5120 | 0.0593 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.072 | 0.7584 | 5140 | 0.0575 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0663 | 0.7613 | 5160 | 0.0559 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0416 | 0.7643 | 5180 | 0.0563 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.0462 | 0.7672 | 5200 | 0.0577 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0268 | 0.7702 | 5220 | 0.0585 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0406 | 0.7731 | 5240 | 0.0560 | 0.9768 | 0.9798 | 0.9417 | 0.9604 |
| 0.0304 | 0.7761 | 5260 | 0.0592 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0758 | 0.7790 | 5280 | 0.0580 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0623 | 0.7820 | 5300 | 0.0582 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0472 | 0.7850 | 5320 | 0.0586 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0349 | 0.7879 | 5340 | 0.0558 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.0311 | 0.7909 | 5360 | 0.0561 | 0.9812 | 0.9898 | 0.9466 | 0.9677 |
| 0.0943 | 0.7938 | 5380 | 0.0580 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0458 | 0.7968 | 5400 | 0.0579 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.0744 | 0.7997 | 5420 | 0.0576 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.1293 | 0.8027 | 5440 | 0.0562 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0526 | 0.8056 | 5460 | 0.0557 | 0.9812 | 0.9898 | 0.9466 | 0.9677 |
| 0.0609 | 0.8086 | 5480 | 0.0571 | 0.9812 | 0.9898 | 0.9466 | 0.9677 |
| 0.0684 | 0.8115 | 5500 | 0.0583 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0579 | 0.8145 | 5520 | 0.0594 | 0.9768 | 0.9847 | 0.9369 | 0.9602 |
| 0.0728 | 0.8174 | 5540 | 0.0592 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0304 | 0.8204 | 5560 | 0.0592 | 0.9754 | 0.9797 | 0.9369 | 0.9578 |
| 0.0519 | 0.8233 | 5580 | 0.0575 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0091 | 0.8263 | 5600 | 0.0568 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0306 | 0.8292 | 5620 | 0.0574 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0112 | 0.8322 | 5640 | 0.0587 | 0.9783 | 0.9897 | 0.9369 | 0.9626 |
| 0.0214 | 0.8351 | 5660 | 0.0576 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.02 | 0.8381 | 5680 | 0.0564 | 0.9812 | 0.9898 | 0.9466 | 0.9677 |
| 0.0866 | 0.8410 | 5700 | 0.0559 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0269 | 0.8440 | 5720 | 0.0576 | 0.9797 | 0.9898 | 0.9417 | 0.9652 |
| 0.0441 | 0.8469 | 5740 | 0.0572 | 0.9812 | 0.9898 | 0.9466 | 0.9677 |
| 0.0413 | 0.8499 | 5760 | 0.0545 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.061 | 0.8528 | 5780 | 0.0542 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0699 | 0.8558 | 5800 | 0.0544 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0674 | 0.8587 | 5820 | 0.0548 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0402 | 0.8617 | 5840 | 0.0571 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0226 | 0.8646 | 5860 | 0.0577 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.0419 | 0.8676 | 5880 | 0.0577 | 0.9783 | 0.9848 | 0.9417 | 0.9628 |
| 0.089 | 0.8705 | 5900 | 0.0567 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.1096 | 0.8735 | 5920 | 0.0562 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0933 | 0.8764 | 5940 | 0.0551 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0482 | 0.8794 | 5960 | 0.0552 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.1723 | 0.8823 | 5980 | 0.0557 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0259 | 0.8853 | 6000 | 0.0557 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0291 | 0.8882 | 6020 | 0.0557 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0872 | 0.8912 | 6040 | 0.0560 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0819 | 0.8941 | 6060 | 0.0557 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0912 | 0.8971 | 6080 | 0.0558 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0307 | 0.9000 | 6100 | 0.0563 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0271 | 0.9030 | 6120 | 0.0554 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0405 | 0.9059 | 6140 | 0.0555 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0175 | 0.9089 | 6160 | 0.0550 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0459 | 0.9118 | 6180 | 0.0549 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0631 | 0.9148 | 6200 | 0.0554 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0737 | 0.9177 | 6220 | 0.0552 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.042 | 0.9207 | 6240 | 0.0559 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0505 | 0.9236 | 6260 | 0.0557 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0424 | 0.9266 | 6280 | 0.0552 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0506 | 0.9295 | 6300 | 0.0560 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0238 | 0.9325 | 6320 | 0.0557 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0703 | 0.9354 | 6340 | 0.0550 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0772 | 0.9384 | 6360 | 0.0556 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0697 | 0.9414 | 6380 | 0.0557 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0446 | 0.9443 | 6400 | 0.0558 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0323 | 0.9473 | 6420 | 0.0550 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0496 | 0.9502 | 6440 | 0.0553 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0809 | 0.9532 | 6460 | 0.0553 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0706 | 0.9561 | 6480 | 0.0549 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0716 | 0.9591 | 6500 | 0.0555 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.067 | 0.9620 | 6520 | 0.0552 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0626 | 0.9650 | 6540 | 0.0551 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0275 | 0.9679 | 6560 | 0.0553 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0196 | 0.9709 | 6580 | 0.0551 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0986 | 0.9738 | 6600 | 0.0555 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0275 | 0.9768 | 6620 | 0.0559 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0358 | 0.9797 | 6640 | 0.0552 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0266 | 0.9827 | 6660 | 0.0550 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0759 | 0.9856 | 6680 | 0.0559 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0396 | 0.9886 | 6700 | 0.0549 | 0.9797 | 0.9848 | 0.9466 | 0.9653 |
| 0.0172 | 0.9915 | 6720 | 0.0554 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0444 | 0.9945 | 6740 | 0.0552 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
| 0.0481 | 0.9974 | 6760 | 0.0549 | 0.9783 | 0.9799 | 0.9466 | 0.9630 |
### Framework versions
- PEFT 0.13.2
- Transformers 4.46.0
- Pytorch 2.5.1+cu124
- Datasets 3.1.0
- Tokenizers 0.20.3 |