--- tags: - masked-auto-encoding - generated_from_trainer model-index: - name: zh_wiki_small results: [] --- # zh_wiki_small This model is a fine-tuned version of [](https://huggingface.co/) on the wikipedia dataset. It achieves the following results on the evaluation set: - Loss: 0.4159 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.00015 - train_batch_size: 32 - eval_batch_size: 4 - seed: 42 - distributed_type: multi-GPU - num_devices: 4 - gradient_accumulation_steps: 2 - total_train_batch_size: 256 - total_eval_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_ratio: 0.05 - training_steps: 500000 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:------:|:---------------:| | 0.4793 | 0.14 | 1000 | 0.4540 | | 0.4701 | 0.28 | 2000 | 0.4467 | | 0.4673 | 0.42 | 3000 | 0.4304 | | 0.4669 | 0.55 | 4000 | 0.4413 | | 0.4668 | 0.69 | 5000 | 0.4368 | | 0.4676 | 0.83 | 6000 | 0.4358 | | 0.4691 | 0.97 | 7000 | 0.4367 | | 0.4693 | 1.11 | 8000 | 0.4429 | | 0.4709 | 1.25 | 9000 | 0.4388 | | 0.4722 | 1.39 | 10000 | 0.4453 | | 0.4729 | 1.53 | 11000 | 0.4415 | | 0.4732 | 1.67 | 12000 | 0.4510 | | 0.4751 | 1.8 | 13000 | 0.4461 | | 0.4765 | 1.94 | 14000 | 0.4448 | | 0.477 | 2.08 | 15000 | 0.4498 | | 0.4779 | 2.22 | 16000 | 0.4447 | | 0.4795 | 2.36 | 17000 | 0.4430 | | 0.481 | 2.5 | 18000 | 0.4499 | | 0.4821 | 2.64 | 19000 | 0.4551 | | 0.4829 | 2.78 | 20000 | 0.4519 | | 0.4838 | 2.91 | 21000 | 0.4520 | | 0.4856 | 3.05 | 22000 | 0.4633 | | 0.4857 | 3.19 | 23000 | 0.4576 | | 0.4869 | 3.33 | 24000 | 0.4485 | | 0.4882 | 3.47 | 25000 | 0.4591 | | 0.4883 | 3.61 | 26000 | 0.4645 | | 0.4889 | 3.75 | 27000 | 0.4570 | | 0.4884 | 3.89 | 28000 | 0.4572 | | 0.4897 | 4.02 | 29000 | 0.4553 | | 0.4883 | 4.16 | 30000 | 0.4534 | | 0.4881 | 4.3 | 31000 | 0.4587 | | 0.4889 | 4.44 | 32000 | 0.4632 | | 0.4886 | 4.58 | 33000 | 0.4587 | | 0.4883 | 4.72 | 34000 | 0.4621 | | 0.4876 | 4.86 | 35000 | 0.4522 | | 0.4878 | 5.0 | 36000 | 0.4560 | | 0.4883 | 5.13 | 37000 | 0.4579 | | 0.4882 | 5.27 | 38000 | 0.4554 | | 0.4883 | 5.41 | 39000 | 0.4588 | | 0.4872 | 5.55 | 40000 | 0.4561 | | 0.4868 | 5.69 | 41000 | 0.4614 | | 0.4875 | 5.83 | 42000 | 0.4584 | | 0.4868 | 5.97 | 43000 | 0.4619 | | 0.4874 | 6.11 | 44000 | 0.4519 | | 0.4874 | 6.24 | 45000 | 0.4625 | | 0.487 | 6.38 | 46000 | 0.4579 | | 0.4872 | 6.52 | 47000 | 0.4534 | | 0.4872 | 6.66 | 48000 | 0.4516 | | 0.4865 | 6.8 | 49000 | 0.4635 | | 0.4865 | 6.94 | 50000 | 0.4610 | | 0.4863 | 7.08 | 51000 | 0.4515 | | 0.4861 | 7.22 | 52000 | 0.4584 | | 0.4866 | 7.35 | 53000 | 0.4541 | | 0.4862 | 7.49 | 54000 | 0.4508 | | 0.4863 | 7.63 | 55000 | 0.4565 | | 0.486 | 7.77 | 56000 | 0.4665 | | 0.486 | 7.91 | 57000 | 0.4565 | | 0.4861 | 8.05 | 58000 | 0.4580 | | 0.4852 | 8.19 | 59000 | 0.4596 | | 0.4846 | 8.33 | 60000 | 0.4527 | | 0.4848 | 8.46 | 61000 | 0.4505 | | 0.4849 | 8.6 | 62000 | 0.4407 | | 0.4851 | 8.74 | 63000 | 0.4579 | | 0.4848 | 8.88 | 64000 | 0.4559 | | 0.4851 | 9.02 | 65000 | 0.4505 | | 0.4846 | 9.16 | 66000 | 0.4615 | | 0.4842 | 9.3 | 67000 | 0.4618 | | 0.484 | 9.44 | 68000 | 0.4559 | | 0.4841 | 9.57 | 69000 | 0.4613 | | 0.484 | 9.71 | 70000 | 0.4527 | | 0.4842 | 9.85 | 71000 | 0.4483 | | 0.4842 | 9.99 | 72000 | 0.4585 | | 0.4837 | 10.13 | 73000 | 0.4585 | | 0.4833 | 10.27 | 74000 | 0.4541 | | 0.4836 | 10.41 | 75000 | 0.4528 | | 0.4832 | 10.55 | 76000 | 0.4475 | | 0.4836 | 10.68 | 77000 | 0.4525 | | 0.4826 | 10.82 | 78000 | 0.4562 | | 0.4824 | 10.96 | 79000 | 0.4502 | | 0.4828 | 11.1 | 80000 | 0.4529 | | 0.4829 | 11.24 | 81000 | 0.4524 | | 0.4823 | 11.38 | 82000 | 0.4506 | | 0.4827 | 11.52 | 83000 | 0.4511 | | 0.4823 | 11.66 | 84000 | 0.4506 | | 0.4827 | 11.79 | 85000 | 0.4561 | | 0.4832 | 11.93 | 86000 | 0.4471 | | 0.482 | 12.07 | 87000 | 0.4479 | | 0.4819 | 12.21 | 88000 | 0.4561 | | 0.4816 | 12.35 | 89000 | 0.4590 | | 0.4818 | 12.49 | 90000 | 0.4469 | | 0.4815 | 12.63 | 91000 | 0.4633 | | 0.4822 | 12.77 | 92000 | 0.4566 | | 0.4816 | 12.9 | 93000 | 0.4548 | | 0.4824 | 13.04 | 94000 | 0.4548 | | 0.4812 | 13.18 | 95000 | 0.4533 | | 0.4809 | 13.32 | 96000 | 0.4546 | | 0.481 | 13.46 | 97000 | 0.4590 | | 0.4807 | 13.6 | 98000 | 0.4465 | | 0.4808 | 13.74 | 99000 | 0.4531 | | 0.4806 | 13.88 | 100000 | 0.4459 | | 0.4809 | 14.01 | 101000 | 0.4517 | | 0.4801 | 14.15 | 102000 | 0.4519 | | 0.4801 | 14.29 | 103000 | 0.4547 | | 0.4805 | 14.43 | 104000 | 0.4517 | | 0.4799 | 14.57 | 105000 | 0.4491 | | 0.4805 | 14.71 | 106000 | 0.4559 | | 0.48 | 14.85 | 107000 | 0.4551 | | 0.4796 | 14.99 | 108000 | 0.4537 | | 0.4801 | 15.12 | 109000 | 0.4509 | | 0.4797 | 15.26 | 110000 | 0.4482 | | 0.4798 | 15.4 | 111000 | 0.4466 | | 0.4789 | 15.54 | 112000 | 0.4445 | | 0.4808 | 15.68 | 113000 | 0.4493 | | 0.4789 | 15.82 | 114000 | 0.4475 | | 0.4792 | 15.96 | 115000 | 0.4543 | | 0.4787 | 16.1 | 116000 | 0.4471 | | 0.4796 | 16.23 | 117000 | 0.4565 | | 0.4787 | 16.37 | 118000 | 0.4515 | | 0.4788 | 16.51 | 119000 | 0.4449 | | 0.4783 | 16.65 | 120000 | 0.4454 | | 0.4787 | 16.79 | 121000 | 0.4486 | | 0.4789 | 16.93 | 122000 | 0.4480 | | 0.4782 | 17.07 | 123000 | 0.4529 | | 0.4782 | 17.21 | 124000 | 0.4481 | | 0.4777 | 17.34 | 125000 | 0.4528 | | 0.4779 | 17.48 | 126000 | 0.4514 | | 0.4781 | 17.62 | 127000 | 0.4520 | | 0.4776 | 17.76 | 128000 | 0.4495 | | 0.4777 | 17.9 | 129000 | 0.4501 | | 0.4783 | 18.04 | 130000 | 0.4528 | | 0.4771 | 18.18 | 131000 | 0.4498 | | 0.4775 | 18.32 | 132000 | 0.4525 | | 0.4772 | 18.45 | 133000 | 0.4482 | | 0.4775 | 18.59 | 134000 | 0.4532 | | 0.4769 | 18.73 | 135000 | 0.4537 | | 0.4776 | 18.87 | 136000 | 0.4509 | | 0.4775 | 19.01 | 137000 | 0.4464 | | 0.4769 | 19.15 | 138000 | 0.4464 | | 0.4772 | 19.29 | 139000 | 0.4499 | | 0.4766 | 19.43 | 140000 | 0.4428 | | 0.4764 | 19.56 | 141000 | 0.4536 | | 0.477 | 19.7 | 142000 | 0.4444 | | 0.4764 | 19.84 | 143000 | 0.4482 | | 0.4764 | 19.98 | 144000 | 0.4510 | | 0.4763 | 20.12 | 145000 | 0.4519 | | 0.4761 | 20.26 | 146000 | 0.4452 | | 0.4761 | 20.4 | 147000 | 0.4476 | | 0.4756 | 20.54 | 148000 | 0.4494 | | 0.4757 | 20.67 | 149000 | 0.4544 | | 0.4762 | 20.81 | 150000 | 0.4412 | | 0.4757 | 20.95 | 151000 | 0.4459 | | 0.4749 | 21.09 | 152000 | 0.4532 | | 0.4752 | 21.23 | 153000 | 0.4477 | | 0.4749 | 21.37 | 154000 | 0.4396 | | 0.4764 | 21.51 | 155000 | 0.4466 | | 0.4753 | 21.65 | 156000 | 0.4523 | | 0.4755 | 21.78 | 157000 | 0.4582 | | 0.4749 | 21.92 | 158000 | 0.4539 | | 0.475 | 22.06 | 159000 | 0.4539 | | 0.4747 | 22.2 | 160000 | 0.4519 | | 0.4745 | 22.34 | 161000 | 0.4370 | | 0.4748 | 22.48 | 162000 | 0.4449 | | 0.4743 | 22.62 | 163000 | 0.4484 | | 0.4745 | 22.76 | 164000 | 0.4471 | | 0.4739 | 22.89 | 165000 | 0.4480 | | 0.4746 | 23.03 | 166000 | 0.4519 | | 0.4739 | 23.17 | 167000 | 0.4478 | | 0.4739 | 23.31 | 168000 | 0.4497 | | 0.4738 | 23.45 | 169000 | 0.4462 | | 0.474 | 23.59 | 170000 | 0.4430 | | 0.4737 | 23.73 | 171000 | 0.4483 | | 0.4737 | 23.87 | 172000 | 0.4508 | | 0.474 | 24.0 | 173000 | 0.4439 | | 0.4729 | 24.14 | 174000 | 0.4426 | | 0.4735 | 24.28 | 175000 | 0.4433 | | 0.4722 | 24.42 | 176000 | 0.4483 | | 0.4728 | 24.56 | 177000 | 0.4496 | | 0.4727 | 24.7 | 178000 | 0.4473 | | 0.4729 | 24.84 | 179000 | 0.4404 | | 0.4722 | 24.98 | 180000 | 0.4426 | | 0.4724 | 25.11 | 181000 | 0.4479 | | 0.4739 | 25.25 | 182000 | 0.4430 | | 0.4723 | 25.39 | 183000 | 0.4418 | | 0.4724 | 25.53 | 184000 | 0.4371 | | 0.472 | 25.67 | 185000 | 0.4456 | | 0.4726 | 25.81 | 186000 | 0.4419 | | 0.4721 | 25.95 | 187000 | 0.4417 | | 0.4722 | 26.09 | 188000 | 0.4475 | | 0.4715 | 26.22 | 189000 | 0.4389 | | 0.4717 | 26.36 | 190000 | 0.4451 | | 0.4716 | 26.5 | 191000 | 0.4440 | | 0.4714 | 26.64 | 192000 | 0.4399 | | 0.4712 | 26.78 | 193000 | 0.4398 | | 0.4709 | 26.92 | 194000 | 0.4424 | | 0.4714 | 27.06 | 195000 | 0.4533 | | 0.4706 | 27.2 | 196000 | 0.4394 | | 0.471 | 27.33 | 197000 | 0.4436 | | 0.4707 | 27.47 | 198000 | 0.4421 | | 0.471 | 27.61 | 199000 | 0.4459 | | 0.4707 | 27.75 | 200000 | 0.4439 | | 0.471 | 27.89 | 201000 | 0.4467 | | 0.471 | 28.03 | 202000 | 0.4439 | | 0.4704 | 28.17 | 203000 | 0.4445 | | 0.4705 | 28.31 | 204000 | 0.4429 | | 0.4706 | 28.44 | 205000 | 0.4382 | | 0.4703 | 28.58 | 206000 | 0.4425 | | 0.4695 | 28.72 | 207000 | 0.4414 | | 0.4696 | 28.86 | 208000 | 0.4405 | | 0.4696 | 29.0 | 209000 | 0.4460 | | 0.4701 | 29.14 | 210000 | 0.4460 | | 0.4696 | 29.28 | 211000 | 0.4397 | | 0.4693 | 29.42 | 212000 | 0.4439 | | 0.4694 | 29.55 | 213000 | 0.4495 | | 0.469 | 29.69 | 214000 | 0.4466 | | 0.4691 | 29.83 | 215000 | 0.4336 | | 0.4694 | 29.97 | 216000 | 0.4377 | | 0.4698 | 30.11 | 217000 | 0.4356 | | 0.4689 | 30.25 | 218000 | 0.4381 | | 0.4685 | 30.39 | 219000 | 0.4431 | | 0.4688 | 30.53 | 220000 | 0.4411 | | 0.4687 | 30.66 | 221000 | 0.4445 | | 0.4685 | 30.8 | 222000 | 0.4432 | | 0.4687 | 30.94 | 223000 | 0.4383 | | 0.4681 | 31.08 | 224000 | 0.4371 | | 0.4683 | 31.22 | 225000 | 0.4384 | | 0.4678 | 31.36 | 226000 | 0.4396 | | 0.4682 | 31.5 | 227000 | 0.4387 | | 0.4671 | 31.64 | 228000 | 0.4382 | | 0.4676 | 31.77 | 229000 | 0.4410 | | 0.4681 | 31.91 | 230000 | 0.4391 | | 0.4676 | 32.05 | 231000 | 0.4429 | | 0.4673 | 32.19 | 232000 | 0.4395 | | 0.4669 | 32.33 | 233000 | 0.4389 | | 0.4675 | 32.47 | 234000 | 0.4452 | | 0.4667 | 32.61 | 235000 | 0.4395 | | 0.4667 | 32.75 | 236000 | 0.4460 | | 0.4672 | 32.88 | 237000 | 0.4404 | | 0.4667 | 33.02 | 238000 | 0.4372 | | 0.4663 | 33.16 | 239000 | 0.4362 | | 0.4669 | 33.3 | 240000 | 0.4428 | | 0.4662 | 33.44 | 241000 | 0.4370 | | 0.4662 | 33.58 | 242000 | 0.4382 | | 0.466 | 33.72 | 243000 | 0.4395 | | 0.4661 | 33.86 | 244000 | 0.4418 | | 0.4663 | 33.99 | 245000 | 0.4407 | | 0.4661 | 34.13 | 246000 | 0.4346 | | 0.4652 | 34.27 | 247000 | 0.4392 | | 0.4662 | 34.41 | 248000 | 0.4396 | | 0.4655 | 34.55 | 249000 | 0.4427 | | 0.4657 | 34.69 | 250000 | 0.4484 | | 0.4654 | 34.83 | 251000 | 0.4268 | | 0.4655 | 34.97 | 252000 | 0.4384 | | 0.4649 | 35.1 | 253000 | 0.4383 | | 0.465 | 35.24 | 254000 | 0.4368 | | 0.4648 | 35.38 | 255000 | 0.4327 | | 0.4647 | 35.52 | 256000 | 0.4416 | | 0.4652 | 35.66 | 257000 | 0.4390 | | 0.4646 | 35.8 | 258000 | 0.4450 | | 0.4651 | 35.94 | 259000 | 0.4354 | | 0.4643 | 36.08 | 260000 | 0.4473 | | 0.464 | 36.21 | 261000 | 0.4423 | | 0.4638 | 36.35 | 262000 | 0.4339 | | 0.464 | 36.49 | 263000 | 0.4438 | | 0.464 | 36.63 | 264000 | 0.4398 | | 0.4637 | 36.77 | 265000 | 0.4352 | | 0.4641 | 36.91 | 266000 | 0.4352 | | 0.4651 | 37.05 | 267000 | 0.4324 | | 0.4637 | 37.19 | 268000 | 0.4341 | | 0.4633 | 37.32 | 269000 | 0.4331 | | 0.4639 | 37.46 | 270000 | 0.4391 | | 0.463 | 37.6 | 271000 | 0.4380 | | 0.4635 | 37.74 | 272000 | 0.4355 | | 0.4631 | 37.88 | 273000 | 0.4397 | | 0.464 | 38.02 | 274000 | 0.4336 | | 0.4629 | 38.16 | 275000 | 0.4339 | | 0.4634 | 38.3 | 276000 | 0.4355 | | 0.4632 | 38.43 | 277000 | 0.4388 | | 0.4628 | 38.57 | 278000 | 0.4341 | | 0.4621 | 38.71 | 279000 | 0.4337 | | 0.4626 | 38.85 | 280000 | 0.4340 | | 0.462 | 38.99 | 281000 | 0.4306 | | 0.8286 | 39.13 | 282000 | 0.4504 | | 0.4624 | 39.27 | 283000 | 0.4399 | | 0.4621 | 39.41 | 284000 | 0.4351 | | 0.4622 | 39.54 | 285000 | 0.4304 | | 0.4619 | 39.68 | 286000 | 0.4329 | | 0.4618 | 39.82 | 287000 | 0.4208 | | 0.462 | 39.96 | 288000 | 0.4414 | | 0.4615 | 40.1 | 289000 | 0.4353 | | 0.4614 | 40.24 | 290000 | 0.4398 | | 0.4611 | 40.38 | 291000 | 0.4371 | | 0.4608 | 40.52 | 292000 | 0.4326 | | 0.4611 | 40.65 | 293000 | 0.4332 | | 0.4614 | 40.79 | 294000 | 0.4343 | | 0.4609 | 40.93 | 295000 | 0.4306 | | 0.4608 | 41.07 | 296000 | 0.4323 | | 0.4608 | 41.21 | 297000 | 0.4321 | | 0.4601 | 41.35 | 298000 | 0.4330 | | 0.4606 | 41.49 | 299000 | 0.4361 | | 0.4606 | 41.63 | 300000 | 0.4367 | | 0.46 | 41.76 | 301000 | 0.4327 | | 0.4596 | 41.9 | 302000 | 0.4306 | | 0.46 | 42.04 | 303000 | 0.4352 | | 0.46 | 42.18 | 304000 | 0.4338 | | 0.4597 | 42.32 | 305000 | 0.4333 | | 0.4596 | 42.46 | 306000 | 0.4334 | | 0.4591 | 42.6 | 307000 | 0.4334 | | 0.4597 | 42.74 | 308000 | 0.4319 | | 0.4586 | 42.87 | 309000 | 0.4268 | | 0.4593 | 43.01 | 310000 | 0.4366 | | 0.4591 | 43.15 | 311000 | 0.4283 | | 0.4587 | 43.29 | 312000 | 0.4289 | | 0.4594 | 43.43 | 313000 | 0.4332 | | 0.459 | 43.57 | 314000 | 0.4326 | | 0.4586 | 43.71 | 315000 | 0.4356 | | 0.4581 | 43.85 | 316000 | 0.4271 | | 0.4584 | 43.98 | 317000 | 0.4325 | | 0.4586 | 44.12 | 318000 | 0.4350 | | 0.4584 | 44.26 | 319000 | 0.4273 | | 0.4576 | 44.4 | 320000 | 0.4284 | | 0.458 | 44.54 | 321000 | 0.4331 | | 0.4581 | 44.68 | 322000 | 0.4263 | | 0.4579 | 44.82 | 323000 | 0.4283 | | 0.4583 | 44.96 | 324000 | 0.4362 | | 0.4571 | 45.1 | 325000 | 0.4330 | | 0.4566 | 45.23 | 326000 | 0.4300 | | 0.4572 | 45.37 | 327000 | 0.4258 | | 0.4574 | 45.51 | 328000 | 0.4200 | | 0.4573 | 45.65 | 329000 | 0.4299 | | 0.4578 | 45.79 | 330000 | 0.4319 | | 0.4576 | 45.93 | 331000 | 0.4352 | | 0.4574 | 46.07 | 332000 | 0.4278 | | 0.4572 | 46.21 | 333000 | 0.4326 | | 0.4568 | 46.34 | 334000 | 0.4295 | | 0.4569 | 46.48 | 335000 | 0.4300 | | 0.4566 | 46.62 | 336000 | 0.4333 | | 0.4567 | 46.76 | 337000 | 0.4262 | | 0.4564 | 46.9 | 338000 | 0.4354 | | 0.4574 | 47.04 | 339000 | 0.4357 | | 0.4564 | 47.18 | 340000 | 0.4308 | | 0.4554 | 47.32 | 341000 | 0.4350 | | 0.456 | 47.45 | 342000 | 0.4400 | | 0.456 | 47.59 | 343000 | 0.4237 | | 0.4559 | 47.73 | 344000 | 0.4236 | | 0.4559 | 47.87 | 345000 | 0.4305 | | 0.4559 | 48.01 | 346000 | 0.4245 | | 0.4549 | 48.15 | 347000 | 0.4182 | | 0.4556 | 48.29 | 348000 | 0.4330 | | 0.4551 | 48.43 | 349000 | 0.4397 | | 0.455 | 48.56 | 350000 | 0.4252 | | 0.4548 | 48.7 | 351000 | 0.4246 | | 0.4551 | 48.84 | 352000 | 0.4291 | | 0.4554 | 48.98 | 353000 | 0.4286 | | 0.4547 | 49.12 | 354000 | 0.4336 | | 0.4548 | 49.26 | 355000 | 0.4324 | | 0.4545 | 49.4 | 356000 | 0.4236 | | 0.4547 | 49.54 | 357000 | 0.4345 | | 0.4542 | 49.67 | 358000 | 0.4329 | | 0.4545 | 49.81 | 359000 | 0.4241 | | 0.4541 | 49.95 | 360000 | 0.4177 | | 0.454 | 50.09 | 361000 | 0.4244 | | 0.4538 | 50.23 | 362000 | 0.4190 | | 0.4535 | 50.37 | 363000 | 0.4331 | | 0.4545 | 50.51 | 364000 | 0.4252 | | 0.454 | 50.65 | 365000 | 0.4315 | | 0.4536 | 50.78 | 366000 | 0.4301 | | 0.4534 | 50.92 | 367000 | 0.4357 | | 0.4537 | 51.06 | 368000 | 0.4334 | | 0.4535 | 51.2 | 369000 | 0.4200 | | 0.4538 | 51.34 | 370000 | 0.4274 | | 0.4536 | 51.48 | 371000 | 0.4178 | | 0.4534 | 51.62 | 372000 | 0.4181 | | 0.4533 | 51.76 | 373000 | 0.4211 | | 0.4535 | 51.89 | 374000 | 0.4290 | | 0.4535 | 52.03 | 375000 | 0.4201 | | 0.4526 | 52.17 | 376000 | 0.4263 | | 0.4526 | 52.31 | 377000 | 0.4237 | | 0.4524 | 52.45 | 378000 | 0.4254 | | 0.4529 | 52.59 | 379000 | 0.4260 | | 0.4531 | 52.73 | 380000 | 0.4202 | | 0.4523 | 52.87 | 381000 | 0.4223 | | 0.4523 | 53.0 | 382000 | 0.4271 | | 0.4522 | 53.14 | 383000 | 0.4286 | | 0.4524 | 53.28 | 384000 | 0.4256 | | 0.4515 | 53.42 | 385000 | 0.4221 | | 0.4513 | 53.56 | 386000 | 0.4255 | | 0.452 | 53.7 | 387000 | 0.4270 | | 0.4519 | 53.84 | 388000 | 0.4222 | | 0.4518 | 53.98 | 389000 | 0.4233 | | 0.4513 | 54.11 | 390000 | 0.4233 | | 0.4517 | 54.25 | 391000 | 0.4239 | | 0.4518 | 54.39 | 392000 | 0.4273 | | 0.4508 | 54.53 | 393000 | 0.4200 | | 0.4511 | 54.67 | 394000 | 0.4236 | | 0.4508 | 54.81 | 395000 | 0.4193 | | 0.4507 | 54.95 | 396000 | 0.4293 | | 0.4508 | 55.09 | 397000 | 0.4187 | | 0.4504 | 55.22 | 398000 | 0.4283 | | 0.4512 | 55.36 | 399000 | 0.4239 | | 0.4504 | 55.5 | 400000 | 0.4269 | | 0.4506 | 55.64 | 401000 | 0.4291 | | 0.4504 | 55.78 | 402000 | 0.4238 | | 0.4503 | 55.92 | 403000 | 0.4200 | | 0.4506 | 56.06 | 404000 | 0.4186 | | 0.4507 | 56.2 | 405000 | 0.4260 | | 0.4504 | 56.33 | 406000 | 0.4188 | | 0.4503 | 56.47 | 407000 | 0.4231 | | 0.4498 | 56.61 | 408000 | 0.4148 | | 0.4499 | 56.75 | 409000 | 0.4182 | | 0.4498 | 56.89 | 410000 | 0.4229 | | 0.4501 | 57.03 | 411000 | 0.4252 | | 0.4497 | 57.17 | 412000 | 0.4220 | | 0.45 | 57.31 | 413000 | 0.4181 | | 0.4497 | 57.44 | 414000 | 0.4270 | | 0.4497 | 57.58 | 415000 | 0.4208 | | 0.4499 | 57.72 | 416000 | 0.4224 | | 0.4496 | 57.86 | 417000 | 0.4207 | | 0.4494 | 58.0 | 418000 | 0.4268 | | 0.4499 | 58.14 | 419000 | 0.4240 | | 0.4495 | 58.28 | 420000 | 0.4294 | | 0.4487 | 58.42 | 421000 | 0.4207 | | 0.4495 | 58.55 | 422000 | 0.4246 | | 0.4491 | 58.69 | 423000 | 0.4213 | | 0.4492 | 58.83 | 424000 | 0.4241 | | 0.4486 | 58.97 | 425000 | 0.4247 | | 0.4485 | 59.11 | 426000 | 0.4163 | | 0.4489 | 59.25 | 427000 | 0.4239 | | 0.4483 | 59.39 | 428000 | 0.4240 | | 0.4491 | 59.53 | 429000 | 0.4214 | | 0.4485 | 59.66 | 430000 | 0.4285 | | 0.449 | 59.8 | 431000 | 0.4265 | | 0.4484 | 59.94 | 432000 | 0.4188 | | 0.4484 | 60.08 | 433000 | 0.4176 | | 0.4488 | 60.22 | 434000 | 0.4200 | | 0.448 | 60.36 | 435000 | 0.4116 | | 0.4477 | 60.5 | 436000 | 0.4215 | | 0.4484 | 60.64 | 437000 | 0.4204 | | 0.448 | 60.77 | 438000 | 0.4093 | | 0.4479 | 60.91 | 439000 | 0.4181 | | 0.4481 | 61.05 | 440000 | 0.4232 | | 0.4477 | 61.19 | 441000 | 0.4202 | | 0.4478 | 61.33 | 442000 | 0.4167 | | 0.4481 | 61.47 | 443000 | 0.4173 | | 0.4483 | 61.61 | 444000 | 0.4158 | | 0.4473 | 61.75 | 445000 | 0.4174 | | 0.4474 | 61.88 | 446000 | 0.4266 | | 0.4477 | 62.02 | 447000 | 0.4242 | | 0.4476 | 62.16 | 448000 | 0.4240 | | 0.4478 | 62.3 | 449000 | 0.4286 | | 0.4474 | 62.44 | 450000 | 0.4294 | | 0.4482 | 62.58 | 451000 | 0.4144 | | 0.4471 | 62.72 | 452000 | 0.4316 | | 0.448 | 62.86 | 453000 | 0.4228 | | 0.4474 | 62.99 | 454000 | 0.4242 | | 0.447 | 63.13 | 455000 | 0.4231 | | 0.4475 | 63.27 | 456000 | 0.4235 | | 0.4475 | 63.41 | 457000 | 0.4279 | | 0.4476 | 63.55 | 458000 | 0.4230 | | 0.4464 | 63.69 | 459000 | 0.4145 | | 0.4467 | 63.83 | 460000 | 0.4230 | | 0.4465 | 63.97 | 461000 | 0.4208 | | 0.4466 | 64.1 | 462000 | 0.4243 | | 0.447 | 64.24 | 463000 | 0.4220 | | 0.4473 | 64.38 | 464000 | 0.4253 | | 0.4471 | 64.52 | 465000 | 0.4194 | | 0.447 | 64.66 | 466000 | 0.4262 | | 0.447 | 64.8 | 467000 | 0.4245 | | 0.4468 | 64.94 | 468000 | 0.4143 | | 0.4463 | 65.08 | 469000 | 0.4187 | | 0.4465 | 65.21 | 470000 | 0.4185 | | 0.4465 | 65.35 | 471000 | 0.4244 | | 0.4467 | 65.49 | 472000 | 0.4201 | | 0.4465 | 65.63 | 473000 | 0.4160 | | 0.4467 | 65.77 | 474000 | 0.4273 | | 0.4465 | 65.91 | 475000 | 0.4183 | | 0.4467 | 66.05 | 476000 | 0.4227 | | 0.4469 | 66.19 | 477000 | 0.4166 | | 0.4467 | 66.32 | 478000 | 0.4199 | | 0.4464 | 66.46 | 479000 | 0.4181 | | 0.4463 | 66.6 | 480000 | 0.4217 | | 0.4464 | 66.74 | 481000 | 0.4158 | | 0.4468 | 66.88 | 482000 | 0.4191 | | 0.447 | 67.02 | 483000 | 0.4248 | | 0.4465 | 67.16 | 484000 | 0.4234 | | 0.4463 | 67.3 | 485000 | 0.4238 | | 0.446 | 67.43 | 486000 | 0.4162 | | 0.4462 | 67.57 | 487000 | 0.4202 | | 0.4462 | 67.71 | 488000 | 0.4177 | | 0.4455 | 67.85 | 489000 | 0.4228 | | 0.4463 | 67.99 | 490000 | 0.4146 | | 0.4454 | 68.13 | 491000 | 0.4190 | | 0.446 | 68.27 | 492000 | 0.4219 | | 0.4461 | 68.41 | 493000 | 0.4250 | | 0.4462 | 68.54 | 494000 | 0.4172 | | 0.4464 | 68.68 | 495000 | 0.4122 | | 0.4459 | 68.82 | 496000 | 0.4178 | | 0.4459 | 68.96 | 497000 | 0.4095 | | 0.4458 | 69.1 | 498000 | 0.4124 | | 0.4458 | 69.24 | 499000 | 0.4182 | | 0.4458 | 69.38 | 500000 | 0.4177 | ### Framework versions - Transformers 4.17.0 - Pytorch 1.12.0 - Datasets 2.0.0 - Tokenizers 0.13.2