Repository: tensorflow
Issue: Based on FP16, the network training of resnet50 was carried out, and the existence accuracy randomly converged to 0.76
Labels: stat:awaiting tensorflower,type:support,comp:model,TF 2.11

### Issue type

Support

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

tf2.11

### Custom code

Yes

### OS platform and distribution

centos7.6

### Mobile device

_No response_

### Python version

3.7、3.8、3,。9

### Bazel version

5.3.0

### GCC/compiler version

9.4.0

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Using the inagenet dataset, on the resnet50 network, based on FP16 training with the model-2.11.0 code,  the network has random convergence, is this phenomenon known, is there any solution at present?  Is there any good way to debug this problem? Thanks.

### Standalone code to reproduce the issue

```shell
python3  /public/home/TF_test/rocm5.4/tf2.11/models-master_fp16/official/benchmark/models/resnet_imagenet_main.py      --num_gpus=4     --batch_size=512  --train_epochs=90  --use_synthetic_data=false    --data_dir=/public/software/apps/DeepLearning/Data/ImageNet-tensorflow/  --enable_checkpoint_and_export  --dtype=fp16   --model_dir=./Checkpoint/
```


### Relevant log output

```shell
I0717 10:42:45.093785 47025319389696 controller.py:291]  eval | step:  10009 | eval time:   74.7 sec | output: {'test_accuracy': 0.08882, 'test_loss': 1.2578498}
I0717 11:19:43.333361 47025319389696 controller.py:291]  eval | step:  20018 | eval time:   47.9 sec | output: {'test_accuracy': 0.18752, 'test_loss': 1.0495228}
I0717 11:56:49.300491 47025319389696 controller.py:291]  eval | step:  30027 | eval time:   48.0 sec | output: {'test_accuracy': 0.2584, 'test_loss': 0.9102226}
I0717 12:33:43.809209 47025319389696 controller.py:291]  eval | step:  40036 | eval time:   47.3 sec | output: {'test_accuracy': 0.33168, 'test_loss': 0.77455497}
I0717 13:10:37.399015 47025319389696 controller.py:291]  eval | step:  50045 | eval time:   48.6 sec | output: {'test_accuracy': 0.32974, 'test_loss': 0.7866249}
I0717 13:47:31.637742 47025319389696 controller.py:291]  eval | step:  60054 | eval time:   48.0 sec | output: {'test_accuracy': 0.32832, 'test_loss': 0.7928696}
I0717 14:24:25.244748 47025319389696 controller.py:291]  eval | step:  70063 | eval time:   47.4 sec | output: {'test_accuracy': 0.36952, 'test_loss': 0.7357325}
I0717 15:01:26.831945 47025319389696 controller.py:291]  eval | step:  80072 | eval time:   49.6 sec | output: {'test_accuracy': 0.33716, 'test_loss': 0.7839137}
I0717 15:39:10.716763 47025319389696 controller.py:291]  eval | step:  90081 | eval time:   51.2 sec | output: {'test_accuracy': 0.32314, 'test_loss': 0.82485163}
I0717 16:17:14.804115 47025319389696 controller.py:291]  eval | step:  100090 | eval time:   49.1 sec | output: {'test_accuracy': 0.36102, 'test_loss': 0.7599265}
I0717 16:54:38.272225 47025319389696 controller.py:291]  eval | step:  110099 | eval time:   49.5 sec | output: {'test_accuracy': 0.42732, 'test_loss': 0.6477533}
I0717 17:32:14.305328 47025319389696 controller.py:291]  eval | step:  120108 | eval time:   49.1 sec | output: {'test_accuracy': 0.3769, 'test_loss': 0.745124}
I0717 18:19:19.995545 47025319389696 controller.py:291]  eval | step:  130117 | eval time:  196.2 sec | output: {'test_accuracy': 0.43222, 'test_loss': 0.63742775}
I0717 19:01:38.542571 47025319389696 controller.py:291]  eval | step:  140126 | eval time:   49.1 sec | output: {'test_accuracy': 0.43536, 'test_loss': 0.63410264}
I0717 19:39:52.303068 47025319389696 controller.py:291]  eval | step:  150135 | eval time:   46.7 sec | output: {'test_accuracy': 0.43782, 'test_loss': 0.6309778}
I0717 20:17:52.557711 47025319389696 controller.py:291]  eval | step:  160144 | eval time:   49.5 sec | output: {'test_accuracy': 0.44036, 'test_loss': 0.6212317}
I0717 20:55:54.236690 47025319389696 controller.py:291]  eval | step:  170153 | eval time:   47.7 sec | output: {'test_accuracy': 0.4211, 'test_loss': 0.6638591}
I0717 21:33:46.721763 47025319389696 controller.py:291]  eval | step:  180162 | eval time:   48.7 sec | output: {'test_accuracy': 0.41044, 'test_loss': 0.6806779}
I0717 22:11:56.432586 47025319389696 controller.py:291]  eval | step:  190171 | eval time:   48.9 sec | output: {'test_accuracy': 0.4227, 'test_loss': 0.6592407}
I0717 22:49:48.895009 47025319389696 controller.py:291]  eval | step:  200180 | eval time:   47.1 sec | output: {'test_accuracy': 0.41614, 'test_loss': 0.6689654}
I0717 23:27:56.641305 47025319389696 controller.py:291]  eval | step:  210189 | eval time:   49.0 sec | output: {'test_accuracy': 0.43086, 'test_loss': 0.6471556}
I0718 00:06:08.226011 47025319389696 controller.py:291]  eval | step:  220198 | eval time:   49.0 sec | output: {'test_accuracy': 0.43288, 'test_loss': 0.651372}
I0718 00:43:44.846733 47025319389696 controller.py:291]  eval | step:  230207 | eval time:   48.6 sec | output: {'test_accuracy': 0.44876, 'test_loss': 0.61514467}
I0718 01:21:01.860804 47025319389696 controller.py:291]  eval | step:  240216 | eval time:   47.8 sec | output: {'test_accuracy': 0.39994, 'test_loss': 0.6896438}
I0718 01:58:26.185002 47025319389696 controller.py:291]  eval | step:  250225 | eval time:   49.3 sec | output: {'test_accuracy': 0.43974, 'test_loss': 0.63887787}
I0718 02:35:51.395891 47025319389696 controller.py:291]  eval | step:  260234 | eval time:   48.7 sec | output: {'test_accuracy': 0.42148, 'test_loss': 0.65387714}
I0718 03:12:58.957236 47025319389696 controller.py:291]  eval | step:  270243 | eval time:   48.0 sec | output: {'test_accuracy': 0.42988, 'test_loss': 0.64366835}
I0718 03:50:05.619722 47025319389696 controller.py:291]  eval | step:  280252 | eval time:   49.1 sec | output: {'test_accuracy': 0.4292, 'test_loss': 0.64544094}
I0718 04:27:19.267962 47025319389696 controller.py:291]  eval | step:  290261 | eval time:   47.7 sec | output: {'test_accuracy': 0.43688, 'test_loss': 0.633886}
I0718 05:04:16.388976 47025319389696 controller.py:291]  eval | step:  300270 | eval time:   47.0 sec | output: {'test_accuracy': 0.4378, 'test_loss': 0.6427796}
I0718 05:41:28.633920 47025319389696 controller.py:291]  eval | step:  310279 | eval time:   47.1 sec | output: {'test_accuracy': 0.60604, 'test_loss': 0.41581097}
I0718 06:18:23.006313 47025319389696 controller.py:291]  eval | step:  320288 | eval time:   47.6 sec | output: {'test_accuracy': 0.60958, 'test_loss': 0.41247183}
I0718 06:55:19.940962 47025319389696 controller.py:291]  eval | step:  330297 | eval time:   47.5 sec | output: {'test_accuracy': 0.62418, 'test_loss': 0.3925642}
I0718 07:32:11.406366 47025319389696 controller.py:291]  eval | step:  340306 | eval time:   49.3 sec | output: {'test_accuracy': 0.60878, 'test_loss': 0.41140762}
I0718 08:09:13.098378 47025319389696 controller.py:291]  eval | step:  350315 | eval time:   46.3 sec | output: {'test_accuracy': 0.61416, 'test_loss': 0.40241298}
I0718 08:46:11.470759 47025319389696 controller.py:291]  eval | step:  360324 | eval time:   48.7 sec | output: {'test_accuracy': 0.61666, 'test_loss': 0.40103617}
I0718 09:23:19.292029 47025319389696 controller.py:291]  eval | step:  370333 | eval time:   47.6 sec | output: {'test_accuracy': 0.64238, 'test_loss': 0.3717267}
I0718 09:59:58.652599 47025319389696 controller.py:291]  eval | step:  380342 | eval time:   46.9 sec | output: {'test_accuracy': 0.63504, 'test_loss': 0.37905875}
I0718 10:36:30.269412 47025319389696 controller.py:291]  eval | step:  390351 | eval time:   46.7 sec | output: {'test_accuracy': 0.62588, 'test_loss': 0.38983408}
I0718 11:12:59.710509 47025319389696 controller.py:291]  eval | step:  400360 | eval time:   47.5 sec | output: {'test_accuracy': 0.62752, 'test_loss': 0.39042756}
I0718 11:49:34.702973 47025319389696 controller.py:291]  eval | step:  410369 | eval time:   47.6 sec | output: {'test_accuracy': 0.63086, 'test_loss': 0.38619643}
I0718 12:26:10.658050 47025319389696 controller.py:291]  eval | step:  420378 | eval time:   46.4 sec | output: {'test_accuracy': 0.61492, 'test_loss': 0.40776753}
I0718 13:02:54.373109 47025319389696 controller.py:291]  eval | step:  430387 | eval time:   47.1 sec | output: {'test_accuracy': 0.61482, 'test_loss': 0.40698445}
I0718 13:39:56.424064 47025319389696 controller.py:291]  eval | step:  440396 | eval time:   46.6 sec | output: {'test_accuracy': 0.61064, 'test_loss': 0.40885204}
I0718 14:16:56.972168 47025319389696 controller.py:291]  eval | step:  450405 | eval time:   47.2 sec | output: {'test_accuracy': 0.6101, 'test_loss': 0.41232613}
I0718 14:53:49.710101 47025319389696 controller.py:291]  eval | step:  460414 | eval time:   50.2 sec | output: {'test_accuracy': 0.62022, 'test_loss': 0.39717302}
I0718 15:31:00.055065 47025319389696 controller.py:291]  eval | step:  470423 | eval time:   47.8 sec | output: {'test_accuracy': 0.63738, 'test_loss': 0.3755849}
I0718 16:26:43.115331 47025319389696 controller.py:291]  eval | step:  480432 | eval time:   48.6 sec | output: {'test_accuracy': 0.63156, 'test_loss': 0.38420147}
I0718 17:14:57.232034 47025319389696 controller.py:291]  eval | step:  490441 | eval time:   47.9 sec | output: {'test_accuracy': 0.61762, 'test_loss': 0.40265194}
I0718 18:08:04.532505 47025319389696 controller.py:291]  eval | step:  500450 | eval time:   47.4 sec | output: {'test_accuracy': 0.63422, 'test_loss': 0.3809521}
I0718 18:45:18.862020 47025319389696 controller.py:291]  eval | step:  510459 | eval time:   50.4 sec | output: {'test_accuracy': 0.62098, 'test_loss': 0.39487627}
I0718 19:22:18.786072 47025319389696 controller.py:291]  eval | step:  520468 | eval time:   47.2 sec | output: {'test_accuracy': 0.61512, 'test_loss': 0.40484685}
I0718 19:59:17.472421 47025319389696 controller.py:291]  eval | step:  530477 | eval time:   47.3 sec | output: {'test_accuracy': 0.63776, 'test_loss': 0.37440324}
I0718 20:46:59.059379 47025319389696 controller.py:291]  eval | step:  540486 | eval time:  203.9 sec | output: {'test_accuracy': 0.62722, 'test_loss': 0.39110208}
I0718 21:45:11.991321 47025319389696 controller.py:291]  eval | step:  550495 | eval time:   47.9 sec | output: {'test_accuracy': 0.63562, 'test_loss': 0.38335004}
I0718 22:22:17.513960 47025319389696 controller.py:291]  eval | step:  560504 | eval time:   46.5 sec | output: {'test_accuracy': 0.61826, 'test_loss': 0.40100682}
I0718 22:59:05.393987 47025319389696 controller.py:291]  eval | step:  570513 | eval time:   46.1 sec | output: {'test_accuracy': 0.63496, 'test_loss': 0.3817627}
I0718 23:36:01.059101 47025319389696 controller.py:291]  eval | step:  580522 | eval time:   47.2 sec | output: {'test_accuracy': 0.61354, 'test_loss': 0.40860355}
I0719 00:12:57.767053 47025319389696 controller.py:291]  eval | step:  590531 | eval time:   48.7 sec | output: {'test_accuracy': 0.62766, 'test_loss': 0.3902482}
I0719 00:50:08.519172 47025319389696 controller.py:291]  eval | step:  600540 | eval time:   47.3 sec | output: {'test_accuracy': 0.62992, 'test_loss': 0.38722458}
I0719 01:27:14.241234 47025319389696 controller.py:291]  eval | step:  610549 | eval time:   49.5 sec | output: {'test_accuracy': 0.6785, 'test_loss': 0.3319877}
I0719 02:04:12.080565 47025319389696 controller.py:291]  eval | step:  620558 | eval time:   47.3 sec | output: {'test_accuracy': 0.68068, 'test_loss': 0.3297141}
I0719 02:41:31.931695 47025319389696 controller.py:291]  eval | step:  630567 | eval time:   47.5 sec | output: {'test_accuracy': 0.67544, 'test_loss': 0.33620217}
I0719 03:18:53.615640 47025319389696 controller.py:291]  eval | step:  640576 | eval time:   47.8 sec | output: {'test_accuracy': 0.67902, 'test_loss': 0.32960042}
I0719 03:56:03.865268 47025319389696 controller.py:291]  eval | step:  650585 | eval time:   47.2 sec | output: {'test_accuracy': 0.68312, 'test_loss': 0.32697994}
I0719 04:33:13.600398 47025319389696 controller.py:291]  eval | step:  660594 | eval time:   49.1 sec | output: {'test_accuracy': 0.68692, 'test_loss': 0.3218242}
I0719 05:10:26.761108 47025319389696 controller.py:291]  eval | step:  670603 | eval time:   47.5 sec | output: {'test_accuracy': 0.68098, 'test_loss': 0.33020604}
I0719 05:47:24.072698 47025319389696 controller.py:291]  eval | step:  680612 | eval time:   47.8 sec | output: {'test_accuracy': 0.68784, 'test_loss': 0.32333767}
I0719 06:24:23.891359 47025319389696 controller.py:291]  eval | step:  690621 | eval time:   47.1 sec | output: {'test_accuracy': 0.68598, 'test_loss': 0.32332215}
I0719 07:01:34.794637 47025319389696 controller.py:291]  eval | step:  700630 | eval time:   48.9 sec | output: {'test_accuracy': 0.68566, 'test_loss': 0.3250208}
I0719 07:38:47.115363 47025319389696 controller.py:291]  eval | step:  710639 | eval time:   47.2 sec | output: {'test_accuracy': 0.68696, 'test_loss': 0.32350788}
I0719 08:15:57.504423 47025319389696 controller.py:291]  eval | step:  720648 | eval time:   52.0 sec | output: {'test_accuracy': 0.68832, 'test_loss': 0.3217886}
I0719 08:53:14.727922 47025319389696 controller.py:291]  eval | step:  730657 | eval time:   49.9 sec | output: {'test_accuracy': 0.6852, 'test_loss': 0.32565758}
I0719 09:30:20.804430 47025319389696 controller.py:291]  eval | step:  740666 | eval time:   49.2 sec | output: {'test_accuracy': 0.68326, 'test_loss': 0.33065847}
I0719 10:07:20.642869 47025319389696 controller.py:291]  eval | step:  750675 | eval time:   48.0 sec | output: {'test_accuracy': 0.68372, 'test_loss': 0.32719776}
I0719 10:46:23.875730 47025319389696 controller.py:291]  eval | step:  760684 | eval time:   47.9 sec | output: {'test_accuracy': 0.68192, 'test_loss': 0.32962936}
I0719 11:23:17.199512 47025319389696 controller.py:291]  eval | step:  770693 | eval time:   47.2 sec | output: {'test_accuracy': 0.68262, 'test_loss': 0.32934195}
I0719 12:00:04.367842 47025319389696 controller.py:291]  eval | step:  780702 | eval time:   46.9 sec | output: {'test_accuracy': 0.67706, 'test_loss': 0.3367395}
I0719 12:37:04.218576 47025319389696 controller.py:291]  eval | step:  790711 | eval time:   49.9 sec | output: {'test_accuracy': 0.68592, 'test_loss': 0.32515442}
I0719 13:14:02.169136 47025319389696 controller.py:291]  eval | step:  800720 | eval time:   49.5 sec | output: {'test_accuracy': 0.6774, 'test_loss': 0.33403492}
I0719 13:51:13.950588 47025319389696 controller.py:291]  eval | step:  810729 | eval time:   48.3 sec | output: {'test_accuracy': 0.68796, 'test_loss': 0.32154444}
I0719 14:28:19.452734 47025319389696 controller.py:291]  eval | step:  820738 | eval time:   47.6 sec | output: {'test_accuracy': 0.68378, 'test_loss': 0.32751894}
I0719 15:05:49.973363 47025319389696 controller.py:291]  eval | step:  830747 | eval time:   47.5 sec | output: {'test_accuracy': 0.6856, 'test_loss': 0.3268724}
I0719 15:42:51.213348 47025319389696 controller.py:291]  eval | step:  840756 | eval time:   46.4 sec | output: {'test_accuracy': 0.68698, 'test_loss': 0.32407942}
I0719 16:19:54.159197 47025319389696 controller.py:291]  eval | step:  850765 | eval time:   48.8 sec | output: {'test_accuracy': 0.6875, 'test_loss': 0.32387343}
I0719 16:57:06.514332 47025319389696 controller.py:291]  eval | step:  860774 | eval time:   48.9 sec | output: {'test_accuracy': 0.6877, 'test_loss': 0.32420784}
I0719 17:34:12.212647 47025319389696 controller.py:291]  eval | step:  870783 | eval time:   48.5 sec | output: {'test_accuracy': 0.688, 'test_loss': 0.3242049}
I0719 18:11:14.854687 47025319389696 controller.py:291]  eval | step:  880792 | eval time:   48.6 sec | output: {'test_accuracy': 0.6882, 'test_loss': 0.324394}
I0719 18:48:31.401537 47025319389696 controller.py:291]  eval | step:  890801 | eval time:   49.7 sec | output: {'test_accuracy': 0.68516, 'test_loss': 0.32722312}
I0719 19:25:37.033689 47025319389696 controller.py:291]  eval | step:  900810 | eval time:   48.7 sec | output: {'test_accuracy': 0.68694, 'test_loss': 0.32629344}
 eval | step:  10009 | eval time:   74.7 sec | output: {'test_accuracy': 0.08882, 'test_loss': 1.2578498}
 eval | step:  20018 | eval time:   47.9 sec | output: {'test_accuracy': 0.18752, 'test_loss': 1.0495228}
 eval | step:  30027 | eval time:   48.0 sec | output: {'test_accuracy': 0.2584, 'test_loss': 0.9102226}
 eval | step:  40036 | eval time:   47.3 sec | output: {'test_accuracy': 0.33168, 'test_loss': 0.77455497}
 eval | step:  50045 | eval time:   48.6 sec | output: {'test_accuracy': 0.32974, 'test_loss': 0.7866249}
 eval | step:  60054 | eval time:   48.0 sec | output: {'test_accuracy': 0.32832, 'test_loss': 0.7928696}
 eval | step:  70063 | eval time:   47.4 sec | output: {'test_accuracy': 0.36952, 'test_loss': 0.7357325}
 eval | step:  80072 | eval time:   49.6 sec | output: {'test_accuracy': 0.33716, 'test_loss': 0.7839137}
 eval | step:  90081 | eval time:   51.2 sec | output: {'test_accuracy': 0.32314, 'test_loss': 0.82485163}
 eval | step:  100090 | eval time:   49.1 sec | output: {'test_accuracy': 0.36102, 'test_loss': 0.7599265}
 eval | step:  110099 | eval time:   49.5 sec | output: {'test_accuracy': 0.42732, 'test_loss': 0.6477533}
 eval | step:  120108 | eval time:   49.1 sec | output: {'test_accuracy': 0.3769, 'test_loss': 0.745124}
 eval | step:  130117 | eval time:  196.2 sec | output: {'test_accuracy': 0.43222, 'test_loss': 0.63742775}
 eval | step:  140126 | eval time:   49.1 sec | output: {'test_accuracy': 0.43536, 'test_loss': 0.63410264}
 eval | step:  150135 | eval time:   46.7 sec | output: {'test_accuracy': 0.43782, 'test_loss': 0.6309778}
 eval | step:  160144 | eval time:   49.5 sec | output: {'test_accuracy': 0.44036, 'test_loss': 0.6212317}
 eval | step:  170153 | eval time:   47.7 sec | output: {'test_accuracy': 0.4211, 'test_loss': 0.6638591}
 eval | step:  180162 | eval time:   48.7 sec | output: {'test_accuracy': 0.41044, 'test_loss': 0.6806779}
 eval | step:  190171 | eval time:   48.9 sec | output: {'test_accuracy': 0.4227, 'test_loss': 0.6592407}
 eval | step:  200180 | eval time:   47.1 sec | output: {'test_accuracy': 0.41614, 'test_loss': 0.6689654}
 eval | step:  210189 | eval time:   49.0 sec | output: {'test_accuracy': 0.43086, 'test_loss': 0.6471556}
 eval | step:  220198 | eval time:   49.0 sec | output: {'test_accuracy': 0.43288, 'test_loss': 0.651372}
 eval | step:  230207 | eval time:   48.6 sec | output: {'test_accuracy': 0.44876, 'test_loss': 0.61514467}
 eval | step:  240216 | eval time:   47.8 sec | output: {'test_accuracy': 0.39994, 'test_loss': 0.6896438}
 eval | step:  250225 | eval time:   49.3 sec | output: {'test_accuracy': 0.43974, 'test_loss': 0.63887787}
 eval | step:  260234 | eval time:   48.7 sec | output: {'test_accuracy': 0.42148, 'test_loss': 0.65387714}
 eval | step:  270243 | eval time:   48.0 sec | output: {'test_accuracy': 0.42988, 'test_loss': 0.64366835}
 eval | step:  280252 | eval time:   49.1 sec | output: {'test_accuracy': 0.4292, 'test_loss': 0.64544094}
 eval | step:  290261 | eval time:   47.7 sec | output: {'test_accuracy': 0.43688, 'test_loss': 0.633886}
 eval | step:  300270 | eval time:   47.0 sec | output: {'test_accuracy': 0.4378, 'test_loss': 0.6427796}
 eval | step:  310279 | eval time:   47.1 sec | output: {'test_accuracy': 0.60604, 'test_loss': 0.41581097}
 eval | step:  320288 | eval time:   47.6 sec | output: {'test_accuracy': 0.60958, 'test_loss': 0.41247183}
 eval | step:  330297 | eval time:   47.5 sec | output: {'test_accuracy': 0.62418, 'test_loss': 0.3925642}
 eval | step:  340306 | eval time:   49.3 sec | output: {'test_accuracy': 0.60878, 'test_loss': 0.41140762}
 eval | step:  350315 | eval time:   46.3 sec | output: {'test_accuracy': 0.61416, 'test_loss': 0.40241298}
 eval | step:  360324 | eval time:   48.7 sec | output: {'test_accuracy': 0.61666, 'test_loss': 0.40103617}
 eval | step:  370333 | eval time:   47.6 sec | output: {'test_accuracy': 0.64238, 'test_loss': 0.3717267}
 eval | step:  380342 | eval time:   46.9 sec | output: {'test_accuracy': 0.63504, 'test_loss': 0.37905875}
 eval | step:  390351 | eval time:   46.7 sec | output: {'test_accuracy': 0.62588, 'test_loss': 0.38983408}
 eval | step:  400360 | eval time:   47.5 sec | output: {'test_accuracy': 0.62752, 'test_loss': 0.39042756}
 eval | step:  410369 | eval time:   47.6 sec | output: {'test_accuracy': 0.63086, 'test_loss': 0.38619643}
 eval | step:  420378 | eval time:   46.4 sec | output: {'test_accuracy': 0.61492, 'test_loss': 0.40776753}
 eval | step:  430387 | eval time:   47.1 sec | output: {'test_accuracy': 0.61482, 'test_loss': 0.40698445}
 eval | step:  440396 | eval time:   46.6 sec | output: {'test_accuracy': 0.61064, 'test_loss': 0.40885204}
 eval | step:  450405 | eval time:   47.2 sec | output: {'test_accuracy': 0.6101, 'test_loss': 0.41232613}
 eval | step:  460414 | eval time:   50.2 sec | output: {'test_accuracy': 0.62022, 'test_loss': 0.39717302}
 eval | step:  470423 | eval time:   47.8 sec | output: {'test_accuracy': 0.63738, 'test_loss': 0.3755849}
 eval | step:  480432 | eval time:   48.6 sec | output: {'test_accuracy': 0.63156, 'test_loss': 0.38420147}
 eval | step:  490441 | eval time:   47.9 sec | output: {'test_accuracy': 0.61762, 'test_loss': 0.40265194}
 eval | step:  500450 | eval time:   47.4 sec | output: {'test_accuracy': 0.63422, 'test_loss': 0.3809521}
 eval | step:  510459 | eval time:   50.4 sec | output: {'test_accuracy': 0.62098, 'test_loss': 0.39487627}
 eval | step:  520468 | eval time:   47.2 sec | output: {'test_accuracy': 0.61512, 'test_loss': 0.40484685}
 eval | step:  530477 | eval time:   47.3 sec | output: {'test_accuracy': 0.63776, 'test_loss': 0.37440324}
 eval | step:  540486 | eval time:  203.9 sec | output: {'test_accuracy': 0.62722, 'test_loss': 0.39110208}
 eval | step:  550495 | eval time:   47.9 sec | output: {'test_accuracy': 0.63562, 'test_loss': 0.38335004}
 eval | step:  560504 | eval time:   46.5 sec | output: {'test_accuracy': 0.61826, 'test_loss': 0.40100682}
 eval | step:  570513 | eval time:   46.1 sec | output: {'test_accuracy': 0.63496, 'test_loss': 0.3817627}
 eval | step:  580522 | eval time:   47.2 sec | output: {'test_accuracy': 0.61354, 'test_loss': 0.40860355}
 eval | step:  590531 | eval time:   48.7 sec | output: {'test_accuracy': 0.62766, 'test_loss': 0.3902482}
 eval | step:  600540 | eval time:   47.3 sec | output: {'test_accuracy': 0.62992, 'test_loss': 0.38722458}
 eval | step:  610549 | eval time:   49.5 sec | output: {'test_accuracy': 0.6785, 'test_loss': 0.3319877}
 eval | step:  620558 | eval time:   47.3 sec | output: {'test_accuracy': 0.68068, 'test_loss': 0.3297141}
 eval | step:  630567 | eval time:   47.5 sec | output: {'test_accuracy': 0.67544, 'test_loss': 0.33620217}
 eval | step:  640576 | eval time:   47.8 sec | output: {'test_accuracy': 0.67902, 'test_loss': 0.32960042}
 eval | step:  650585 | eval time:   47.2 sec | output: {'test_accuracy': 0.68312, 'test_loss': 0.32697994}
 eval | step:  660594 | eval time:   49.1 sec | output: {'test_accuracy': 0.68692, 'test_loss': 0.3218242}
 eval | step:  670603 | eval time:   47.5 sec | output: {'test_accuracy': 0.68098, 'test_loss': 0.33020604}
 eval | step:  680612 | eval time:   47.8 sec | output: {'test_accuracy': 0.68784, 'test_loss': 0.32333767}
 eval | step:  690621 | eval time:   47.1 sec | output: {'test_accuracy': 0.68598, 'test_loss': 0.32332215}
 eval | step:  700630 | eval time:   48.9 sec | output: {'test_accuracy': 0.68566, 'test_loss': 0.3250208}
 eval | step:  710639 | eval time:   47.2 sec | output: {'test_accuracy': 0.68696, 'test_loss': 0.32350788}
 eval | step:  720648 | eval time:   52.0 sec | output: {'test_accuracy': 0.68832, 'test_loss': 0.3217886}
 eval | step:  730657 | eval time:   49.9 sec | output: {'test_accuracy': 0.6852, 'test_loss': 0.32565758}
 eval | step:  740666 | eval time:   49.2 sec | output: {'test_accuracy': 0.68326, 'test_loss': 0.33065847}
 eval | step:  750675 | eval time:   48.0 sec | output: {'test_accuracy': 0.68372, 'test_loss': 0.32719776}
 eval | step:  760684 | eval time:   47.9 sec | output: {'test_accuracy': 0.68192, 'test_loss': 0.32962936}
 eval | step:  770693 | eval time:   47.2 sec | output: {'test_accuracy': 0.68262, 'test_loss': 0.32934195}
 eval | step:  780702 | eval time:   46.9 sec | output: {'test_accuracy': 0.67706, 'test_loss': 0.3367395}
 eval | step:  790711 | eval time:   49.9 sec | output: {'test_accuracy': 0.68592, 'test_loss': 0.32515442}
 eval | step:  800720 | eval time:   49.5 sec | output: {'test_accuracy': 0.6774, 'test_loss': 0.33403492}
 eval | step:  810729 | eval time:   48.3 sec | output: {'test_accuracy': 0.68796, 'test_loss': 0.32154444}
 eval | step:  820738 | eval time:   47.6 sec | output: {'test_accuracy': 0.68378, 'test_loss': 0.32751894}
 eval | step:  830747 | eval time:   47.5 sec | output: {'test_accuracy': 0.6856, 'test_loss': 0.3268724}
 eval | step:  840756 | eval time:   46.4 sec | output: {'test_accuracy': 0.68698, 'test_loss': 0.32407942}
 eval | step:  850765 | eval time:   48.8 sec | output: {'test_accuracy': 0.6875, 'test_loss': 0.32387343}
 eval | step:  860774 | eval time:   48.9 sec | output: {'test_accuracy': 0.6877, 'test_loss': 0.32420784}
 eval | step:  870783 | eval time:   48.5 sec | output: {'test_accuracy': 0.688, 'test_loss': 0.3242049}
 eval | step:  880792 | eval time:   48.6 sec | output: {'test_accuracy': 0.6882, 'test_loss': 0.324394}
 eval | step:  890801 | eval time:   49.7 sec | output: {'test_accuracy': 0.68516, 'test_loss': 0.32722312}
 eval | step:  900810 | eval time:   48.7 sec | output: {'test_accuracy': 0.68694, 'test_loss': 0.32629344}
```