coco_zero_shot.log

(coco step1: zero-shot evaluation) load pretrained model from: 
/share/home/qxnie/anaconda3/lib/python3.9/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
args.rank 1 args.world_size 4 args.gpu 1
get_rank() 0
args.rank 3 args.world_size 4 args.gpu 3
get_rank() 0
args.rank 0 args.world_size 4 args.gpu 0
get_rank() 0
args.rank 2 args.world_size 4 args.gpu 2
get_rank() 0
| distributed init (rank 2): env://
| distributed init (rank 1): env://
| distributed init (rank 3): env://
| distributed init (rank 0): env://
Creating retrieval dataset
{'image_root': '/share/common/ImageDatasets/coco2017', 'ann_root': 'annotation', 'dataset': 'coco', 'pretrained': 'output/zero_shot_coco_checkpoint_4m.pth', 'vit': 'base', 'batch_size_train': 24, 'batch_size_test': 48, 'vit_grad_ckpt': True, 'vit_ckpt_layer': 4, 'init_lr': 1e-05, 'image_size': 384, 'queue_size': 57600, 'alpha': 0.4, 'k_test': 256, 'negative_all_rank': True, 'weight_decay': 0.05, 'min_lr': 0, 'max_epoch': 6}
Creating model
load checkpoint for output/zero_shot_coco_checkpoint_4m.pth
reshape position embedding from 196 to 576
reshape position embedding from 196 to 576
load checkpoint from output/zero_shot_coco_checkpoint_4m.pth
missing keys:
['idx_queue', 'ptr_queue']
Start training
Computing features for evaluation...
Evaluation:  [   0/1251]  eta: 0:02:21    time: 0.1129  data: 0.0010  max mem: 6164
Evaluation:  [  50/1251]  eta: 0:10:41    time: 0.5462  data: 0.0000  max mem: 6190
Evaluation:  [ 100/1251]  eta: 0:10:19    time: 0.5431  data: 0.0000  max mem: 6190
Evaluation:  [ 150/1251]  eta: 0:09:55    time: 0.5438  data: 0.0000  max mem: 6190
Evaluation:  [ 200/1251]  eta: 0:09:29    time: 0.5459  data: 0.0000  max mem: 6190
Evaluation:  [ 250/1251]  eta: 0:09:03    time: 0.5463  data: 0.0000  max mem: 6190
Evaluation:  [ 300/1251]  eta: 0:08:37    time: 0.5463  data: 0.0000  max mem: 6190
Evaluation:  [ 350/1251]  eta: 0:08:10    time: 0.5479  data: 0.0000  max mem: 6190
Evaluation:  [ 400/1251]  eta: 0:07:43    time: 0.5483  data: 0.0000  max mem: 6190
Evaluation:  [ 450/1251]  eta: 0:07:16    time: 0.5475  data: 0.0000  max mem: 6190
Evaluation:  [ 500/1251]  eta: 0:06:49    time: 0.5470  data: 0.0000  max mem: 6190
Evaluation:  [ 550/1251]  eta: 0:06:22    time: 0.5445  data: 0.0000  max mem: 6190
Evaluation:  [ 600/1251]  eta: 0:05:54    time: 0.5444  data: 0.0000  max mem: 6190
Evaluation:  [ 650/1251]  eta: 0:05:27    time: 0.5464  data: 0.0000  max mem: 6190
Evaluation:  [ 700/1251]  eta: 0:05:00    time: 0.5464  data: 0.0000  max mem: 6190
Evaluation:  [ 750/1251]  eta: 0:04:33    time: 0.5475  data: 0.0000  max mem: 6190
Evaluation:  [ 800/1251]  eta: 0:04:06    time: 0.5483  data: 0.0000  max mem: 6190
Evaluation:  [ 850/1251]  eta: 0:03:38    time: 0.5470  data: 0.0000  max mem: 6190
Evaluation:  [ 900/1251]  eta: 0:03:11    time: 0.5459  data: 0.0000  max mem: 6190
Evaluation:  [ 950/1251]  eta: 0:02:44    time: 0.5458  data: 0.0000  max mem: 6190
Evaluation:  [1000/1251]  eta: 0:02:16    time: 0.5449  data: 0.0000  max mem: 6190
Evaluation:  [1050/1251]  eta: 0:01:49    time: 0.5457  data: 0.0000  max mem: 6190
Evaluation:  [1100/1251]  eta: 0:01:22    time: 0.5462  data: 0.0000  max mem: 6190
Evaluation:  [1150/1251]  eta: 0:00:55    time: 0.5483  data: 0.0000  max mem: 6190
Evaluation:  [1200/1251]  eta: 0:00:27    time: 0.5480  data: 0.0000  max mem: 6190
Evaluation:  [1250/1251]  eta: 0:00:00    time: 0.5483  data: 0.0000  max mem: 6190
Evaluation: Total time: 0:11:23 (0.5461 s / it)
Evaluation:  [   0/6253]  eta: 0:22:58    time: 0.2205  data: 0.0040  max mem: 6668
Evaluation:  [  50/6253]  eta: 1:03:44    time: 0.6158  data: 0.0000  max mem: 6668
Evaluation:  [ 100/6253]  eta: 1:03:07    time: 0.6123  data: 0.0000  max mem: 6668
Evaluation:  [ 150/6253]  eta: 1:02:35    time: 0.6154  data: 0.0000  max mem: 6668
Evaluation:  [ 200/6253]  eta: 1:02:01    time: 0.6113  data: 0.0000  max mem: 6668
Evaluation:  [ 250/6253]  eta: 1:01:30    time: 0.6135  data: 0.0000  max mem: 6668
Evaluation:  [ 300/6253]  eta: 1:00:56    time: 0.6103  data: 0.0000  max mem: 6668
Evaluation:  [ 350/6253]  eta: 1:00:24    time: 0.6109  data: 0.0000  max mem: 6668
Evaluation:  [ 400/6253]  eta: 0:59:53    time: 0.6137  data: 0.0000  max mem: 6668
Evaluation:  [ 450/6253]  eta: 0:59:20    time: 0.6117  data: 0.0000  max mem: 6668
Evaluation:  [ 500/6253]  eta: 0:58:49    time: 0.6127  data: 0.0000  max mem: 6668
Evaluation:  [ 550/6253]  eta: 0:58:17    time: 0.6091  data: 0.0000  max mem: 6668
Evaluation:  [ 600/6253]  eta: 0:57:47    time: 0.6138  data: 0.0000  max mem: 6668
Evaluation:  [ 650/6253]  eta: 0:57:16    time: 0.6123  data: 0.0000  max mem: 6668
Evaluation:  [ 700/6253]  eta: 0:56:45    time: 0.6090  data: 0.0000  max mem: 6668
Evaluation:  [ 750/6253]  eta: 0:56:14    time: 0.6093  data: 0.0000  max mem: 6668
Evaluation:  [ 800/6253]  eta: 0:55:42    time: 0.6124  data: 0.0000  max mem: 6668
Evaluation:  [ 850/6253]  eta: 0:55:11    time: 0.6091  data: 0.0000  max mem: 6668
Evaluation:  [ 900/6253]  eta: 0:54:41    time: 0.6151  data: 0.0000  max mem: 6668
Evaluation:  [ 950/6253]  eta: 0:54:11    time: 0.6139  data: 0.0000  max mem: 6668
Evaluation:  [1000/6253]  eta: 0:53:39    time: 0.6102  data: 0.0000  max mem: 6668
Evaluation:  [1050/6253]  eta: 0:53:09    time: 0.6097  data: 0.0000  max mem: 6668
Evaluation:  [1100/6253]  eta: 0:52:39    time: 0.6188  data: 0.0000  max mem: 6668
Evaluation:  [1150/6253]  eta: 0:52:08    time: 0.6135  data: 0.0000  max mem: 6668
Evaluation:  [1200/6253]  eta: 0:51:37    time: 0.6169  data: 0.0000  max mem: 6668
Evaluation:  [1250/6253]  eta: 0:51:06    time: 0.6117  data: 0.0000  max mem: 6668
Evaluation:  [1300/6253]  eta: 0:50:35    time: 0.6129  data: 0.0000  max mem: 6668
Evaluation:  [1350/6253]  eta: 0:50:04    time: 0.6131  data: 0.0000  max mem: 6668
Evaluation:  [1400/6253]  eta: 0:49:32    time: 0.6129  data: 0.0000  max mem: 6668
Evaluation:  [1450/6253]  eta: 0:49:01    time: 0.6158  data: 0.0000  max mem: 6668
Evaluation:  [1500/6253]  eta: 0:48:31    time: 0.6153  data: 0.0000  max mem: 6668
Evaluation:  [1550/6253]  eta: 0:48:00    time: 0.6088  data: 0.0000  max mem: 6668
Evaluation:  [1600/6253]  eta: 0:47:30    time: 0.6084  data: 0.0000  max mem: 6668
Evaluation:  [1650/6253]  eta: 0:46:59    time: 0.6117  data: 0.0000  max mem: 6668
Evaluation:  [1700/6253]  eta: 0:46:29    time: 0.6195  data: 0.0000  max mem: 6668
Evaluation:  [1750/6253]  eta: 0:45:58    time: 0.6138  data: 0.0000  max mem: 6668
Evaluation:  [1800/6253]  eta: 0:45:28    time: 0.6171  data: 0.0000  max mem: 6668
Evaluation:  [1850/6253]  eta: 0:44:57    time: 0.6095  data: 0.0000  max mem: 6668
Evaluation:  [1900/6253]  eta: 0:44:26    time: 0.6098  data: 0.0000  max mem: 6668
Evaluation:  [1950/6253]  eta: 0:43:56    time: 0.6080  data: 0.0000  max mem: 6668
Evaluation:  [2000/6253]  eta: 0:43:25    time: 0.6107  data: 0.0000  max mem: 6668
Evaluation:  [2050/6253]  eta: 0:42:54    time: 0.6115  data: 0.0000  max mem: 6668
Evaluation:  [2100/6253]  eta: 0:42:24    time: 0.6127  data: 0.0000  max mem: 6668
Evaluation:  [2150/6253]  eta: 0:41:53    time: 0.6105  data: 0.0000  max mem: 6668
Evaluation:  [2200/6253]  eta: 0:41:22    time: 0.6181  data: 0.0000  max mem: 6668
Evaluation:  [2250/6253]  eta: 0:40:52    time: 0.6170  data: 0.0000  max mem: 6668
Evaluation:  [2300/6253]  eta: 0:40:21    time: 0.6139  data: 0.0000  max mem: 6668
Evaluation:  [2350/6253]  eta: 0:39:50    time: 0.6094  data: 0.0000  max mem: 6668
Evaluation:  [2400/6253]  eta: 0:39:20    time: 0.6099  data: 0.0000  max mem: 6668
Evaluation:  [2450/6253]  eta: 0:38:49    time: 0.6116  data: 0.0000  max mem: 6668
Evaluation:  [2500/6253]  eta: 0:38:18    time: 0.6106  data: 0.0000  max mem: 6668
Evaluation:  [2550/6253]  eta: 0:37:48    time: 0.6111  data: 0.0000  max mem: 6668
Evaluation:  [2600/6253]  eta: 0:37:17    time: 0.6103  data: 0.0000  max mem: 6668
Evaluation:  [2650/6253]  eta: 0:36:46    time: 0.6070  data: 0.0000  max mem: 6668
Evaluation:  [2700/6253]  eta: 0:36:16    time: 0.6225  data: 0.0000  max mem: 6668
Evaluation:  [2750/6253]  eta: 0:35:45    time: 0.6148  data: 0.0000  max mem: 6668
Evaluation:  [2800/6253]  eta: 0:35:15    time: 0.6145  data: 0.0000  max mem: 6668
Evaluation:  [2850/6253]  eta: 0:34:44    time: 0.6129  data: 0.0000  max mem: 6668
Evaluation:  [2900/6253]  eta: 0:34:14    time: 0.6167  data: 0.0000  max mem: 6668
Evaluation:  [2950/6253]  eta: 0:33:43    time: 0.6077  data: 0.0000  max mem: 6668
Evaluation:  [3000/6253]  eta: 0:33:12    time: 0.6097  data: 0.0000  max mem: 6668
Evaluation:  [3050/6253]  eta: 0:32:41    time: 0.6089  data: 0.0000  max mem: 6668
Evaluation:  [3100/6253]  eta: 0:32:11    time: 0.6119  data: 0.0000  max mem: 6668
Evaluation:  [3150/6253]  eta: 0:31:40    time: 0.6100  data: 0.0000  max mem: 6668
Evaluation:  [3200/6253]  eta: 0:31:10    time: 0.6164  data: 0.0000  max mem: 6668
Evaluation:  [3250/6253]  eta: 0:30:39    time: 0.6106  data: 0.0000  max mem: 6668
Evaluation:  [3300/6253]  eta: 0:30:08    time: 0.6143  data: 0.0000  max mem: 6668
Evaluation:  [3350/6253]  eta: 0:29:38    time: 0.6126  data: 0.0000  max mem: 6668
Evaluation:  [3400/6253]  eta: 0:29:07    time: 0.6086  data: 0.0000  max mem: 6668
Evaluation:  [3450/6253]  eta: 0:28:36    time: 0.6103  data: 0.0000  max mem: 6668
Evaluation:  [3500/6253]  eta: 0:28:06    time: 0.6089  data: 0.0000  max mem: 6668
Evaluation:  [3550/6253]  eta: 0:27:35    time: 0.6113  data: 0.0000  max mem: 6668
Evaluation:  [3600/6253]  eta: 0:27:05    time: 0.6153  data: 0.0000  max mem: 6668
Evaluation:  [3650/6253]  eta: 0:26:34    time: 0.6157  data: 0.0000  max mem: 6668
Evaluation:  [3700/6253]  eta: 0:26:03    time: 0.6159  data: 0.0000  max mem: 6668
Evaluation:  [3750/6253]  eta: 0:25:33    time: 0.6149  data: 0.0000  max mem: 6668
Evaluation:  [3800/6253]  eta: 0:25:02    time: 0.6110  data: 0.0000  max mem: 6668
Evaluation:  [3850/6253]  eta: 0:24:31    time: 0.6160  data: 0.0000  max mem: 6668
Evaluation:  [3900/6253]  eta: 0:24:01    time: 0.6098  data: 0.0000  max mem: 6668
Evaluation:  [3950/6253]  eta: 0:23:30    time: 0.6084  data: 0.0000  max mem: 6668
Evaluation:  [4000/6253]  eta: 0:22:59    time: 0.6098  data: 0.0000  max mem: 6668
Evaluation:  [4050/6253]  eta: 0:22:29    time: 0.6152  data: 0.0000  max mem: 6668
Evaluation:  [4100/6253]  eta: 0:21:58    time: 0.6154  data: 0.0000  max mem: 6668
Evaluation:  [4150/6253]  eta: 0:21:28    time: 0.6139  data: 0.0000  max mem: 6668
Evaluation:  [4200/6253]  eta: 0:20:57    time: 0.6121  data: 0.0000  max mem: 6668
Evaluation:  [4250/6253]  eta: 0:20:26    time: 0.6136  data: 0.0000  max mem: 6668
Evaluation:  [4300/6253]  eta: 0:19:56    time: 0.6101  data: 0.0000  max mem: 6668
Evaluation:  [4350/6253]  eta: 0:19:25    time: 0.6138  data: 0.0000  max mem: 6668
Evaluation:  [4400/6253]  eta: 0:18:54    time: 0.6115  data: 0.0000  max mem: 6668
Evaluation:  [4450/6253]  eta: 0:18:24    time: 0.6114  data: 0.0000  max mem: 6668
Evaluation:  [4500/6253]  eta: 0:17:53    time: 0.6083  data: 0.0000  max mem: 6668
Evaluation:  [4550/6253]  eta: 0:17:22    time: 0.6082  data: 0.0000  max mem: 6668
Evaluation:  [4600/6253]  eta: 0:16:52    time: 0.6145  data: 0.0000  max mem: 6668
Evaluation:  [4650/6253]  eta: 0:16:21    time: 0.6145  data: 0.0000  max mem: 6668
Evaluation:  [4700/6253]  eta: 0:15:51    time: 0.6065  data: 0.0000  max mem: 6668
Evaluation:  [4750/6253]  eta: 0:15:20    time: 0.6085  data: 0.0000  max mem: 6668
Evaluation:  [4800/6253]  eta: 0:14:49    time: 0.6093  data: 0.0000  max mem: 6668
Evaluation:  [4850/6253]  eta: 0:14:19    time: 0.6097  data: 0.0000  max mem: 6668
Evaluation:  [4900/6253]  eta: 0:13:48    time: 0.6131  data: 0.0000  max mem: 6668
Evaluation:  [4950/6253]  eta: 0:13:17    time: 0.6141  data: 0.0000  max mem: 6668
Evaluation:  [5000/6253]  eta: 0:12:47    time: 0.6140  data: 0.0000  max mem: 6668
Evaluation:  [5050/6253]  eta: 0:12:16    time: 0.6086  data: 0.0000  max mem: 6668
Evaluation:  [5100/6253]  eta: 0:11:46    time: 0.6112  data: 0.0000  max mem: 6668
Evaluation:  [5150/6253]  eta: 0:11:15    time: 0.6109  data: 0.0000  max mem: 6668
Evaluation:  [5200/6253]  eta: 0:10:44    time: 0.6091  data: 0.0000  max mem: 6668
Evaluation:  [5250/6253]  eta: 0:10:14    time: 0.6108  data: 0.0000  max mem: 6668
Evaluation:  [5300/6253]  eta: 0:09:43    time: 0.6118  data: 0.0000  max mem: 6668
Evaluation:  [5350/6253]  eta: 0:09:12    time: 0.6120  data: 0.0000  max mem: 6668
Evaluation:  [5400/6253]  eta: 0:08:42    time: 0.6114  data: 0.0000  max mem: 6668
Evaluation:  [5450/6253]  eta: 0:08:11    time: 0.6124  data: 0.0000  max mem: 6668
Evaluation:  [5500/6253]  eta: 0:07:41    time: 0.6148  data: 0.0000  max mem: 6668
Evaluation:  [5550/6253]  eta: 0:07:10    time: 0.6136  data: 0.0000  max mem: 6668
Evaluation:  [5600/6253]  eta: 0:06:39    time: 0.6103  data: 0.0000  max mem: 6668
Evaluation:  [5650/6253]  eta: 0:06:09    time: 0.6138  data: 0.0000  max mem: 6668
Evaluation:  [5700/6253]  eta: 0:05:38    time: 0.6124  data: 0.0000  max mem: 6668
Evaluation:  [5750/6253]  eta: 0:05:07    time: 0.6180  data: 0.0000  max mem: 6668
Evaluation:  [5800/6253]  eta: 0:04:37    time: 0.6118  data: 0.0000  max mem: 6668
Evaluation:  [5850/6253]  eta: 0:04:06    time: 0.6153  data: 0.0000  max mem: 6668
Evaluation:  [5900/6253]  eta: 0:03:36    time: 0.6109  data: 0.0000  max mem: 6668
Evaluation:  [5950/6253]  eta: 0:03:05    time: 0.6107  data: 0.0000  max mem: 6668
Evaluation:  [6000/6253]  eta: 0:02:34    time: 0.6170  data: 0.0000  max mem: 6668
Evaluation:  [6050/6253]  eta: 0:02:04    time: 0.6168  data: 0.0000  max mem: 6668
Evaluation:  [6100/6253]  eta: 0:01:33    time: 0.6123  data: 0.0000  max mem: 6668
Evaluation:  [6150/6253]  eta: 0:01:03    time: 0.6119  data: 0.0000  max mem: 6668
Evaluation:  [6200/6253]  eta: 0:00:32    time: 0.6105  data: 0.0000  max mem: 6668
Evaluation:  [6250/6253]  eta: 0:00:01    time: 0.6111  data: 0.0000  max mem: 6668
Evaluation:  [6252/6253]  eta: 0:00:00    time: 0.6115  data: 0.0000  max mem: 6668
Evaluation: Total time: 1:03:49 (0.6125 s / it)
Evaluation time 1:19:18
Computing features for evaluation...
Evaluation:  [   0/1251]  eta: 0:02:30    time: 0.1199  data: 0.0010  max mem: 6668
Evaluation:  [  50/1251]  eta: 0:10:44    time: 0.5472  data: 0.0000  max mem: 6668
Evaluation:  [ 100/1251]  eta: 0:10:22    time: 0.5453  data: 0.0000  max mem: 6668
Evaluation:  [ 150/1251]  eta: 0:09:58    time: 0.5502  data: 0.0000  max mem: 6668
Evaluation:  [ 200/1251]  eta: 0:09:31    time: 0.5462  data: 0.0000  max mem: 6668
Evaluation:  [ 250/1251]  eta: 0:09:05    time: 0.5452  data: 0.0000  max mem: 6668
Evaluation:  [ 300/1251]  eta: 0:08:38    time: 0.5455  data: 0.0000  max mem: 6668
Evaluation:  [ 350/1251]  eta: 0:08:11    time: 0.5454  data: 0.0000  max mem: 6668
Evaluation:  [ 400/1251]  eta: 0:07:44    time: 0.5460  data: 0.0000  max mem: 6668
Evaluation:  [ 450/1251]  eta: 0:07:17    time: 0.5470  data: 0.0000  max mem: 6668
Evaluation:  [ 500/1251]  eta: 0:06:49    time: 0.5465  data: 0.0000  max mem: 6668
Evaluation:  [ 550/1251]  eta: 0:06:22    time: 0.5446  data: 0.0000  max mem: 6668
Evaluation:  [ 600/1251]  eta: 0:05:55    time: 0.5474  data: 0.0000  max mem: 6668
Evaluation:  [ 650/1251]  eta: 0:05:28    time: 0.5464  data: 0.0000  max mem: 6668
Evaluation:  [ 700/1251]  eta: 0:05:01    time: 0.5484  data: 0.0000  max mem: 6668
Evaluation:  [ 750/1251]  eta: 0:04:33    time: 0.5490  data: 0.0000  max mem: 6668
Evaluation:  [ 800/1251]  eta: 0:04:06    time: 0.5477  data: 0.0000  max mem: 6668
Evaluation:  [ 850/1251]  eta: 0:03:39    time: 0.5505  data: 0.0000  max mem: 6668
Evaluation:  [ 900/1251]  eta: 0:03:11    time: 0.5493  data: 0.0000  max mem: 6668
Evaluation:  [ 950/1251]  eta: 0:02:44    time: 0.5470  data: 0.0000  max mem: 6668
Evaluation:  [1000/1251]  eta: 0:02:17    time: 0.5491  data: 0.0000  max mem: 6668
Evaluation:  [1050/1251]  eta: 0:01:49    time: 0.5481  data: 0.0000  max mem: 6668
Evaluation:  [1100/1251]  eta: 0:01:22    time: 0.5467  data: 0.0000  max mem: 6668
Evaluation:  [1150/1251]  eta: 0:00:55    time: 0.5444  data: 0.0000  max mem: 6668
Evaluation:  [1200/1251]  eta: 0:00:27    time: 0.5452  data: 0.0000  max mem: 6668
Evaluation:  [1250/1251]  eta: 0:00:00    time: 0.5465  data: 0.0000  max mem: 6668
Evaluation: Total time: 0:11:23 (0.5466 s / it)
Evaluation:  [   0/6253]  eta: 0:26:56    time: 0.2585  data: 0.0041  max mem: 6668
Evaluation:  [  50/6253]  eta: 1:04:00    time: 0.6132  data: 0.0000  max mem: 6668
Evaluation:  [ 100/6253]  eta: 1:03:28    time: 0.6168  data: 0.0000  max mem: 6668
Evaluation:  [ 150/6253]  eta: 1:02:50    time: 0.6153  data: 0.0000  max mem: 6668
Evaluation:  [ 200/6253]  eta: 1:02:13    time: 0.6114  data: 0.0000  max mem: 6668
Evaluation:  [ 250/6253]  eta: 1:01:37    time: 0.6102  data: 0.0000  max mem: 6668
Evaluation:  [ 300/6253]  eta: 1:01:04    time: 0.6135  data: 0.0000  max mem: 6668
Evaluation:  [ 350/6253]  eta: 1:00:30    time: 0.6121  data: 0.0000  max mem: 6668
Evaluation:  [ 400/6253]  eta: 0:59:58    time: 0.6144  data: 0.0000  max mem: 6668
Evaluation:  [ 450/6253]  eta: 0:59:27    time: 0.6160  data: 0.0000  max mem: 6668
Evaluation:  [ 500/6253]  eta: 0:58:56    time: 0.6145  data: 0.0000  max mem: 6668
Evaluation:  [ 550/6253]  eta: 0:58:24    time: 0.6161  data: 0.0000  max mem: 6668
Evaluation:  [ 600/6253]  eta: 0:57:52    time: 0.6083  data: 0.0000  max mem: 6668
Evaluation:  [ 650/6253]  eta: 0:57:22    time: 0.6162  data: 0.0000  max mem: 6668
Evaluation:  [ 700/6253]  eta: 0:56:50    time: 0.6160  data: 0.0000  max mem: 6668
Evaluation:  [ 750/6253]  eta: 0:56:19    time: 0.6143  data: 0.0000  max mem: 6668
Evaluation:  [ 800/6253]  eta: 0:55:48    time: 0.6122  data: 0.0000  max mem: 6668
Evaluation:  [ 850/6253]  eta: 0:55:17    time: 0.6112  data: 0.0000  max mem: 6668
Evaluation:  [ 900/6253]  eta: 0:54:46    time: 0.6166  data: 0.0000  max mem: 6668
Evaluation:  [ 950/6253]  eta: 0:54:15    time: 0.6136  data: 0.0000  max mem: 6668
Evaluation:  [1000/6253]  eta: 0:53:44    time: 0.6163  data: 0.0000  max mem: 6668
Evaluation:  [1050/6253]  eta: 0:53:13    time: 0.6143  data: 0.0000  max mem: 6668
Evaluation:  [1100/6253]  eta: 0:52:43    time: 0.6134  data: 0.0000  max mem: 6668
Evaluation:  [1150/6253]  eta: 0:52:12    time: 0.6131  data: 0.0000  max mem: 6668
Evaluation:  [1200/6253]  eta: 0:51:41    time: 0.6112  data: 0.0000  max mem: 6668
Evaluation:  [1250/6253]  eta: 0:51:11    time: 0.6114  data: 0.0000  max mem: 6668
Evaluation:  [1300/6253]  eta: 0:50:40    time: 0.6117  data: 0.0000  max mem: 6668
Evaluation:  [1350/6253]  eta: 0:50:09    time: 0.6091  data: 0.0000  max mem: 6668
Evaluation:  [1400/6253]  eta: 0:49:39    time: 0.6157  data: 0.0000  max mem: 6668
Evaluation:  [1450/6253]  eta: 0:49:07    time: 0.6137  data: 0.0000  max mem: 6668
Evaluation:  [1500/6253]  eta: 0:48:37    time: 0.6139  data: 0.0000  max mem: 6668
Evaluation:  [1550/6253]  eta: 0:48:06    time: 0.6158  data: 0.0000  max mem: 6668
Evaluation:  [1600/6253]  eta: 0:47:35    time: 0.6123  data: 0.0000  max mem: 6668
Evaluation:  [1650/6253]  eta: 0:47:05    time: 0.6137  data: 0.0000  max mem: 6668
Evaluation:  [1700/6253]  eta: 0:46:34    time: 0.6150  data: 0.0000  max mem: 6668
Evaluation:  [1750/6253]  eta: 0:46:03    time: 0.6178  data: 0.0000  max mem: 6668
Evaluation:  [1800/6253]  eta: 0:45:33    time: 0.6169  data: 0.0000  max mem: 6668
Evaluation:  [1850/6253]  eta: 0:45:02    time: 0.6128  data: 0.0000  max mem: 6668
Evaluation:  [1900/6253]  eta: 0:44:31    time: 0.6146  data: 0.0000  max mem: 6668
Evaluation:  [1950/6253]  eta: 0:44:01    time: 0.6148  data: 0.0000  max mem: 6668
Evaluation:  [2000/6253]  eta: 0:43:30    time: 0.6141  data: 0.0000  max mem: 6668
Evaluation:  [2050/6253]  eta: 0:42:59    time: 0.6186  data: 0.0000  max mem: 6668
Evaluation:  [2100/6253]  eta: 0:42:28    time: 0.6173  data: 0.0000  max mem: 6668
Evaluation:  [2150/6253]  eta: 0:41:58    time: 0.6164  data: 0.0000  max mem: 6668
Evaluation:  [2200/6253]  eta: 0:41:27    time: 0.6194  data: 0.0000  max mem: 6668
Evaluation:  [2250/6253]  eta: 0:40:57    time: 0.6145  data: 0.0000  max mem: 6668
Evaluation:  [2300/6253]  eta: 0:40:26    time: 0.6140  data: 0.0000  max mem: 6668
Evaluation:  [2350/6253]  eta: 0:39:55    time: 0.6080  data: 0.0000  max mem: 6668
Evaluation:  [2400/6253]  eta: 0:39:25    time: 0.6176  data: 0.0000  max mem: 6668
Evaluation:  [2450/6253]  eta: 0:38:54    time: 0.6210  data: 0.0000  max mem: 6668
Evaluation:  [2500/6253]  eta: 0:38:23    time: 0.6173  data: 0.0000  max mem: 6668
Evaluation:  [2550/6253]  eta: 0:37:53    time: 0.6178  data: 0.0000  max mem: 6668
Evaluation:  [2600/6253]  eta: 0:37:22    time: 0.6099  data: 0.0000  max mem: 6668
Evaluation:  [2650/6253]  eta: 0:36:51    time: 0.6164  data: 0.0000  max mem: 6668
Evaluation:  [2700/6253]  eta: 0:36:21    time: 0.6199  data: 0.0000  max mem: 6668
Evaluation:  [2750/6253]  eta: 0:35:50    time: 0.6168  data: 0.0000  max mem: 6668
Evaluation:  [2800/6253]  eta: 0:35:19    time: 0.6168  data: 0.0000  max mem: 6668
Evaluation:  [2850/6253]  eta: 0:34:49    time: 0.6174  data: 0.0000  max mem: 6668
Evaluation:  [2900/6253]  eta: 0:34:18    time: 0.6134  data: 0.0000  max mem: 6668
Evaluation:  [2950/6253]  eta: 0:33:47    time: 0.6120  data: 0.0000  max mem: 6668
Evaluation:  [3000/6253]  eta: 0:33:16    time: 0.6120  data: 0.0000  max mem: 6668
Evaluation:  [3050/6253]  eta: 0:32:46    time: 0.6180  data: 0.0000  max mem: 6668
Evaluation:  [3100/6253]  eta: 0:32:15    time: 0.6135  data: 0.0000  max mem: 6668
Evaluation:  [3150/6253]  eta: 0:31:44    time: 0.6173  data: 0.0000  max mem: 6668
Evaluation:  [3200/6253]  eta: 0:31:14    time: 0.6186  data: 0.0000  max mem: 6668
Evaluation:  [3250/6253]  eta: 0:30:43    time: 0.6202  data: 0.0000  max mem: 6668
Evaluation:  [3300/6253]  eta: 0:30:12    time: 0.6187  data: 0.0000  max mem: 6668
Evaluation:  [3350/6253]  eta: 0:29:42    time: 0.6158  data: 0.0000  max mem: 6668
Evaluation:  [3400/6253]  eta: 0:29:11    time: 0.6168  data: 0.0000  max mem: 6668
Evaluation:  [3450/6253]  eta: 0:28:40    time: 0.6077  data: 0.0000  max mem: 6668
Evaluation:  [3500/6253]  eta: 0:28:10    time: 0.6126  data: 0.0000  max mem: 6668
Evaluation:  [3550/6253]  eta: 0:27:39    time: 0.6106  data: 0.0000  max mem: 6668
Evaluation:  [3600/6253]  eta: 0:27:08    time: 0.6084  data: 0.0000  max mem: 6668
Evaluation:  [3650/6253]  eta: 0:26:38    time: 0.6114  data: 0.0000  max mem: 6668
Evaluation:  [3700/6253]  eta: 0:26:07    time: 0.6117  data: 0.0000  max mem: 6668
Evaluation:  [3750/6253]  eta: 0:25:36    time: 0.6158  data: 0.0000  max mem: 6668
Evaluation:  [3800/6253]  eta: 0:25:05    time: 0.6135  data: 0.0000  max mem: 6668
Evaluation:  [3850/6253]  eta: 0:24:35    time: 0.6111  data: 0.0000  max mem: 6668
Evaluation:  [3900/6253]  eta: 0:24:04    time: 0.6136  data: 0.0000  max mem: 6668
Evaluation:  [3950/6253]  eta: 0:23:33    time: 0.6114  data: 0.0000  max mem: 6668
Evaluation:  [4000/6253]  eta: 0:23:03    time: 0.6191  data: 0.0000  max mem: 6668
Evaluation:  [4050/6253]  eta: 0:22:32    time: 0.6125  data: 0.0000  max mem: 6668
Evaluation:  [4100/6253]  eta: 0:22:01    time: 0.6098  data: 0.0000  max mem: 6668
Evaluation:  [4150/6253]  eta: 0:21:31    time: 0.6150  data: 0.0000  max mem: 6668
Evaluation:  [4200/6253]  eta: 0:21:00    time: 0.6198  data: 0.0000  max mem: 6668
Evaluation:  [4250/6253]  eta: 0:20:29    time: 0.6172  data: 0.0000  max mem: 6668
Evaluation:  [4300/6253]  eta: 0:19:59    time: 0.6113  data: 0.0000  max mem: 6668
Evaluation:  [4350/6253]  eta: 0:19:28    time: 0.6098  data: 0.0000  max mem: 6668
Evaluation:  [4400/6253]  eta: 0:18:57    time: 0.6144  data: 0.0000  max mem: 6668
Evaluation:  [4450/6253]  eta: 0:18:26    time: 0.6154  data: 0.0000  max mem: 6668
Evaluation:  [4500/6253]  eta: 0:17:56    time: 0.6140  data: 0.0000  max mem: 6668
Evaluation:  [4550/6253]  eta: 0:17:25    time: 0.6095  data: 0.0000  max mem: 6668
Evaluation:  [4600/6253]  eta: 0:16:54    time: 0.6093  data: 0.0000  max mem: 6668
Evaluation:  [4650/6253]  eta: 0:16:24    time: 0.6150  data: 0.0000  max mem: 6668
Evaluation:  [4700/6253]  eta: 0:15:53    time: 0.6154  data: 0.0000  max mem: 6668
Evaluation:  [4750/6253]  eta: 0:15:22    time: 0.6115  data: 0.0000  max mem: 6668
Evaluation:  [4800/6253]  eta: 0:14:51    time: 0.6104  data: 0.0000  max mem: 6668
Evaluation:  [4850/6253]  eta: 0:14:21    time: 0.6118  data: 0.0000  max mem: 6668
Evaluation:  [4900/6253]  eta: 0:13:50    time: 0.6164  data: 0.0000  max mem: 6668
Evaluation:  [4950/6253]  eta: 0:13:19    time: 0.6099  data: 0.0000  max mem: 6668
Evaluation:  [5000/6253]  eta: 0:12:49    time: 0.6153  data: 0.0000  max mem: 6668
Evaluation:  [5050/6253]  eta: 0:12:18    time: 0.6140  data: 0.0000  max mem: 6668
Evaluation:  [5100/6253]  eta: 0:11:47    time: 0.6097  data: 0.0000  max mem: 6668
Evaluation:  [5150/6253]  eta: 0:11:17    time: 0.6142  data: 0.0000  max mem: 6668
Evaluation:  [5200/6253]  eta: 0:10:46    time: 0.6148  data: 0.0000  max mem: 6668
Evaluation:  [5250/6253]  eta: 0:10:15    time: 0.6097  data: 0.0000  max mem: 6668
Evaluation:  [5300/6253]  eta: 0:09:44    time: 0.6116  data: 0.0000  max mem: 6668
Evaluation:  [5350/6253]  eta: 0:09:14    time: 0.6098  data: 0.0000  max mem: 6668
Evaluation:  [5400/6253]  eta: 0:08:43    time: 0.6106  data: 0.0000  max mem: 6668
Evaluation:  [5450/6253]  eta: 0:08:12    time: 0.6114  data: 0.0000  max mem: 6668
Evaluation:  [5500/6253]  eta: 0:07:42    time: 0.6157  data: 0.0000  max mem: 6668
Evaluation:  [5550/6253]  eta: 0:07:11    time: 0.6118  data: 0.0000  max mem: 6668
Evaluation:  [5600/6253]  eta: 0:06:40    time: 0.6151  data: 0.0000  max mem: 6668
Evaluation:  [5650/6253]  eta: 0:06:10    time: 0.6106  data: 0.0000  max mem: 6668
Evaluation:  [5700/6253]  eta: 0:05:39    time: 0.6125  data: 0.0000  max mem: 6668
Evaluation:  [5750/6253]  eta: 0:05:08    time: 0.6152  data: 0.0000  max mem: 6668
Evaluation:  [5800/6253]  eta: 0:04:38    time: 0.6116  data: 0.0000  max mem: 6668
Evaluation:  [5850/6253]  eta: 0:04:07    time: 0.6088  data: 0.0000  max mem: 6668
Evaluation:  [5900/6253]  eta: 0:03:36    time: 0.6142  data: 0.0000  max mem: 6668
Evaluation:  [5950/6253]  eta: 0:03:05    time: 0.6111  data: 0.0000  max mem: 6668
Evaluation:  [6000/6253]  eta: 0:02:35    time: 0.6113  data: 0.0000  max mem: 6668
Evaluation:  [6050/6253]  eta: 0:02:04    time: 0.6163  data: 0.0000  max mem: 6668
Evaluation:  [6100/6253]  eta: 0:01:33    time: 0.6180  data: 0.0000  max mem: 6668
Evaluation:  [6150/6253]  eta: 0:01:03    time: 0.6225  data: 0.0000  max mem: 6668
Evaluation:  [6200/6253]  eta: 0:00:32    time: 0.6146  data: 0.0000  max mem: 6668
Evaluation:  [6250/6253]  eta: 0:00:01    time: 0.6117  data: 0.0000  max mem: 6668
Evaluation:  [6252/6253]  eta: 0:00:00    time: 0.6120  data: 0.0000  max mem: 6668
Evaluation: Total time: 1:03:57 (0.6137 s / it)
Evaluation time 1:19:16
{'txt_r1': 74.44, 'txt_r5': 92.74, 'txt_r10': 96.38, 'txt_r_mean': 87.85333333333334, 'img_r1': 52.483006797281085, 'img_r5': 77.44102359056377, 'img_r10': 85.4938024790084, 'img_r_mean': 71.80594428895108, 'r_mean': 79.82963881114222}
{'txt_r1': 73.64, 'txt_r5': 92.18, 'txt_r10': 96.1, 'txt_r_mean': 87.30666666666666, 'img_r1': 52.00719712115154, 'img_r5': 77.03718512594962, 'img_r10': 84.75809676129548, 'img_r_mean': 71.26749300279887, 'r_mean': 79.28707983473276}
Training time 2:38:53