01-18 00:33:02 INFO [logger.py:80]: Initialized logger with log file in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR. 01-18 00:33:02 INFO [logger.py:80]: Initialized logger with log file in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR. 01-18 00:33:02 INFO [logger.py:80]: Initialized logger with log file in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR. 01-18 00:33:02 INFO [logger.py:80]: Initialized logger with log file in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR. 01-18 00:33:32 INFO [logging.py:61]: Configuration file is saved to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/config__2024_01_18--00_33_25.toml. 01-18 00:33:32 INFO [logging.py:61]: Environment information: - `Accelerate` version: 0.26.1 - Platform: Linux-5.14.0-362.13.1.el9_3.x86_64-x86_64-with-glibc2.34 - Python version: 3.10.13 - Numpy version: 1.26.3 - PyTorch version (GPU?): 2.1.2 (True) - System RAM: 503.48 GB - GPU Available: True - GPU IDs: 4 - GPU type: NVIDIA A100-SXM4-80GB 01-18 00:33:32 INFO [logging.py:61]: =============================================================================================== Layer (type:depth-idx) Param # =============================================================================================== DistributedDataParallel -- ├─Model: 1-1 -- │ └─EncodecModel: 2-1 -- │ │ └─EncodecEncoder: 3-1 (7,425,792) │ │ └─EncodecDecoder: 3-2 (7,426,018) │ │ └─EncodecResidualVectorQuantizer: 3-3 -- │ └─TokenEmbedding: 2-2 -- │ │ └─Dropout: 3-4 -- │ │ └─Embedding: 3-5 524,800 │ └─Identity: 2-3 -- │ └─SinePositionalEmbedding: 2-4 1 │ │ └─Dropout: 3-6 -- │ └─TransformerEncoder: 2-5 -- │ │ └─ModuleList: 3-7 37,828,608 │ │ └─LayerNorm: 3-8 1,024 │ └─Linear: 2-6 524,800 │ └─MulticlassAccuracy: 2-7 -- │ └─TokenEmbedding: 2-8 -- │ │ └─Dropout: 3-9 -- │ │ └─Embedding: 3-10 524,288 │ └─ModuleList: 2-9 -- │ │ └─TokenEmbedding: 3-11 524,800 │ │ └─TokenEmbedding: 3-12 524,288 │ │ └─TokenEmbedding: 3-13 524,288 │ │ └─TokenEmbedding: 3-14 524,288 │ │ └─TokenEmbedding: 3-15 524,288 │ │ └─TokenEmbedding: 3-16 524,288 │ │ └─TokenEmbedding: 3-17 524,288 │ │ └─TokenEmbedding: 3-18 524,288 │ └─Identity: 2-10 -- │ └─SinePositionalEmbedding: 2-11 1 │ │ └─Dropout: 3-19 -- │ └─TransformerEncoder: 2-12 -- │ │ └─ModuleList: 3-20 50,436,096 │ │ └─AdaptiveLayerNorm: 3-21 526,336 │ └─ModuleList: 2-13 -- │ │ └─Linear: 3-22 524,288 │ │ └─Linear: 3-23 524,288 │ │ └─Linear: 3-24 524,288 │ │ └─Linear: 3-25 524,288 │ │ └─Linear: 3-26 524,288 │ │ └─Linear: 3-27 524,288 │ │ └─Linear: 3-28 524,288 │ └─ModuleList: 2-14 -- │ │ └─TokenEmbedding: 3-29 512 │ │ └─TokenEmbedding: 3-30 512 │ │ └─TokenEmbedding: 3-31 512 │ │ └─TokenEmbedding: 3-32 512 │ │ └─TokenEmbedding: 3-33 512 │ │ └─TokenEmbedding: 3-34 512 │ │ └─TokenEmbedding: 3-35 512 │ └─MulticlassAccuracy: 2-15 -- =============================================================================================== Total params: 113,086,180 Trainable params: 98,234,369 Non-trainable params: 14,851,811 =============================================================================================== 01-18 00:33:32 INFO [logging.py:61]: Training control variables: 01-18 00:33:32 INFO [logging.py:61]: `steps_per_epoch`: 500 01-18 00:33:32 INFO [logging.py:61]: Gradient accumulation steps: 1 01-18 00:33:32 INFO [logging.py:61]: `update_steps_per_epoch`: 500 01-18 00:33:32 INFO [logging.py:61]: `max_steps`: 500000 01-18 00:33:32 INFO [logging.py:61]: `max_epochs`: 1000 01-18 00:33:32 INFO [logging.py:61]: warmup_steps=1000. warmup_ratio will be ignored. 01-18 00:33:32 INFO [logging.py:61]: ========= Epoch 1 out of 1000 ========= 01-18 00:33:32 INFO [logging.py:61]: Begin training... 01-18 00:48:19 INFO [logging.py:61]: Loss 'loss' on epoch 1: 41.1013298034668 01-18 00:48:19 INFO [logging.py:61]: Loss 'ar_loss' on epoch 1: 3.8041999340057373 01-18 00:48:19 INFO [logging.py:61]: Loss 'nar_loss' on epoch 1: 37.2971305847168 01-18 00:48:19 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 1: 0.5283006429672241 01-18 00:48:19 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 1: 0.14399056136608124 01-18 00:48:19 INFO [logging.py:61]: ========= Epoch 2 out of 1000 ========= 01-18 00:48:19 INFO [logging.py:61]: Begin training... 01-18 01:03:02 INFO [logging.py:61]: Loss 'loss' on epoch 2: 71.26728057861328 01-18 01:03:02 INFO [logging.py:61]: Loss 'ar_loss' on epoch 2: 3.0147104263305664 01-18 01:03:02 INFO [logging.py:61]: Loss 'nar_loss' on epoch 2: 68.25257110595703 01-18 01:03:02 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 2: 0.6516079306602478 01-18 01:03:02 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 2: 0.14630988240242004 01-18 01:03:02 INFO [logging.py:61]: ========= Epoch 3 out of 1000 ========= 01-18 01:03:02 INFO [logging.py:61]: Begin training... 01-18 01:17:44 INFO [logging.py:61]: Loss 'loss' on epoch 3: 80.09443664550781 01-18 01:17:44 INFO [logging.py:61]: Loss 'ar_loss' on epoch 3: 2.8742563724517822 01-18 01:17:44 INFO [logging.py:61]: Loss 'nar_loss' on epoch 3: 77.22017669677734 01-18 01:17:44 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 3: 0.6769989132881165 01-18 01:17:44 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 3: 0.1532195657491684 01-18 01:17:44 INFO [logging.py:61]: ========= Epoch 4 out of 1000 ========= 01-18 01:17:44 INFO [logging.py:61]: Begin training... 01-18 01:32:28 INFO [logging.py:61]: Loss 'loss' on epoch 4: 64.45104217529297 01-18 01:32:28 INFO [logging.py:61]: Loss 'ar_loss' on epoch 4: 2.756040096282959 01-18 01:32:28 INFO [logging.py:61]: Loss 'nar_loss' on epoch 4: 61.69499969482422 01-18 01:32:28 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 4: 0.6978757381439209 01-18 01:32:28 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 4: 0.1739574521780014 01-18 01:32:28 INFO [logging.py:61]: ========= Epoch 5 out of 1000 ========= 01-18 01:32:28 INFO [logging.py:61]: Begin training... 01-18 01:47:11 INFO [logging.py:61]: Loss 'loss' on epoch 5: 50.534027099609375 01-18 01:47:11 INFO [logging.py:61]: Loss 'ar_loss' on epoch 5: 2.4202704429626465 01-18 01:47:11 INFO [logging.py:61]: Loss 'nar_loss' on epoch 5: 48.1137580871582 01-18 01:47:11 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 5: 0.7520415186882019 01-18 01:47:11 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 5: 0.2114027887582779 01-18 01:47:11 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0005 01-18 01:47:12 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0005/pytorch_model.bin 01-18 01:47:13 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0005/optimizer.bin 01-18 01:47:13 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0005/scheduler.bin 01-18 01:47:13 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0005/sampler.bin 01-18 01:47:13 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0005/sampler_1.bin 01-18 01:47:13 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0005/random_states_0.pkl 01-18 01:47:13 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0005/custom_checkpoint_0.pkl 01-18 01:47:14 INFO [logging.py:61]: ========= Epoch 6 out of 1000 ========= 01-18 01:47:14 INFO [logging.py:61]: Begin training... 01-18 02:02:24 INFO [logging.py:61]: Loss 'loss' on epoch 6: 37.49298858642578 01-18 02:02:24 INFO [logging.py:61]: Loss 'ar_loss' on epoch 6: 1.8649616241455078 01-18 02:02:24 INFO [logging.py:61]: Loss 'nar_loss' on epoch 6: 35.628028869628906 01-18 02:02:24 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 6: 0.8278111219406128 01-18 02:02:24 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 6: 0.22959543764591217 01-18 02:02:24 INFO [logging.py:61]: ========= Epoch 7 out of 1000 ========= 01-18 02:02:24 INFO [logging.py:61]: Begin training... 01-18 02:17:03 INFO [logging.py:61]: Loss 'loss' on epoch 7: 28.96417236328125 01-18 02:17:03 INFO [logging.py:61]: Loss 'ar_loss' on epoch 7: 1.677205204963684 01-18 02:17:03 INFO [logging.py:61]: Loss 'nar_loss' on epoch 7: 27.286962509155273 01-18 02:17:03 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 7: 0.8506313562393188 01-18 02:17:03 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 7: 0.2341448813676834 01-18 02:17:03 INFO [logging.py:61]: ========= Epoch 8 out of 1000 ========= 01-18 02:17:03 INFO [logging.py:61]: Begin training... 01-18 02:31:48 INFO [logging.py:61]: Loss 'loss' on epoch 8: 28.844099044799805 01-18 02:31:48 INFO [logging.py:61]: Loss 'ar_loss' on epoch 8: 1.6051831245422363 01-18 02:31:48 INFO [logging.py:61]: Loss 'nar_loss' on epoch 8: 27.238914489746094 01-18 02:31:48 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 8: 0.85962975025177 01-18 02:31:48 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 8: 0.2343209981918335 01-18 02:31:48 INFO [logging.py:61]: ========= Epoch 9 out of 1000 ========= 01-18 02:31:48 INFO [logging.py:61]: Begin training... 01-18 02:46:32 INFO [logging.py:61]: Loss 'loss' on epoch 9: 18.423540115356445 01-18 02:46:32 INFO [logging.py:61]: Loss 'ar_loss' on epoch 9: 1.5607959032058716 01-18 02:46:32 INFO [logging.py:61]: Loss 'nar_loss' on epoch 9: 16.862743377685547 01-18 02:46:32 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 9: 0.8656294345855713 01-18 02:46:32 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 9: 0.268953800201416 01-18 02:46:32 INFO [logging.py:61]: ========= Epoch 10 out of 1000 ========= 01-18 02:46:32 INFO [logging.py:61]: Begin training... 01-18 03:01:15 INFO [logging.py:61]: Loss 'loss' on epoch 10: 17.509979248046875 01-18 03:01:15 INFO [logging.py:61]: Loss 'ar_loss' on epoch 10: 1.516535997390747 01-18 03:01:15 INFO [logging.py:61]: Loss 'nar_loss' on epoch 10: 15.993441581726074 01-18 03:01:15 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 10: 0.8709884881973267 01-18 03:01:15 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 10: 0.27383264899253845 01-18 03:01:15 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0010 01-18 03:01:16 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0010/pytorch_model.bin 01-18 03:01:18 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0010/optimizer.bin 01-18 03:01:18 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0010/scheduler.bin 01-18 03:01:18 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0010/sampler.bin 01-18 03:01:18 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0010/sampler_1.bin 01-18 03:01:18 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0010/random_states_0.pkl 01-18 03:01:18 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0010/custom_checkpoint_0.pkl 01-18 03:01:18 INFO [logging.py:61]: ========= Epoch 11 out of 1000 ========= 01-18 03:01:18 INFO [logging.py:61]: Begin training... 01-18 03:16:19 INFO [logging.py:61]: Loss 'loss' on epoch 11: 20.488168716430664 01-18 03:16:19 INFO [logging.py:61]: Loss 'ar_loss' on epoch 11: 1.492824673652649 01-18 03:16:19 INFO [logging.py:61]: Loss 'nar_loss' on epoch 11: 18.995346069335938 01-18 03:16:19 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 11: 0.8740840554237366 01-18 03:16:19 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 11: 0.2761929929256439 01-18 03:16:19 INFO [logging.py:61]: ========= Epoch 12 out of 1000 ========= 01-18 03:16:19 INFO [logging.py:61]: Begin training... 01-18 03:31:06 INFO [logging.py:61]: Loss 'loss' on epoch 12: 12.799497604370117 01-18 03:31:06 INFO [logging.py:61]: Loss 'ar_loss' on epoch 12: 1.4752053022384644 01-18 03:31:06 INFO [logging.py:61]: Loss 'nar_loss' on epoch 12: 11.324292182922363 01-18 03:31:06 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 12: 0.8769260048866272 01-18 03:31:06 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 12: 0.29501354694366455 01-18 03:31:06 INFO [logging.py:61]: ========= Epoch 13 out of 1000 ========= 01-18 03:31:06 INFO [logging.py:61]: Begin training... 01-18 03:45:51 INFO [logging.py:61]: Loss 'loss' on epoch 13: 9.103281021118164 01-18 03:45:51 INFO [logging.py:61]: Loss 'ar_loss' on epoch 13: 1.456277847290039 01-18 03:45:51 INFO [logging.py:61]: Loss 'nar_loss' on epoch 13: 7.647003650665283 01-18 03:45:51 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 13: 0.8797268867492676 01-18 03:45:51 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 13: 0.3158681094646454 01-18 03:45:51 INFO [logging.py:61]: ========= Epoch 14 out of 1000 ========= 01-18 03:45:51 INFO [logging.py:61]: Begin training... 01-18 04:00:36 INFO [logging.py:61]: Loss 'loss' on epoch 14: 13.451669692993164 01-18 04:00:36 INFO [logging.py:61]: Loss 'ar_loss' on epoch 14: 1.4416464567184448 01-18 04:00:36 INFO [logging.py:61]: Loss 'nar_loss' on epoch 14: 12.01002311706543 01-18 04:00:36 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 14: 0.8816895484924316 01-18 04:00:36 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 14: 0.2968295216560364 01-18 04:00:36 INFO [logging.py:61]: ========= Epoch 15 out of 1000 ========= 01-18 04:00:36 INFO [logging.py:61]: Begin training... 01-18 04:15:15 INFO [logging.py:61]: Loss 'loss' on epoch 15: 10.637478828430176 01-18 04:15:15 INFO [logging.py:61]: Loss 'ar_loss' on epoch 15: 1.4250620603561401 01-18 04:15:15 INFO [logging.py:61]: Loss 'nar_loss' on epoch 15: 9.21241569519043 01-18 04:15:15 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 15: 0.8841312527656555 01-18 04:15:15 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 15: 0.3035244345664978 01-18 04:15:15 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0015 01-18 04:15:16 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0015/pytorch_model.bin 01-18 04:15:17 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0015/optimizer.bin 01-18 04:15:17 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0015/scheduler.bin 01-18 04:15:17 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0015/sampler.bin 01-18 04:15:17 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0015/sampler_1.bin 01-18 04:15:17 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0015/random_states_0.pkl 01-18 04:15:17 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0015/custom_checkpoint_0.pkl 01-18 04:15:17 INFO [logging.py:61]: ========= Epoch 16 out of 1000 ========= 01-18 04:15:17 INFO [logging.py:61]: Begin training... 01-18 04:30:27 INFO [logging.py:61]: Loss 'loss' on epoch 16: 7.80450439453125 01-18 04:30:27 INFO [logging.py:61]: Loss 'ar_loss' on epoch 16: 1.4189400672912598 01-18 04:30:27 INFO [logging.py:61]: Loss 'nar_loss' on epoch 16: 6.385564804077148 01-18 04:30:27 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 16: 0.8853126764297485 01-18 04:30:27 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 16: 0.3228910267353058 01-18 04:30:27 INFO [logging.py:61]: ========= Epoch 17 out of 1000 ========= 01-18 04:30:27 INFO [logging.py:61]: Begin training... 01-18 04:45:12 INFO [logging.py:61]: Loss 'loss' on epoch 17: 7.98758602142334 01-18 04:45:12 INFO [logging.py:61]: Loss 'ar_loss' on epoch 17: 1.402923583984375 01-18 04:45:12 INFO [logging.py:61]: Loss 'nar_loss' on epoch 17: 6.584662914276123 01-18 04:45:12 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 17: 0.8873386383056641 01-18 04:45:12 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 17: 0.3134208917617798 01-18 04:45:12 INFO [logging.py:61]: ========= Epoch 18 out of 1000 ========= 01-18 04:45:12 INFO [logging.py:61]: Begin training... 01-18 04:59:55 INFO [logging.py:61]: Loss 'loss' on epoch 18: 8.954412460327148 01-18 04:59:55 INFO [logging.py:61]: Loss 'ar_loss' on epoch 18: 1.3915390968322754 01-18 04:59:55 INFO [logging.py:61]: Loss 'nar_loss' on epoch 18: 7.562872886657715 01-18 04:59:55 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 18: 0.8887759447097778 01-18 04:59:55 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 18: 0.3137945532798767 01-18 04:59:55 INFO [logging.py:61]: ========= Epoch 19 out of 1000 ========= 01-18 04:59:55 INFO [logging.py:61]: Begin training... 01-18 05:14:30 INFO [logging.py:61]: Loss 'loss' on epoch 19: 6.779003620147705 01-18 05:14:30 INFO [logging.py:61]: Loss 'ar_loss' on epoch 19: 1.3836216926574707 01-18 05:14:30 INFO [logging.py:61]: Loss 'nar_loss' on epoch 19: 5.395382404327393 01-18 05:14:30 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 19: 0.8901618719100952 01-18 05:14:30 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 19: 0.3347846567630768 01-18 05:14:30 INFO [logging.py:61]: ========= Epoch 20 out of 1000 ========= 01-18 05:14:30 INFO [logging.py:61]: Begin training... 01-18 05:29:19 INFO [logging.py:61]: Loss 'loss' on epoch 20: 8.779007911682129 01-18 05:29:19 INFO [logging.py:61]: Loss 'ar_loss' on epoch 20: 1.3702212572097778 01-18 05:29:19 INFO [logging.py:61]: Loss 'nar_loss' on epoch 20: 7.408785343170166 01-18 05:29:19 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 20: 0.8919013738632202 01-18 05:29:19 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 20: 0.3212190866470337 01-18 05:29:19 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0020 01-18 05:29:19 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0020/pytorch_model.bin 01-18 05:29:21 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0020/optimizer.bin 01-18 05:29:21 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0020/scheduler.bin 01-18 05:29:21 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0020/sampler.bin 01-18 05:29:21 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0020/sampler_1.bin 01-18 05:29:21 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0020/random_states_0.pkl 01-18 05:29:21 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0020/custom_checkpoint_0.pkl 01-18 05:29:21 INFO [logging.py:61]: ========= Epoch 21 out of 1000 ========= 01-18 05:29:21 INFO [logging.py:61]: Begin training... 01-18 05:44:34 INFO [logging.py:61]: Loss 'loss' on epoch 21: 6.515992164611816 01-18 05:44:34 INFO [logging.py:61]: Loss 'ar_loss' on epoch 21: 1.363883376121521 01-18 05:44:34 INFO [logging.py:61]: Loss 'nar_loss' on epoch 21: 5.152108669281006 01-18 05:44:34 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 21: 0.8928155899047852 01-18 05:44:34 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 21: 0.3437054753303528 01-18 05:44:34 INFO [logging.py:61]: ========= Epoch 22 out of 1000 ========= 01-18 05:44:34 INFO [logging.py:61]: Begin training... 01-18 05:59:19 INFO [logging.py:61]: Loss 'loss' on epoch 22: 6.411158084869385 01-18 05:59:19 INFO [logging.py:61]: Loss 'ar_loss' on epoch 22: 1.3568339347839355 01-18 05:59:19 INFO [logging.py:61]: Loss 'nar_loss' on epoch 22: 5.054324150085449 01-18 05:59:19 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 22: 0.8938457369804382 01-18 05:59:19 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 22: 0.3443514406681061 01-18 05:59:19 INFO [logging.py:61]: ========= Epoch 23 out of 1000 ========= 01-18 05:59:19 INFO [logging.py:61]: Begin training... 01-18 06:14:01 INFO [logging.py:61]: Loss 'loss' on epoch 23: 6.536004066467285 01-18 06:14:01 INFO [logging.py:61]: Loss 'ar_loss' on epoch 23: 1.3483623266220093 01-18 06:14:01 INFO [logging.py:61]: Loss 'nar_loss' on epoch 23: 5.187641620635986 01-18 06:14:01 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 23: 0.8951892256736755 01-18 06:14:01 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 23: 0.3462466299533844 01-18 06:14:01 INFO [logging.py:61]: ========= Epoch 24 out of 1000 ========= 01-18 06:14:01 INFO [logging.py:61]: Begin training... 01-18 06:28:46 INFO [logging.py:61]: Loss 'loss' on epoch 24: 6.32097053527832 01-18 06:28:46 INFO [logging.py:61]: Loss 'ar_loss' on epoch 24: 1.3559930324554443 01-18 06:28:46 INFO [logging.py:61]: Loss 'nar_loss' on epoch 24: 4.9649786949157715 01-18 06:28:46 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 24: 0.8945856094360352 01-18 06:28:46 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 24: 0.3438716530799866 01-18 06:28:46 INFO [logging.py:61]: ========= Epoch 25 out of 1000 ========= 01-18 06:28:46 INFO [logging.py:61]: Begin training... 01-18 06:43:30 INFO [logging.py:61]: Loss 'loss' on epoch 25: 6.4575934410095215 01-18 06:43:30 INFO [logging.py:61]: Loss 'ar_loss' on epoch 25: 1.3440008163452148 01-18 06:43:30 INFO [logging.py:61]: Loss 'nar_loss' on epoch 25: 5.113592147827148 01-18 06:43:30 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 25: 0.8960744142532349 01-18 06:43:30 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 25: 0.3424939811229706 01-18 06:43:30 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0025 01-18 06:43:31 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0025/pytorch_model.bin 01-18 06:43:32 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0025/optimizer.bin 01-18 06:43:32 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0025/scheduler.bin 01-18 06:43:32 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0025/sampler.bin 01-18 06:43:32 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0025/sampler_1.bin 01-18 06:43:32 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0025/random_states_0.pkl 01-18 06:43:32 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0025/custom_checkpoint_0.pkl 01-18 06:43:32 INFO [logging.py:61]: ========= Epoch 26 out of 1000 ========= 01-18 06:43:32 INFO [logging.py:61]: Begin training... 01-18 06:59:08 INFO [logging.py:61]: Loss 'loss' on epoch 26: 5.9814558029174805 01-18 06:59:08 INFO [logging.py:61]: Loss 'ar_loss' on epoch 26: 1.3335331678390503 01-18 06:59:08 INFO [logging.py:61]: Loss 'nar_loss' on epoch 26: 4.647922992706299 01-18 06:59:08 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 26: 0.8972698450088501 01-18 06:59:08 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 26: 0.3597102165222168 01-18 06:59:08 INFO [logging.py:61]: ========= Epoch 27 out of 1000 ========= 01-18 06:59:08 INFO [logging.py:61]: Begin training... 01-18 07:13:48 INFO [logging.py:61]: Loss 'loss' on epoch 27: 5.9211649894714355 01-18 07:13:48 INFO [logging.py:61]: Loss 'ar_loss' on epoch 27: 1.3315753936767578 01-18 07:13:48 INFO [logging.py:61]: Loss 'nar_loss' on epoch 27: 4.589588642120361 01-18 07:13:48 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 27: 0.8978341817855835 01-18 07:13:48 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 27: 0.3617432415485382 01-18 07:13:48 INFO [logging.py:61]: ========= Epoch 28 out of 1000 ========= 01-18 07:13:48 INFO [logging.py:61]: Begin training... 01-18 07:28:33 INFO [logging.py:61]: Loss 'loss' on epoch 28: 5.8742170333862305 01-18 07:28:33 INFO [logging.py:61]: Loss 'ar_loss' on epoch 28: 1.3296784162521362 01-18 07:28:33 INFO [logging.py:61]: Loss 'nar_loss' on epoch 28: 4.544538497924805 01-18 07:28:33 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 28: 0.8983495831489563 01-18 07:28:33 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 28: 0.366417795419693 01-18 07:28:33 INFO [logging.py:61]: ========= Epoch 29 out of 1000 ========= 01-18 07:28:33 INFO [logging.py:61]: Begin training... 01-18 07:43:17 INFO [logging.py:61]: Loss 'loss' on epoch 29: 5.860039710998535 01-18 07:43:17 INFO [logging.py:61]: Loss 'ar_loss' on epoch 29: 1.3166179656982422 01-18 07:43:17 INFO [logging.py:61]: Loss 'nar_loss' on epoch 29: 4.543421268463135 01-18 07:43:17 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 29: 0.8997495174407959 01-18 07:43:17 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 29: 0.3649654686450958 01-18 07:43:17 INFO [logging.py:61]: ========= Epoch 30 out of 1000 ========= 01-18 07:43:17 INFO [logging.py:61]: Begin training... 01-18 07:58:02 INFO [logging.py:61]: Loss 'loss' on epoch 30: 5.796247959136963 01-18 07:58:02 INFO [logging.py:61]: Loss 'ar_loss' on epoch 30: 1.3036959171295166 01-18 07:58:02 INFO [logging.py:61]: Loss 'nar_loss' on epoch 30: 4.492552280426025 01-18 07:58:02 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 30: 0.90115886926651 01-18 07:58:02 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 30: 0.37101566791534424 01-18 07:58:02 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0030 01-18 07:58:03 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0030/pytorch_model.bin 01-18 07:58:04 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0030/optimizer.bin 01-18 07:58:04 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0030/scheduler.bin 01-18 07:58:04 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0030/sampler.bin 01-18 07:58:04 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0030/sampler_1.bin 01-18 07:58:04 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0030/random_states_0.pkl 01-18 07:58:04 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0030/custom_checkpoint_0.pkl 01-18 07:58:04 INFO [logging.py:61]: ========= Epoch 31 out of 1000 ========= 01-18 07:58:04 INFO [logging.py:61]: Begin training... 01-18 08:12:58 INFO [logging.py:61]: Loss 'loss' on epoch 31: 5.9744038581848145 01-18 08:12:58 INFO [logging.py:61]: Loss 'ar_loss' on epoch 31: 1.311854362487793 01-18 08:12:58 INFO [logging.py:61]: Loss 'nar_loss' on epoch 31: 4.662549018859863 01-18 08:12:58 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 31: 0.9008342623710632 01-18 08:12:58 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 31: 0.366784006357193 01-18 08:12:58 INFO [logging.py:61]: ========= Epoch 32 out of 1000 ========= 01-18 08:12:58 INFO [logging.py:61]: Begin training... 01-18 08:27:46 INFO [logging.py:61]: Loss 'loss' on epoch 32: 5.735446453094482 01-18 08:27:46 INFO [logging.py:61]: Loss 'ar_loss' on epoch 32: 1.3046653270721436 01-18 08:27:46 INFO [logging.py:61]: Loss 'nar_loss' on epoch 32: 4.43078088760376 01-18 08:27:46 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 32: 0.9014154076576233 01-18 08:27:46 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 32: 0.37591466307640076 01-18 08:27:46 INFO [logging.py:61]: ========= Epoch 33 out of 1000 ========= 01-18 08:27:46 INFO [logging.py:61]: Begin training... 01-18 08:42:30 INFO [logging.py:61]: Loss 'loss' on epoch 33: 5.872527122497559 01-18 08:42:30 INFO [logging.py:61]: Loss 'ar_loss' on epoch 33: 1.2990329265594482 01-18 08:42:30 INFO [logging.py:61]: Loss 'nar_loss' on epoch 33: 4.5734944343566895 01-18 08:42:30 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 33: 0.9022935032844543 01-18 08:42:30 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 33: 0.37081286311149597 01-18 08:42:30 INFO [logging.py:61]: ========= Epoch 34 out of 1000 ========= 01-18 08:42:30 INFO [logging.py:61]: Begin training... 01-18 08:57:15 INFO [logging.py:61]: Loss 'loss' on epoch 34: 5.6954474449157715 01-18 08:57:15 INFO [logging.py:61]: Loss 'ar_loss' on epoch 34: 1.3018174171447754 01-18 08:57:15 INFO [logging.py:61]: Loss 'nar_loss' on epoch 34: 4.393630504608154 01-18 08:57:15 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 34: 0.9023982882499695 01-18 08:57:15 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 34: 0.38097846508026123 01-18 08:57:15 INFO [logging.py:61]: ========= Epoch 35 out of 1000 ========= 01-18 08:57:15 INFO [logging.py:61]: Begin training... 01-18 09:11:52 INFO [logging.py:61]: Loss 'loss' on epoch 35: 5.682955741882324 01-18 09:11:52 INFO [logging.py:61]: Loss 'ar_loss' on epoch 35: 1.2978262901306152 01-18 09:11:52 INFO [logging.py:61]: Loss 'nar_loss' on epoch 35: 4.385129451751709 01-18 09:11:52 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 35: 0.9027104377746582 01-18 09:11:52 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 35: 0.3820667564868927 01-18 09:11:52 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0035 01-18 09:11:53 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0035/pytorch_model.bin 01-18 09:11:54 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0035/optimizer.bin 01-18 09:11:54 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0035/scheduler.bin 01-18 09:11:54 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0035/sampler.bin 01-18 09:11:54 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0035/sampler_1.bin 01-18 09:11:54 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0035/random_states_0.pkl 01-18 09:11:54 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0035/custom_checkpoint_0.pkl 01-18 09:11:54 INFO [logging.py:61]: ========= Epoch 36 out of 1000 ========= 01-18 09:11:54 INFO [logging.py:61]: Begin training... 01-18 09:27:23 INFO [logging.py:61]: Loss 'loss' on epoch 36: 5.689979553222656 01-18 09:27:23 INFO [logging.py:61]: Loss 'ar_loss' on epoch 36: 1.2983156442642212 01-18 09:27:23 INFO [logging.py:61]: Loss 'nar_loss' on epoch 36: 4.391663551330566 01-18 09:27:23 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 36: 0.90284264087677 01-18 09:27:23 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 36: 0.3806702196598053 01-18 09:27:23 INFO [logging.py:61]: ========= Epoch 37 out of 1000 ========= 01-18 09:27:23 INFO [logging.py:61]: Begin training... 01-18 09:42:07 INFO [logging.py:61]: Loss 'loss' on epoch 37: 5.661297798156738 01-18 09:42:07 INFO [logging.py:61]: Loss 'ar_loss' on epoch 37: 1.2922143936157227 01-18 09:42:07 INFO [logging.py:61]: Loss 'nar_loss' on epoch 37: 4.369083404541016 01-18 09:42:07 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 37: 0.9038951396942139 01-18 09:42:07 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 37: 0.3834979236125946 01-18 09:42:07 INFO [logging.py:61]: ========= Epoch 38 out of 1000 ========= 01-18 09:42:07 INFO [logging.py:61]: Begin training... 01-18 09:57:28 INFO [logging.py:61]: Loss 'loss' on epoch 38: 5.638166427612305 01-18 09:57:28 INFO [logging.py:61]: Loss 'ar_loss' on epoch 38: 1.2806264162063599 01-18 09:57:28 INFO [logging.py:61]: Loss 'nar_loss' on epoch 38: 4.357540130615234 01-18 09:57:28 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 38: 0.9050056338310242 01-18 09:57:28 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 38: 0.38519880175590515 01-18 09:57:28 INFO [logging.py:61]: ========= Epoch 39 out of 1000 ========= 01-18 09:57:28 INFO [logging.py:61]: Begin training... 01-18 10:12:01 INFO [logging.py:61]: Loss 'loss' on epoch 39: 5.627877712249756 01-18 10:12:01 INFO [logging.py:61]: Loss 'ar_loss' on epoch 39: 1.2748196125030518 01-18 10:12:01 INFO [logging.py:61]: Loss 'nar_loss' on epoch 39: 4.353058338165283 01-18 10:12:01 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 39: 0.9056810140609741 01-18 10:12:01 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 39: 0.3871024549007416 01-18 10:12:01 INFO [logging.py:61]: ========= Epoch 40 out of 1000 ========= 01-18 10:12:01 INFO [logging.py:61]: Begin training... 01-18 10:26:42 INFO [logging.py:61]: Loss 'loss' on epoch 40: 5.619797229766846 01-18 10:26:42 INFO [logging.py:61]: Loss 'ar_loss' on epoch 40: 1.2779885530471802 01-18 10:26:42 INFO [logging.py:61]: Loss 'nar_loss' on epoch 40: 4.341809272766113 01-18 10:26:42 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 40: 0.9054821729660034 01-18 10:26:42 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 40: 0.38915959000587463 01-18 10:26:42 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0040 01-18 10:26:43 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0040/pytorch_model.bin 01-18 10:26:45 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0040/optimizer.bin 01-18 10:26:45 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0040/scheduler.bin 01-18 10:26:45 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0040/sampler.bin 01-18 10:26:45 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0040/sampler_1.bin 01-18 10:26:45 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0040/random_states_0.pkl 01-18 10:26:45 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0040/custom_checkpoint_0.pkl 01-18 10:26:45 INFO [logging.py:61]: ========= Epoch 41 out of 1000 ========= 01-18 10:26:45 INFO [logging.py:61]: Begin training... 01-18 10:42:20 INFO [logging.py:61]: Loss 'loss' on epoch 41: 5.582706928253174 01-18 10:42:20 INFO [logging.py:61]: Loss 'ar_loss' on epoch 41: 1.2725111246109009 01-18 10:42:20 INFO [logging.py:61]: Loss 'nar_loss' on epoch 41: 4.3101959228515625 01-18 10:42:20 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 41: 0.9060384035110474 01-18 10:42:20 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 41: 0.39349165558815 01-18 10:42:20 INFO [logging.py:61]: ========= Epoch 42 out of 1000 ========= 01-18 10:42:20 INFO [logging.py:61]: Begin training... 01-18 10:57:24 INFO [logging.py:61]: Loss 'loss' on epoch 42: 5.580837726593018 01-18 10:57:24 INFO [logging.py:61]: Loss 'ar_loss' on epoch 42: 1.2710316181182861 01-18 10:57:24 INFO [logging.py:61]: Loss 'nar_loss' on epoch 42: 4.309806823730469 01-18 10:57:24 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 42: 0.9061719179153442 01-18 10:57:24 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 42: 0.39457419514656067 01-18 10:57:24 INFO [logging.py:61]: ========= Epoch 43 out of 1000 ========= 01-18 10:57:24 INFO [logging.py:61]: Begin training... 01-18 11:13:05 INFO [logging.py:61]: Loss 'loss' on epoch 43: 5.569582939147949 01-18 11:13:05 INFO [logging.py:61]: Loss 'ar_loss' on epoch 43: 1.2762430906295776 01-18 11:13:05 INFO [logging.py:61]: Loss 'nar_loss' on epoch 43: 4.293339252471924 01-18 11:13:05 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 43: 0.9061148166656494 01-18 11:13:05 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 43: 0.39825910329818726 01-18 11:13:05 INFO [logging.py:61]: ========= Epoch 44 out of 1000 ========= 01-18 11:13:05 INFO [logging.py:61]: Begin training... 01-18 11:27:38 INFO [logging.py:61]: Loss 'loss' on epoch 44: 5.717195987701416 01-18 11:27:38 INFO [logging.py:61]: Loss 'ar_loss' on epoch 44: 1.2707628011703491 01-18 11:27:38 INFO [logging.py:61]: Loss 'nar_loss' on epoch 44: 4.446432590484619 01-18 11:27:38 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 44: 0.9066540598869324 01-18 11:27:38 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 44: 0.3859277665615082 01-18 11:27:38 INFO [logging.py:61]: ========= Epoch 45 out of 1000 ========= 01-18 11:27:38 INFO [logging.py:61]: Begin training... 01-18 11:42:11 INFO [logging.py:61]: Loss 'loss' on epoch 45: 5.571887969970703 01-18 11:42:11 INFO [logging.py:61]: Loss 'ar_loss' on epoch 45: 1.2563157081604004 01-18 11:42:11 INFO [logging.py:61]: Loss 'nar_loss' on epoch 45: 4.315572261810303 01-18 11:42:11 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 45: 0.9081159830093384 01-18 11:42:11 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 45: 0.39590778946876526 01-18 11:42:11 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0045 01-18 11:42:12 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0045/pytorch_model.bin 01-18 11:42:13 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0045/optimizer.bin 01-18 11:42:13 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0045/scheduler.bin 01-18 11:42:13 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0045/sampler.bin 01-18 11:42:13 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0045/sampler_1.bin 01-18 11:42:13 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0045/random_states_0.pkl 01-18 11:42:13 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0045/custom_checkpoint_0.pkl 01-18 11:42:13 INFO [logging.py:61]: ========= Epoch 46 out of 1000 ========= 01-18 11:42:13 INFO [logging.py:61]: Begin training... 01-18 11:56:47 INFO [logging.py:61]: Loss 'loss' on epoch 46: 5.5488386154174805 01-18 11:56:47 INFO [logging.py:61]: Loss 'ar_loss' on epoch 46: 1.270042061805725 01-18 11:56:47 INFO [logging.py:61]: Loss 'nar_loss' on epoch 46: 4.278796195983887 01-18 11:56:47 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 46: 0.9069695472717285 01-18 11:56:47 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 46: 0.40045443177223206 01-18 11:56:47 INFO [logging.py:61]: ========= Epoch 47 out of 1000 ========= 01-18 11:56:47 INFO [logging.py:61]: Begin training... 01-18 12:11:19 INFO [logging.py:61]: Loss 'loss' on epoch 47: 5.518341541290283 01-18 12:11:19 INFO [logging.py:61]: Loss 'ar_loss' on epoch 47: 1.2568832635879517 01-18 12:11:19 INFO [logging.py:61]: Loss 'nar_loss' on epoch 47: 4.261458396911621 01-18 12:11:19 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 47: 0.9085185527801514 01-18 12:11:19 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 47: 0.4032013416290283 01-18 12:11:19 INFO [logging.py:61]: ========= Epoch 48 out of 1000 ========= 01-18 12:11:19 INFO [logging.py:61]: Begin training... 01-18 12:25:53 INFO [logging.py:61]: Loss 'loss' on epoch 48: 5.5060343742370605 01-18 12:25:54 INFO [logging.py:61]: Loss 'ar_loss' on epoch 48: 1.258938193321228 01-18 12:25:54 INFO [logging.py:61]: Loss 'nar_loss' on epoch 48: 4.247096538543701 01-18 12:25:54 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 48: 0.9083271622657776 01-18 12:25:54 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 48: 0.40573224425315857 01-18 12:25:54 INFO [logging.py:61]: ========= Epoch 49 out of 1000 ========= 01-18 12:25:54 INFO [logging.py:61]: Begin training... 01-18 12:40:27 INFO [logging.py:61]: Loss 'loss' on epoch 49: 5.473796844482422 01-18 12:40:27 INFO [logging.py:61]: Loss 'ar_loss' on epoch 49: 1.2464219331741333 01-18 12:40:27 INFO [logging.py:61]: Loss 'nar_loss' on epoch 49: 4.22737455368042 01-18 12:40:27 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 49: 0.9096682071685791 01-18 12:40:27 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 49: 0.40877988934516907 01-18 12:40:27 INFO [logging.py:61]: ========= Epoch 50 out of 1000 ========= 01-18 12:40:27 INFO [logging.py:61]: Begin training... 01-18 12:54:59 INFO [logging.py:61]: Loss 'loss' on epoch 50: 5.809966564178467 01-18 12:54:59 INFO [logging.py:61]: Loss 'ar_loss' on epoch 50: 1.2504818439483643 01-18 12:54:59 INFO [logging.py:61]: Loss 'nar_loss' on epoch 50: 4.559484958648682 01-18 12:54:59 INFO [logging.py:61]: Loss 'ar_accuracy_metric' on epoch 50: 0.9094463586807251 01-18 12:54:59 INFO [logging.py:61]: Loss 'nar_acc_metric' on epoch 50: 0.38511520624160767 01-18 12:54:59 INFO [logging.py:61]: Saving current state to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0050 01-18 12:55:00 INFO [logging.py:61]: Model weights saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0050/pytorch_model.bin 01-18 12:55:01 INFO [logging.py:61]: Optimizer state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0050/optimizer.bin 01-18 12:55:01 INFO [logging.py:61]: Scheduler state saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0050/scheduler.bin 01-18 12:55:01 INFO [logging.py:61]: Sampler state for dataloader 0 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0050/sampler.bin 01-18 12:55:01 INFO [logging.py:61]: Sampler state for dataloader 1 saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0050/sampler_1.bin 01-18 12:55:01 INFO [logging.py:61]: Random states saved in /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0050/random_states_0.pkl 01-18 12:55:01 INFO [logging.py:61]: Saving the state of TrainerState to /fred/oz325/xhao/proj/audiozen/recipes/librimix_sot/tokenizer_separation/exp/swin_default_LR1e-2_AR-NAR/checkpoints/epoch_0050/custom_checkpoint_0.pkl 01-18 12:55:01 INFO [logging.py:61]: ========= Epoch 51 out of 1000 ========= 01-18 12:55:01 INFO [logging.py:61]: Begin training...