Starting... 
2022-04-11 16:28:44.758017: Using splits from existing split file: /home/woody/iwi5/iwi5039h/nnUNet_data/nnUNet_preprocessed/Task500_Glacier_zonefronts/splits_final.pkl 
2022-04-11 16:28:44.794525: The split file contains 5 splits. 
2022-04-11 16:28:44.795276: Desired fold for training: 2 
2022-04-11 16:28:44.796228: This split has 447 training and 112 validation cases. 
2022-04-11 16:28:45.320519: TRAINING KEYS:
 odict_keys(['Crane_2002-11-09_ERS_20_2_061', 'Crane_2002-11-18_ERS_20_3_195', 'Crane_2003-01-18_ERS_20_1_061', 'Crane_2007-02-05_ERS_20_1_195', 'Crane_2007-03-31_ENVISAT_20_3_467', 'Crane_2007-05-05_ENVISAT_20_1_467', 'Crane_2007-09-22_ENVISAT_20_1_467', 'Crane_2007-09-24_PALSAR_17_1_137', 'Crane_2007-10-27_ENVISAT_20_2_467', 'Crane_2007-11-21_PALSAR_17_2_135', 'Crane_2008-10-24_TSX_7_2_034', 'Crane_2008-11-23_PALSAR_17_2_135', 'Crane_2009-09-29_PALSAR_17_1_137', 'Crane_2009-11-02_TSX_7_3_034', 'Crane_2009-11-13_TSX_7_1_034', 'Crane_2010-07-03_ENVISAT_20_2_467', 'Crane_2010-09-15_PALSAR_17_4_138', 'Crane_2010-10-14_PALSAR_17_2_135', 'Crane_2010-11-05_TSX_7_2_110', 'Crane_2010-11-16_TSX_7_2_110', 'Crane_2010-11-27_TSX_7_1_110', 'Crane_2010-11-29_PALSAR_17_1_135', 'Crane_2010-12-08_TSX_7_1_110', 'Crane_2010-12-19_TSX_7_1_110', 'Crane_2010-12-25_TSX_7_1_034', 'Crane_2010-12-30_TSX_7_1_110', 'Crane_2011-06-02_TSX_7_2_110', 'Crane_2011-06-24_TSX_7_1_110', 'Crane_2011-07-16_TSX_7_1_110', 'Crane_2011-08-13_TSX_7_1_034', 'Crane_2011-09-04_TSX_7_1_034', 'Crane_2011-12-01_TSX_7_1_034', 'Crane_2011-12-12_TSX_7_2_034', 'Crane_2012-06-16_TSX_7_2_034', 'Crane_2012-07-25_TSX_7_2_125', 'Crane_2012-08-10_TSX_7_3_034', 'Crane_2012-08-21_TSX_7_3_034', 'Crane_2012-09-01_TSX_7_1_034', 'Crane_2012-11-06_TSX_7_2_034', 'Crane_2013-03-07_TSX_7_2_034', 'Crane_2013-03-29_TSX_7_2_034', 'Crane_2013-06-30_TSX_7_2_110', 'Crane_2013-07-17_TSX_7_3_034', 'Crane_2013-07-22_TSX_7_3_110', 'Crane_2013-08-02_TSX_7_3_110', 'Crane_2013-08-19_TSX_7_4_034', 'Crane_2013-11-04_TSX_7_4_034', 'Crane_2013-11-15_TSX_7_3_034', 'Crane_2013-11-26_TSX_7_4_034', 'Crane_2013-12-18_TSX_7_4_034', 'Crane_2013-12-29_TSX_7_4_034', 'Crane_2014-01-20_TSX_7_2_034', 'Crane_2014-02-11_TSX_7_3_034', 'DBE_1995-11-16_ERS_20_1_152', 'DBE_1996-01-25_ERS_20_1_152', 'DBE_1997-03-05_ERS_20_1_424', 'DBE_1998-02-18_ERS_20_1_424', 'DBE_1999-11-10_ERS_20_4_424', 'DBE_2000-09-10_RSAT_20_3_282', 'DBE_2000-10-04_RSAT_20_4_282', 'DBE_2000-10-06_RSAT_12_3_306', 'DBE_2000-10-13_RSAT_20_4_063', 'DBE_2002-02-27_ERS_20_1_424', 'DBE_2002-11-16_ERS_20_3_161', 'DBE_2002-12-21_ERS_20_4_161', 'DBE_2003-01-08_ERS_20_4_424', 'DBE_2003-02-12_ERS_20_3_424', 'DBE_2003-11-19_ERS_20_3_424', 'DBE_2004-03-03_ENVISAT_20_3_424', 'DBE_2004-03-03_ERS_20_3_424', 'DBE_2004-04-07_ENVISAT_20_3_424', 'DBE_2004-06-16_ENVISAT_20_3_424', 'DBE_2004-07-21_ENVISAT_20_3_424', 'DBE_2004-08-25_ENVISAT_20_3_424', 'DBE_2004-09-13_RSAT_12_3_282', 'DBE_2004-09-29_ENVISAT_20_3_424', 'DBE_2004-10-07_RSAT_12_4_282', 'DBE_2004-10-16_RSAT_12_3_063', 'DBE_2004-11-03_ENVISAT_20_4_424', 'DBE_2004-11-09_RSAT_12_3_063', 'DBE_2004-11-24_RSAT_12_3_282', 'DBE_2004-12-08_ENVISAT_20_5_424', 'DBE_2004-12-08_ERS_20_5_424', 'DBE_2004-12-11_RSAT_12_4_182', 'DBE_2004-12-18_RSAT_12_3_282', 'DBE_2005-01-12_ERS_20_6_424', 'DBE_2005-03-23_ENVISAT_20_1_424', 'DBE_2005-04-27_ENVISAT_20_2_424', 'DBE_2005-05-11_RSAT_12_3_282', 'DBE_2005-06-01_ENVISAT_20_2_424', 'DBE_2005-07-06_ENVISAT_20_3_424', 'DBE_2005-08-10_ENVISAT_20_2_424', 'DBE_2006-01-06_RSAT_12_4_282', 'DBE_2006-04-12_ENVISAT_20_1_424', 'DBE_2006-04-12_RSAT_12_1_282', 'DBE_2006-05-06_RSAT_12_3_282', 'DBE_2006-05-18_PALSAR_17_2_135', 'DBE_2006-06-21_ENVISAT_20_2_424', 'DBE_2006-06-21_PALSAR_17_2_137', 'DBE_2006-07-03_PALSAR_17_3_135', 'DBE_2006-07-26_ENVISAT_20_3_424', 'DBE_2006-08-06_PALSAR_17_2_137', 'DBE_2006-08-10_RSAT_20_4_282', 'DBE_2006-08-30_ENVISAT_20_2_424', 'DBE_2006-09-03_RSAT_20_5_282', 'DBE_2006-11-06_PALSAR_17_3_137', 'DBE_2006-11-08_ENVISAT_20_3_424', 'DBE_2007-01-01_RSAT_12_5_282', 'DBE_2007-01-25_RSAT_12_5_282', 'DBE_2007-05-01_RSAT_12_4_282', 'DBE_2007-08-15_ENVISAT_20_5_424', 'DBE_2007-09-19_ENVISAT_20_6_424', 'DBE_2007-11-11_PALSAR_17_4_129', 'DBE_2007-12-03_RSAT_12_4_282', 'DBE_2007-12-27_PALSAR_17_5_129', 'DBE_2008-09-28_PALSAR_17_5_129', 'DBE_2008-10-14_TSX_7_3_049', 'DBE_2008-11-05_TSX_7_2_049', 'DBE_2008-11-13_PALSAR_17_5_129', 'DBE_2008-12-29_PALSAR_17_5_129', 'DBE_2009-10-23_TSX_7_2_049', 'DBE_2009-11-03_TSX_7_2_049', 'DBE_2009-12-03_PALSAR_17_5_132', 'DBE_2010-01-18_PALSAR_17_5_132', 'DBE_2010-01-22_ERS_20_6_152', 'DBE_2010-03-17_ENVISAT_20_6_424', 'DBE_2010-04-21_ENVISAT_20_6_424', 'DBE_2010-10-10_TSX_7_2_049', 'DBE_2010-10-16_PALSAR_17_4_127', 'DBE_2010-10-21_PALSAR_17_4_132', 'DBE_2010-11-12_TSX_7_4_049', 'DBE_2010-11-23_TSX_7_5_049', 'DBE_2010-12-01_PALSAR_17_4_127', 'DBE_2010-12-06_PALSAR_17_3_132', 'DBE_2011-01-16_PALSAR_17_4_127', 'DBE_2011-01-17_TSX_7_4_049', 'DBE_2011-01-28_TSX_7_2_049', 'DBE_2011-07-05_TSX_7_1_110', 'DBE_2011-07-27_TSX_7_1_110', 'DBE_2011-09-09_TSX_7_1_110', 'DBE_2011-09-20_TSX_7_1_110', 'DBE_2011-12-06_TSX_7_5_110', 'DBE_2011-12-28_TSX_7_2_110', 'DBE_2012-03-14_TSX_7_1_110', 'DBE_2012-04-05_TSX_7_1_110', 'DBE_2012-05-08_TSX_7_1_110', 'DBE_2012-06-10_TSX_7_1_110', 'DBE_2012-06-21_TSX_7_1_110', 'DBE_2012-08-15_TSX_7_1_110', 'DBE_2012-11-11_TSX_7_1_110', 'DBE_2013-01-16_TSX_7_5_110', 'DBE_2013-03-12_TSX_7_1_110', 'DBE_2013-03-23_TSX_7_1_110', 'DBE_2013-05-28_TSX_7_1_110', 'DBE_2013-06-19_TSX_7_1_110', 'DBE_2013-07-11_TSX_7_1_110', 'DBE_2013-09-04_TSX_7_1_110', 'DBE_2013-10-07_TSX_7_1_110', 'DBE_2013-11-09_TSX_7_1_110', 'DBE_2013-12-01_TSX_7_2_110', 'DBE_2013-12-12_TSX_7_3_110', 'DBE_2013-12-23_TSX_7_2_110', 'DBE_2014-01-03_TSX_7_5_110', 'DBE_2014-02-05_TSX_7_1_110', 'DBE_2014-03-21_TSX_7_2_110', 'JAC_2009-04-27_TSX_6_1_005', 'JAC_2009-05-08_TSX_6_1_005', 'JAC_2009-05-19_TSX_6_1_005', 'JAC_2009-05-30_TSX_6_1_005', 'JAC_2009-06-10_TSX_6_1_005', 'JAC_2009-06-21_TSX_6_1_005', 'JAC_2009-07-13_TSX_6_1_005', 'JAC_2009-07-24_TSX_6_1_005', 'JAC_2009-08-04_TSX_6_1_005', 'JAC_2009-08-26_TSX_6_1_005', 'JAC_2009-09-06_TSX_6_1_005', 'JAC_2009-09-17_TSX_6_1_005', 'JAC_2009-10-09_TSX_6_1_005', 'JAC_2009-10-20_TSX_6_1_005', 'JAC_2009-10-31_TSX_6_1_005', 'JAC_2009-11-11_TSX_6_1_005', 'JAC_2009-12-03_TSX_6_1_005', 'JAC_2009-12-14_TSX_6_1_005', 'JAC_2009-12-25_TSX_6_1_005', 'JAC_2010-01-05_TSX_6_1_005', 'JAC_2010-01-16_TSX_6_1_005', 'JAC_2010-02-07_TSX_6_1_005', 'JAC_2010-03-23_TSX_6_1_005', 'JAC_2010-04-14_TSX_6_1_005', 'JAC_2010-04-25_TSX_6_1_005', 'JAC_2010-05-06_TSX_6_1_005', 'JAC_2010-05-17_TSX_6_1_005', 'JAC_2010-06-19_TSX_6_1_005', 'JAC_2010-07-11_TSX_6_1_005', 'JAC_2010-07-22_TSX_6_1_005', 'JAC_2010-09-04_TSX_6_1_005', 'JAC_2010-09-15_TSX_6_1_005', 'JAC_2010-10-07_TSX_6_1_005', 'JAC_2010-11-09_TSX_6_1_005', 'JAC_2010-11-20_TSX_6_1_005', 'JAC_2010-12-01_TSX_6_1_005', 'JAC_2010-12-23_TSX_6_1_005', 'JAC_2011-01-03_TSX_6_1_005', 'JAC_2011-01-25_TSX_6_1_005', 'JAC_2011-02-16_TSX_6_1_005', 'JAC_2011-03-10_TSX_6_1_005', 'JAC_2011-03-21_TSX_6_1_005', 'JAC_2011-04-12_TSX_6_1_005', 'JAC_2011-04-23_TSX_6_1_005', 'JAC_2011-05-15_TSX_6_1_005', 'JAC_2011-06-17_TSX_6_1_005', 'JAC_2011-08-23_TSX_6_1_027', 'JAC_2011-09-03_TSX_6_1_027', 'JAC_2011-09-14_TSX_6_1_027', 'JAC_2011-10-16_TSX_6_1_005', 'JAC_2011-11-08_TSX_6_1_027', 'JAC_2011-12-10_TSX_6_1_005', 'JAC_2011-12-21_TSX_6_1_005', 'JAC_2012-01-13_TSX_6_1_027', 'JAC_2012-01-23_TSX_6_1_005', 'JAC_2012-02-14_TSX_6_1_005', 'JAC_2012-03-07_TSX_6_1_005', 'JAC_2012-03-29_TSX_6_1_005', 'JAC_2012-04-20_TSX_6_1_005', 'JAC_2012-05-24_TSX_6_1_027', 'JAC_2012-06-03_TSX_6_1_005', 'JAC_2012-06-14_TSX_6_1_005', 'JAC_2012-07-07_TSX_6_1_027', 'JAC_2012-07-18_TSX_6_1_027', 'JAC_2012-08-09_TSX_6_1_027', 'JAC_2012-08-20_TSX_6_1_027', 'JAC_2012-09-11_TSX_6_1_027', 'JAC_2012-10-03_TSX_6_1_027', 'JAC_2012-10-14_TSX_6_1_027', 'JAC_2012-11-15_TSX_6_1_005', 'JAC_2012-12-07_TSX_6_1_005', 'JAC_2013-01-20_TSX_6_1_005', 'JAC_2013-02-11_TSX_6_1_005', 'JAC_2013-02-22_TSX_6_1_005', 'JAC_2013-03-06_TSX_6_1_027', 'JAC_2013-03-17_TSX_6_1_027', 'JAC_2013-04-07_TSX_6_1_005', 'JAC_2013-04-18_TSX_6_1_005', 'JAC_2013-05-10_TSX_6_1_005', 'JAC_2013-05-22_TSX_6_1_027', 'JAC_2013-06-13_TSX_6_1_027', 'JAC_2013-06-24_TSX_6_1_027', 'JAC_2013-07-05_TSX_6_1_027', 'JAC_2013-08-17_TSX_6_1_005', 'JAC_2013-09-08_TSX_6_1_005', 'JAC_2013-09-20_TSX_6_1_027', 'JAC_2013-10-23_TSX_6_1_027', 'JAC_2013-11-14_TSX_6_1_027', 'JAC_2013-11-25_TSX_6_1_027', 'JAC_2013-12-06_TSX_6_1_027', 'JAC_2013-12-17_TSX_6_1_027', 'JAC_2014-01-29_TSX_6_1_005', 'JAC_2014-03-03_TSX_6_1_005', 'JAC_2014-03-14_TSX_6_1_005', 'JAC_2014-03-25_TSX_6_1_005', 'JAC_2014-04-05_TSX_6_1_005', 'JAC_2014-04-16_TSX_6_1_005', 'JAC_2014-04-27_TSX_6_1_005', 'JAC_2014-06-10_TSX_6_1_005', 'JAC_2014-06-21_TSX_6_1_005', 'JAC_2014-07-03_TSX_6_1_027', 'JAC_2014-07-14_TSX_6_1_027', 'JAC_2014-07-25_TSX_6_1_027', 'JAC_2014-09-18_TSX_6_1_027', 'JAC_2014-11-11_TSX_6_1_005', 'JAC_2014-12-03_TSX_6_1_005', 'JAC_2014-12-25_TSX_6_1_005', 'JAC_2015-01-05_TSX_6_1_005', 'JAC_2015-01-16_TSX_6_1_005', 'JAC_2015-02-07_TSX_6_1_005', 'JAC_2015-02-19_TSX_6_1_027', 'JAC_2015-03-12_TSX_6_1_005', 'JAC_2015-03-23_TSX_6_1_005', 'JAC_2015-04-25_TSX_6_1_005', 'JAC_2015-05-06_TSX_6_1_005', 'JAC_2015-05-17_TSX_6_1_005', 'JAC_2015-06-08_TSX_6_1_005', 'JAC_2015-06-19_TSX_6_1_005', 'JAC_2015-07-11_TSX_6_1_005', 'JAC_2015-07-22_TSX_6_1_005', 'JAC_2015-08-02_TSX_6_1_005', 'JAC_2015-08-13_TSX_6_1_005', 'JAC_2015-09-04_TSX_6_1_005', 'JAC_2015-10-18_TSX_6_1_005', 'JAC_2015-10-29_TSX_6_1_005', 'JAC_2015-11-09_TSX_6_1_005', 'JAC_2015-11-20_TSX_6_1_005', 'JAC_2015-12-12_TSX_6_1_005', 'JAC_2015-12-23_TSX_6_1_005', 'Jorum_2003-01-27_ERS_20_1_195', 'Jorum_2006-11-27_ERS_20_3_195', 'Jorum_2007-03-31_ENVISAT_20_2_467', 'Jorum_2007-05-05_ENVISAT_20_2_467', 'Jorum_2007-06-09_ENVISAT_20_2_467', 'Jorum_2007-09-22_ENVISAT_20_3_467', 'Jorum_2007-10-27_ENVISAT_20_3_467', 'Jorum_2007-11-21_PALSAR_17_2_135', 'Jorum_2008-01-05_ENVISAT_20_2_467', 'Jorum_2008-11-23_PALSAR_17_2_135', 'Jorum_2009-12-25_PALSAR_17_1_136', 'Jorum_2010-03-20_ENVISAT_20_2_467', 'Jorum_2010-09-15_PALSAR_17_2_138', 'Jorum_2010-11-05_TSX_7_1_110', 'Jorum_2010-11-16_TSX_7_2_110', 'Jorum_2010-11-27_TSX_7_2_110', 'Jorum_2010-11-29_PALSAR_17_1_135', 'Jorum_2010-12-08_TSX_7_2_110', 'Jorum_2010-12-14_TSX_7_1_034', 'Jorum_2010-12-19_TSX_7_1_110', 'Jorum_2010-12-25_TSX_7_1_034', 'Jorum_2010-12-30_TSX_7_1_110', 'Jorum_2011-06-24_TSX_7_2_110', 'Jorum_2011-07-16_TSX_7_3_110', 'Jorum_2011-08-13_TSX_7_2_034', 'Jorum_2011-09-04_TSX_7_4_034', 'Jorum_2011-10-18_TSX_7_3_034', 'Jorum_2012-05-14_TSX_7_2_034', 'Jorum_2012-06-16_TSX_7_2_034', 'Jorum_2012-06-27_TSX_7_2_034', 'Jorum_2012-07-25_TSX_7_2_125', 'Jorum_2012-07-30_TSX_7_2_034', 'Jorum_2012-08-10_TSX_7_2_034', 'Jorum_2012-08-21_TSX_7_2_034', 'Jorum_2012-10-04_TSX_7_2_034', 'Jorum_2013-02-24_TSX_7_3_034', 'Jorum_2013-03-29_TSX_7_2_034', 'Jorum_2013-06-30_TSX_7_2_110', 'Jorum_2013-07-17_TSX_7_1_034', 'Jorum_2013-07-22_TSX_7_1_110', 'Jorum_2013-08-02_TSX_7_2_110', 'Jorum_2013-08-19_TSX_7_2_034', 'Jorum_2013-11-26_TSX_7_5_034', 'Jorum_2013-12-07_TSX_7_5_034', 'Jorum_2013-12-18_TSX_7_5_034', 'Jorum_2013-12-29_TSX_7_4_034', 'Jorum_2014-01-20_TSX_7_2_034', 'Jorum_2014-02-11_TSX_7_2_034', 'Jorum_2016-07-03_S1_20_2_009', 'Jorum_2016-09-01_S1_20_2_009', 'Jorum_2016-12-30_S1_20_2_009', 'Jorum_2017-06-16_S1_20_2_009', 'Jorum_2017-09-08_S1_20_2_009', 'Jorum_2018-01-06_S1_20_3_009', 'Jorum_2018-03-07_S1_20_3_009', 'Jorum_2018-06-11_S1_20_2_009', 'Jorum_2018-09-03_S1_20_3_009', 'Jorum_2019-03-02_S1_20_3_009', 'Jorum_2020-06-12_S1_20_3_009', 'SI_1995-11-13_ERS_20_3_109', 'SI_1995-11-14_ERS_20_4_118', 'SI_1995-11-15_ERS_20_3_118', 'SI_1995-11-16_ERS_20_4_152', 'SI_1996-01-22_ERS_20_3_109', 'SI_1996-02-12_ERS_20_2_390', 'SI_1996-02-29_ERS_20_2_152', 'SI_1996-03-18_ERS_20_2_390', 'SI_1997-01-27_ERS_20_4_390', 'SI_1997-03-03_ERS_20_4_390', 'SI_1999-11-07_ERS_20_2_381', 'SI_2000-02-20_ERS_20_3_381', 'SI_2000-09-10_RSAT_20_5_282', 'SI_2000-09-19_RSAT_20_5_063', 'SI_2000-10-04_RSAT_20_5_282', 'SI_2002-01-21_ERS_20_2_390', 'SI_2002-02-25_ERS_20_3_390', 'SI_2002-11-12_ERS_20_4_109', 'SI_2002-12-21_ERS_20_3_161', 'SI_2003-01-25_ERS_20_3_161', 'SI_2004-02-13_ENVISAT_20_4_152', 'SI_2004-07-02_ENVISAT_20_4_152', 'SI_2004-08-06_ENVISAT_20_4_152', 'SI_2004-09-13_RSAT_12_5_282', 'SI_2004-11-09_RSAT_12_5_063', 'SI_2004-11-19_ERS_20_4_152', 'SI_2004-11-24_RSAT_12_4_282', 'SI_2004-12-03_RSAT_12_5_063', 'SI_2004-12-18_RSAT_12_4_282', 'SI_2004-12-24_ERS_20_5_152', 'SI_2005-01-28_ERS_20_5_152', 'SI_2005-02-28_RSAT_12_5_282', 'SI_2005-07-22_ENVISAT_20_2_152', 'SI_2005-08-26_ENVISAT_20_2_152', 'SI_2006-01-06_RSAT_12_2_282', 'SI_2006-01-29_ENVISAT_20_3_381', 'SI_2006-02-23_RSAT_12_1_282', 'SI_2006-03-05_ENVISAT_20_2_381', 'SI_2006-04-12_RSAT_12_1_282', 'SI_2006-05-06_RSAT_12_3_282', 'SI_2006-05-18_PALSAR_17_2_135', 'SI_2006-07-03_PALSAR_17_3_135', 'SI_2006-08-27_ENVISAT_20_3_381', 'SI_2006-09-03_RSAT_12_2_282', 'SI_2006-09-03_RSAT_20_2_282', 'SI_2006-10-18_ERS_20_4_118', 'SI_2007-01-01_RSAT_12_3_282', 'SI_2007-01-14_ENVISAT_20_3_381', 'SI_2007-01-25_RSAT_12_1_282', 'SI_2007-03-14_RSAT_12_2_282', 'SI_2007-03-25_ENVISAT_20_3_381', 'SI_2007-05-01_RSAT_12_2_282', 'SI_2007-06-03_ENVISAT_20_3_381', 'SI_2007-07-08_ENVISAT_20_3_381', 'SI_2007-09-16_ENVISAT_20_3_381', 'SI_2007-12-27_RSAT_12_2_282', 'SI_2008-01-20_RSAT_12_2_282', 'SI_2008-04-13_ENVISAT_20_2_381', 'SI_2008-05-18_ENVISAT_20_3_381', 'SI_2008-07-27_ENVISAT_20_4_381', 'SI_2008-08-31_ENVISAT_20_4_381', 'SI_2009-02-22_ENVISAT_20_2_381', 'SI_2009-03-29_ENVISAT_20_4_381', 'SI_2009-07-12_ENVISAT_20_3_381', 'SI_2009-08-16_ENVISAT_20_3_381', 'SI_2010-01-13_PALSAR_17_2_127', 'SI_2010-04-02_ERS_20_2_152', 'SI_2010-12-01_PALSAR_17_3_127', 'SI_2011-01-16_PALSAR_17_2_127', 'SI_2011-03-03_PALSAR_17_2_127', 'SI_2011-06-14_TSX_7_1_125', 'SI_2011-07-28_TSX_7_1_125', 'SI_2011-09-10_TSX_7_1_125', 'SI_2011-09-21_TSX_7_1_125', 'SI_2011-11-26_TSX_7_2_125', 'SI_2012-01-09_TSX_7_1_125', 'SI_2012-03-04_TSX_7_1_125', 'SI_2012-03-15_TSX_7_1_125', 'SI_2012-04-28_TSX_7_1_125', 'SI_2012-06-22_TSX_7_1_125', 'SI_2012-07-03_TSX_7_1_125', 'SI_2012-08-05_TSX_7_1_125', 'SI_2012-08-16_TSX_7_1_125', 'SI_2013-07-12_TSX_7_1_125', 'SI_2013-08-14_TSX_7_1_125', 'SI_2013-10-19_TSX_7_1_125', 'SI_2013-11-08_TSX_7_2_104', 'SI_2013-11-21_TSX_7_1_125', 'SI_2013-12-02_TSX_7_1_125', 'SI_2013-12-13_TSX_7_2_125', 'SI_2014-01-04_TSX_7_2_125', 'SI_2014-03-22_TSX_7_1_125', 'SI_2014-04-02_TSX_7_1_125', 'SI_2014-12-22_TSX_7_2_125']) 
2022-04-11 16:28:45.321667: VALIDATION KEYS:
 odict_keys(['Crane_2006-11-27_ERS_20_1_195', 'Crane_2007-06-09_ENVISAT_20_1_467', 'Crane_2007-07-14_ENVISAT_20_1_467', 'Crane_2008-01-05_ENVISAT_20_2_467', 'Crane_2008-10-13_TSX_7_3_034', 'Crane_2008-11-04_TSX_7_1_034', 'Crane_2010-05-29_ENVISAT_20_3_467', 'Crane_2010-12-14_TSX_7_1_034', 'Crane_2011-10-18_TSX_7_1_034', 'Crane_2012-05-14_TSX_7_2_034', 'Crane_2012-06-27_TSX_7_2_034', 'Crane_2012-07-03_TSX_7_1_125', 'Crane_2012-07-30_TSX_7_2_034', 'Crane_2012-10-04_TSX_7_3_034', 'Crane_2013-02-24_TSX_7_2_034', 'Crane_2013-12-07_TSX_7_5_034', 'DBE_1997-01-29_ERS_20_2_424', 'DBE_2004-05-12_ENVISAT_20_3_424', 'DBE_2005-02-16_ENVISAT_20_1_424', 'DBE_2006-02-23_RSAT_12_3_282', 'DBE_2006-03-08_ENVISAT_20_1_424', 'DBE_2006-03-19_RSAT_12_4_282', 'DBE_2006-10-04_ENVISAT_20_3_424', 'DBE_2007-02-18_RSAT_12_4_282', 'DBE_2007-03-14_RSAT_12_3_282', 'DBE_2007-12-27_RSAT_12_5_282', 'DBE_2008-01-20_RSAT_12_4_282', 'DBE_2008-03-30_TSX_7_3_049', 'DBE_2008-10-25_TSX_7_2_049', 'DBE_2009-02-13_PALSAR_17_5_129', 'DBE_2010-11-19_PALSAR_17_4_129', 'DBE_2011-01-04_PALSAR_17_4_129', 'DBE_2011-10-23_TSX_7_1_110', 'DBE_2011-11-14_TSX_7_1_110', 'DBE_2012-08-04_TSX_7_1_110', 'DBE_2012-10-31_TSX_7_1_110', 'DBE_2013-04-03_TSX_7_1_110', 'JAC_2009-04-16_TSX_6_1_005', 'JAC_2009-07-02_TSX_6_1_005', 'JAC_2009-08-15_TSX_6_1_005', 'JAC_2009-09-28_TSX_6_1_005', 'JAC_2009-11-22_TSX_6_1_005', 'JAC_2010-02-18_TSX_6_1_005', 'JAC_2010-03-12_TSX_6_1_005', 'JAC_2010-06-30_TSX_6_1_005', 'JAC_2010-10-18_TSX_6_1_005', 'JAC_2011-05-26_TSX_6_1_005', 'JAC_2011-07-20_TSX_6_1_005', 'JAC_2011-10-05_TSX_6_1_005', 'JAC_2011-11-18_TSX_6_1_005', 'JAC_2012-02-25_TSX_6_1_005', 'JAC_2012-09-21_TSX_6_1_005', 'JAC_2012-11-04_TSX_6_1_005', 'JAC_2012-12-19_TSX_6_1_027', 'JAC_2013-01-09_TSX_6_1_005', 'JAC_2013-07-27_TSX_6_1_027', 'JAC_2013-08-28_TSX_6_1_005', 'JAC_2013-10-12_TSX_6_1_027', 'JAC_2014-07-02_TSX_6_1_005', 'JAC_2014-08-04_TSX_6_1_005', 'JAC_2014-09-28_TSX_6_1_005', 'JAC_2014-10-09_TSX_6_1_005', 'JAC_2014-10-20_TSX_6_1_005', 'JAC_2014-12-14_TSX_6_1_005', 'JAC_2015-01-27_TSX_6_1_005', 'JAC_2015-04-14_TSX_6_1_005', 'JAC_2015-09-26_TSX_6_1_005', 'Jorum_2007-02-05_ERS_20_1_195', 'Jorum_2007-07-14_ENVISAT_20_2_467', 'Jorum_2010-05-29_ENVISAT_20_2_467', 'Jorum_2010-07-03_ENVISAT_20_2_467', 'Jorum_2010-10-14_PALSAR_17_2_135', 'Jorum_2011-06-02_TSX_7_3_110', 'Jorum_2011-12-01_TSX_7_1_034', 'Jorum_2011-12-12_TSX_7_3_034', 'Jorum_2012-07-03_TSX_7_2_125', 'Jorum_2012-09-01_TSX_7_2_034', 'Jorum_2012-11-06_TSX_7_2_034', 'Jorum_2013-03-07_TSX_7_2_034', 'Jorum_2013-11-04_TSX_7_2_034', 'Jorum_2013-11-15_TSX_7_2_034', 'Jorum_2019-01-01_S1_20_3_009', 'Jorum_2019-09-22_S1_20_3_009', 'Jorum_2020-01-08_S1_20_3_009', 'Jorum_2020-03-08_S1_20_3_009', 'SI_1996-01-23_ERS_20_5_109', 'SI_1996-01-24_ERS_20_4_118', 'SI_2000-10-13_RSAT_20_5_063', 'SI_2003-12-05_ENVISAT_20_4_152', 'SI_2004-10-15_ERS_20_5_152', 'SI_2004-12-24_ENVISAT_20_5_152', 'SI_2005-03-04_ENVISAT_20_3_152', 'SI_2005-04-08_ENVISAT_20_2_152', 'SI_2005-04-17_RSAT_12_3_282', 'SI_2005-05-11_RSAT_12_2_282', 'SI_2006-03-19_RSAT_12_2_282', 'SI_2006-07-23_ENVISAT_20_4_381', 'SI_2006-08-10_RSAT_20_3_282', 'SI_2006-11-21_ERS_20_3_109', 'SI_2007-10-21_ENVISAT_20_3_381', 'SI_2007-12-03_RSAT_12_4_282', 'SI_2010-01-22_ERS_20_3_152', 'SI_2010-02-26_ERS_20_2_152', 'SI_2010-10-16_PALSAR_17_3_127', 'SI_2011-07-06_TSX_7_1_125', 'SI_2011-08-19_TSX_7_1_125', 'SI_2011-12-29_TSX_7_1_125', 'SI_2012-03-26_TSX_7_1_125', 'SI_2012-05-20_TSX_7_1_125', 'SI_2013-09-16_TSX_7_1_125', 'SI_2013-10-28_TSX_7_1_104', 'SI_2014-12-11_TSX_7_4_125']) 
2022-04-11 16:29:02.311103: lr: 0.01 
2022-04-11 16:29:21.970881: Unable to plot network architecture: 
2022-04-11 16:29:22.138290: No module named 'hiddenlayer' 
2022-04-11 16:29:22.279533: 
printing the network instead:
 
2022-04-11 16:29:22.407020: Generic_UNet(
  (conv_blocks_localization): ModuleList(
    (0): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (1): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (2): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (3): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (4): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(512, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (5): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(256, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (6): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(128, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (7): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(64, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
  )
  (conv_blocks_context): ModuleList(
    (0): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(1, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (1): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(32, 64, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (2): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(64, 128, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (3): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(128, 256, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (4): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(256, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (5): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (6): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (7): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (8): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
  )
  (td): ModuleList()
  (tu): ModuleList(
    (0): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (1): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (2): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (3): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (4): ConvTranspose2d(480, 256, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (5): ConvTranspose2d(256, 128, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (6): ConvTranspose2d(128, 64, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (7): ConvTranspose2d(64, 32, kernel_size=[2, 2], stride=[2, 2], bias=False)
  )
  (seg_outputs): ModuleList(
    (0): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (1): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (2): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (3): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (4): Conv2d(256, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (5): Conv2d(128, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (6): Conv2d(64, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (7): Conv2d(32, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
  )
) 
2022-04-11 16:29:22.568178: 
 
2022-04-11 16:29:22.720524: svmem(total=99932160000, available=85939302400, percent=14.0, used=11108339712, free=82753105920, active=10908237824, inactive=4518129664, buffers=11083776, cached=6059630592, shared=1102602240, slab=680423424) 
2022-04-11 16:29:22.891376: 
epoch:  0 
2022-04-11 16:32:18.620029: train loss : 0.8383 
2022-04-11 16:32:30.366750: validation loss: 0.7132 
2022-04-11 16:32:30.372205: Average global foreground Dice: [0.0, 0.6944, 0.002, 0.0] 
2022-04-11 16:32:30.378840: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:32:32.159397: lr: 0.009982 
2022-04-11 16:32:32.160324: saving scheduled checkpoint file... 
2022-04-11 16:32:32.333378: saving checkpoint... 
2022-04-11 16:32:36.706608: done, saving took 4.55 seconds 
2022-04-11 16:32:36.752414: done 
2022-04-11 16:32:36.757953: This epoch took 193.702975 s
 
2022-04-11 16:32:36.762106: svmem(total=99932160000, available=82068475904, percent=17.9, used=14032879616, free=63497764864, active=22821175296, inactive=11797909504, buffers=12124160, cached=22389391360, shared=2029879296, slab=728453120) 
2022-04-11 16:32:36.766997: 
epoch:  1 
2022-04-11 16:34:28.137769: train loss : 0.6358 
2022-04-11 16:34:39.541886: validation loss: 0.5909 
2022-04-11 16:34:39.555787: Average global foreground Dice: [0.0127, 0.6198, 0.5431, 0.0] 
2022-04-11 16:34:39.558937: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:34:40.538138: lr: 0.009964 
2022-04-11 16:34:40.539016: saving scheduled checkpoint file... 
2022-04-11 16:34:40.676353: saving checkpoint... 
2022-04-11 16:34:44.991780: done, saving took 4.45 seconds 
2022-04-11 16:34:45.018047: done 
2022-04-11 16:34:45.173623: saving checkpoint... 
2022-04-11 16:34:51.502676: done, saving took 6.48 seconds 
2022-04-11 16:34:51.780866: This epoch took 135.006552 s
 
2022-04-11 16:34:51.969091: svmem(total=99932160000, available=81727209472, percent=18.2, used=14245724160, free=58060582912, active=29426511872, inactive=10606313472, buffers=12156928, cached=27613696000, shared=2152267776, slab=752168960) 
2022-04-11 16:34:52.092752: 
epoch:  2 
2022-04-11 16:36:42.645243: train loss : 0.5418 
2022-04-11 16:36:54.706502: validation loss: 0.5038 
2022-04-11 16:36:54.709538: Average global foreground Dice: [0.3021, 0.677, 0.5582, 0.0] 
2022-04-11 16:36:54.714470: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:36:55.354076: lr: 0.009946 
2022-04-11 16:36:55.355220: saving scheduled checkpoint file... 
2022-04-11 16:36:55.484973: saving checkpoint... 
2022-04-11 16:36:59.793855: done, saving took 4.44 seconds 
2022-04-11 16:36:59.815878: done 
2022-04-11 16:36:59.973000: saving checkpoint... 
2022-04-11 16:37:05.968635: done, saving took 6.15 seconds 
2022-04-11 16:37:06.354204: This epoch took 134.109006 s
 
2022-04-11 16:37:06.488575: svmem(total=99932160000, available=81502392320, percent=18.4, used=14443917312, free=55984324608, active=33066758144, inactive=9032454144, buffers=12292096, cached=29491625984, shared=2176741376, slab=758013952) 
2022-04-11 16:37:06.612522: 
epoch:  3 
2022-04-11 16:38:57.942476: train loss : 0.4260 
2022-04-11 16:39:09.580778: validation loss: 0.4502 
2022-04-11 16:39:09.585446: Average global foreground Dice: [0.3913, 0.7044, 0.4113, 0.0] 
2022-04-11 16:39:09.586855: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:39:10.599400: lr: 0.009928 
2022-04-11 16:39:10.600310: saving scheduled checkpoint file... 
2022-04-11 16:39:10.761249: saving checkpoint... 
2022-04-11 16:39:15.275614: done, saving took 4.67 seconds 
2022-04-11 16:39:15.297841: done 
2022-04-11 16:39:15.443369: saving checkpoint... 
2022-04-11 16:39:23.171240: done, saving took 7.87 seconds 
2022-04-11 16:39:23.185701: This epoch took 136.439317 s
 
2022-04-11 16:39:23.186834: svmem(total=99932160000, available=81479114752, percent=18.5, used=14466318336, free=55163805696, active=35064827904, inactive=7881318400, buffers=12341248, cached=30289694720, shared=2176741376, slab=748388352) 
2022-04-11 16:39:23.187587: 
epoch:  4 
2022-04-11 16:41:12.486218: train loss : 0.3930 
2022-04-11 16:41:24.883153: validation loss: 0.3187 
2022-04-11 16:41:24.884849: Average global foreground Dice: [0.3792, 0.7446, 0.6179, 0.0] 
2022-04-11 16:41:24.885910: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:41:25.618442: lr: 0.00991 
2022-04-11 16:41:25.619349: saving scheduled checkpoint file... 
2022-04-11 16:41:25.782159: saving checkpoint... 
2022-04-11 16:41:30.304513: done, saving took 4.68 seconds 
2022-04-11 16:41:30.318978: done 
2022-04-11 16:41:30.365493: saving checkpoint... 
2022-04-11 16:41:37.593537: done, saving took 7.27 seconds 
2022-04-11 16:41:37.615170: This epoch took 134.426736 s
 
2022-04-11 16:41:37.616170: svmem(total=99932160000, available=81214849024, percent=18.7, used=14730006528, free=54483496960, active=36364800000, inactive=7260372992, buffers=12374016, cached=30706282496, shared=2176741376, slab=748867584) 
2022-04-11 16:41:37.616845: 
epoch:  5 
2022-04-11 16:43:27.516055: train loss : 0.3247 
2022-04-11 16:43:39.403166: validation loss: 0.2913 
2022-04-11 16:43:39.414866: Average global foreground Dice: [0.5024, 0.6905, 0.6954, 0.0] 
2022-04-11 16:43:39.415762: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:43:40.371935: lr: 0.009892 
2022-04-11 16:43:40.372980: saving scheduled checkpoint file... 
2022-04-11 16:43:40.530972: saving checkpoint... 
2022-04-11 16:43:44.884229: done, saving took 4.51 seconds 
2022-04-11 16:43:44.913023: done 
2022-04-11 16:43:45.060934: saving checkpoint... 
2022-04-11 16:43:52.741711: done, saving took 7.83 seconds 
2022-04-11 16:43:52.764571: This epoch took 135.147149 s
 
2022-04-11 16:43:52.765695: svmem(total=99932160000, available=81247326208, percent=18.7, used=14697349120, free=54205628416, active=36976738304, inactive=6924636160, buffers=12406784, cached=31016775680, shared=2176741376, slab=748040192) 
2022-04-11 16:43:52.766554: 
epoch:  6 
2022-04-11 16:45:43.486783: train loss : 0.2841 
2022-04-11 16:45:55.587921: validation loss: 0.3800 
2022-04-11 16:45:55.590691: Average global foreground Dice: [0.403, 0.6846, 0.6232, 0.0003] 
2022-04-11 16:45:55.592041: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:45:56.274127: lr: 0.009874 
2022-04-11 16:45:56.274945: saving scheduled checkpoint file... 
2022-04-11 16:45:56.407263: saving checkpoint... 
2022-04-11 16:46:00.804041: done, saving took 4.53 seconds 
2022-04-11 16:46:00.827181: done 
2022-04-11 16:46:00.962447: saving checkpoint... 
2022-04-11 16:46:08.852499: done, saving took 8.02 seconds 
2022-04-11 16:46:08.873817: This epoch took 136.106552 s
 
2022-04-11 16:46:08.874954: svmem(total=99932160000, available=80838062080, percent=19.1, used=15106457600, free=53648084992, active=37889077248, inactive=6565842944, buffers=12468224, cached=31165149184, shared=2176741376, slab=747859968) 
2022-04-11 16:46:08.875713: 
epoch:  7 
2022-04-11 16:48:00.378889: train loss : 0.2836 
2022-04-11 16:48:13.406800: validation loss: 0.1969 
2022-04-11 16:48:13.414472: Average global foreground Dice: [0.4222, 0.7256, 0.7557, 0.0007] 
2022-04-11 16:48:13.415601: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:48:14.135707: lr: 0.009856 
2022-04-11 16:48:14.136525: saving scheduled checkpoint file... 
2022-04-11 16:48:14.260251: saving checkpoint... 
2022-04-11 16:48:19.547201: done, saving took 5.41 seconds 
2022-04-11 16:48:19.895602: done 
2022-04-11 16:48:20.166358: saving checkpoint... 
2022-04-11 16:48:26.662812: done, saving took 6.63 seconds 
2022-04-11 16:48:27.012335: This epoch took 138.135899 s
 
2022-04-11 16:48:27.146479: svmem(total=99932160000, available=80644329472, percent=19.3, used=15300014080, free=53441540096, active=38611701760, inactive=6039908352, buffers=12402688, cached=31178203136, shared=2176741376, slab=762716160) 
2022-04-11 16:48:27.279247: 
epoch:  8 
2022-04-11 16:50:17.649608: train loss : 0.2460 
2022-04-11 16:50:29.697590: validation loss: 0.1389 
2022-04-11 16:50:29.715172: Average global foreground Dice: [0.507, 0.7575, 0.7225, 0.0007] 
2022-04-11 16:50:29.717262: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:50:30.351159: lr: 0.009838 
2022-04-11 16:50:30.352123: saving scheduled checkpoint file... 
2022-04-11 16:50:30.477761: saving checkpoint... 
2022-04-11 16:50:35.893661: done, saving took 5.54 seconds 
2022-04-11 16:50:36.242221: done 
2022-04-11 16:50:36.498096: saving checkpoint... 
2022-04-11 16:50:42.943192: done, saving took 6.58 seconds 
2022-04-11 16:50:43.221102: This epoch took 135.825356 s
 
2022-04-11 16:50:43.364026: svmem(total=99932160000, available=80585244672, percent=19.4, used=15359098880, free=53473447936, active=39260164096, inactive=5358718976, buffers=12443648, cached=31087169536, shared=2176741376, slab=763191296) 
2022-04-11 16:50:43.488315: 
epoch:  9 
2022-04-11 16:52:32.582656: train loss : 0.2146 
2022-04-11 16:52:45.068069: validation loss: 0.1891 
2022-04-11 16:52:45.069990: Average global foreground Dice: [0.451, 0.77, 0.7121, 0.0035] 
2022-04-11 16:52:45.070942: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:52:45.979302: lr: 0.00982 
2022-04-11 16:52:45.980196: saving scheduled checkpoint file... 
2022-04-11 16:52:46.111447: saving checkpoint... 
2022-04-11 16:52:51.455339: done, saving took 5.47 seconds 
2022-04-11 16:52:51.795504: done 
2022-04-11 16:52:52.038329: saving checkpoint... 
2022-04-11 16:52:58.546830: done, saving took 6.64 seconds 
2022-04-11 16:52:58.878459: This epoch took 135.265339 s
 
2022-04-11 16:52:59.003027: svmem(total=99932160000, available=80423473152, percent=19.5, used=15520915456, free=53297934336, active=39673171968, inactive=5125767168, buffers=12476416, cached=31100833792, shared=2176741376, slab=761810944) 
2022-04-11 16:52:59.148776: 
epoch:  10 
2022-04-11 16:54:50.813915: train loss : 0.2205 
2022-04-11 16:55:02.422807: validation loss: 0.2414 
2022-04-11 16:55:02.433598: Average global foreground Dice: [0.4035, 0.7376, 0.7648, 0.0052] 
2022-04-11 16:55:02.434442: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:55:03.272655: lr: 0.009802 
2022-04-11 16:55:03.273656: saving scheduled checkpoint file... 
2022-04-11 16:55:03.417008: saving checkpoint... 
2022-04-11 16:55:07.798768: done, saving took 4.52 seconds 
2022-04-11 16:55:07.820100: done 
2022-04-11 16:55:07.943480: saving checkpoint... 
2022-04-11 16:55:15.221753: done, saving took 7.40 seconds 
2022-04-11 16:55:15.243127: This epoch took 135.964637 s
 
2022-04-11 16:55:15.244159: svmem(total=99932160000, available=80358567936, percent=19.6, used=15585705984, free=53400592384, active=39886299136, inactive=4810670080, buffers=12550144, cached=30933311488, shared=2176741376, slab=749621248) 
2022-04-11 16:55:15.244933: 
epoch:  11 
2022-04-11 16:57:03.065999: train loss : 0.1415 
2022-04-11 16:57:15.346260: validation loss: 0.0764 
2022-04-11 16:57:15.348165: Average global foreground Dice: [0.4835, 0.7948, 0.8104, 0.0059] 
2022-04-11 16:57:15.349193: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:57:15.976434: lr: 0.009784 
2022-04-11 16:57:15.977364: saving scheduled checkpoint file... 
2022-04-11 16:57:16.066934: saving checkpoint... 
2022-04-11 16:57:20.310653: done, saving took 4.33 seconds 
2022-04-11 16:57:20.324946: done 
2022-04-11 16:57:20.373294: saving checkpoint... 
2022-04-11 16:57:24.636313: done, saving took 4.31 seconds 
2022-04-11 16:57:24.650393: This epoch took 129.404654 s
 
2022-04-11 16:57:24.651708: svmem(total=99932160000, available=80444608512, percent=19.5, used=15499546624, free=53553885184, active=39689162752, inactive=4867719168, buffers=12570624, cached=30866157568, shared=2176741376, slab=748748800) 
2022-04-11 16:57:24.652543: 
epoch:  12 
2022-04-11 16:59:13.830305: train loss : 0.1190 
2022-04-11 16:59:26.085970: validation loss: 0.0806 
2022-04-11 16:59:26.087827: Average global foreground Dice: [0.4783, 0.7912, 0.8138, 0.0009] 
2022-04-11 16:59:26.088846: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 16:59:26.964910: lr: 0.009766 
2022-04-11 16:59:26.965954: saving scheduled checkpoint file... 
2022-04-11 16:59:27.077009: saving checkpoint... 
2022-04-11 16:59:31.496207: done, saving took 4.53 seconds 
2022-04-11 16:59:31.521912: done 
2022-04-11 16:59:31.568365: saving checkpoint... 
2022-04-11 16:59:36.034108: done, saving took 4.51 seconds 
2022-04-11 16:59:36.045140: This epoch took 131.391899 s
 
2022-04-11 16:59:36.046377: svmem(total=99932160000, available=80202440704, percent=19.7, used=15741689856, free=53428174848, active=39755771904, inactive=4926009344, buffers=9449472, cached=30752845824, shared=2176741376, slab=750268416) 
2022-04-11 16:59:36.047213: 
epoch:  13 
2022-04-11 17:01:24.825757: train loss : 0.0839 
2022-04-11 17:01:36.191854: validation loss: 0.1541 
2022-04-11 17:01:36.196495: Average global foreground Dice: [0.4653, 0.8043, 0.7763, 0.014] 
2022-04-11 17:01:36.197667: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:01:37.065439: lr: 0.009748 
2022-04-11 17:01:37.066352: saving scheduled checkpoint file... 
2022-04-11 17:01:37.134460: saving checkpoint... 
2022-04-11 17:01:41.502199: done, saving took 4.43 seconds 
2022-04-11 17:01:41.516398: done 
2022-04-11 17:01:41.579885: saving checkpoint... 
2022-04-11 17:01:45.928214: done, saving took 4.41 seconds 
2022-04-11 17:01:45.949211: This epoch took 129.901156 s
 
2022-04-11 17:01:45.950520: svmem(total=99932160000, available=80092258304, percent=19.9, used=15852011520, free=53897637888, active=39632306176, inactive=4571983872, buffers=9510912, cached=30172999680, shared=2176741376, slab=753008640) 
2022-04-11 17:01:45.951542: 
epoch:  14 
2022-04-11 17:03:35.556691: train loss : 0.1555 
2022-04-11 17:03:48.272540: validation loss: 0.0834 
2022-04-11 17:03:48.274387: Average global foreground Dice: [0.4438, 0.8066, 0.7466, 0.0023] 
2022-04-11 17:03:48.275354: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:03:49.138830: lr: 0.00973 
2022-04-11 17:03:49.139895: saving scheduled checkpoint file... 
2022-04-11 17:03:49.258554: saving checkpoint... 
2022-04-11 17:03:53.607033: done, saving took 4.47 seconds 
2022-04-11 17:03:53.620690: done 
2022-04-11 17:03:53.699992: saving checkpoint... 
2022-04-11 17:03:59.861936: done, saving took 6.24 seconds 
2022-04-11 17:04:00.193324: This epoch took 134.240984 s
 
2022-04-11 17:04:00.327151: svmem(total=99932160000, available=80090370048, percent=19.9, used=15853895680, free=53577572352, active=39788568576, inactive=4728938496, buffers=9543680, cached=30491148288, shared=2176741376, slab=762818560) 
2022-04-11 17:04:00.460663: 
epoch:  15 
2022-04-11 17:05:50.712062: train loss : 0.0690 
2022-04-11 17:06:02.457979: validation loss: 0.0573 
2022-04-11 17:06:02.459675: Average global foreground Dice: [0.5576, 0.7947, 0.7042, 0.0035] 
2022-04-11 17:06:02.460576: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:06:03.385564: lr: 0.009712 
2022-04-11 17:06:03.386735: saving scheduled checkpoint file... 
2022-04-11 17:06:03.511280: saving checkpoint... 
2022-04-11 17:06:07.968447: done, saving took 4.58 seconds 
2022-04-11 17:06:07.988547: done 
2022-04-11 17:06:08.084845: saving checkpoint... 
2022-04-11 17:06:15.758411: done, saving took 7.77 seconds 
2022-04-11 17:06:16.019409: This epoch took 135.416072 s
 
2022-04-11 17:06:16.020797: svmem(total=99932160000, available=79975960576, percent=20.0, used=15968157696, free=53419659264, active=39982460928, inactive=4692889600, buffers=9625600, cached=30534717440, shared=2176741376, slab=763142144) 
2022-04-11 17:06:16.021628: 
epoch:  16 
2022-04-11 17:08:05.711457: train loss : 0.0850 
2022-04-11 17:08:17.350441: validation loss: 0.1282 
2022-04-11 17:08:17.356658: Average global foreground Dice: [0.5066, 0.8209, 0.7088, 0.0173] 
2022-04-11 17:08:17.357989: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:08:18.070345: lr: 0.009693 
2022-04-11 17:08:18.071199: saving scheduled checkpoint file... 
2022-04-11 17:08:18.189613: saving checkpoint... 
2022-04-11 17:08:22.637767: done, saving took 4.57 seconds 
2022-04-11 17:08:22.658099: done 
2022-04-11 17:08:22.767280: saving checkpoint... 
2022-04-11 17:08:28.155891: done, saving took 5.50 seconds 
2022-04-11 17:08:28.513254: This epoch took 132.490921 s
 
2022-04-11 17:08:28.639047: svmem(total=99932160000, available=79840915456, percent=20.1, used=16103251968, free=53262864384, active=40172126208, inactive=4656844800, buffers=9678848, cached=30556364800, shared=2176741376, slab=763895808) 
2022-04-11 17:08:28.780650: 
epoch:  17 
2022-04-11 17:10:18.798044: train loss : 0.0241 
2022-04-11 17:10:30.934109: validation loss: -0.0185 
2022-04-11 17:10:30.945080: Average global foreground Dice: [0.5679, 0.8152, 0.8264, 0.0182] 
2022-04-11 17:10:30.945982: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:10:32.069904: lr: 0.009675 
2022-04-11 17:10:32.070898: saving scheduled checkpoint file... 
2022-04-11 17:10:32.188094: saving checkpoint... 
2022-04-11 17:10:36.594737: done, saving took 4.52 seconds 
2022-04-11 17:10:36.613117: done 
2022-04-11 17:10:36.679364: saving checkpoint... 
2022-04-11 17:10:41.384459: done, saving took 4.77 seconds 
2022-04-11 17:10:41.724480: This epoch took 132.818515 s
 
2022-04-11 17:10:41.850417: svmem(total=99932160000, available=80092262400, percent=19.9, used=15851790336, free=53559996416, active=39647866880, inactive=4889632768, buffers=9719808, cached=30510653440, shared=2176741376, slab=763949056) 
2022-04-11 17:10:41.992156: 
epoch:  18 
2022-04-11 17:12:30.192746: train loss : -0.0028 
2022-04-11 17:12:42.267686: validation loss: -0.0014 
2022-04-11 17:12:42.269645: Average global foreground Dice: [0.5523, 0.8278, 0.814, 0.0197] 
2022-04-11 17:12:42.271551: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:12:43.200034: lr: 0.009657 
2022-04-11 17:12:43.201229: saving scheduled checkpoint file... 
2022-04-11 17:12:43.321431: saving checkpoint... 
2022-04-11 17:12:47.656323: done, saving took 4.45 seconds 
2022-04-11 17:12:47.680110: done 
2022-04-11 17:12:47.761659: saving checkpoint... 
2022-04-11 17:12:52.640848: done, saving took 4.96 seconds 
2022-04-11 17:12:52.980912: This epoch took 130.864134 s
 
2022-04-11 17:12:53.115172: svmem(total=99932160000, available=80108302336, percent=19.8, used=15835815936, free=53592854528, active=39737450496, inactive=4762378240, buffers=9752576, cached=30493736960, shared=2176741376, slab=763990016) 
2022-04-11 17:12:53.230497: 
epoch:  19 
2022-04-11 17:14:42.352023: train loss : -0.0007 
2022-04-11 17:14:54.513379: validation loss: -0.0653 
2022-04-11 17:14:54.515137: Average global foreground Dice: [0.5946, 0.8358, 0.8671, 0.0196] 
2022-04-11 17:14:54.516787: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:14:55.391411: lr: 0.009639 
2022-04-11 17:14:55.392390: saving scheduled checkpoint file... 
2022-04-11 17:14:55.527889: saving checkpoint... 
2022-04-11 17:14:59.855733: done, saving took 4.46 seconds 
2022-04-11 17:14:59.876522: done 
2022-04-11 17:14:59.977354: saving checkpoint... 
2022-04-11 17:15:07.528285: done, saving took 7.65 seconds 
2022-04-11 17:15:07.796789: This epoch took 134.432788 s
 
2022-04-11 17:15:07.904507: svmem(total=99932160000, available=80065179648, percent=19.9, used=15878995968, free=53530169344, active=39928692736, inactive=4632068096, buffers=9809920, cached=30513184768, shared=2176741376, slab=761933824) 
2022-04-11 17:15:07.914895: 
epoch:  20 
2022-04-11 17:16:57.713430: train loss : -0.0237 
2022-04-11 17:17:10.100204: validation loss: -0.0513 
2022-04-11 17:17:10.102030: Average global foreground Dice: [0.5357, 0.8327, 0.8615, 0.0327] 
2022-04-11 17:17:10.103093: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:17:10.740647: lr: 0.009621 
2022-04-11 17:17:10.741664: saving scheduled checkpoint file... 
2022-04-11 17:17:10.854535: saving checkpoint... 
2022-04-11 17:17:17.112521: done, saving took 6.37 seconds 
2022-04-11 17:17:17.533105: done 
2022-04-11 17:17:17.780490: saving checkpoint... 
2022-04-11 17:17:24.301504: done, saving took 6.63 seconds 
2022-04-11 17:17:24.721561: This epoch took 136.805703 s
 
2022-04-11 17:17:24.846983: svmem(total=99932160000, available=80032976896, percent=19.9, used=15911194624, free=53487419392, active=39987073024, inactive=4619493376, buffers=9883648, cached=30523662336, shared=2176741376, slab=761585664) 
2022-04-11 17:17:24.988778: 
epoch:  21 
2022-04-11 17:19:15.541661: train loss : -0.0466 
2022-04-11 17:19:26.841453: validation loss: -0.0949 
2022-04-11 17:19:26.852394: Average global foreground Dice: [0.5883, 0.8515, 0.8637, 0.0133] 
2022-04-11 17:19:26.853574: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:19:27.431586: lr: 0.009603 
2022-04-11 17:19:27.432583: saving scheduled checkpoint file... 
2022-04-11 17:19:27.557918: saving checkpoint... 
2022-04-11 17:19:35.096781: done, saving took 7.66 seconds 
2022-04-11 17:19:35.119835: done 
2022-04-11 17:19:35.254205: saving checkpoint... 
2022-04-11 17:19:42.979483: done, saving took 7.86 seconds 
2022-04-11 17:19:43.003520: This epoch took 137.889827 s
 
2022-04-11 17:19:43.004719: svmem(total=99932160000, available=80139321344, percent=19.8, used=15791153152, free=53571235840, active=39902949376, inactive=4630851584, buffers=9924608, cached=30559846400, shared=2190397440, slab=748982272) 
2022-04-11 17:19:43.005449: 
epoch:  22 
2022-04-11 17:21:32.761564: train loss : -0.0659 
2022-04-11 17:21:44.796192: validation loss: -0.1110 
2022-04-11 17:21:44.802004: Average global foreground Dice: [0.5653, 0.8655, 0.8835, 0.0215] 
2022-04-11 17:21:44.803018: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:21:45.871306: lr: 0.009585 
2022-04-11 17:21:46.012506: saving scheduled checkpoint file... 
2022-04-11 17:21:46.262728: saving checkpoint... 
2022-04-11 17:21:52.795921: done, saving took 6.66 seconds 
2022-04-11 17:21:53.144849: done 
2022-04-11 17:21:53.389305: saving checkpoint... 
2022-04-11 17:21:59.181036: done, saving took 5.91 seconds 
2022-04-11 17:21:59.202549: This epoch took 136.196440 s
 
2022-04-11 17:21:59.203742: svmem(total=99932160000, available=80054931456, percent=19.9, used=15875563520, free=53467832320, active=40003940352, inactive=4631003136, buffers=10354688, cached=30578409472, shared=2190618624, slab=748404736) 
2022-04-11 17:21:59.204435: 
epoch:  23 
2022-04-11 17:23:50.992524: train loss : -0.0616 
2022-04-11 17:24:02.849772: validation loss: -0.0433 
2022-04-11 17:24:02.854854: Average global foreground Dice: [0.5612, 0.8286, 0.8396, 0.0075] 
2022-04-11 17:24:02.855673: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:24:03.709100: lr: 0.009567 
2022-04-11 17:24:03.833583: saving scheduled checkpoint file... 
2022-04-11 17:24:04.033678: saving checkpoint... 
2022-04-11 17:24:10.349413: done, saving took 6.40 seconds 
2022-04-11 17:24:10.363986: done 
2022-04-11 17:24:10.413529: saving checkpoint... 
2022-04-11 17:24:14.711354: done, saving took 4.35 seconds 
2022-04-11 17:24:14.732906: This epoch took 135.527718 s
 
2022-04-11 17:24:14.733961: svmem(total=99932160000, available=79820492800, percent=20.1, used=16104665088, free=53236469760, active=40128978944, inactive=4739092480, buffers=10403840, cached=30580621312, shared=2195603456, slab=748437504) 
2022-04-11 17:24:14.734887: 
epoch:  24 
2022-04-11 17:26:04.071591: train loss : -0.0485 
2022-04-11 17:26:16.500901: validation loss: -0.1258 
2022-04-11 17:26:16.636851: Average global foreground Dice: [0.5697, 0.8633, 0.8952, 0.0213] 
2022-04-11 17:26:16.770446: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:26:17.773479: lr: 0.009549 
2022-04-11 17:26:17.774662: saving scheduled checkpoint file... 
2022-04-11 17:26:17.869101: saving checkpoint... 
2022-04-11 17:26:22.360280: done, saving took 4.58 seconds 
2022-04-11 17:26:22.376608: done 
2022-04-11 17:26:22.425719: saving checkpoint... 
2022-04-11 17:26:26.785501: done, saving took 4.41 seconds 
2022-04-11 17:26:26.800156: This epoch took 132.064413 s
 
2022-04-11 17:26:26.801381: svmem(total=99932160000, available=79859445760, percent=20.1, used=16073908224, free=53255327744, active=40176402432, inactive=4673867776, buffers=10457088, cached=30592466944, shared=2187481088, slab=749211648) 
2022-04-11 17:26:26.801984: 
epoch:  25 
2022-04-11 17:28:15.712314: train loss : -0.1192 
2022-04-11 17:28:27.569278: validation loss: -0.1539 
2022-04-11 17:28:27.579335: Average global foreground Dice: [0.6091, 0.8569, 0.9034, 0.0337] 
2022-04-11 17:28:27.580379: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:28:28.289363: lr: 0.009531 
2022-04-11 17:28:28.290240: saving scheduled checkpoint file... 
2022-04-11 17:28:28.379505: saving checkpoint... 
2022-04-11 17:28:32.813131: done, saving took 4.52 seconds 
2022-04-11 17:28:32.829498: done 
2022-04-11 17:28:32.879285: saving checkpoint... 
2022-04-11 17:28:37.261594: done, saving took 4.43 seconds 
2022-04-11 17:28:37.273144: This epoch took 130.470494 s
 
2022-04-11 17:28:37.274332: svmem(total=99932160000, available=79793893376, percent=20.2, used=16135827456, free=53195010048, active=40254816256, inactive=4653740032, buffers=10489856, cached=30590832640, shared=2190909440, slab=751222784) 
2022-04-11 17:28:37.275262: 
epoch:  26 
2022-04-11 17:30:25.948448: train loss : -0.0717 
2022-04-11 17:30:37.383471: validation loss: -0.0910 
2022-04-11 17:30:37.388302: Average global foreground Dice: [0.5798, 0.8526, 0.8756, 0.0319] 
2022-04-11 17:30:37.389362: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:30:38.352343: lr: 0.009513 
2022-04-11 17:30:38.353359: saving scheduled checkpoint file... 
2022-04-11 17:30:38.426958: saving checkpoint... 
2022-04-11 17:30:42.799635: done, saving took 4.45 seconds 
2022-04-11 17:30:42.818932: done 
2022-04-11 17:30:42.896253: saving checkpoint... 
2022-04-11 17:30:47.340676: done, saving took 4.52 seconds 
2022-04-11 17:30:47.354033: This epoch took 130.078048 s
 
2022-04-11 17:30:47.355217: svmem(total=99932160000, available=79969701888, percent=20.0, used=15960064000, free=53354881024, active=40069017600, inactive=4681125888, buffers=10530816, cached=30606684160, shared=2190950400, slab=750616576) 
2022-04-11 17:30:47.355999: 
epoch:  27 
2022-04-11 17:32:35.520586: train loss : -0.0928 
2022-04-11 17:32:46.824144: validation loss: -0.1246 
2022-04-11 17:32:46.948321: Average global foreground Dice: [0.646, 0.8596, 0.8766, 0.0181] 
2022-04-11 17:32:47.064045: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:32:48.019142: lr: 0.009495 
2022-04-11 17:32:48.020500: saving scheduled checkpoint file... 
2022-04-11 17:32:48.114504: saving checkpoint... 
2022-04-11 17:32:52.465997: done, saving took 4.44 seconds 
2022-04-11 17:32:52.485550: done 
2022-04-11 17:32:52.575041: saving checkpoint... 
2022-04-11 17:32:56.929423: done, saving took 4.44 seconds 
2022-04-11 17:32:56.948479: This epoch took 129.591626 s
 
2022-04-11 17:32:56.949757: svmem(total=99932160000, available=79922532352, percent=20.0, used=16003096576, free=53290283008, active=40149504000, inactive=4660228096, buffers=10563584, cached=30628216832, shared=2195189760, slab=750067712) 
2022-04-11 17:32:56.950728: 
epoch:  28 
2022-04-11 17:34:45.965609: train loss : -0.0420 
2022-04-11 17:34:58.183168: validation loss: -0.1120 
2022-04-11 17:34:58.185290: Average global foreground Dice: [0.6003, 0.8603, 0.89, 0.0176] 
2022-04-11 17:34:58.186235: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:34:59.064234: lr: 0.009476 
2022-04-11 17:34:59.065377: saving scheduled checkpoint file... 
2022-04-11 17:34:59.202975: saving checkpoint... 
2022-04-11 17:35:03.555598: done, saving took 4.49 seconds 
2022-04-11 17:35:03.576909: done 
2022-04-11 17:35:03.679896: saving checkpoint... 
2022-04-11 17:35:08.169672: done, saving took 4.59 seconds 
2022-04-11 17:35:08.190194: This epoch took 131.238769 s
 
2022-04-11 17:35:08.191296: svmem(total=99932160000, available=79856181248, percent=20.1, used=16073908224, free=53204942848, active=40221618176, inactive=4673159168, buffers=10620928, cached=30642688000, shared=2190729216, slab=748519424) 
2022-04-11 17:35:08.191986: 
epoch:  29 
2022-04-11 17:36:56.324367: train loss : -0.1198 
2022-04-11 17:37:06.723888: validation loss: -0.2037 
2022-04-11 17:37:06.877604: Average global foreground Dice: [0.6417, 0.8637, 0.9082, 0.0513] 
2022-04-11 17:37:07.018170: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:37:07.982632: lr: 0.009458 
2022-04-11 17:37:07.983580: saving scheduled checkpoint file... 
2022-04-11 17:37:08.084745: saving checkpoint... 
2022-04-11 17:37:12.561200: done, saving took 4.58 seconds 
2022-04-11 17:37:12.577147: done 
2022-04-11 17:37:12.658896: saving checkpoint... 
2022-04-11 17:37:17.088682: done, saving took 4.50 seconds 
2022-04-11 17:37:17.106342: This epoch took 128.913617 s
 
2022-04-11 17:37:17.107531: svmem(total=99932160000, available=79909937152, percent=20.0, used=16019812352, free=53259444224, active=40113741824, inactive=4728684544, buffers=10674176, cached=30642229248, shared=2190917632, slab=750198784) 
2022-04-11 17:37:17.108247: 
epoch:  30 
2022-04-11 17:39:05.067832: train loss : -0.1542 
2022-04-11 17:39:16.987144: validation loss: -0.2459 
2022-04-11 17:39:17.121413: Average global foreground Dice: [0.672, 0.8921, 0.9155, 0.0568] 
2022-04-11 17:39:17.272040: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:39:18.193395: lr: 0.00944 
2022-04-11 17:39:18.194754: saving scheduled checkpoint file... 
2022-04-11 17:39:18.307282: saving checkpoint... 
2022-04-11 17:39:22.687311: done, saving took 4.49 seconds 
2022-04-11 17:39:22.707944: done 
2022-04-11 17:39:22.792160: saving checkpoint... 
2022-04-11 17:39:27.159605: done, saving took 4.45 seconds 
2022-04-11 17:39:27.180301: This epoch took 130.071377 s
 
2022-04-11 17:39:27.185214: svmem(total=99932160000, available=79653949440, percent=20.3, used=16276070400, free=52996567040, active=40377241600, inactive=4732076032, buffers=10706944, cached=30648815616, shared=2190729216, slab=749166592) 
2022-04-11 17:39:27.186246: 
epoch:  31 
2022-04-11 17:41:16.194634: train loss : -0.1475 
2022-04-11 17:41:28.196286: validation loss: -0.1884 
2022-04-11 17:41:28.333374: Average global foreground Dice: [0.6379, 0.8586, 0.8987, 0.1151] 
2022-04-11 17:41:28.430262: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:41:29.253831: lr: 0.009422 
2022-04-11 17:41:29.254672: saving scheduled checkpoint file... 
2022-04-11 17:41:29.362135: saving checkpoint... 
2022-04-11 17:41:33.688899: done, saving took 4.43 seconds 
2022-04-11 17:41:33.708772: done 
2022-04-11 17:41:33.804230: saving checkpoint... 
2022-04-11 17:41:38.199274: done, saving took 4.49 seconds 
2022-04-11 17:41:38.214114: This epoch took 131.027078 s
 
2022-04-11 17:41:38.215335: svmem(total=99932160000, available=79988953088, percent=20.0, used=15941107712, free=53314121728, active=40076025856, inactive=4712493056, buffers=10747904, cached=30666182656, shared=2190540800, slab=750309376) 
2022-04-11 17:41:38.216099: 
epoch:  32 
2022-04-11 17:43:27.916218: train loss : -0.1702 
2022-04-11 17:43:39.710464: validation loss: -0.2211 
2022-04-11 17:43:39.718107: Average global foreground Dice: [0.6379, 0.8833, 0.9269, 0.1226] 
2022-04-11 17:43:39.719246: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:43:40.596567: lr: 0.009404 
2022-04-11 17:43:40.597480: saving scheduled checkpoint file... 
2022-04-11 17:43:40.731062: saving checkpoint... 
2022-04-11 17:43:45.172635: done, saving took 4.57 seconds 
2022-04-11 17:43:45.196418: done 
2022-04-11 17:43:45.308162: saving checkpoint... 
2022-04-11 17:43:49.698716: done, saving took 4.50 seconds 
2022-04-11 17:43:49.720945: This epoch took 131.504124 s
 
2022-04-11 17:43:49.722125: svmem(total=99932160000, available=79890546688, percent=20.1, used=16039641088, free=53206544384, active=40207495168, inactive=4691505152, buffers=10780672, cached=30675193856, shared=2190467072, slab=748601344) 
2022-04-11 17:43:49.722812: 
epoch:  33 
2022-04-11 17:45:38.813148: train loss : -0.1506 
2022-04-11 17:45:49.515244: validation loss: -0.2575 
2022-04-11 17:45:49.523391: Average global foreground Dice: [0.6852, 0.8793, 0.9208, 0.093] 
2022-04-11 17:45:49.524287: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:45:50.373707: lr: 0.009386 
2022-04-11 17:45:50.384726: saving scheduled checkpoint file... 
2022-04-11 17:45:50.538391: saving checkpoint... 
2022-04-11 17:45:55.048978: done, saving took 4.66 seconds 
2022-04-11 17:45:55.107425: done 
2022-04-11 17:45:55.232438: saving checkpoint... 
2022-04-11 17:45:59.677913: done, saving took 4.57 seconds 
2022-04-11 17:45:59.707698: This epoch took 129.984140 s
 
2022-04-11 17:45:59.709218: svmem(total=99932160000, available=79920259072, percent=20.0, used=15999909888, free=53262090240, active=40140156928, inactive=4699107328, buffers=10846208, cached=30659313664, shared=2200510464, slab=749338624) 
2022-04-11 17:45:59.709978: 
epoch:  34 
2022-04-11 17:47:50.518926: train loss : -0.1898 
2022-04-11 17:48:03.196715: validation loss: -0.1992 
2022-04-11 17:48:03.211408: Average global foreground Dice: [0.6177, 0.8592, 0.9292, 0.0813] 
2022-04-11 17:48:03.212494: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:48:03.917224: lr: 0.009368 
2022-04-11 17:48:03.918342: saving scheduled checkpoint file... 
2022-04-11 17:48:04.046407: saving checkpoint... 
2022-04-11 17:48:08.443705: done, saving took 4.52 seconds 
2022-04-11 17:48:08.467646: done 
2022-04-11 17:48:08.569381: saving checkpoint... 
2022-04-11 17:48:12.943191: done, saving took 4.47 seconds 
2022-04-11 17:48:12.964445: This epoch took 133.253653 s
 
2022-04-11 17:48:12.965667: svmem(total=99932160000, available=79787683840, percent=20.2, used=16155947008, free=53112004608, active=40304455680, inactive=4685148160, buffers=10895360, cached=30653313024, shared=2176851968, slab=749416448) 
2022-04-11 17:48:12.966528: 
epoch:  35 
2022-04-11 17:50:01.568428: train loss : -0.1489 
2022-04-11 17:50:14.108016: validation loss: -0.1756 
2022-04-11 17:50:14.115074: Average global foreground Dice: [0.6538, 0.8545, 0.8959, 0.1401] 
2022-04-11 17:50:14.116234: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:50:14.757838: lr: 0.00935 
2022-04-11 17:50:14.759098: saving scheduled checkpoint file... 
2022-04-11 17:50:14.849431: saving checkpoint... 
2022-04-11 17:50:19.370234: done, saving took 4.61 seconds 
2022-04-11 17:50:19.384979: done 
2022-04-11 17:50:19.435884: saving checkpoint... 
2022-04-11 17:50:23.843127: done, saving took 4.46 seconds 
2022-04-11 17:50:23.859512: This epoch took 130.892308 s
 
2022-04-11 17:50:23.860728: svmem(total=99932160000, available=79913381888, percent=20.0, used=16016400384, free=53234814976, active=40140328960, inactive=4733530112, buffers=10936320, cached=30670008320, shared=2190729216, slab=749203456) 
2022-04-11 17:50:23.861744: 
epoch:  36 
2022-04-11 17:52:12.791188: train loss : -0.2172 
2022-04-11 17:52:23.565217: validation loss: -0.2660 
2022-04-11 17:52:23.572215: Average global foreground Dice: [0.648, 0.8905, 0.9296, 0.1319] 
2022-04-11 17:52:23.573407: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:52:24.550938: lr: 0.009331 
2022-04-11 17:52:24.552175: saving scheduled checkpoint file... 
2022-04-11 17:52:24.637913: saving checkpoint... 
2022-04-11 17:52:29.085109: done, saving took 4.53 seconds 
2022-04-11 17:52:29.099234: done 
2022-04-11 17:52:29.145959: saving checkpoint... 
2022-04-11 17:52:33.515846: done, saving took 4.42 seconds 
2022-04-11 17:52:33.528689: This epoch took 129.666228 s
 
2022-04-11 17:52:33.529688: svmem(total=99932160000, available=79568367616, percent=20.4, used=16374923264, free=52889493504, active=40362979328, inactive=4854628352, buffers=10899456, cached=30656843776, shared=2177347584, slab=750022656) 
2022-04-11 17:52:33.530457: 
epoch:  37 
2022-04-11 17:54:21.777404: train loss : -0.1574 
2022-04-11 17:54:34.453546: validation loss: -0.2502 
2022-04-11 17:54:34.458647: Average global foreground Dice: [0.6485, 0.8762, 0.9135, 0.1625] 
2022-04-11 17:54:34.459774: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:54:35.317976: lr: 0.009313 
2022-04-11 17:54:35.319101: saving scheduled checkpoint file... 
2022-04-11 17:54:35.396176: saving checkpoint... 
2022-04-11 17:54:39.704309: done, saving took 4.38 seconds 
2022-04-11 17:54:39.721591: done 
2022-04-11 17:54:39.791577: saving checkpoint... 
2022-04-11 17:54:44.173209: done, saving took 4.45 seconds 
2022-04-11 17:54:44.204264: This epoch took 130.672939 s
 
2022-04-11 17:54:44.205723: svmem(total=99932160000, available=79898259456, percent=20.0, used=16024453120, free=53202071552, active=40163254272, inactive=4735365120, buffers=10932224, cached=30694703104, shared=2197856256, slab=749850624) 
2022-04-11 17:54:44.206516: 
epoch:  38 
2022-04-11 17:56:33.129577: train loss : -0.2479 
2022-04-11 17:56:45.722909: validation loss: -0.2709 
2022-04-11 17:56:45.731575: Average global foreground Dice: [0.6558, 0.8828, 0.9186, 0.1947] 
2022-04-11 17:56:45.732710: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:56:46.577985: lr: 0.009295 
2022-04-11 17:56:46.579007: saving scheduled checkpoint file... 
2022-04-11 17:56:46.701446: saving checkpoint... 
2022-04-11 17:56:51.205689: done, saving took 4.63 seconds 
2022-04-11 17:56:51.222567: done 
2022-04-11 17:56:51.306959: saving checkpoint... 
2022-04-11 17:56:55.754508: done, saving took 4.53 seconds 
2022-04-11 17:56:55.774802: This epoch took 131.567512 s
 
2022-04-11 17:56:55.776503: svmem(total=99932160000, available=79852994560, percent=20.1, used=16076169216, free=53139443712, active=40246689792, inactive=4721717248, buffers=10985472, cached=30705561600, shared=2191302656, slab=747655168) 
2022-04-11 17:56:55.777590: 
epoch:  39 
2022-04-11 17:58:43.966228: train loss : -0.2523 
2022-04-11 17:58:55.719875: validation loss: -0.3212 
2022-04-11 17:58:55.723001: Average global foreground Dice: [0.7205, 0.8902, 0.9451, 0.1997] 
2022-04-11 17:58:55.724006: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 17:58:56.482720: lr: 0.009277 
2022-04-11 17:58:56.483753: saving scheduled checkpoint file... 
2022-04-11 17:58:56.583149: saving checkpoint... 
2022-04-11 17:59:00.939406: done, saving took 4.45 seconds 
2022-04-11 17:59:00.959960: done 
2022-04-11 17:59:01.059660: saving checkpoint... 
2022-04-11 17:59:05.441554: done, saving took 4.48 seconds 
2022-04-11 17:59:05.461923: This epoch took 129.683524 s
 
2022-04-11 17:59:05.463245: svmem(total=99932160000, available=79816949760, percent=20.1, used=16102465536, free=53095460864, active=40271319040, inactive=4738600960, buffers=11018240, cached=30723215360, shared=2201137152, slab=748310528) 
2022-04-11 17:59:05.464458: 
epoch:  40 
2022-04-11 18:00:54.128830: train loss : -0.2956 
2022-04-11 18:01:06.459257: validation loss: -0.3328 
2022-04-11 18:01:06.461355: Average global foreground Dice: [0.6872, 0.8971, 0.9415, 0.2203] 
2022-04-11 18:01:06.463103: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:01:07.165745: lr: 0.009259 
2022-04-11 18:01:07.167098: saving scheduled checkpoint file... 
2022-04-11 18:01:07.282499: saving checkpoint... 
2022-04-11 18:01:11.618765: done, saving took 4.45 seconds 
2022-04-11 18:01:11.646321: done 
2022-04-11 18:01:11.761791: saving checkpoint... 
2022-04-11 18:01:16.120932: done, saving took 4.47 seconds 
2022-04-11 18:01:16.161144: This epoch took 130.695981 s
 
2022-04-11 18:01:16.162386: svmem(total=99932160000, available=79680802816, percent=20.3, used=16248975360, free=52941762560, active=40418725888, inactive=4745105408, buffers=11075584, cached=30730346496, shared=2190729216, slab=748171264) 
2022-04-11 18:01:16.163194: 
epoch:  41 
2022-04-11 18:03:08.028357: train loss : -0.3033 
2022-04-11 18:03:20.700680: validation loss: -0.3196 
2022-04-11 18:03:20.707918: Average global foreground Dice: [0.6973, 0.8954, 0.938, 0.1892] 
2022-04-11 18:03:20.709382: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:03:21.727026: lr: 0.009241 
2022-04-11 18:03:21.727811: saving scheduled checkpoint file... 
2022-04-11 18:03:21.831546: saving checkpoint... 
2022-04-11 18:03:26.266255: done, saving took 4.54 seconds 
2022-04-11 18:03:26.284009: done 
2022-04-11 18:03:26.366511: saving checkpoint... 
2022-04-11 18:03:30.784654: done, saving took 4.50 seconds 
2022-04-11 18:03:30.802447: This epoch took 134.638590 s
 
2022-04-11 18:03:30.803666: svmem(total=99932160000, available=79883628544, percent=20.1, used=16036175872, free=53127180288, active=40209625088, inactive=4772032512, buffers=11108352, cached=30757695488, shared=2200641536, slab=749694976) 
2022-04-11 18:03:30.804422: 
epoch:  42 
2022-04-11 18:05:19.555938: train loss : -0.2571 
2022-04-11 18:05:30.910809: validation loss: -0.3221 
2022-04-11 18:05:30.936192: Average global foreground Dice: [0.6674, 0.9013, 0.9325, 0.2041] 
2022-04-11 18:05:30.938836: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:05:31.792742: lr: 0.009223 
2022-04-11 18:05:31.795073: saving scheduled checkpoint file... 
2022-04-11 18:05:31.916603: saving checkpoint... 
2022-04-11 18:05:36.491760: done, saving took 4.69 seconds 
2022-04-11 18:05:36.517823: done 
2022-04-11 18:05:36.616285: saving checkpoint... 
2022-04-11 18:05:41.132042: done, saving took 4.61 seconds 
2022-04-11 18:05:41.153010: This epoch took 130.347757 s
 
2022-04-11 18:05:41.154269: svmem(total=99932160000, available=79716724736, percent=20.2, used=16207319040, free=52952604672, active=40386465792, inactive=4769701888, buffers=11161600, cached=30761074688, shared=2196180992, slab=748691456) 
2022-04-11 18:05:41.155024: 
epoch:  43 
2022-04-11 18:07:28.909273: train loss : -0.2976 
2022-04-11 18:07:41.015836: validation loss: -0.2873 
2022-04-11 18:07:41.017850: Average global foreground Dice: [0.649, 0.8944, 0.9344, 0.2118] 
2022-04-11 18:07:41.019704: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:07:41.661348: lr: 0.009204 
2022-04-11 18:07:41.662791: saving scheduled checkpoint file... 
2022-04-11 18:07:41.782575: saving checkpoint... 
2022-04-11 18:07:46.315125: done, saving took 4.65 seconds 
2022-04-11 18:07:46.339929: done 
2022-04-11 18:07:46.456807: saving checkpoint... 
2022-04-11 18:07:50.976947: done, saving took 4.64 seconds 
2022-04-11 18:07:50.994344: This epoch took 129.838613 s
 
2022-04-11 18:07:50.996086: svmem(total=99932160000, available=80165163008, percent=19.8, used=15764430848, free=53383573504, active=39935684608, inactive=4787187712, buffers=11206656, cached=30772948992, shared=2190839808, slab=748908544) 
2022-04-11 18:07:50.996893: 
epoch:  44 
2022-04-11 18:09:40.644806: train loss : -0.3329 
2022-04-11 18:09:52.637951: validation loss: -0.3455 
2022-04-11 18:09:52.644818: Average global foreground Dice: [0.7272, 0.8963, 0.9354, 0.2082] 
2022-04-11 18:09:52.645894: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:09:53.245603: lr: 0.009186 
2022-04-11 18:09:53.246867: saving scheduled checkpoint file... 
2022-04-11 18:09:53.382312: saving checkpoint... 
2022-04-11 18:09:57.748913: done, saving took 4.50 seconds 
2022-04-11 18:09:57.784714: done 
2022-04-11 18:09:57.911573: saving checkpoint... 
2022-04-11 18:10:02.321270: done, saving took 4.54 seconds 
2022-04-11 18:10:02.340304: This epoch took 131.342642 s
 
2022-04-11 18:10:02.341555: svmem(total=99932160000, available=79907651584, percent=20.0, used=16021749760, free=53108555776, active=40194035712, inactive=4804419584, buffers=11247616, cached=30790606848, shared=2190729216, slab=749531136) 
2022-04-11 18:10:02.342539: 
epoch:  45 
2022-04-11 18:11:52.329574: train loss : -0.3032 
2022-04-11 18:12:04.600862: validation loss: -0.3576 
2022-04-11 18:12:04.633094: Average global foreground Dice: [0.6938, 0.8982, 0.9449, 0.227] 
2022-04-11 18:12:04.634259: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:12:05.473109: lr: 0.009168 
2022-04-11 18:12:05.474448: saving scheduled checkpoint file... 
2022-04-11 18:12:05.612965: saving checkpoint... 
2022-04-11 18:12:09.999439: done, saving took 4.52 seconds 
2022-04-11 18:12:10.048984: done 
2022-04-11 18:12:10.190148: saving checkpoint... 
2022-04-11 18:12:14.751710: done, saving took 4.70 seconds 
2022-04-11 18:12:14.780734: This epoch took 132.437454 s
 
2022-04-11 18:12:14.782234: svmem(total=99932160000, available=79993397248, percent=20.0, used=15935844352, free=53186457600, active=40106336256, inactive=4812460032, buffers=11280384, cached=30798577664, shared=2190995456, slab=748134400) 
2022-04-11 18:12:14.783377: 
epoch:  46 
2022-04-11 18:14:05.948243: train loss : -0.2746 
2022-04-11 18:14:17.818054: validation loss: -0.2732 
2022-04-11 18:14:17.819968: Average global foreground Dice: [0.6829, 0.8728, 0.9219, 0.2426] 
2022-04-11 18:14:17.826038: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:14:18.714536: lr: 0.00915 
2022-04-11 18:14:18.715757: saving scheduled checkpoint file... 
2022-04-11 18:14:18.870497: saving checkpoint... 
2022-04-11 18:14:23.349865: done, saving took 4.63 seconds 
2022-04-11 18:14:23.373030: done 
2022-04-11 18:14:23.525639: saving checkpoint... 
2022-04-11 18:14:27.980921: done, saving took 4.61 seconds 
2022-04-11 18:14:28.005311: This epoch took 133.221111 s
 
2022-04-11 18:14:28.006602: svmem(total=99932160000, available=79777280000, percent=20.2, used=16152252416, free=52951678976, active=40319315968, inactive=4830777344, buffers=11329536, cached=30816899072, shared=2190729216, slab=748314624) 
2022-04-11 18:14:28.007437: 
epoch:  47 
2022-04-11 18:16:17.345339: train loss : -0.2575 
2022-04-11 18:16:30.395550: validation loss: -0.3261 
2022-04-11 18:16:30.406255: Average global foreground Dice: [0.7013, 0.8978, 0.9383, 0.2225] 
2022-04-11 18:16:30.407378: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:16:31.409569: lr: 0.009132 
2022-04-11 18:16:31.411433: saving scheduled checkpoint file... 
2022-04-11 18:16:31.534727: saving checkpoint... 
2022-04-11 18:16:35.962150: done, saving took 4.55 seconds 
2022-04-11 18:16:35.976064: done 
2022-04-11 18:16:36.023141: saving checkpoint... 
2022-04-11 18:16:40.550837: done, saving took 4.57 seconds 
2022-04-11 18:16:40.564545: This epoch took 132.556349 s
 
2022-04-11 18:16:40.565499: svmem(total=99932160000, available=79921778688, percent=20.0, used=16007303168, free=53078695936, active=40179265536, inactive=4848308224, buffers=11390976, cached=30834769920, shared=2191224832, slab=748281856) 
2022-04-11 18:16:40.566780: 
epoch:  48 
2022-04-11 18:18:29.454154: train loss : -0.3062 
2022-04-11 18:18:41.419073: validation loss: -0.3566 
2022-04-11 18:18:41.421111: Average global foreground Dice: [0.7144, 0.8948, 0.9473, 0.2476] 
2022-04-11 18:18:41.422583: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:18:42.228663: lr: 0.009114 
2022-04-11 18:18:42.230000: saving scheduled checkpoint file... 
2022-04-11 18:18:42.305202: saving checkpoint... 
2022-04-11 18:18:46.861636: done, saving took 4.63 seconds 
2022-04-11 18:18:46.878791: done 
2022-04-11 18:18:46.949307: saving checkpoint... 
2022-04-11 18:18:51.479402: done, saving took 4.60 seconds 
2022-04-11 18:18:51.511493: This epoch took 130.943942 s
 
2022-04-11 18:18:51.512991: svmem(total=99932160000, available=79760318464, percent=20.2, used=16168878080, free=52899635200, active=40343465984, inactive=4865409024, buffers=11440128, cached=30852206592, shared=2190917632, slab=750542848) 
2022-04-11 18:18:51.513631: 
epoch:  49 
2022-04-11 18:20:40.817448: train loss : -0.3209 
2022-04-11 18:20:52.698920: validation loss: -0.3479 
2022-04-11 18:20:52.701006: Average global foreground Dice: [0.6903, 0.8928, 0.9395, 0.2355] 
2022-04-11 18:20:52.702309: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:20:53.592628: lr: 0.009095 
2022-04-11 18:20:53.593660: saving scheduled checkpoint file... 
2022-04-11 18:20:53.674033: saving checkpoint... 
2022-04-11 18:20:58.181073: done, saving took 4.59 seconds 
2022-04-11 18:20:58.199318: done 
2022-04-11 18:20:58.283829: saving checkpoint... 
2022-04-11 18:21:02.787506: done, saving took 4.59 seconds 
2022-04-11 18:21:02.807529: This epoch took 131.293002 s
 
2022-04-11 18:21:02.808759: svmem(total=99932160000, available=79844966400, percent=20.1, used=16097918976, free=52976480256, active=40267784192, inactive=4859322368, buffers=11481088, cached=30846279680, shared=2177150976, slab=750616576) 
2022-04-11 18:21:02.809770: 
epoch:  50 
2022-04-11 18:22:50.440024: train loss : -0.3290 
2022-04-11 18:23:01.913938: validation loss: -0.3236 
2022-04-11 18:23:01.925612: Average global foreground Dice: [0.6875, 0.8994, 0.9241, 0.2267] 
2022-04-11 18:23:01.926497: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:23:02.796388: lr: 0.009077 
2022-04-11 18:23:02.797408: saving scheduled checkpoint file... 
2022-04-11 18:23:02.907719: saving checkpoint... 
2022-04-11 18:23:07.347059: done, saving took 4.55 seconds 
2022-04-11 18:23:07.367398: done 
2022-04-11 18:23:07.464020: saving checkpoint... 
2022-04-11 18:23:12.059358: done, saving took 4.69 seconds 
2022-04-11 18:23:12.126172: This epoch took 129.315506 s
 
2022-04-11 18:23:12.127367: svmem(total=99932160000, available=79888232448, percent=20.1, used=16041078784, free=53002260480, active=40216285184, inactive=4889018368, buffers=11538432, cached=30877282304, shared=2190729216, slab=749723648) 
2022-04-11 18:23:12.128171: 
epoch:  51 
2022-04-11 18:25:02.687912: train loss : -0.3634 
2022-04-11 18:25:14.880443: validation loss: -0.4012 
2022-04-11 18:25:14.885563: Average global foreground Dice: [0.7243, 0.9172, 0.9406, 0.2608] 
2022-04-11 18:25:14.892920: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:25:15.458791: lr: 0.009059 
2022-04-11 18:25:15.459826: saving scheduled checkpoint file... 
2022-04-11 18:25:15.569287: saving checkpoint... 
2022-04-11 18:25:20.064790: done, saving took 4.60 seconds 
2022-04-11 18:25:20.093359: done 
2022-04-11 18:25:20.202086: saving checkpoint... 
2022-04-11 18:25:24.739528: done, saving took 4.64 seconds 
2022-04-11 18:25:24.766609: This epoch took 132.637280 s
 
2022-04-11 18:25:24.768102: svmem(total=99932160000, available=79837290496, percent=20.1, used=16106405888, free=52933779456, active=40278056960, inactive=4892155904, buffers=11587584, cached=30880387072, shared=2176368640, slab=751030272) 
2022-04-11 18:25:24.769222: 
epoch:  52 
2022-04-11 18:27:13.211792: train loss : -0.3542 
2022-04-11 18:27:25.292295: validation loss: -0.4104 
2022-04-11 18:27:25.297908: Average global foreground Dice: [0.7333, 0.9097, 0.9548, 0.2591] 
2022-04-11 18:27:25.299000: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:27:25.906206: lr: 0.009041 
2022-04-11 18:27:25.907289: saving scheduled checkpoint file... 
2022-04-11 18:27:26.029352: saving checkpoint... 
2022-04-11 18:27:30.633933: done, saving took 4.73 seconds 
2022-04-11 18:27:30.662129: done 
2022-04-11 18:27:30.782799: saving checkpoint... 
2022-04-11 18:27:35.295721: done, saving took 4.63 seconds 
2022-04-11 18:27:35.345610: This epoch took 130.575421 s
 
2022-04-11 18:27:35.346681: svmem(total=99932160000, available=79699656704, percent=20.2, used=16229265408, free=52788436992, active=40400609280, inactive=4914495488, buffers=11620352, cached=30902837248, shared=2191114240, slab=750858240) 
2022-04-11 18:27:35.347330: 
epoch:  53 
2022-04-11 18:29:26.226279: train loss : -0.3174 
2022-04-11 18:29:37.915609: validation loss: -0.1902 
2022-04-11 18:29:37.921908: Average global foreground Dice: [0.6426, 0.8784, 0.8544, 0.2416] 
2022-04-11 18:29:37.923004: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:29:38.930038: lr: 0.009023 
2022-04-11 18:29:38.931223: saving scheduled checkpoint file... 
2022-04-11 18:29:39.037795: saving checkpoint... 
2022-04-11 18:29:43.804264: done, saving took 4.87 seconds 
2022-04-11 18:29:43.829938: done 
2022-04-11 18:29:43.830951: This epoch took 128.482938 s
 
2022-04-11 18:29:43.831762: svmem(total=99932160000, available=80003649536, percent=19.9, used=15915921408, free=53077491712, active=40059842560, inactive=4965748736, buffers=11653120, cached=30927093760, shared=2200420352, slab=751058944) 
2022-04-11 18:29:43.832510: 
epoch:  54 
2022-04-11 18:31:33.316841: train loss : -0.2950 
2022-04-11 18:31:45.612329: validation loss: -0.3619 
2022-04-11 18:31:45.614116: Average global foreground Dice: [0.7267, 0.8876, 0.9426, 0.2579] 
2022-04-11 18:31:45.615275: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:31:46.334899: lr: 0.009004 
2022-04-11 18:31:46.336149: saving scheduled checkpoint file... 
2022-04-11 18:31:46.398931: saving checkpoint... 
2022-04-11 18:31:50.786901: done, saving took 4.45 seconds 
2022-04-11 18:31:50.812150: done 
2022-04-11 18:31:50.861634: saving checkpoint... 
2022-04-11 18:31:55.294380: done, saving took 4.48 seconds 
2022-04-11 18:31:55.304062: This epoch took 131.470731 s
 
2022-04-11 18:31:55.305379: svmem(total=99932160000, available=79823888384, percent=20.1, used=16105828352, free=52880379904, active=40277413888, inactive=4945883136, buffers=11694080, cached=30934257664, shared=2190123008, slab=752246784) 
2022-04-11 18:31:55.306541: 
epoch:  55 
2022-04-11 18:33:43.780884: train loss : -0.2900 
2022-04-11 18:33:56.052660: validation loss: -0.2927 
2022-04-11 18:33:56.065925: Average global foreground Dice: [0.7304, 0.8702, 0.9102, 0.2564] 
2022-04-11 18:33:56.067854: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:33:57.007891: lr: 0.008986 
2022-04-11 18:33:57.008950: saving scheduled checkpoint file... 
2022-04-11 18:33:57.061221: saving checkpoint... 
2022-04-11 18:34:01.136494: done, saving took 4.13 seconds 
2022-04-11 18:34:01.177446: done 
2022-04-11 18:34:01.256102: saving checkpoint... 
2022-04-11 18:34:05.776237: done, saving took 4.60 seconds 
2022-04-11 18:34:05.794544: This epoch took 130.486822 s
 
2022-04-11 18:34:05.795600: svmem(total=99932160000, available=79917670400, percent=20.0, used=16011816960, free=52956692480, active=40185597952, inactive=4963401728, buffers=11726848, cached=30951923712, shared=2190512128, slab=750366720) 
2022-04-11 18:34:05.796435: 
epoch:  56 
2022-04-11 18:35:54.461278: train loss : -0.3402 
2022-04-11 18:36:06.935725: validation loss: -0.3250 
2022-04-11 18:36:06.937874: Average global foreground Dice: [0.7381, 0.8832, 0.9315, 0.2538] 
2022-04-11 18:36:06.939217: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:36:07.712737: lr: 0.008968 
2022-04-11 18:36:07.713959: saving scheduled checkpoint file... 
2022-04-11 18:36:07.804652: saving checkpoint... 
2022-04-11 18:36:12.191068: done, saving took 4.48 seconds 
2022-04-11 18:36:12.210554: done 
2022-04-11 18:36:12.301599: saving checkpoint... 
2022-04-11 18:36:16.784233: done, saving took 4.57 seconds 
2022-04-11 18:36:16.835099: This epoch took 131.037873 s
 
2022-04-11 18:36:16.836496: svmem(total=99932160000, available=79680905216, percent=20.3, used=16248107008, free=52711985152, active=40420089856, inactive=4971601920, buffers=11804672, cached=30960263168, shared=2190917632, slab=751546368) 
2022-04-11 18:36:16.837582: 
epoch:  57 
2022-04-11 18:38:05.917658: train loss : -0.3425 
2022-04-11 18:38:17.739550: validation loss: -0.3862 
2022-04-11 18:38:17.746461: Average global foreground Dice: [0.7115, 0.9068, 0.95, 0.2582] 
2022-04-11 18:38:17.747417: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:38:18.554743: lr: 0.00895 
2022-04-11 18:38:18.555904: saving scheduled checkpoint file... 
2022-04-11 18:38:18.662643: saving checkpoint... 
2022-04-11 18:38:23.134002: done, saving took 4.58 seconds 
2022-04-11 18:38:23.181788: done 
2022-04-11 18:38:23.292351: saving checkpoint... 
2022-04-11 18:38:27.714071: done, saving took 4.53 seconds 
2022-04-11 18:38:27.735439: This epoch took 130.897167 s
 
2022-04-11 18:38:27.736770: svmem(total=99932160000, available=79764774912, percent=20.2, used=16164392960, free=52778516480, active=40339099648, inactive=4988727296, buffers=11943936, cached=30977306624, shared=2190729216, slab=749408256) 
2022-04-11 18:38:27.737526: 
epoch:  58 
2022-04-11 18:40:16.087787: train loss : -0.3693 
2022-04-11 18:40:27.976927: validation loss: -0.3992 
2022-04-11 18:40:27.982015: Average global foreground Dice: [0.7171, 0.9036, 0.9543, 0.2823] 
2022-04-11 18:40:27.983122: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:40:28.798152: lr: 0.008931 
2022-04-11 18:40:28.799332: saving scheduled checkpoint file... 
2022-04-11 18:40:28.951311: saving checkpoint... 
2022-04-11 18:40:33.480633: done, saving took 4.68 seconds 
2022-04-11 18:40:33.506636: done 
2022-04-11 18:40:33.624773: saving checkpoint... 
2022-04-11 18:40:39.323181: done, saving took 5.82 seconds 
2022-04-11 18:40:39.349415: This epoch took 131.610953 s
 
2022-04-11 18:40:39.350735: svmem(total=99932160000, available=79809720320, percent=20.1, used=16119914496, free=52805849088, active=40295858176, inactive=5005606912, buffers=11984896, cached=30994411520, shared=2190200832, slab=750002176) 
2022-04-11 18:40:39.351525: 
epoch:  59 
2022-04-11 18:42:29.380038: train loss : -0.3400 
2022-04-11 18:42:42.404811: validation loss: -0.3447 
2022-04-11 18:42:42.508684: Average global foreground Dice: [0.697, 0.9003, 0.939, 0.2504] 
2022-04-11 18:42:42.509826: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:42:43.990760: lr: 0.008913 
2022-04-11 18:42:43.991994: saving scheduled checkpoint file... 
2022-04-11 18:42:44.081155: saving checkpoint... 
2022-04-11 18:42:48.407389: done, saving took 4.41 seconds 
2022-04-11 18:42:48.420973: done 
2022-04-11 18:42:48.470129: saving checkpoint... 
2022-04-11 18:42:52.877926: done, saving took 4.46 seconds 
2022-04-11 18:42:52.892277: This epoch took 133.540070 s
 
2022-04-11 18:42:52.893361: svmem(total=99932160000, available=79796461568, percent=20.1, used=16132804608, free=52784709632, active=40310804480, inactive=5013766144, buffers=12017664, cached=31002628096, shared=2190540800, slab=749842432) 
2022-04-11 18:42:52.894434: 
epoch:  60 
2022-04-11 18:44:41.677922: train loss : -0.3164 
2022-04-11 18:44:53.993494: validation loss: -0.3522 
2022-04-11 18:44:54.011237: Average global foreground Dice: [0.7444, 0.883, 0.9284, 0.2769] 
2022-04-11 18:44:54.012275: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:44:54.841855: lr: 0.008895 
2022-04-11 18:44:54.843293: saving scheduled checkpoint file... 
2022-04-11 18:44:54.920233: saving checkpoint... 
2022-04-11 18:44:59.232893: done, saving took 4.39 seconds 
2022-04-11 18:44:59.242110: done 
2022-04-11 18:44:59.288929: saving checkpoint... 
2022-04-11 18:45:03.656400: done, saving took 4.41 seconds 
2022-04-11 18:45:03.671207: This epoch took 130.776010 s
 
2022-04-11 18:45:03.672447: svmem(total=99932160000, available=79688646656, percent=20.3, used=16240168960, free=52660232192, active=40359469056, inactive=5089103872, buffers=12058624, cached=31019700224, shared=2190934016, slab=750411776) 
2022-04-11 18:45:03.673056: 
epoch:  61 
2022-04-11 18:46:52.608824: train loss : -0.3591 
2022-04-11 18:47:04.083358: validation loss: -0.4040 
2022-04-11 18:47:04.093926: Average global foreground Dice: [0.7703, 0.9092, 0.9362, 0.2883] 
2022-04-11 18:47:04.094977: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:47:04.984067: lr: 0.008877 
2022-04-11 18:47:04.985166: saving scheduled checkpoint file... 
2022-04-11 18:47:05.062300: saving checkpoint... 
2022-04-11 18:47:09.540766: done, saving took 4.56 seconds 
2022-04-11 18:47:09.554120: done 
2022-04-11 18:47:09.602984: saving checkpoint... 
2022-04-11 18:47:14.059774: done, saving took 4.50 seconds 
2022-04-11 18:47:14.073175: This epoch took 130.399393 s
 
2022-04-11 18:47:14.074266: svmem(total=99932160000, available=79711137792, percent=20.2, used=16217927680, free=52683632640, active=40329515008, inactive=5098708992, buffers=12107776, cached=31018491904, shared=2190729216, slab=749748224) 
2022-04-11 18:47:14.075135: 
epoch:  62 
2022-04-11 18:49:04.051090: train loss : -0.3631 
2022-04-11 18:49:15.411530: validation loss: -0.3372 
2022-04-11 18:49:15.425715: Average global foreground Dice: [0.72, 0.893, 0.9206, 0.2819] 
2022-04-11 18:49:15.426695: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:49:16.299844: lr: 0.008859 
2022-04-11 18:49:16.301068: saving scheduled checkpoint file... 
2022-04-11 18:49:16.403541: saving checkpoint... 
2022-04-11 18:49:20.868148: done, saving took 4.57 seconds 
2022-04-11 18:49:20.896018: done 
2022-04-11 18:49:20.961529: saving checkpoint... 
2022-04-11 18:49:25.423142: done, saving took 4.53 seconds 
2022-04-11 18:49:25.438756: This epoch took 131.362866 s
 
2022-04-11 18:49:25.439951: svmem(total=99932160000, available=79824089088, percent=20.1, used=16104894464, free=52783812608, active=40243359744, inactive=5080940544, buffers=12140544, cached=31031312384, shared=2190794752, slab=749948928) 
2022-04-11 18:49:25.440713: 
epoch:  63 
2022-04-11 18:51:12.533670: train loss : -0.2308 
2022-04-11 18:51:24.043441: validation loss: -0.3317 
2022-04-11 18:51:24.054056: Average global foreground Dice: [0.6861, 0.8889, 0.9406, 0.2739] 
2022-04-11 18:51:24.055115: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:51:24.776662: lr: 0.00884 
2022-04-11 18:51:24.777938: saving scheduled checkpoint file... 
2022-04-11 18:51:24.871399: saving checkpoint... 
2022-04-11 18:51:29.229772: done, saving took 4.45 seconds 
2022-04-11 18:51:29.293340: done 
2022-04-11 18:51:29.382376: saving checkpoint... 
2022-04-11 18:51:33.821740: done, saving took 4.53 seconds 
2022-04-11 18:51:33.838578: This epoch took 128.397100 s
 
2022-04-11 18:51:33.839656: svmem(total=99932160000, available=79772405760, percent=20.2, used=16156700672, free=52712022016, active=40326942720, inactive=5067395072, buffers=12181504, cached=31051255808, shared=2190729216, slab=750608384) 
2022-04-11 18:51:33.840366: 
epoch:  64 
2022-04-11 18:53:22.798474: train loss : -0.3436 
2022-04-11 18:53:34.436456: validation loss: -0.3325 
2022-04-11 18:53:34.439212: Average global foreground Dice: [0.72, 0.885, 0.9378, 0.28] 
2022-04-11 18:53:34.440347: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:53:35.314080: lr: 0.008822 
2022-04-11 18:53:35.319263: saving scheduled checkpoint file... 
2022-04-11 18:53:35.453386: saving checkpoint... 
2022-04-11 18:53:39.984853: done, saving took 4.66 seconds 
2022-04-11 18:53:40.013618: done 
2022-04-11 18:53:40.130311: saving checkpoint... 
2022-04-11 18:53:44.548821: done, saving took 4.53 seconds 
2022-04-11 18:53:44.566462: This epoch took 130.725362 s
 
2022-04-11 18:53:44.567624: svmem(total=99932160000, available=79781126144, percent=20.2, used=16147398656, free=52703285248, active=40319287296, inactive=5084684288, buffers=12214272, cached=31069261824, shared=2191224832, slab=750788608) 
2022-04-11 18:53:44.568347: 
epoch:  65 
2022-04-11 18:55:35.768610: train loss : -0.3584 
2022-04-11 18:55:48.186480: validation loss: -0.4180 
2022-04-11 18:55:48.201849: Average global foreground Dice: [0.7026, 0.922, 0.9502, 0.2818] 
2022-04-11 18:55:48.202832: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:55:49.148475: lr: 0.008804 
2022-04-11 18:55:49.149693: saving scheduled checkpoint file... 
2022-04-11 18:55:49.245367: saving checkpoint... 
2022-04-11 18:55:53.632167: done, saving took 4.48 seconds 
2022-04-11 18:55:53.647583: done 
2022-04-11 18:55:53.696235: saving checkpoint... 
2022-04-11 18:55:58.177927: done, saving took 4.53 seconds 
2022-04-11 18:55:58.203563: This epoch took 133.634508 s
 
2022-04-11 18:55:58.205003: svmem(total=99932160000, available=79984041984, percent=20.0, used=15945101312, free=52888768512, active=40119865344, inactive=5100847104, buffers=12259328, cached=31086030848, shared=2190729216, slab=750276608) 
2022-04-11 18:55:58.205828: 
epoch:  66 
2022-04-11 18:57:46.582463: train loss : -0.4153 
2022-04-11 18:57:58.975185: validation loss: -0.3720 
2022-04-11 18:57:58.978269: Average global foreground Dice: [0.7221, 0.9211, 0.9303, 0.2819] 
2022-04-11 18:57:58.979526: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 18:57:59.788316: lr: 0.008785 
2022-04-11 18:57:59.789510: saving scheduled checkpoint file... 
2022-04-11 18:57:59.849247: saving checkpoint... 
2022-04-11 18:58:04.234985: done, saving took 4.44 seconds 
2022-04-11 18:58:04.274234: done 
2022-04-11 18:58:04.349087: saving checkpoint... 
2022-04-11 18:58:08.813178: done, saving took 4.54 seconds 
2022-04-11 18:58:08.831114: This epoch took 130.624520 s
 
2022-04-11 18:58:08.832569: svmem(total=99932160000, available=79826038784, percent=20.1, used=16092590080, free=52722929664, active=40257286144, inactive=5119102976, buffers=12292096, cached=31104348160, shared=2201214976, slab=750428160) 
2022-04-11 18:58:08.833340: 
epoch:  67 
2022-04-11 18:59:59.077597: train loss : -0.4256 
2022-04-11 19:00:11.068739: validation loss: -0.4299 
2022-04-11 19:00:11.077947: Average global foreground Dice: [0.7414, 0.904, 0.9513, 0.3073] 
2022-04-11 19:00:11.079247: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:00:11.694825: lr: 0.008767 
2022-04-11 19:00:11.696208: saving scheduled checkpoint file... 
2022-04-11 19:00:11.797222: saving checkpoint... 
2022-04-11 19:00:16.204213: done, saving took 4.51 seconds 
2022-04-11 19:00:16.226290: done 
2022-04-11 19:00:16.326380: saving checkpoint... 
2022-04-11 19:00:20.785190: done, saving took 4.56 seconds 
2022-04-11 19:00:20.833007: This epoch took 131.998987 s
 
2022-04-11 19:00:20.834862: svmem(total=99932160000, available=79979835392, percent=20.0, used=15948763136, free=52858322944, active=40122253312, inactive=5123862528, buffers=12349440, cached=31112724480, shared=2191290368, slab=750133248) 
2022-04-11 19:00:20.835988: 
epoch:  68 
2022-04-11 19:02:10.853594: train loss : -0.4414 
2022-04-11 19:02:22.192281: validation loss: -0.4279 
2022-04-11 19:02:22.198862: Average global foreground Dice: [0.6979, 0.9188, 0.9499, 0.2918] 
2022-04-11 19:02:22.201996: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:02:23.222147: lr: 0.008749 
2022-04-11 19:02:23.223394: saving scheduled checkpoint file... 
2022-04-11 19:02:23.337244: saving checkpoint... 
2022-04-11 19:02:27.797032: done, saving took 4.57 seconds 
2022-04-11 19:02:27.860159: done 
2022-04-11 19:02:27.972201: saving checkpoint... 
2022-04-11 19:02:32.405503: done, saving took 4.54 seconds 
2022-04-11 19:02:32.447948: This epoch took 131.610577 s
 
2022-04-11 19:02:32.449674: svmem(total=99932160000, available=79756570624, percent=20.2, used=16172904448, free=52617535488, active=40344162304, inactive=5140148224, buffers=12382208, cached=31129337856, shared=2190233600, slab=752115712) 
2022-04-11 19:02:32.450578: 
epoch:  69 
2022-04-11 19:04:22.663799: train loss : -0.4378 
2022-04-11 19:04:34.711961: validation loss: -0.4407 
2022-04-11 19:04:34.723410: Average global foreground Dice: [0.7514, 0.9127, 0.9537, 0.2943] 
2022-04-11 19:04:34.724487: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:04:35.671135: lr: 0.008731 
2022-04-11 19:04:35.672269: saving scheduled checkpoint file... 
2022-04-11 19:04:35.812718: saving checkpoint... 
2022-04-11 19:04:40.223891: done, saving took 4.55 seconds 
2022-04-11 19:04:40.269451: done 
2022-04-11 19:04:40.409379: saving checkpoint... 
2022-04-11 19:04:44.820016: done, saving took 4.55 seconds 
2022-04-11 19:04:44.843276: This epoch took 132.391685 s
 
2022-04-11 19:04:44.844604: svmem(total=99932160000, available=79874854912, percent=20.1, used=16053727232, free=52728057856, active=40230313984, inactive=5148631040, buffers=12414976, cached=31137959936, shared=2191106048, slab=751529984) 
2022-04-11 19:04:44.845674: 
epoch:  70 
2022-04-11 19:06:35.627851: train loss : -0.3973 
2022-04-11 19:06:47.308837: validation loss: -0.4222 
2022-04-11 19:06:47.317483: Average global foreground Dice: [0.7098, 0.9227, 0.944, 0.2844] 
2022-04-11 19:06:47.318621: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:06:48.208149: lr: 0.008712 
2022-04-11 19:06:48.209240: saving scheduled checkpoint file... 
2022-04-11 19:06:48.328553: saving checkpoint... 
2022-04-11 19:06:52.713453: done, saving took 4.50 seconds 
2022-04-11 19:06:52.736564: done 
2022-04-11 19:06:52.866855: saving checkpoint... 
2022-04-11 19:06:57.251951: done, saving took 4.51 seconds 
2022-04-11 19:06:57.275425: This epoch took 132.428860 s
 
2022-04-11 19:06:57.276793: svmem(total=99932160000, available=79855013888, percent=20.1, used=16073908224, free=52690808832, active=40247767040, inactive=5165600768, buffers=12472320, cached=31154970624, shared=2190618624, slab=751423488) 
2022-04-11 19:06:57.277500: 
epoch:  71 
2022-04-11 19:08:48.559316: train loss : -0.4410 
2022-04-11 19:09:01.294741: validation loss: -0.4220 
2022-04-11 19:09:01.313634: Average global foreground Dice: [0.7434, 0.9188, 0.9478, 0.3244] 
2022-04-11 19:09:01.314880: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:09:02.253683: lr: 0.008694 
2022-04-11 19:09:02.255094: saving scheduled checkpoint file... 
2022-04-11 19:09:02.345276: saving checkpoint... 
2022-04-11 19:09:07.299199: done, saving took 5.04 seconds 
2022-04-11 19:09:07.308671: done 
2022-04-11 19:09:07.357350: saving checkpoint... 
2022-04-11 19:09:11.639551: done, saving took 4.33 seconds 
2022-04-11 19:09:11.676040: This epoch took 134.397553 s
 
2022-04-11 19:09:11.678647: svmem(total=99932160000, available=79912747008, percent=20.0, used=16005918720, free=52786167808, active=39918276608, inactive=5400137728, buffers=12525568, cached=31127547904, shared=2201026560, slab=751845376) 
2022-04-11 19:09:11.680933: 
epoch:  72 
2022-04-11 19:11:01.096949: train loss : -0.4401 
2022-04-11 19:11:12.918258: validation loss: -0.4453 
2022-04-11 19:11:12.921820: Average global foreground Dice: [0.729, 0.9222, 0.9466, 0.3031] 
2022-04-11 19:11:12.922894: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:11:13.847553: lr: 0.008676 
2022-04-11 19:11:13.848822: saving scheduled checkpoint file... 
2022-04-11 19:11:13.938092: saving checkpoint... 
2022-04-11 19:11:18.508667: done, saving took 4.66 seconds 
2022-04-11 19:11:18.548627: done 
2022-04-11 19:11:18.596068: saving checkpoint... 
2022-04-11 19:11:23.052591: done, saving took 4.50 seconds 
2022-04-11 19:11:23.067970: This epoch took 131.384430 s
 
2022-04-11 19:11:23.069133: svmem(total=99932160000, available=79925932032, percent=20.0, used=15992659968, free=52800811008, active=39942610944, inactive=5365710848, buffers=12566528, cached=31126122496, shared=2201137152, slab=751017984) 
2022-04-11 19:11:23.069895: 
epoch:  73 
2022-04-11 19:13:12.954562: train loss : -0.4134 
2022-04-11 19:13:24.362525: validation loss: -0.4424 
2022-04-11 19:13:24.403897: Average global foreground Dice: [0.6817, 0.9155, 0.9462, 0.3428] 
2022-04-11 19:13:24.440742: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:13:25.407308: lr: 0.008658 
2022-04-11 19:13:25.408533: saving scheduled checkpoint file... 
2022-04-11 19:13:25.495282: saving checkpoint... 
2022-04-11 19:13:30.046750: done, saving took 4.64 seconds 
2022-04-11 19:13:30.060266: done 
2022-04-11 19:13:30.125745: saving checkpoint... 
2022-04-11 19:13:34.910530: done, saving took 4.85 seconds 
2022-04-11 19:13:34.942544: This epoch took 131.871965 s
 
2022-04-11 19:13:34.943733: svmem(total=99932160000, available=79807049728, percent=20.1, used=16114909184, free=52659159040, active=40231981056, inactive=5217751040, buffers=12599296, cached=31145492480, shared=2197667840, slab=750616576) 
2022-04-11 19:13:34.944675: 
epoch:  74 
2022-04-11 19:15:23.699051: train loss : -0.3759 
2022-04-11 19:15:35.285985: validation loss: -0.4102 
2022-04-11 19:15:35.298681: Average global foreground Dice: [0.7184, 0.9266, 0.9504, 0.2844] 
2022-04-11 19:15:35.305368: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:15:36.210243: lr: 0.008639 
2022-04-11 19:15:36.211784: saving scheduled checkpoint file... 
2022-04-11 19:15:36.325903: saving checkpoint... 
2022-04-11 19:15:41.076797: done, saving took 4.86 seconds 
2022-04-11 19:15:41.095058: done 
2022-04-11 19:15:41.171995: saving checkpoint... 
2022-04-11 19:15:45.968144: done, saving took 4.87 seconds 
2022-04-11 19:15:45.985987: This epoch took 131.040313 s
 
2022-04-11 19:15:45.987493: svmem(total=99932160000, available=79883902976, percent=20.1, used=16044929024, free=52717436928, active=40206512128, inactive=5183258624, buffers=12648448, cached=31157145600, shared=2190729216, slab=750673920) 
2022-04-11 19:15:45.988255: 
epoch:  75 
2022-04-11 19:17:35.344790: train loss : -0.3316 
2022-04-11 19:17:47.744544: validation loss: -0.3481 
2022-04-11 19:17:47.761646: Average global foreground Dice: [0.6979, 0.8921, 0.9364, 0.3067] 
2022-04-11 19:17:47.763078: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:17:48.554794: lr: 0.008621 
2022-04-11 19:17:48.556699: saving scheduled checkpoint file... 
2022-04-11 19:17:48.668384: saving checkpoint... 
2022-04-11 19:17:53.131718: done, saving took 4.57 seconds 
2022-04-11 19:17:53.150771: done 
2022-04-11 19:17:53.151926: This epoch took 127.162924 s
 
2022-04-11 19:17:53.153116: svmem(total=99932160000, available=79704297472, percent=20.2, used=16232001536, free=52520505344, active=40407158784, inactive=5182107648, buffers=12685312, cached=31166967808, shared=2183262208, slab=750551040) 
2022-04-11 19:17:53.154172: 
epoch:  76 
2022-04-11 19:19:42.567049: train loss : -0.3856 
2022-04-11 19:19:54.882689: validation loss: -0.4529 
2022-04-11 19:19:54.884584: Average global foreground Dice: [0.7743, 0.9236, 0.959, 0.2838] 
2022-04-11 19:19:54.885959: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:19:55.686659: lr: 0.008603 
2022-04-11 19:19:55.687937: saving scheduled checkpoint file... 
2022-04-11 19:19:55.815388: saving checkpoint... 
2022-04-11 19:20:00.328297: done, saving took 4.64 seconds 
2022-04-11 19:20:00.351215: done 
2022-04-11 19:20:00.484995: saving checkpoint... 
2022-04-11 19:20:04.939795: done, saving took 4.59 seconds 
2022-04-11 19:20:04.963440: This epoch took 131.808570 s
 
2022-04-11 19:20:04.965207: svmem(total=99932160000, available=79881256960, percent=20.1, used=16040456192, free=52689592320, active=40215601152, inactive=5202231296, buffers=12726272, cached=31189385216, shared=2197667840, slab=751235072) 
2022-04-11 19:20:04.966195: 
epoch:  77 
2022-04-11 19:21:55.564168: train loss : -0.4108 
2022-04-11 19:22:07.984364: validation loss: -0.4309 
2022-04-11 19:22:07.992920: Average global foreground Dice: [0.7512, 0.9148, 0.9487, 0.3324] 
2022-04-11 19:22:07.995342: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:22:08.889096: lr: 0.008584 
2022-04-11 19:22:08.890610: saving scheduled checkpoint file... 
2022-04-11 19:22:08.979357: saving checkpoint... 
2022-04-11 19:22:13.271720: done, saving took 4.38 seconds 
2022-04-11 19:22:13.311867: done 
2022-04-11 19:22:13.360156: saving checkpoint... 
2022-04-11 19:22:17.644489: done, saving took 4.33 seconds 
2022-04-11 19:22:17.662059: This epoch took 132.695061 s
 
2022-04-11 19:22:17.663178: svmem(total=99932160000, available=79969144832, percent=20.0, used=15952535552, free=52757200896, active=40125267968, inactive=5219688448, buffers=12783616, cached=31209639936, shared=2197778432, slab=751476736) 
2022-04-11 19:22:17.664119: 
epoch:  78 
2022-04-11 19:24:06.489694: train loss : -0.4463 
2022-04-11 19:24:18.792219: validation loss: -0.4259 
2022-04-11 19:24:18.794644: Average global foreground Dice: [0.756, 0.9211, 0.9522, 0.3079] 
2022-04-11 19:24:18.795815: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:24:19.705538: lr: 0.008566 
2022-04-11 19:24:19.706643: saving scheduled checkpoint file... 
2022-04-11 19:24:19.788782: saving checkpoint... 
2022-04-11 19:24:24.091992: done, saving took 4.38 seconds 
2022-04-11 19:24:24.124182: done 
2022-04-11 19:24:24.172054: saving checkpoint... 
2022-04-11 19:24:28.490698: done, saving took 4.37 seconds 
2022-04-11 19:24:28.500968: This epoch took 130.836061 s
 
2022-04-11 19:24:28.502596: svmem(total=99932160000, available=79765438464, percent=20.2, used=16163446784, free=52536328192, active=40338694144, inactive=5229584384, buffers=12816384, cached=31219568640, shared=2190422016, slab=750559232) 
2022-04-11 19:24:28.503475: 
epoch:  79 
2022-04-11 19:26:17.090495: train loss : -0.4287 
2022-04-11 19:26:29.690371: validation loss: -0.4452 
2022-04-11 19:26:29.693670: Average global foreground Dice: [0.7483, 0.907, 0.9584, 0.3361] 
2022-04-11 19:26:29.695044: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:26:30.530449: lr: 0.008548 
2022-04-11 19:26:30.531762: saving scheduled checkpoint file... 
2022-04-11 19:26:30.588653: saving checkpoint... 
2022-04-11 19:26:35.086165: done, saving took 4.55 seconds 
2022-04-11 19:26:35.099966: done 
2022-04-11 19:26:35.151117: saving checkpoint... 
2022-04-11 19:26:39.806222: done, saving took 4.71 seconds 
2022-04-11 19:26:39.816671: This epoch took 131.311970 s
 
2022-04-11 19:26:39.818281: svmem(total=99932160000, available=79695081472, percent=20.3, used=16223395840, free=52457848832, active=40402538496, inactive=5248090112, buffers=12865536, cached=31238049792, shared=2200948736, slab=752193536) 
2022-04-11 19:26:39.819216: 
epoch:  80 
2022-04-11 19:28:27.636253: train loss : -0.4576 
2022-04-11 19:28:40.577753: validation loss: -0.4641 
2022-04-11 19:28:40.583248: Average global foreground Dice: [0.7857, 0.9139, 0.9596, 0.3075] 
2022-04-11 19:28:40.584465: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:28:41.389392: lr: 0.008529 
2022-04-11 19:28:41.391198: saving scheduled checkpoint file... 
2022-04-11 19:28:41.476132: saving checkpoint... 
2022-04-11 19:28:45.950191: done, saving took 4.56 seconds 
2022-04-11 19:28:45.966151: done 
2022-04-11 19:28:46.030195: saving checkpoint... 
2022-04-11 19:28:50.786183: done, saving took 4.82 seconds 
2022-04-11 19:28:50.850151: This epoch took 131.030026 s
 
2022-04-11 19:28:50.851656: svmem(total=99932160000, available=79808212992, percent=20.1, used=16120602624, free=52552548352, active=40294531072, inactive=5256171520, buffers=12898304, cached=31246110720, shared=2190729216, slab=752066560) 
2022-04-11 19:28:50.852681: 
epoch:  81 
2022-04-11 19:30:40.788688: train loss : -0.3600 
2022-04-11 19:30:52.069865: validation loss: -0.2520 
2022-04-11 19:30:52.071894: Average global foreground Dice: [0.6728, 0.8845, 0.8731, 0.2692] 
2022-04-11 19:30:52.072870: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:30:52.961364: lr: 0.008511 
2022-04-11 19:30:52.962746: saving scheduled checkpoint file... 
2022-04-11 19:30:53.072965: saving checkpoint... 
2022-04-11 19:30:57.791932: done, saving took 4.83 seconds 
2022-04-11 19:30:57.810042: done 
2022-04-11 19:30:57.811156: This epoch took 126.957357 s
 
2022-04-11 19:30:57.812045: svmem(total=99932160000, available=79627644928, percent=20.3, used=16301047808, free=52359671808, active=40475148288, inactive=5268357120, buffers=12939264, cached=31258501120, shared=2190651392, slab=751489024) 
2022-04-11 19:30:57.812900: 
epoch:  82 
2022-04-11 19:32:48.256374: train loss : -0.3436 
2022-04-11 19:33:00.158393: validation loss: -0.3643 
2022-04-11 19:33:00.166117: Average global foreground Dice: [0.7478, 0.8926, 0.9312, 0.31] 
2022-04-11 19:33:00.167212: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:33:00.982320: lr: 0.008493 
2022-04-11 19:33:00.983391: saving scheduled checkpoint file... 
2022-04-11 19:33:01.111238: saving checkpoint... 
2022-04-11 19:33:05.713918: done, saving took 4.73 seconds 
2022-04-11 19:33:05.751968: done 
2022-04-11 19:33:05.753059: This epoch took 127.939397 s
 
2022-04-11 19:33:05.754202: svmem(total=99932160000, available=79930032128, percent=20.0, used=15998578688, free=52644511744, active=40170983424, inactive=5285814272, buffers=12972032, cached=31276097536, shared=2190696448, slab=751210496) 
2022-04-11 19:33:05.754968: 
epoch:  83 
2022-04-11 19:34:54.288779: train loss : -0.3970 
2022-04-11 19:35:05.551187: validation loss: -0.4134 
2022-04-11 19:35:05.563288: Average global foreground Dice: [0.7306, 0.9044, 0.9556, 0.2945] 
2022-04-11 19:35:05.564813: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:35:06.729835: lr: 0.008474 
2022-04-11 19:35:06.731178: saving scheduled checkpoint file... 
2022-04-11 19:35:06.843199: saving checkpoint... 
2022-04-11 19:35:11.802370: done, saving took 5.07 seconds 
2022-04-11 19:35:11.818347: done 
2022-04-11 19:35:11.819731: This epoch took 126.063767 s
 
2022-04-11 19:35:11.820912: svmem(total=99932160000, available=79896788992, percent=20.0, used=16031600640, free=52603457536, active=40208752640, inactive=5293899776, buffers=13029376, cached=31284072448, shared=2191028224, slab=751943680) 
2022-04-11 19:35:11.821617: 
epoch:  84 
2022-04-11 19:37:01.572476: train loss : -0.4251 
2022-04-11 19:37:13.437613: validation loss: -0.4857 
2022-04-11 19:37:13.439928: Average global foreground Dice: [0.763, 0.9259, 0.9628, 0.3428] 
2022-04-11 19:37:13.441168: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:37:14.281657: lr: 0.008456 
2022-04-11 19:37:14.282959: saving scheduled checkpoint file... 
2022-04-11 19:37:14.391652: saving checkpoint... 
2022-04-11 19:37:18.825608: done, saving took 4.54 seconds 
2022-04-11 19:37:18.843890: done 
2022-04-11 19:37:18.844848: This epoch took 127.022279 s
 
2022-04-11 19:37:18.845818: svmem(total=99932160000, available=79833350144, percent=20.1, used=16084828160, free=52522532864, active=40257724416, inactive=5321392128, buffers=13082624, cached=31311716352, shared=2201137152, slab=752189440) 
2022-04-11 19:37:18.847115: 
epoch:  85 
2022-04-11 19:39:08.158489: train loss : -0.4347 
2022-04-11 19:39:20.515566: validation loss: -0.4445 
2022-04-11 19:39:20.538625: Average global foreground Dice: [0.7624, 0.9185, 0.944, 0.3333] 
2022-04-11 19:39:20.540266: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:39:21.429280: lr: 0.008438 
2022-04-11 19:39:21.447976: saving scheduled checkpoint file... 
2022-04-11 19:39:21.579726: saving checkpoint... 
2022-04-11 19:39:26.084039: done, saving took 4.60 seconds 
2022-04-11 19:39:26.129845: done 
2022-04-11 19:39:26.222409: saving checkpoint... 
2022-04-11 19:39:30.791637: done, saving took 4.66 seconds 
2022-04-11 19:39:30.810283: This epoch took 131.962336 s
 
2022-04-11 19:39:30.811574: svmem(total=99932160000, available=79953838080, percent=20.0, used=15974580224, free=52625608704, active=40153276416, inactive=5328470016, buffers=13115392, cached=31318855680, shared=2190839808, slab=751890432) 
2022-04-11 19:39:30.812686: 
epoch:  86 
2022-04-11 19:41:18.961611: train loss : -0.4630 
2022-04-11 19:41:30.463061: validation loss: -0.4516 
2022-04-11 19:41:30.465165: Average global foreground Dice: [0.7784, 0.9233, 0.9557, 0.298] 
2022-04-11 19:41:30.466349: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:41:31.029016: lr: 0.008419 
2022-04-11 19:41:31.030437: saving scheduled checkpoint file... 
2022-04-11 19:41:31.134289: saving checkpoint... 
2022-04-11 19:41:35.647794: done, saving took 4.62 seconds 
2022-04-11 19:41:35.670600: done 
2022-04-11 19:41:35.783644: saving checkpoint... 
2022-04-11 19:41:40.323191: done, saving took 4.65 seconds 
2022-04-11 19:41:40.344730: This epoch took 129.531061 s
 
2022-04-11 19:41:40.346515: svmem(total=99932160000, available=79648481280, percent=20.3, used=16282984448, free=52312444928, active=40460005376, inactive=5332942848, buffers=13156352, cached=31323574272, shared=2187681792, slab=753393664) 
2022-04-11 19:41:40.347778: 
epoch:  87 
2022-04-11 19:43:28.919549: train loss : -0.4594 
2022-04-11 19:43:41.180722: validation loss: -0.4972 
2022-04-11 19:43:41.192438: Average global foreground Dice: [0.7449, 0.9348, 0.9629, 0.3372] 
2022-04-11 19:43:41.193697: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:43:42.029339: lr: 0.008401 
2022-04-11 19:43:42.030731: saving scheduled checkpoint file... 
2022-04-11 19:43:42.179013: saving checkpoint... 
2022-04-11 19:43:46.696348: done, saving took 4.66 seconds 
2022-04-11 19:43:46.723328: done 
2022-04-11 19:43:46.844866: saving checkpoint... 
2022-04-11 19:43:52.321544: done, saving took 5.60 seconds 
2022-04-11 19:43:52.343965: This epoch took 131.995378 s
 
2022-04-11 19:43:52.345257: svmem(total=99932160000, available=79927582720, percent=20.0, used=16002637824, free=52573986816, active=40182067200, inactive=5351669760, buffers=13189120, cached=31342346240, shared=2188935168, slab=752611328) 
2022-04-11 19:43:52.346204: 
epoch:  88 
2022-04-11 19:45:42.959710: train loss : -0.4510 
2022-04-11 19:45:55.740217: validation loss: -0.4075 
2022-04-11 19:45:55.754058: Average global foreground Dice: [0.734, 0.9172, 0.944, 0.316] 
2022-04-11 19:45:55.755268: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:45:56.549409: lr: 0.008383 
2022-04-11 19:45:56.550666: saving scheduled checkpoint file... 
2022-04-11 19:45:56.679492: saving checkpoint... 
2022-04-11 19:46:01.122812: done, saving took 4.57 seconds 
2022-04-11 19:46:01.151486: done 
2022-04-11 19:46:01.291653: saving checkpoint... 
2022-04-11 19:46:05.875231: done, saving took 4.72 seconds 
2022-04-11 19:46:05.895168: This epoch took 133.548157 s
 
2022-04-11 19:46:05.896800: svmem(total=99932160000, available=79727710208, percent=20.2, used=16200790016, free=52357459968, active=40372719616, inactive=5370839040, buffers=13234176, cached=31360675840, shared=2190618624, slab=751812608) 
2022-04-11 19:46:05.897758: 
epoch:  89 
2022-04-11 19:47:57.516755: train loss : -0.4166 
2022-04-11 19:48:10.345619: validation loss: -0.4315 
2022-04-11 19:48:10.347998: Average global foreground Dice: [0.739, 0.9237, 0.9493, 0.2896] 
2022-04-11 19:48:10.349365: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:48:11.106747: lr: 0.008364 
2022-04-11 19:48:11.108303: saving scheduled checkpoint file... 
2022-04-11 19:48:11.206079: saving checkpoint... 
2022-04-11 19:48:15.833690: done, saving took 4.72 seconds 
2022-04-11 19:48:15.852101: done 
2022-04-11 19:48:15.853369: This epoch took 129.954684 s
 
2022-04-11 19:48:15.854333: svmem(total=99932160000, available=79789084672, percent=20.2, used=16139104256, free=52411113472, active=40313311232, inactive=5377699840, buffers=13283328, cached=31368658944, shared=2190696448, slab=752959488) 
2022-04-11 19:48:15.855162: 
epoch:  90 
2022-04-11 19:50:04.594330: train loss : -0.4305 
2022-04-11 19:50:16.908813: validation loss: -0.4126 
2022-04-11 19:50:16.910526: Average global foreground Dice: [0.7651, 0.9217, 0.9146, 0.3204] 
2022-04-11 19:50:16.911248: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:50:17.786272: lr: 0.008346 
2022-04-11 19:50:17.787788: saving scheduled checkpoint file... 
2022-04-11 19:50:17.863137: saving checkpoint... 
2022-04-11 19:50:22.912194: done, saving took 5.12 seconds 
2022-04-11 19:50:22.926160: done 
2022-04-11 19:50:22.988713: saving checkpoint... 
2022-04-11 19:50:27.802487: done, saving took 4.88 seconds 
2022-04-11 19:50:27.828764: This epoch took 131.972866 s
 
2022-04-11 19:50:27.830261: svmem(total=99932160000, available=79848239104, percent=20.1, used=16074907648, free=52452782080, active=40250990592, inactive=5400023040, buffers=13324288, cached=31391145984, shared=2195742720, slab=752361472) 
2022-04-11 19:50:27.831079: 
epoch:  91 
2022-04-11 19:52:16.189468: train loss : -0.4784 
2022-04-11 19:52:28.774900: validation loss: -0.5245 
2022-04-11 19:52:28.838062: Average global foreground Dice: [0.7794, 0.9413, 0.9693, 0.3531] 
2022-04-11 19:52:28.839561: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:52:29.564855: lr: 0.008328 
2022-04-11 19:52:29.566262: saving scheduled checkpoint file... 
2022-04-11 19:52:29.641610: saving checkpoint... 
2022-04-11 19:52:34.340420: done, saving took 4.77 seconds 
2022-04-11 19:52:34.358958: done 
2022-04-11 19:52:34.443138: saving checkpoint... 
2022-04-11 19:52:38.917607: done, saving took 4.56 seconds 
2022-04-11 19:52:38.964340: This epoch took 131.131869 s
 
2022-04-11 19:52:38.965850: svmem(total=99932160000, available=79805902848, percent=20.1, used=16122281984, free=52392869888, active=40300290048, inactive=5412536320, buffers=13357056, cached=31403651072, shared=2190729216, slab=751976448) 
2022-04-11 19:52:38.966961: 
epoch:  92 
2022-04-11 19:54:26.707017: train loss : -0.4776 
2022-04-11 19:54:39.632468: validation loss: -0.5032 
2022-04-11 19:54:39.634376: Average global foreground Dice: [0.7812, 0.9363, 0.9608, 0.3512] 
2022-04-11 19:54:39.635463: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:54:40.398957: lr: 0.008309 
2022-04-11 19:54:40.400272: saving scheduled checkpoint file... 
2022-04-11 19:54:40.507959: saving checkpoint... 
2022-04-11 19:54:45.036362: done, saving took 4.64 seconds 
2022-04-11 19:54:45.052702: done 
2022-04-11 19:54:45.147595: saving checkpoint... 
2022-04-11 19:54:49.794691: done, saving took 4.74 seconds 
2022-04-11 19:54:49.814700: This epoch took 130.847052 s
 
2022-04-11 19:54:49.816205: svmem(total=99932160000, available=79822712832, percent=20.1, used=16105484288, free=52392300544, active=40284106752, inactive=5429911552, buffers=13389824, cached=31420985344, shared=2190729216, slab=751575040) 
2022-04-11 19:54:49.816959: 
epoch:  93 
2022-04-11 19:56:39.553763: train loss : -0.4750 
2022-04-11 19:56:51.748650: validation loss: -0.5076 
2022-04-11 19:56:51.758039: Average global foreground Dice: [0.7844, 0.9292, 0.9655, 0.3609] 
2022-04-11 19:56:51.758816: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:56:52.281888: lr: 0.008291 
2022-04-11 19:56:52.282988: saving scheduled checkpoint file... 
2022-04-11 19:56:52.396893: saving checkpoint... 
2022-04-11 19:56:56.869599: done, saving took 4.59 seconds 
2022-04-11 19:56:56.916936: done 
2022-04-11 19:56:57.028143: saving checkpoint... 
2022-04-11 19:57:01.627964: done, saving took 4.71 seconds 
2022-04-11 19:57:01.649259: This epoch took 131.831567 s
 
2022-04-11 19:57:01.650545: svmem(total=99932160000, available=79737712640, percent=20.2, used=16190476288, free=52299489280, active=40368902144, inactive=5437632512, buffers=13438976, cached=31428755456, shared=2190729216, slab=752689152) 
2022-04-11 19:57:01.651245: 
epoch:  94 
2022-04-11 19:58:50.244973: train loss : -0.4508 
2022-04-11 19:59:01.650753: validation loss: -0.4956 
2022-04-11 19:59:01.660589: Average global foreground Dice: [0.7272, 0.9348, 0.9631, 0.3328] 
2022-04-11 19:59:01.662066: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 19:59:02.493473: lr: 0.008272 
2022-04-11 19:59:02.494920: saving scheduled checkpoint file... 
2022-04-11 19:59:02.638256: saving checkpoint... 
2022-04-11 19:59:07.159783: done, saving took 4.66 seconds 
2022-04-11 19:59:07.183091: done 
2022-04-11 19:59:07.307317: saving checkpoint... 
2022-04-11 19:59:11.732294: done, saving took 4.55 seconds 
2022-04-11 19:59:11.755785: This epoch took 130.103949 s
 
2022-04-11 19:59:11.757192: svmem(total=99932160000, available=79835758592, percent=20.1, used=16104472576, free=52380045312, active=40280543232, inactive=5441130496, buffers=13488128, cached=31434153984, shared=2178744320, slab=752828416) 
2022-04-11 19:59:11.757947: 
epoch:  95 
2022-04-11 20:01:02.174555: train loss : -0.4574 
2022-04-11 20:01:15.418384: validation loss: -0.5022 
2022-04-11 20:01:15.425785: Average global foreground Dice: [0.7776, 0.9339, 0.9655, 0.345] 
2022-04-11 20:01:15.429042: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:01:16.243087: lr: 0.008254 
2022-04-11 20:01:16.244700: saving scheduled checkpoint file... 
2022-04-11 20:01:16.339079: saving checkpoint... 
2022-04-11 20:01:20.840303: done, saving took 4.59 seconds 
2022-04-11 20:01:20.856107: done 
2022-04-11 20:01:20.923468: saving checkpoint... 
2022-04-11 20:01:25.672117: done, saving took 4.81 seconds 
2022-04-11 20:01:25.690191: This epoch took 133.931485 s
 
2022-04-11 20:01:25.691272: svmem(total=99932160000, available=79966879744, percent=20.0, used=15961006080, free=52493701120, active=40134991872, inactive=5472661504, buffers=13529088, cached=31463923712, shared=2190995456, slab=753504256) 
2022-04-11 20:01:25.692038: 
epoch:  96 
2022-04-11 20:03:15.487056: train loss : -0.4875 
2022-04-11 20:03:27.024831: validation loss: -0.5252 
2022-04-11 20:03:27.033628: Average global foreground Dice: [0.7514, 0.9432, 0.9591, 0.3573] 
2022-04-11 20:03:27.035560: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:03:27.930416: lr: 0.008236 
2022-04-11 20:03:27.931415: saving scheduled checkpoint file... 
2022-04-11 20:03:28.039379: saving checkpoint... 
2022-04-11 20:03:32.513817: done, saving took 4.58 seconds 
2022-04-11 20:03:32.532259: done 
2022-04-11 20:03:32.612505: saving checkpoint... 
2022-04-11 20:03:37.141274: done, saving took 4.61 seconds 
2022-04-11 20:03:37.158606: This epoch took 131.465746 s
 
2022-04-11 20:03:37.160041: svmem(total=99932160000, available=80125939712, percent=19.8, used=15851778048, free=52644978688, active=40033288192, inactive=5430616064, buffers=13561856, cached=31421841408, shared=2141171712, slab=752824320) 
2022-04-11 20:03:37.160873: 
epoch:  97 
2022-04-11 20:05:28.225470: train loss : -0.4898 
2022-04-11 20:05:39.723708: validation loss: -0.5272 
2022-04-11 20:05:39.730157: Average global foreground Dice: [0.8045, 0.9319, 0.9657, 0.3569] 
2022-04-11 20:05:39.731495: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:05:40.414049: lr: 0.008217 
2022-04-11 20:05:40.415428: saving scheduled checkpoint file... 
2022-04-11 20:05:40.505463: saving checkpoint... 
2022-04-11 20:05:45.106361: done, saving took 4.69 seconds 
2022-04-11 20:05:45.177929: done 
2022-04-11 20:05:45.272574: saving checkpoint... 
2022-04-11 20:05:49.821238: done, saving took 4.64 seconds 
2022-04-11 20:05:49.840283: This epoch took 132.678647 s
 
2022-04-11 20:05:49.841653: svmem(total=99932160000, available=80048914432, percent=19.9, used=15893663744, free=52556107776, active=40024416256, inactive=5522948096, buffers=13606912, cached=31468781568, shared=2176585728, slab=752136192) 
2022-04-11 20:05:49.842587: 
epoch:  98 
2022-04-11 20:07:37.594597: train loss : -0.4731 
2022-04-11 20:07:48.359886: validation loss: -0.4686 
2022-04-11 20:07:48.366635: Average global foreground Dice: [0.7554, 0.9214, 0.9626, 0.3257] 
2022-04-11 20:07:48.368142: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:07:49.138749: lr: 0.008199 
2022-04-11 20:07:49.140047: saving scheduled checkpoint file... 
2022-04-11 20:07:49.249209: saving checkpoint... 
2022-04-11 20:07:53.856808: done, saving took 4.72 seconds 
2022-04-11 20:07:53.896279: done 
2022-04-11 20:07:53.897628: This epoch took 124.054301 s
 
2022-04-11 20:07:53.899038: svmem(total=99932160000, available=80154509312, percent=19.8, used=15810015232, free=52645310464, active=39986184192, inactive=5474725888, buffers=13651968, cached=31463182336, shared=2154283008, slab=752685056) 
2022-04-11 20:07:53.899865: 
epoch:  99 
2022-04-11 20:09:45.380586: train loss : -0.4613 
2022-04-11 20:09:57.630492: validation loss: -0.4375 
2022-04-11 20:09:57.645493: Average global foreground Dice: [0.7303, 0.9199, 0.9543, 0.3381] 
2022-04-11 20:09:57.646502: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:09:58.430693: lr: 0.008181 
2022-04-11 20:09:58.432065: saving scheduled checkpoint file... 
2022-04-11 20:09:58.549209: saving checkpoint... 
2022-04-11 20:10:03.122119: done, saving took 4.69 seconds 
2022-04-11 20:10:03.150090: done 
2022-04-11 20:10:03.151522: This epoch took 129.250812 s
 
2022-04-11 20:10:03.152513: svmem(total=99932160000, available=80245936128, percent=19.7, used=15804383232, free=52728848384, active=39986487296, inactive=5394046976, buffers=13692928, cached=31385235456, shared=2068455424, slab=751624192) 
2022-04-11 20:10:03.153251: 
epoch:  100 
2022-04-11 20:11:54.156168: train loss : -0.4705 
2022-04-11 20:12:06.448020: validation loss: -0.4594 
2022-04-11 20:12:06.455801: Average global foreground Dice: [0.7771, 0.9209, 0.9547, 0.3235] 
2022-04-11 20:12:06.457087: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:12:07.154581: lr: 0.008162 
2022-04-11 20:12:07.156078: saving scheduled checkpoint file... 
2022-04-11 20:12:07.266361: saving checkpoint... 
2022-04-11 20:12:11.719616: done, saving took 4.56 seconds 
2022-04-11 20:12:11.741328: done 
2022-04-11 20:12:11.742734: This epoch took 128.588322 s
 
2022-04-11 20:12:11.743738: svmem(total=99932160000, available=79956234240, percent=20.0, used=15968346112, free=52444536832, active=40137515008, inactive=5526097920, buffers=13742080, cached=31505534976, shared=2194165760, slab=751697920) 
2022-04-11 20:12:11.744879: 
epoch:  101 
2022-04-11 20:14:02.660084: train loss : -0.4506 
2022-04-11 20:14:14.847087: validation loss: -0.3601 
2022-04-11 20:14:14.854236: Average global foreground Dice: [0.7607, 0.8634, 0.8773, 0.3137] 
2022-04-11 20:14:14.856170: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:14:15.500042: lr: 0.008144 
2022-04-11 20:14:15.501829: saving scheduled checkpoint file... 
2022-04-11 20:14:15.594999: saving checkpoint... 
2022-04-11 20:14:19.939562: done, saving took 4.44 seconds 
2022-04-11 20:14:19.953116: done 
2022-04-11 20:14:19.954345: This epoch took 128.208613 s
 
2022-04-11 20:14:19.955836: svmem(total=99932160000, available=79864406016, percent=20.1, used=16063729664, free=52335443968, active=40243974144, inactive=5527633920, buffers=13774848, cached=31519211520, shared=2190618624, slab=752091136) 
2022-04-11 20:14:19.956620: 
epoch:  102 
2022-04-11 20:16:08.550638: train loss : -0.3958 
2022-04-11 20:16:20.442806: validation loss: -0.3533 
2022-04-11 20:16:20.457721: Average global foreground Dice: [0.7471, 0.8819, 0.9189, 0.3135] 
2022-04-11 20:16:20.458638: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:16:21.329217: lr: 0.008125 
2022-04-11 20:16:21.335183: saving scheduled checkpoint file... 
2022-04-11 20:16:21.384580: saving checkpoint... 
2022-04-11 20:16:25.860296: done, saving took 4.52 seconds 
2022-04-11 20:16:25.875388: done 
2022-04-11 20:16:25.876389: This epoch took 125.918662 s
 
2022-04-11 20:16:25.877307: svmem(total=99932160000, available=79769796608, percent=20.2, used=16157691904, free=52170706944, active=40342032384, inactive=5592199168, buffers=15802368, cached=31587958784, shared=2190729216, slab=754061312) 
2022-04-11 20:16:25.877985: 
epoch:  103 
2022-04-11 20:18:13.529655: train loss : -0.4019 
2022-04-11 20:18:25.134830: validation loss: -0.4870 
2022-04-11 20:18:25.142641: Average global foreground Dice: [0.7746, 0.9293, 0.964, 0.3556] 
2022-04-11 20:18:25.143852: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:18:26.164173: lr: 0.008107 
2022-04-11 20:18:26.165416: saving scheduled checkpoint file... 
2022-04-11 20:18:26.253996: saving checkpoint... 
2022-04-11 20:18:31.234210: done, saving took 5.07 seconds 
2022-04-11 20:18:31.248818: done 
2022-04-11 20:18:31.249894: This epoch took 125.371134 s
 
2022-04-11 20:18:31.250807: svmem(total=99932160000, available=79834193920, percent=20.1, used=16093274112, free=52228505600, active=40280256512, inactive=5598785536, buffers=15847424, cached=31594532864, shared=2190729216, slab=753745920) 
2022-04-11 20:18:31.251544: 
epoch:  104 
2022-04-11 20:20:20.509098: train loss : -0.4567 
2022-04-11 20:20:32.063931: validation loss: -0.4947 
2022-04-11 20:20:32.081561: Average global foreground Dice: [0.786, 0.9277, 0.9499, 0.345] 
2022-04-11 20:20:32.082766: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:20:32.842141: lr: 0.008088 
2022-04-11 20:20:32.843688: saving scheduled checkpoint file... 
2022-04-11 20:20:32.897511: saving checkpoint... 
2022-04-11 20:20:37.748630: done, saving took 4.90 seconds 
2022-04-11 20:20:37.833988: done 
2022-04-11 20:20:37.835390: This epoch took 126.583206 s
 
2022-04-11 20:20:37.836660: svmem(total=99932160000, available=79906979840, percent=20.0, used=16020430848, free=52317802496, active=39958355968, inactive=5829087232, buffers=15888384, cached=31578038272, shared=2190729216, slab=753389568) 
2022-04-11 20:20:37.837782: 
epoch:  105 
2022-04-11 20:22:26.949100: train loss : -0.4317 
2022-04-11 20:22:38.736835: validation loss: -0.4201 
2022-04-11 20:22:38.745961: Average global foreground Dice: [0.6999, 0.9176, 0.9375, 0.2932] 
2022-04-11 20:22:38.747175: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:22:39.566898: lr: 0.00807 
2022-04-11 20:22:39.567961: saving scheduled checkpoint file... 
2022-04-11 20:22:39.650843: saving checkpoint... 
2022-04-11 20:22:44.163937: done, saving took 4.59 seconds 
2022-04-11 20:22:44.181406: done 
2022-04-11 20:22:44.182283: This epoch took 126.343640 s
 
2022-04-11 20:22:44.183224: svmem(total=99932160000, available=79840514048, percent=20.1, used=16086700032, free=52233854976, active=40157925376, inactive=5710143488, buffers=15929344, cached=31595675648, shared=2190917632, slab=753999872) 
2022-04-11 20:22:44.184354: 
epoch:  106 
2022-04-11 20:24:33.878404: train loss : -0.4765 
2022-04-11 20:24:44.655604: validation loss: -0.4711 
2022-04-11 20:24:44.663757: Average global foreground Dice: [0.7703, 0.9288, 0.9508, 0.3409] 
2022-04-11 20:24:44.665174: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:24:45.569830: lr: 0.008052 
2022-04-11 20:24:45.571244: saving scheduled checkpoint file... 
2022-04-11 20:24:45.691104: saving checkpoint... 
2022-04-11 20:24:50.027073: done, saving took 4.45 seconds 
2022-04-11 20:24:50.052392: done 
2022-04-11 20:24:50.053779: This epoch took 125.868697 s
 
2022-04-11 20:24:50.054680: svmem(total=99932160000, available=79856508928, percent=20.1, used=16070832128, free=52243091456, active=40193339392, inactive=5667872768, buffers=15962112, cached=31602274304, shared=2190729216, slab=752439296) 
2022-04-11 20:24:50.055377: 
epoch:  107 
2022-04-11 20:26:38.681958: train loss : -0.4402 
2022-04-11 20:26:49.402807: validation loss: -0.4886 
2022-04-11 20:26:49.406861: Average global foreground Dice: [0.8019, 0.93, 0.9614, 0.3197] 
2022-04-11 20:26:49.407951: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:26:50.538100: lr: 0.008033 
2022-04-11 20:26:50.539226: saving scheduled checkpoint file... 
2022-04-11 20:26:50.629257: saving checkpoint... 
2022-04-11 20:26:55.082357: done, saving took 4.54 seconds 
2022-04-11 20:26:55.125304: done 
2022-04-11 20:26:55.126331: This epoch took 125.070278 s
 
2022-04-11 20:26:55.127635: svmem(total=99932160000, available=80044990464, percent=19.9, used=15886168064, free=52414222336, active=40037412864, inactive=5652672512, buffers=16027648, cached=31615741952, shared=2186952704, slab=753766400) 
2022-04-11 20:26:55.128677: 
epoch:  108 
2022-04-11 20:28:43.709960: train loss : -0.4647 
2022-04-11 20:28:55.863921: validation loss: -0.5043 
2022-04-11 20:28:55.887970: Average global foreground Dice: [0.7903, 0.9336, 0.9573, 0.3287] 
2022-04-11 20:28:55.889798: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:28:56.673846: lr: 0.008015 
2022-04-11 20:28:56.675273: saving scheduled checkpoint file... 
2022-04-11 20:28:56.771098: saving checkpoint... 
2022-04-11 20:29:01.289440: done, saving took 4.61 seconds 
2022-04-11 20:29:01.310188: done 
2022-04-11 20:29:01.311425: This epoch took 126.181885 s
 
2022-04-11 20:29:01.312328: svmem(total=99932160000, available=79704444928, percent=20.2, used=16222769152, free=52090425344, active=40108228608, inactive=5906620416, buffers=15835136, cached=31603130368, shared=2190729216, slab=753795072) 
2022-04-11 20:29:01.313186: 
epoch:  109 
2022-04-11 20:30:49.609626: train loss : -0.4650 
2022-04-11 20:31:01.605074: validation loss: -0.5021 
2022-04-11 20:31:01.610671: Average global foreground Dice: [0.7691, 0.9208, 0.9687, 0.3575] 
2022-04-11 20:31:01.611875: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:31:02.523199: lr: 0.007996 
2022-04-11 20:31:02.524850: saving scheduled checkpoint file... 
2022-04-11 20:31:02.626150: saving checkpoint... 
2022-04-11 20:31:10.841174: done, saving took 8.31 seconds 
2022-04-11 20:31:10.866663: done 
2022-04-11 20:31:10.867922: This epoch took 129.553728 s
 
2022-04-11 20:31:10.869049: svmem(total=99932160000, available=79757119488, percent=20.2, used=16170340352, free=52115156992, active=40249466880, inactive=5736075264, buffers=15900672, cached=31630761984, shared=2190585856, slab=753856512) 
2022-04-11 20:31:10.869734: 
epoch:  110 
2022-04-11 20:33:00.737396: train loss : -0.5008 
2022-04-11 20:33:12.691877: validation loss: -0.5243 
2022-04-11 20:33:12.705660: Average global foreground Dice: [0.8115, 0.9304, 0.9667, 0.3514] 
2022-04-11 20:33:12.707099: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:33:13.574967: lr: 0.007978 
2022-04-11 20:33:13.575913: saving scheduled checkpoint file... 
2022-04-11 20:33:13.703668: saving checkpoint... 
2022-04-11 20:33:18.453929: done, saving took 4.88 seconds 
2022-04-11 20:33:18.472259: done 
2022-04-11 20:33:18.587283: saving checkpoint... 
2022-04-11 20:33:23.016218: done, saving took 4.54 seconds 
2022-04-11 20:33:23.035502: This epoch took 132.164946 s
 
2022-04-11 20:33:23.036991: svmem(total=99932160000, available=79855865856, percent=20.1, used=16071294976, free=52141387776, active=40270753792, inactive=5690290176, buffers=15949824, cached=31703527424, shared=2190807040, slab=754688000) 
2022-04-11 20:33:23.038115: 
epoch:  111 
2022-04-11 20:35:11.728208: train loss : -0.4693 
2022-04-11 20:35:22.762720: validation loss: -0.5172 
2022-04-11 20:35:22.764205: Average global foreground Dice: [0.7611, 0.9329, 0.9642, 0.3556] 
2022-04-11 20:35:22.764856: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:35:23.681098: lr: 0.007959 
2022-04-11 20:35:23.682624: saving scheduled checkpoint file... 
2022-04-11 20:35:23.823926: saving checkpoint... 
2022-04-11 20:35:28.665348: done, saving took 4.98 seconds 
2022-04-11 20:35:28.701298: done 
2022-04-11 20:35:28.837433: saving checkpoint... 
2022-04-11 20:35:33.643267: done, saving took 4.93 seconds 
2022-04-11 20:35:33.667382: This epoch took 130.628507 s
 
2022-04-11 20:35:33.668496: svmem(total=99932160000, available=79983403008, percent=20.0, used=15943553024, free=52251635712, active=40147931136, inactive=5703266304, buffers=15990784, cached=31720980480, shared=2190839808, slab=753696768) 
2022-04-11 20:35:33.669954: 
epoch:  112 
2022-04-11 20:37:24.024893: train loss : -0.4986 
2022-04-11 20:37:35.644712: validation loss: -0.5238 
2022-04-11 20:37:35.647140: Average global foreground Dice: [0.8093, 0.932, 0.9711, 0.3313] 
2022-04-11 20:37:35.648673: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:37:36.760351: lr: 0.007941 
2022-04-11 20:37:36.761789: saving scheduled checkpoint file... 
2022-04-11 20:37:36.915326: saving checkpoint... 
2022-04-11 20:37:41.425952: done, saving took 4.66 seconds 
2022-04-11 20:37:41.449857: done 
2022-04-11 20:37:41.572745: saving checkpoint... 
2022-04-11 20:37:46.118853: done, saving took 4.67 seconds 
2022-04-11 20:37:46.141796: This epoch took 132.470944 s
 
2022-04-11 20:37:46.143200: svmem(total=99932160000, available=79808798720, percent=20.1, used=16118185984, free=52068990976, active=40329752576, inactive=5705035776, buffers=16044032, cached=31728939008, shared=2190917632, slab=753147904) 
2022-04-11 20:37:46.144306: 
epoch:  113 
2022-04-11 20:39:37.328717: train loss : -0.5152 
2022-04-11 20:39:50.273916: validation loss: -0.5203 
2022-04-11 20:39:50.282852: Average global foreground Dice: [0.786, 0.9384, 0.9653, 0.3595] 
2022-04-11 20:39:50.284112: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:39:50.988056: lr: 0.007922 
2022-04-11 20:39:50.989180: saving scheduled checkpoint file... 
2022-04-11 20:39:51.069639: saving checkpoint... 
2022-04-11 20:39:55.529816: done, saving took 4.54 seconds 
2022-04-11 20:39:55.543067: done 
2022-04-11 20:39:55.590078: saving checkpoint... 
2022-04-11 20:39:59.946940: done, saving took 4.40 seconds 
2022-04-11 20:39:59.958117: This epoch took 133.812752 s
 
2022-04-11 20:39:59.959905: svmem(total=99932160000, available=79833489408, percent=20.1, used=16093655040, free=52076261376, active=40303894528, inactive=5722185728, buffers=16076800, cached=31746166784, shared=2190729216, slab=754675712) 
2022-04-11 20:39:59.960988: 
epoch:  114 
2022-04-11 20:41:49.022459: train loss : -0.5175 
2022-04-11 20:42:00.850754: validation loss: -0.5242 
2022-04-11 20:42:00.854076: Average global foreground Dice: [0.7914, 0.9377, 0.9664, 0.3237] 
2022-04-11 20:42:00.855190: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:42:01.752965: lr: 0.007904 
2022-04-11 20:42:01.754499: saving scheduled checkpoint file... 
2022-04-11 20:42:01.829385: saving checkpoint... 
2022-04-11 20:42:06.511139: done, saving took 4.76 seconds 
2022-04-11 20:42:06.551041: done 
2022-04-11 20:42:06.602673: saving checkpoint... 
2022-04-11 20:42:10.969068: done, saving took 4.42 seconds 
2022-04-11 20:42:10.977591: This epoch took 131.015121 s
 
2022-04-11 20:42:10.979136: svmem(total=99932160000, available=79678115840, percent=20.3, used=16256434176, free=51903299584, active=40461705216, inactive=5732155392, buffers=16117760, cached=31756308480, shared=2183294976, slab=753676288) 
2022-04-11 20:42:10.980310: 
epoch:  115 
2022-04-11 20:44:00.078024: train loss : -0.5041 
2022-04-11 20:44:12.720658: validation loss: -0.4680 
2022-04-11 20:44:12.729506: Average global foreground Dice: [0.7727, 0.9215, 0.9315, 0.323] 
2022-04-11 20:44:12.730851: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:44:13.559659: lr: 0.007885 
2022-04-11 20:44:13.560949: saving scheduled checkpoint file... 
2022-04-11 20:44:13.635590: saving checkpoint... 
2022-04-11 20:44:18.052719: done, saving took 4.49 seconds 
2022-04-11 20:44:18.070196: done 
2022-04-11 20:44:18.071510: This epoch took 127.090358 s
 
2022-04-11 20:44:18.072382: svmem(total=99932160000, available=79622578176, percent=20.3, used=16304406528, free=51839979520, active=40518545408, inactive=5747453952, buffers=16166912, cached=31771607040, shared=2190839808, slab=755355648) 
2022-04-11 20:44:18.073095: 
epoch:  116 
2022-04-11 20:46:07.599499: train loss : -0.4720 
2022-04-11 20:46:19.545973: validation loss: -0.5204 
2022-04-11 20:46:19.553638: Average global foreground Dice: [0.7787, 0.9407, 0.9712, 0.3325] 
2022-04-11 20:46:19.554677: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:46:20.392932: lr: 0.007867 
2022-04-11 20:46:20.394334: saving scheduled checkpoint file... 
2022-04-11 20:46:20.485367: saving checkpoint... 
2022-04-11 20:46:24.993556: done, saving took 4.60 seconds 
2022-04-11 20:46:25.059394: done 
2022-04-11 20:46:25.060317: This epoch took 126.986329 s
 
2022-04-11 20:46:25.061277: svmem(total=99932160000, available=79613616128, percent=20.3, used=16313409536, free=51813531648, active=40526168064, inactive=5764743168, buffers=16211968, cached=31789006848, shared=2190729216, slab=755318784) 
2022-04-11 20:46:25.062568: 
epoch:  117 
2022-04-11 20:48:13.666996: train loss : -0.5140 
2022-04-11 20:48:26.044814: validation loss: -0.4882 
2022-04-11 20:48:26.046916: Average global foreground Dice: [0.7937, 0.9326, 0.9511, 0.3346] 
2022-04-11 20:48:26.047921: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:48:26.738734: lr: 0.007848 
2022-04-11 20:48:26.740403: saving scheduled checkpoint file... 
2022-04-11 20:48:26.873365: saving checkpoint... 
2022-04-11 20:48:31.479702: done, saving took 4.74 seconds 
2022-04-11 20:48:31.499865: done 
2022-04-11 20:48:31.593272: saving checkpoint... 
2022-04-11 20:48:36.424332: done, saving took 4.92 seconds 
2022-04-11 20:48:36.441614: This epoch took 131.378248 s
 
2022-04-11 20:48:36.442821: svmem(total=99932160000, available=79786172416, percent=20.2, used=16140914688, free=51968786432, active=40355258368, inactive=5779935232, buffers=16244736, cached=31806214144, shared=2190696448, slab=755576832) 
2022-04-11 20:48:36.443940: 
epoch:  118 
2022-04-11 20:50:25.090008: train loss : -0.3760 
2022-04-11 20:50:36.978390: validation loss: -0.4539 
2022-04-11 20:50:36.989411: Average global foreground Dice: [0.6936, 0.9221, 0.9592, 0.3447] 
2022-04-11 20:50:36.990404: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:50:38.016005: lr: 0.00783 
2022-04-11 20:50:38.017113: saving scheduled checkpoint file... 
2022-04-11 20:50:38.111369: saving checkpoint... 
2022-04-11 20:50:42.590677: done, saving took 4.57 seconds 
2022-04-11 20:50:42.611813: done 
2022-04-11 20:50:42.613117: This epoch took 126.168366 s
 
2022-04-11 20:50:42.614441: svmem(total=99932160000, available=79743463424, percent=20.2, used=16183525376, free=51910004736, active=40352313344, inactive=5842952192, buffers=16285696, cached=31822344192, shared=2190729216, slab=753340416) 
2022-04-11 20:50:42.615263: 
epoch:  119 
2022-04-11 20:52:32.623685: train loss : -0.4976 
2022-04-11 20:52:45.012023: validation loss: -0.5341 
2022-04-11 20:52:45.014153: Average global foreground Dice: [0.7553, 0.9389, 0.9643, 0.3849] 
2022-04-11 20:52:45.015589: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:52:45.806695: lr: 0.007811 
2022-04-11 20:52:45.808154: saving scheduled checkpoint file... 
2022-04-11 20:52:45.897978: saving checkpoint... 
2022-04-11 20:52:50.563635: done, saving took 4.75 seconds 
2022-04-11 20:52:50.615624: done 
2022-04-11 20:52:50.617126: This epoch took 128.001170 s
 
2022-04-11 20:52:50.618359: svmem(total=99932160000, available=79761813504, percent=20.2, used=16165117952, free=51921588224, active=40380440576, inactive=5805412352, buffers=16318464, cached=31829135360, shared=2190729216, slab=755671040) 
2022-04-11 20:52:50.619269: 
epoch:  120 
2022-04-11 20:54:39.074921: train loss : -0.4620 
2022-04-11 20:54:50.816090: validation loss: -0.4692 
2022-04-11 20:54:50.818139: Average global foreground Dice: [0.7329, 0.9243, 0.9496, 0.3669] 
2022-04-11 20:54:50.825394: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:54:51.767787: lr: 0.007793 
2022-04-11 20:54:51.769156: saving scheduled checkpoint file... 
2022-04-11 20:54:51.884606: saving checkpoint... 
2022-04-11 20:54:57.113736: done, saving took 5.34 seconds 
2022-04-11 20:54:57.175580: done 
2022-04-11 20:54:57.177207: This epoch took 126.557127 s
 
2022-04-11 20:54:57.178369: svmem(total=99932160000, available=79907614720, percent=20.0, used=16019025920, free=52050055168, active=40232853504, inactive=5820727296, buffers=16351232, cached=31846727680, shared=2191114240, slab=755896320) 
2022-04-11 20:54:57.179596: 
epoch:  121 
2022-04-11 20:56:47.119507: train loss : -0.4478 
2022-04-11 20:56:59.219937: validation loss: -0.4621 
2022-04-11 20:56:59.223314: Average global foreground Dice: [0.7968, 0.9197, 0.9488, 0.3311] 
2022-04-11 20:56:59.224308: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:57:00.152165: lr: 0.007774 
2022-04-11 20:57:00.153455: saving scheduled checkpoint file... 
2022-04-11 20:57:00.230032: saving checkpoint... 
2022-04-11 20:57:04.962904: done, saving took 4.81 seconds 
2022-04-11 20:57:04.983374: done 
2022-04-11 20:57:04.984826: This epoch took 127.803999 s
 
2022-04-11 20:57:04.985746: svmem(total=99932160000, available=79826468864, percent=20.1, used=16100020224, free=51979767808, active=40204550144, inactive=5913952256, buffers=16416768, cached=31835955200, shared=2191159296, slab=756224000) 
2022-04-11 20:57:04.986548: 
epoch:  122 
2022-04-11 20:58:53.453725: train loss : -0.4551 
2022-04-11 20:59:04.725389: validation loss: -0.4651 
2022-04-11 20:59:04.732685: Average global foreground Dice: [0.7871, 0.9196, 0.9424, 0.3361] 
2022-04-11 20:59:04.734362: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 20:59:05.665434: lr: 0.007756 
2022-04-11 20:59:05.666556: saving scheduled checkpoint file... 
2022-04-11 20:59:05.761812: saving checkpoint... 
2022-04-11 20:59:10.505135: done, saving took 4.84 seconds 
2022-04-11 20:59:10.597411: done 
2022-04-11 20:59:10.598679: This epoch took 125.611365 s
 
2022-04-11 20:59:10.599636: svmem(total=99932160000, available=79660699648, percent=20.3, used=16259293184, free=51806044160, active=40442982400, inactive=5850873856, buffers=16449536, cached=31850373120, shared=2197745664, slab=755560448) 
2022-04-11 20:59:10.600399: 
epoch:  123 
2022-04-11 21:00:59.790795: train loss : -0.4427 
2022-04-11 21:01:11.558867: validation loss: -0.4555 
2022-04-11 21:01:11.569568: Average global foreground Dice: [0.7623, 0.9268, 0.9556, 0.3094] 
2022-04-11 21:01:11.570910: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:01:12.526236: lr: 0.007737 
2022-04-11 21:01:12.527423: saving scheduled checkpoint file... 
2022-04-11 21:01:12.642876: saving checkpoint... 
2022-04-11 21:01:17.437054: done, saving took 4.91 seconds 
2022-04-11 21:01:17.457519: done 
2022-04-11 21:01:17.458691: This epoch took 126.857260 s
 
2022-04-11 21:01:17.460360: svmem(total=99932160000, available=79888424960, percent=20.1, used=16038363136, free=52016263168, active=40247570432, inactive=5835632640, buffers=16506880, cached=31861026816, shared=2190807040, slab=756371456) 
2022-04-11 21:01:17.461165: 
epoch:  124 
2022-04-11 21:03:07.026683: train loss : -0.4838 
2022-04-11 21:03:18.736803: validation loss: -0.5027 
2022-04-11 21:03:18.753578: Average global foreground Dice: [0.7906, 0.9341, 0.9625, 0.3457] 
2022-04-11 21:03:18.754870: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:03:19.668893: lr: 0.007719 
2022-04-11 21:03:19.670033: saving scheduled checkpoint file... 
2022-04-11 21:03:19.795735: saving checkpoint... 
2022-04-11 21:03:24.256666: done, saving took 4.59 seconds 
2022-04-11 21:03:24.281983: done 
2022-04-11 21:03:24.283119: This epoch took 126.821137 s
 
2022-04-11 21:03:24.284366: svmem(total=99932160000, available=79827320832, percent=20.1, used=16102662144, free=51938713600, active=40313266176, inactive=5848764416, buffers=16539648, cached=31874244608, shared=2187567104, slab=756350976) 
2022-04-11 21:03:24.285372: 
epoch:  125 
2022-04-11 21:05:14.932934: train loss : -0.4752 
2022-04-11 21:05:27.946927: validation loss: -0.4478 
2022-04-11 21:05:27.949568: Average global foreground Dice: [0.7791, 0.9252, 0.9588, 0.3309] 
2022-04-11 21:05:27.951827: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:05:28.631814: lr: 0.0077 
2022-04-11 21:05:28.633273: saving scheduled checkpoint file... 
2022-04-11 21:05:28.722670: saving checkpoint... 
2022-04-11 21:05:33.036278: done, saving took 4.40 seconds 
2022-04-11 21:05:33.054218: done 
2022-04-11 21:05:33.055571: This epoch took 128.768800 s
 
2022-04-11 21:05:33.057081: svmem(total=99932160000, available=79622467584, percent=20.3, used=16304046080, free=51726159872, active=40521355264, inactive=5858238464, buffers=16580608, cached=31885373440, shared=2190917632, slab=755920896) 
2022-04-11 21:05:33.057913: 
epoch:  126 
2022-04-11 21:07:23.491228: train loss : -0.4838 
2022-04-11 21:07:36.288207: validation loss: -0.5212 
2022-04-11 21:07:36.293319: Average global foreground Dice: [0.7746, 0.9349, 0.9694, 0.3488] 
2022-04-11 21:07:36.294309: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:07:37.077576: lr: 0.007682 
2022-04-11 21:07:37.079068: saving scheduled checkpoint file... 
2022-04-11 21:07:37.131421: saving checkpoint... 
2022-04-11 21:07:41.483829: done, saving took 4.40 seconds 
2022-04-11 21:07:41.500539: done 
2022-04-11 21:07:41.547991: saving checkpoint... 
2022-04-11 21:07:45.904351: done, saving took 4.40 seconds 
2022-04-11 21:07:45.919465: This epoch took 132.860420 s
 
2022-04-11 21:07:45.922951: svmem(total=99932160000, available=79669145600, percent=20.3, used=16257560576, free=51727028224, active=40474062848, inactive=5903822848, buffers=16637952, cached=31930933248, shared=2190729216, slab=755970048) 
2022-04-11 21:07:45.924100: 
epoch:  127 
2022-04-11 21:09:34.282719: train loss : -0.4890 
2022-04-11 21:09:45.726859: validation loss: -0.5449 
2022-04-11 21:09:45.729558: Average global foreground Dice: [0.8132, 0.937, 0.9661, 0.3767] 
2022-04-11 21:09:45.731097: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:09:46.466645: lr: 0.007663 
2022-04-11 21:09:46.468047: saving scheduled checkpoint file... 
2022-04-11 21:09:46.534495: saving checkpoint... 
2022-04-11 21:09:51.241809: done, saving took 4.77 seconds 
2022-04-11 21:09:51.257202: done 
2022-04-11 21:09:51.308445: saving checkpoint... 
2022-04-11 21:09:56.073354: done, saving took 4.81 seconds 
2022-04-11 21:09:56.091770: This epoch took 130.166888 s
 
2022-04-11 21:09:56.093506: svmem(total=99932160000, available=79698579456, percent=20.2, used=16228188160, free=51740598272, active=40359890944, inactive=6000218112, buffers=16670720, cached=31946702848, shared=2190729216, slab=756858880) 
2022-04-11 21:09:56.094312: 
epoch:  128 
2022-04-11 21:11:44.672387: train loss : -0.5265 
2022-04-11 21:11:57.164944: validation loss: -0.5253 
2022-04-11 21:11:57.166610: Average global foreground Dice: [0.791, 0.9334, 0.9691, 0.3551] 
2022-04-11 21:11:57.167819: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:11:57.997966: lr: 0.007645 
2022-04-11 21:11:57.999125: saving scheduled checkpoint file... 
2022-04-11 21:11:58.073431: saving checkpoint... 
2022-04-11 21:12:02.569022: done, saving took 4.57 seconds 
2022-04-11 21:12:02.609003: done 
2022-04-11 21:12:02.675307: saving checkpoint... 
2022-04-11 21:12:07.133076: done, saving took 4.52 seconds 
2022-04-11 21:12:07.150094: This epoch took 131.055007 s
 
2022-04-11 21:12:07.151146: svmem(total=99932160000, available=79490162688, percent=20.5, used=16436584448, free=51522387968, active=40619315200, inactive=5954760704, buffers=16711680, cached=31956475904, shared=2190807040, slab=757878784) 
2022-04-11 21:12:07.152369: 
epoch:  129 
2022-04-11 21:13:55.043395: train loss : -0.5496 
2022-04-11 21:14:06.650923: validation loss: -0.5708 
2022-04-11 21:14:06.661996: Average global foreground Dice: [0.8165, 0.9475, 0.9737, 0.3885] 
2022-04-11 21:14:06.663021: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:14:07.515053: lr: 0.007626 
2022-04-11 21:14:07.516424: saving scheduled checkpoint file... 
2022-04-11 21:14:07.613060: saving checkpoint... 
2022-04-11 21:14:12.233228: done, saving took 4.72 seconds 
2022-04-11 21:14:12.297081: done 
2022-04-11 21:14:12.379787: saving checkpoint... 
2022-04-11 21:14:16.944088: done, saving took 4.65 seconds 
2022-04-11 21:14:16.962485: This epoch took 129.805798 s
 
2022-04-11 21:14:16.963584: svmem(total=99932160000, available=79722381312, percent=20.2, used=16204238848, free=51738247168, active=40403431424, inactive=5959823360, buffers=16760832, cached=31972913152, shared=2190729216, slab=756588544) 
2022-04-11 21:14:16.964404: 
epoch:  130 
2022-04-11 21:16:05.365438: train loss : -0.5213 
2022-04-11 21:16:16.680643: validation loss: -0.5165 
2022-04-11 21:16:16.690904: Average global foreground Dice: [0.7988, 0.9307, 0.9651, 0.3568] 
2022-04-11 21:16:16.694297: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:16:17.539469: lr: 0.007608 
2022-04-11 21:16:17.541055: saving scheduled checkpoint file... 
2022-04-11 21:16:17.651053: saving checkpoint... 
2022-04-11 21:16:23.800020: done, saving took 6.26 seconds 
2022-04-11 21:16:23.820652: done 
2022-04-11 21:16:23.914999: saving checkpoint... 
2022-04-11 21:16:28.396893: done, saving took 4.57 seconds 
2022-04-11 21:16:28.422866: This epoch took 131.457701 s
 
2022-04-11 21:16:28.424318: svmem(total=99932160000, available=79685279744, percent=20.3, used=16237023232, free=51682533376, active=40442191872, inactive=5976424448, buffers=16805888, cached=31995797504, shared=2195189760, slab=756277248) 
2022-04-11 21:16:28.425193: 
epoch:  131 
2022-04-11 21:18:17.819119: train loss : -0.4770 
2022-04-11 21:18:30.387132: validation loss: -0.4935 
2022-04-11 21:18:30.389716: Average global foreground Dice: [0.773, 0.9333, 0.9612, 0.353] 
2022-04-11 21:18:30.390686: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:18:31.215630: lr: 0.007589 
2022-04-11 21:18:31.217124: saving scheduled checkpoint file... 
2022-04-11 21:18:31.309376: saving checkpoint... 
2022-04-11 21:18:35.812831: done, saving took 4.59 seconds 
2022-04-11 21:18:35.830607: done 
2022-04-11 21:18:35.831999: This epoch took 127.405347 s
 
2022-04-11 21:18:35.833239: svmem(total=99932160000, available=79986376704, percent=20.0, used=15950872576, free=51976921088, active=40165376000, inactive=5959716864, buffers=16846848, cached=31987519488, shared=2180218880, slab=756686848) 
2022-04-11 21:18:35.833943: 
epoch:  132 
2022-04-11 21:20:24.092654: train loss : -0.4982 
2022-04-11 21:20:36.335370: validation loss: -0.5431 
2022-04-11 21:20:36.342860: Average global foreground Dice: [0.8192, 0.937, 0.9702, 0.3453] 
2022-04-11 21:20:36.343776: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:20:37.243002: lr: 0.007571 
2022-04-11 21:20:37.245043: saving scheduled checkpoint file... 
2022-04-11 21:20:37.348537: saving checkpoint... 
2022-04-11 21:20:41.706118: done, saving took 4.46 seconds 
2022-04-11 21:20:41.724053: done 
2022-04-11 21:20:41.802200: saving checkpoint... 
2022-04-11 21:20:46.241164: done, saving took 4.52 seconds 
2022-04-11 21:20:46.260596: This epoch took 130.425958 s
 
2022-04-11 21:20:46.261600: svmem(total=99932160000, available=79839383552, percent=20.1, used=16087343104, free=51812405248, active=40300711936, inactive=5987401728, buffers=16887808, cached=32015523840, shared=2190729216, slab=757080064) 
2022-04-11 21:20:46.262243: 
epoch:  133 
2022-04-11 21:22:33.765862: train loss : -0.5190 
2022-04-11 21:22:46.187103: validation loss: -0.5369 
2022-04-11 21:22:46.205490: Average global foreground Dice: [0.8313, 0.9436, 0.9742, 0.3724] 
2022-04-11 21:22:46.207181: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:22:46.921143: lr: 0.007552 
2022-04-11 21:22:46.922742: saving scheduled checkpoint file... 
2022-04-11 21:22:47.012590: saving checkpoint... 
2022-04-11 21:22:51.618001: done, saving took 4.69 seconds 
2022-04-11 21:22:51.639824: done 
2022-04-11 21:22:51.733075: saving checkpoint... 
2022-04-11 21:22:56.292573: done, saving took 4.65 seconds 
2022-04-11 21:22:56.312586: This epoch took 130.049355 s
 
2022-04-11 21:22:56.314042: svmem(total=99932160000, available=79857315840, percent=20.1, used=16069283840, free=51813105664, active=40281116672, inactive=6004412416, buffers=16928768, cached=32032841728, shared=2190729216, slab=757940224) 
2022-04-11 21:22:56.314895: 
epoch:  134 
2022-04-11 21:24:45.354506: train loss : -0.5146 
2022-04-11 21:24:57.374978: validation loss: -0.5084 
2022-04-11 21:24:57.380705: Average global foreground Dice: [0.7983, 0.9337, 0.9652, 0.3349] 
2022-04-11 21:24:57.381780: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:24:58.275944: lr: 0.007533 
2022-04-11 21:24:58.276947: saving scheduled checkpoint file... 
2022-04-11 21:24:58.410283: saving checkpoint... 
2022-04-11 21:25:02.797368: done, saving took 4.52 seconds 
2022-04-11 21:25:02.820180: done 
2022-04-11 21:25:02.821300: This epoch took 126.505643 s
 
2022-04-11 21:25:02.822924: svmem(total=99932160000, available=79985692672, percent=20.0, used=15940517888, free=51923750912, active=40149569536, inactive=6022328320, buffers=16969728, cached=32050921472, shared=2191151104, slab=757276672) 
2022-04-11 21:25:02.823892: 
epoch:  135 
2022-04-11 21:26:51.717065: train loss : -0.5137 
2022-04-11 21:27:03.661451: validation loss: -0.5421 
2022-04-11 21:27:03.673467: Average global foreground Dice: [0.7504, 0.9417, 0.9656, 0.3823] 
2022-04-11 21:27:03.674857: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:27:04.534894: lr: 0.007515 
2022-04-11 21:27:04.536218: saving scheduled checkpoint file... 
2022-04-11 21:27:04.686117: saving checkpoint... 
2022-04-11 21:27:09.215473: done, saving took 4.68 seconds 
2022-04-11 21:27:09.238835: done 
2022-04-11 21:27:09.240149: This epoch took 126.415427 s
 
2022-04-11 21:27:09.241325: svmem(total=99932160000, available=79798624256, percent=20.1, used=16128135168, free=51728990208, active=40339333120, inactive=6029557760, buffers=17006592, cached=32058028032, shared=2190618624, slab=757358592) 
2022-04-11 21:27:09.242124: 
epoch:  136 
2022-04-11 21:29:00.133990: train loss : -0.5264 
2022-04-11 21:29:13.152523: validation loss: -0.5713 
2022-04-11 21:29:13.175565: Average global foreground Dice: [0.823, 0.9428, 0.975, 0.3526] 
2022-04-11 21:29:13.177644: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:29:13.708312: lr: 0.007496 
2022-04-11 21:29:13.709645: saving scheduled checkpoint file... 
2022-04-11 21:29:13.835225: saving checkpoint... 
2022-04-11 21:29:18.276312: done, saving took 4.57 seconds 
2022-04-11 21:29:18.301133: done 
2022-04-11 21:29:18.436695: saving checkpoint... 
2022-04-11 21:29:22.860926: done, saving took 4.56 seconds 
2022-04-11 21:29:22.882227: This epoch took 133.639131 s
 
2022-04-11 21:29:22.883469: svmem(total=99932160000, available=79928262656, percent=20.0, used=15993901056, free=51841110016, active=40210108416, inactive=6051581952, buffers=17055744, cached=32080093184, shared=2195234816, slab=757874688) 
2022-04-11 21:29:22.884016: 
epoch:  137 
2022-04-11 21:31:13.941482: train loss : -0.5245 
2022-04-11 21:31:26.915444: validation loss: -0.5024 
2022-04-11 21:31:26.924468: Average global foreground Dice: [0.8036, 0.9324, 0.9633, 0.3717] 
2022-04-11 21:31:26.926321: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:31:27.884184: lr: 0.007478 
2022-04-11 21:31:27.885664: saving scheduled checkpoint file... 
2022-04-11 21:31:27.994128: saving checkpoint... 
2022-04-11 21:31:32.924759: done, saving took 5.04 seconds 
2022-04-11 21:31:32.937167: done 
2022-04-11 21:31:32.985847: saving checkpoint... 
2022-04-11 21:31:37.429532: done, saving took 4.49 seconds 
2022-04-11 21:31:37.445146: This epoch took 134.560565 s
 
2022-04-11 21:31:37.446795: svmem(total=99932160000, available=80043651072, percent=19.9, used=15945945088, free=51939106816, active=40159178752, inactive=6001496064, buffers=17096704, cached=32030011392, shared=2127716352, slab=757026816) 
2022-04-11 21:31:37.447633: 
epoch:  138 
2022-04-11 21:33:27.134992: train loss : -0.5135 
2022-04-11 21:33:39.026887: validation loss: -0.5275 
2022-04-11 21:33:39.033253: Average global foreground Dice: [0.7921, 0.9383, 0.96, 0.3515] 
2022-04-11 21:33:39.034290: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:33:39.983031: lr: 0.007459 
2022-04-11 21:33:39.984415: saving scheduled checkpoint file... 
2022-04-11 21:33:40.060502: saving checkpoint... 
2022-04-11 21:33:44.578304: done, saving took 4.59 seconds 
2022-04-11 21:33:44.641688: done 
2022-04-11 21:33:44.643040: This epoch took 127.194515 s
 
2022-04-11 21:33:44.644829: svmem(total=99932160000, available=80079167488, percent=19.9, used=15948636160, free=51966627840, active=40166367232, inactive=5971009536, buffers=17129472, cached=31999766528, shared=2089381888, slab=756817920) 
2022-04-11 21:33:44.646026: 
epoch:  139 
2022-04-11 21:35:34.236834: train loss : -0.5202 
2022-04-11 21:35:45.228526: validation loss: -0.5511 
2022-04-11 21:35:45.233631: Average global foreground Dice: [0.8151, 0.9388, 0.974, 0.4032] 
2022-04-11 21:35:45.235113: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:35:45.863705: lr: 0.00744 
2022-04-11 21:35:45.864898: saving scheduled checkpoint file... 
2022-04-11 21:35:45.943728: saving checkpoint... 
2022-04-11 21:35:50.767547: done, saving took 4.90 seconds 
2022-04-11 21:35:50.785686: done 
2022-04-11 21:35:50.873014: saving checkpoint... 
2022-04-11 21:35:58.323320: done, saving took 7.54 seconds 
2022-04-11 21:35:58.744198: This epoch took 134.097441 s
 
2022-04-11 21:35:58.860235: svmem(total=99932160000, available=79934353408, percent=20.0, used=16006037504, free=51804647424, active=40204521472, inactive=6075478016, buffers=17178624, cached=32104296448, shared=2176741376, slab=770547712) 
2022-04-11 21:35:58.966848: 
epoch:  140 
2022-04-11 21:37:49.145314: train loss : -0.5490 
2022-04-11 21:37:59.559361: validation loss: -0.5597 
2022-04-11 21:37:59.561729: Average global foreground Dice: [0.7991, 0.9446, 0.9669, 0.3651] 
2022-04-11 21:37:59.563195: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:38:00.677482: lr: 0.007422 
2022-04-11 21:38:00.679443: saving scheduled checkpoint file... 
2022-04-11 21:38:00.801085: saving checkpoint... 
2022-04-11 21:38:08.489938: done, saving took 7.81 seconds 
2022-04-11 21:38:08.509435: done 
2022-04-11 21:38:08.605495: saving checkpoint... 
2022-04-11 21:38:13.342294: done, saving took 4.83 seconds 
2022-04-11 21:38:13.361795: This epoch took 134.296326 s
 
2022-04-11 21:38:13.362849: svmem(total=99932160000, available=79866310656, percent=20.1, used=16053805056, free=51717844992, active=40268271616, inactive=6114213888, buffers=17383424, cached=32143126528, shared=2196983808, slab=757702656) 
2022-04-11 21:38:13.363517: 
epoch:  141 
2022-04-11 21:40:02.926967: train loss : -0.5101 
2022-04-11 21:40:14.573580: validation loss: -0.5253 
2022-04-11 21:40:14.575504: Average global foreground Dice: [0.7673, 0.9275, 0.9571, 0.365] 
2022-04-11 21:40:14.576455: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:40:15.446933: lr: 0.007403 
2022-04-11 21:40:15.448372: saving scheduled checkpoint file... 
2022-04-11 21:40:15.571075: saving checkpoint... 
2022-04-11 21:40:21.571926: done, saving took 6.12 seconds 
2022-04-11 21:40:21.594354: done 
2022-04-11 21:40:21.595461: This epoch took 128.231304 s
 
2022-04-11 21:40:21.596325: svmem(total=99932160000, available=79708610560, percent=20.2, used=16214413312, free=51542544384, active=40429645824, inactive=6128709632, buffers=17424384, cached=32157777920, shared=2194055168, slab=758681600) 
2022-04-11 21:40:21.597158: 
epoch:  142 
2022-04-11 21:42:10.932013: train loss : -0.4528 
2022-04-11 21:42:23.346739: validation loss: -0.4843 
2022-04-11 21:42:23.350590: Average global foreground Dice: [0.7689, 0.9287, 0.9597, 0.3389] 
2022-04-11 21:42:23.351536: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:42:24.227372: lr: 0.007385 
2022-04-11 21:42:24.228552: saving scheduled checkpoint file... 
2022-04-11 21:42:24.372536: saving checkpoint... 
2022-04-11 21:42:30.257170: done, saving took 6.03 seconds 
2022-04-11 21:42:30.280462: done 
2022-04-11 21:42:30.281455: This epoch took 128.683582 s
 
2022-04-11 21:42:30.284025: svmem(total=99932160000, available=79700140032, percent=20.2, used=16226254848, free=51527340032, active=40437907456, inactive=6132056064, buffers=17457152, cached=32161107968, shared=2190729216, slab=757927936) 
2022-04-11 21:42:30.285079: 
epoch:  143 
2022-04-11 21:44:21.700887: train loss : -0.4796 
2022-04-11 21:44:34.776132: validation loss: -0.5046 
2022-04-11 21:44:34.785860: Average global foreground Dice: [0.7618, 0.9278, 0.9614, 0.3913] 
2022-04-11 21:44:34.786894: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:44:35.620004: lr: 0.007366 
2022-04-11 21:44:35.621538: saving scheduled checkpoint file... 
2022-04-11 21:44:35.706299: saving checkpoint... 
2022-04-11 21:44:40.048582: done, saving took 4.43 seconds 
2022-04-11 21:44:40.066191: done 
2022-04-11 21:44:40.067786: This epoch took 129.781374 s
 
2022-04-11 21:44:40.069708: svmem(total=99932160000, available=79690723328, percent=20.3, used=16235507712, free=51500470272, active=40446930944, inactive=6149320704, buffers=17489920, cached=32178692096, shared=2190729216, slab=756805632) 
2022-04-11 21:44:40.070610: 
epoch:  144 
2022-04-11 21:46:30.155665: train loss : -0.4821 
2022-04-11 21:46:41.473310: validation loss: -0.5099 
2022-04-11 21:46:41.476077: Average global foreground Dice: [0.7927, 0.9291, 0.9688, 0.3494] 
2022-04-11 21:46:41.477062: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:46:42.246569: lr: 0.007347 
2022-04-11 21:46:42.247992: saving scheduled checkpoint file... 
2022-04-11 21:46:42.354752: saving checkpoint... 
2022-04-11 21:46:47.228336: done, saving took 4.98 seconds 
2022-04-11 21:46:47.246173: done 
2022-04-11 21:46:47.247923: This epoch took 127.176386 s
 
2022-04-11 21:46:47.249230: svmem(total=99932160000, available=79916531712, percent=20.0, used=16009719808, free=51708669952, active=40221114368, inactive=6166843392, buffers=17539072, cached=32196231168, shared=2190729216, slab=757157888) 
2022-04-11 21:46:47.250567: 
epoch:  145 
2022-04-11 21:48:36.018191: train loss : -0.5205 
2022-04-11 21:48:48.133562: validation loss: -0.5416 
2022-04-11 21:48:48.136468: Average global foreground Dice: [0.8232, 0.9415, 0.9613, 0.364] 
2022-04-11 21:48:48.137444: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:48:49.031531: lr: 0.007329 
2022-04-11 21:48:49.032706: saving scheduled checkpoint file... 
2022-04-11 21:48:49.138496: saving checkpoint... 
2022-04-11 21:48:53.742308: done, saving took 4.71 seconds 
2022-04-11 21:48:53.770981: done 
2022-04-11 21:48:53.772171: This epoch took 126.520971 s
 
2022-04-11 21:48:53.773286: svmem(total=99932160000, available=79770488832, percent=20.2, used=16155623424, free=51554820096, active=40370839552, inactive=6174621696, buffers=17571840, cached=32204144640, shared=2190823424, slab=758247424) 
2022-04-11 21:48:53.774065: 
epoch:  146 
2022-04-11 21:50:42.535598: train loss : -0.5176 
2022-04-11 21:50:54.966973: validation loss: -0.5101 
2022-04-11 21:50:54.969281: Average global foreground Dice: [0.8019, 0.9308, 0.958, 0.3719] 
2022-04-11 21:50:54.970520: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:50:55.744149: lr: 0.00731 
2022-04-11 21:50:55.745677: saving scheduled checkpoint file... 
2022-04-11 21:50:55.864055: saving checkpoint... 
2022-04-11 21:51:00.884702: done, saving took 5.14 seconds 
2022-04-11 21:51:00.963385: done 
2022-04-11 21:51:00.964708: This epoch took 127.189897 s
 
2022-04-11 21:51:00.966070: svmem(total=99932160000, available=79920238592, percent=20.0, used=16019742720, free=51687170048, active=40234885120, inactive=6177955840, buffers=17612800, cached=32207634432, shared=2176851968, slab=758046720) 
2022-04-11 21:51:00.967000: 
epoch:  147 
2022-04-11 21:52:49.574767: train loss : -0.5266 
2022-04-11 21:53:02.046369: validation loss: -0.5483 
2022-04-11 21:53:02.057840: Average global foreground Dice: [0.7981, 0.9362, 0.9663, 0.3577] 
2022-04-11 21:53:02.059339: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:53:02.708509: lr: 0.007291 
2022-04-11 21:53:02.710145: saving scheduled checkpoint file... 
2022-04-11 21:53:02.827418: saving checkpoint... 
2022-04-11 21:53:07.566075: done, saving took 4.85 seconds 
2022-04-11 21:53:07.580736: done 
2022-04-11 21:53:07.582090: This epoch took 126.614322 s
 
2022-04-11 21:53:07.583502: svmem(total=99932160000, available=79621038080, percent=20.3, used=16300527616, free=51370442752, active=40509960192, inactive=6213718016, buffers=17645568, cached=32243544064, shared=2195189760, slab=756903936) 
2022-04-11 21:53:07.584363: 
epoch:  148 
2022-04-11 21:54:58.435889: train loss : -0.5616 
2022-04-11 21:55:11.245640: validation loss: -0.5920 
2022-04-11 21:55:11.249202: Average global foreground Dice: [0.8186, 0.9459, 0.9764, 0.3875] 
2022-04-11 21:55:11.250270: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:55:12.140867: lr: 0.007273 
2022-04-11 21:55:12.142491: saving scheduled checkpoint file... 
2022-04-11 21:55:12.288580: saving checkpoint... 
2022-04-11 21:55:17.050925: done, saving took 4.91 seconds 
2022-04-11 21:55:17.116960: done 
2022-04-11 21:55:17.252766: saving checkpoint... 
2022-04-11 21:55:21.759166: done, saving took 4.64 seconds 
2022-04-11 21:55:21.819887: This epoch took 134.234752 s
 
2022-04-11 21:55:21.821489: svmem(total=99932160000, available=80005058560, percent=19.9, used=15920697344, free=51746770944, active=40136097792, inactive=6217355264, buffers=17702912, cached=32246988800, shared=2191077376, slab=756494336) 
2022-04-11 21:55:21.822237: 
epoch:  149 
2022-04-11 21:57:13.200539: train loss : -0.5451 
2022-04-11 21:57:25.465148: validation loss: -0.5310 
2022-04-11 21:57:25.467590: Average global foreground Dice: [0.8136, 0.9366, 0.9622, 0.3861] 
2022-04-11 21:57:25.468769: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:57:26.437207: lr: 0.007254 
2022-04-11 21:57:26.438581: saving scheduled checkpoint file... 
2022-04-11 21:57:26.525044: saving checkpoint... 
2022-04-11 21:57:30.926903: done, saving took 4.49 seconds 
2022-04-11 21:57:30.945374: done 
2022-04-11 21:57:30.993025: saving checkpoint... 
2022-04-11 21:57:35.523047: done, saving took 4.58 seconds 
2022-04-11 21:57:35.564432: This epoch took 133.741475 s
 
2022-04-11 21:57:35.565435: svmem(total=99932160000, available=79873089536, percent=20.1, used=16052912128, free=51597410304, active=40265449472, inactive=6234460160, buffers=17739776, cached=32264097792, shared=2190839808, slab=757805056) 
2022-04-11 21:57:35.566521: 
epoch:  150 
2022-04-11 21:59:23.730780: train loss : -0.5418 
2022-04-11 21:59:35.287916: validation loss: -0.5463 
2022-04-11 21:59:35.289775: Average global foreground Dice: [0.8173, 0.9425, 0.9697, 0.3962] 
2022-04-11 21:59:35.290657: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 21:59:36.365626: lr: 0.007236 
2022-04-11 21:59:36.367182: saving scheduled checkpoint file... 
2022-04-11 21:59:36.420047: saving checkpoint... 
2022-04-11 21:59:41.133649: done, saving took 4.77 seconds 
2022-04-11 21:59:41.170415: done 
2022-04-11 21:59:41.235209: saving checkpoint... 
2022-04-11 21:59:46.064826: done, saving took 4.89 seconds 
2022-04-11 21:59:46.101812: This epoch took 130.533984 s
 
2022-04-11 21:59:46.112020: svmem(total=99932160000, available=79802626048, percent=20.1, used=16123424768, free=51509383168, active=40336916480, inactive=6251753472, buffers=17772544, cached=32281579520, shared=2190729216, slab=759803904) 
2022-04-11 21:59:46.113413: 
epoch:  151 
2022-04-11 22:01:36.136136: train loss : -0.5426 
2022-04-11 22:01:48.051510: validation loss: -0.5702 
2022-04-11 22:01:48.053413: Average global foreground Dice: [0.8102, 0.9448, 0.9733, 0.3815] 
2022-04-11 22:01:48.055094: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:01:48.956052: lr: 0.007217 
2022-04-11 22:01:48.957465: saving scheduled checkpoint file... 
2022-04-11 22:01:49.048166: saving checkpoint... 
2022-04-11 22:01:53.458514: done, saving took 4.50 seconds 
2022-04-11 22:01:53.483305: done 
2022-04-11 22:01:53.563411: saving checkpoint... 
2022-04-11 22:01:58.464748: done, saving took 4.98 seconds 
2022-04-11 22:01:58.535519: This epoch took 132.421157 s
 
2022-04-11 22:01:58.537306: svmem(total=99932160000, available=79799595008, percent=20.1, used=16126377984, free=51498684416, active=40341204992, inactive=6259257344, buffers=17813504, cached=32289284096, shared=2190618624, slab=757243904) 
2022-04-11 22:01:58.538399: 
epoch:  152 
2022-04-11 22:03:45.829601: train loss : -0.5763 
2022-04-11 22:03:57.693565: validation loss: -0.5580 
2022-04-11 22:03:57.699131: Average global foreground Dice: [0.8171, 0.9393, 0.9725, 0.3632] 
2022-04-11 22:03:57.701541: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:03:58.282432: lr: 0.007198 
2022-04-11 22:03:58.283523: saving scheduled checkpoint file... 
2022-04-11 22:03:58.380529: saving checkpoint... 
2022-04-11 22:04:02.954560: done, saving took 4.67 seconds 
2022-04-11 22:04:03.136057: done 
2022-04-11 22:04:03.228382: saving checkpoint... 
2022-04-11 22:04:07.698401: done, saving took 4.56 seconds 
2022-04-11 22:04:07.718576: This epoch took 129.179475 s
 
2022-04-11 22:04:07.720176: svmem(total=99932160000, available=79803842560, percent=20.1, used=16111611904, free=51485302784, active=40317370368, inactive=6287417344, buffers=17981440, cached=32317263872, shared=2201292800, slab=757989376) 
2022-04-11 22:04:07.721484: 
epoch:  153 
2022-04-11 22:05:56.369341: train loss : -0.5636 
2022-04-11 22:06:08.613092: validation loss: -0.5651 
2022-04-11 22:06:08.623914: Average global foreground Dice: [0.8285, 0.9369, 0.97, 0.3594] 
2022-04-11 22:06:08.624977: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:06:09.346218: lr: 0.00718 
2022-04-11 22:06:09.347501: saving scheduled checkpoint file... 
2022-04-11 22:06:09.462734: saving checkpoint... 
2022-04-11 22:06:14.198704: done, saving took 4.85 seconds 
2022-04-11 22:06:14.220632: done 
2022-04-11 22:06:14.329392: saving checkpoint... 
2022-04-11 22:06:18.923684: done, saving took 4.70 seconds 
2022-04-11 22:06:18.977297: This epoch took 131.254966 s
 
2022-04-11 22:06:18.978362: svmem(total=99932160000, available=79781330944, percent=20.2, used=16144613376, free=51444924416, active=40360513536, inactive=6294380544, buffers=18337792, cached=32324284416, shared=2190807040, slab=758042624) 
2022-04-11 22:06:18.979479: 
epoch:  154 
2022-04-11 22:08:07.805672: train loss : -0.5440 
2022-04-11 22:08:19.765985: validation loss: -0.5425 
2022-04-11 22:08:19.769843: Average global foreground Dice: [0.8014, 0.9399, 0.9624, 0.3599] 
2022-04-11 22:08:19.770690: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:08:20.617732: lr: 0.007161 
2022-04-11 22:08:20.618991: saving scheduled checkpoint file... 
2022-04-11 22:08:20.735162: saving checkpoint... 
2022-04-11 22:08:25.354142: done, saving took 4.73 seconds 
2022-04-11 22:08:25.372988: done 
2022-04-11 22:08:25.373962: This epoch took 126.393790 s
 
2022-04-11 22:08:25.375385: svmem(total=99932160000, available=79934124032, percent=20.0, used=15994753024, free=51580133376, active=40209993728, inactive=6308745216, buffers=18386944, cached=32338886656, shared=2187755520, slab=758222848) 
2022-04-11 22:08:25.376689: 
epoch:  155 
2022-04-11 22:10:15.637335: train loss : -0.5311 
2022-04-11 22:10:28.273785: validation loss: -0.5311 
2022-04-11 22:10:28.493567: Average global foreground Dice: [0.7967, 0.9472, 0.9607, 0.323] 
2022-04-11 22:10:28.494960: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:10:29.066099: lr: 0.007142 
2022-04-11 22:10:29.067385: saving scheduled checkpoint file... 
2022-04-11 22:10:29.144941: saving checkpoint... 
2022-04-11 22:10:33.820069: done, saving took 4.75 seconds 
2022-04-11 22:10:33.838538: done 
2022-04-11 22:10:33.840057: This epoch took 128.462511 s
 
2022-04-11 22:10:33.841403: svmem(total=99932160000, available=79818129408, percent=20.1, used=16107581440, free=51452432384, active=40366522368, inactive=6278914048, buffers=18841600, cached=32353304576, shared=2190577664, slab=759455744) 
2022-04-11 22:10:33.842265: 
epoch:  156 
2022-04-11 22:12:22.489950: train loss : -0.5154 
2022-04-11 22:12:34.566956: validation loss: -0.4925 
2022-04-11 22:12:34.569861: Average global foreground Dice: [0.7849, 0.9313, 0.958, 0.3313] 
2022-04-11 22:12:34.571048: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:12:35.490834: lr: 0.007123 
2022-04-11 22:12:35.492445: saving scheduled checkpoint file... 
2022-04-11 22:12:35.597922: saving checkpoint... 
2022-04-11 22:12:40.269949: done, saving took 4.78 seconds 
2022-04-11 22:12:40.288609: done 
2022-04-11 22:12:40.289613: This epoch took 126.446492 s
 
2022-04-11 22:12:40.290648: svmem(total=99932160000, available=79954690048, percent=20.0, used=15970820096, free=51609726976, active=40043347968, inactive=6447009792, buffers=19525632, cached=32332087296, shared=2190729216, slab=759513088) 
2022-04-11 22:12:40.291348: 
epoch:  157 
2022-04-11 22:14:30.119114: train loss : -0.5210 
2022-04-11 22:14:42.376108: validation loss: -0.5245 
2022-04-11 22:14:42.380383: Average global foreground Dice: [0.7816, 0.9337, 0.972, 0.391] 
2022-04-11 22:14:42.381720: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:14:43.288635: lr: 0.007105 
2022-04-11 22:14:43.289937: saving scheduled checkpoint file... 
2022-04-11 22:14:43.381496: saving checkpoint... 
2022-04-11 22:14:47.902708: done, saving took 4.61 seconds 
2022-04-11 22:14:47.922432: done 
2022-04-11 22:14:47.924005: This epoch took 127.631892 s
 
2022-04-11 22:14:47.925265: svmem(total=99932160000, available=79745462272, percent=20.2, used=16180289536, free=51347263488, active=40414724096, inactive=6334988288, buffers=19558400, cached=32385048576, shared=2190532608, slab=759640064) 
2022-04-11 22:14:47.926224: 
epoch:  158 
2022-04-11 22:16:37.466659: train loss : -0.5142 
2022-04-11 22:16:48.447760: validation loss: -0.4753 
2022-04-11 22:16:48.461305: Average global foreground Dice: [0.807, 0.9197, 0.9342, 0.3691] 
2022-04-11 22:16:48.462517: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:16:49.014259: lr: 0.007086 
2022-04-11 22:16:49.016161: saving scheduled checkpoint file... 
2022-04-11 22:16:49.134419: saving checkpoint... 
2022-04-11 22:16:53.575252: done, saving took 4.56 seconds 
2022-04-11 22:16:53.599000: done 
2022-04-11 22:16:53.600752: This epoch took 125.673627 s
 
2022-04-11 22:16:53.602262: svmem(total=99932160000, available=79876792320, percent=20.1, used=16048726016, free=51470651392, active=40303345664, inactive=6319022080, buffers=19734528, cached=32393048064, shared=2190729216, slab=757878784) 
2022-04-11 22:16:53.603118: 
epoch:  159 
2022-04-11 22:18:42.797202: train loss : -0.4884 
2022-04-11 22:18:55.011771: validation loss: -0.5361 
2022-04-11 22:18:55.014859: Average global foreground Dice: [0.8058, 0.9453, 0.9602, 0.3767] 
2022-04-11 22:18:55.025554: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:18:55.594632: lr: 0.007067 
2022-04-11 22:18:55.595968: saving scheduled checkpoint file... 
2022-04-11 22:18:55.712925: saving checkpoint... 
2022-04-11 22:19:01.916435: done, saving took 6.32 seconds 
2022-04-11 22:19:02.001060: done 
2022-04-11 22:19:02.004079: This epoch took 128.400042 s
 
2022-04-11 22:19:02.007384: svmem(total=99932160000, available=79576432640, percent=20.4, used=16338640896, free=51152797696, active=40599863296, inactive=6346833920, buffers=19771392, cached=32420950016, shared=2201137152, slab=756981760) 
2022-04-11 22:19:02.008734: 
epoch:  160 
2022-04-11 22:20:52.729971: train loss : -0.5591 
2022-04-11 22:21:05.935733: validation loss: -0.5635 
2022-04-11 22:21:05.938838: Average global foreground Dice: [0.7958, 0.9472, 0.9691, 0.3826] 
2022-04-11 22:21:05.939768: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:21:06.743979: lr: 0.007049 
2022-04-11 22:21:06.745795: saving scheduled checkpoint file... 
2022-04-11 22:21:06.861084: saving checkpoint... 
2022-04-11 22:21:11.367598: done, saving took 4.62 seconds 
2022-04-11 22:21:11.393902: done 
2022-04-11 22:21:11.395101: This epoch took 129.385217 s
 
2022-04-11 22:21:11.396561: svmem(total=99932160000, available=79809716224, percent=20.1, used=16105373696, free=51368628224, active=40365219840, inactive=6363721728, buffers=19828736, cached=32438329344, shared=2201104384, slab=757133312) 
2022-04-11 22:21:11.397700: 
epoch:  161 
2022-04-11 22:23:01.543208: train loss : -0.5758 
2022-04-11 22:23:13.879993: validation loss: -0.5916 
2022-04-11 22:23:13.910783: Average global foreground Dice: [0.8487, 0.9565, 0.9749, 0.3796] 
2022-04-11 22:23:13.912293: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:23:14.719367: lr: 0.00703 
2022-04-11 22:23:14.720573: saving scheduled checkpoint file... 
2022-04-11 22:23:14.812060: saving checkpoint... 
2022-04-11 22:23:19.155065: done, saving took 4.43 seconds 
2022-04-11 22:23:19.171465: done 
2022-04-11 22:23:19.173055: This epoch took 127.774516 s
 
2022-04-11 22:23:19.174165: svmem(total=99932160000, available=79719292928, percent=20.2, used=16206151680, free=51270483968, active=40471048192, inactive=6361034752, buffers=19877888, cached=32435646464, shared=2190729216, slab=757825536) 
2022-04-11 22:23:19.175113: 
epoch:  162 
2022-04-11 22:25:08.856876: train loss : -0.5850 
2022-04-11 22:25:20.773796: validation loss: -0.5790 
2022-04-11 22:25:20.784052: Average global foreground Dice: [0.7906, 0.9528, 0.9781, 0.3781] 
2022-04-11 22:25:20.785005: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:25:21.601505: lr: 0.007011 
2022-04-11 22:25:21.602922: saving scheduled checkpoint file... 
2022-04-11 22:25:21.651506: saving checkpoint... 
2022-04-11 22:25:26.152475: done, saving took 4.55 seconds 
2022-04-11 22:25:26.168302: done 
2022-04-11 22:25:26.216529: saving checkpoint... 
2022-04-11 22:25:30.669146: done, saving took 4.50 seconds 
2022-04-11 22:25:30.684761: This epoch took 131.508678 s
 
2022-04-11 22:25:30.685907: svmem(total=99932160000, available=79673577472, percent=20.3, used=16252121088, free=51207168000, active=40511492096, inactive=6378287104, buffers=20054016, cached=32452816896, shared=2190540800, slab=757653504) 
2022-04-11 22:25:30.686785: 
epoch:  163 
2022-04-11 22:27:18.600034: train loss : -0.5803 
2022-04-11 22:27:30.271957: validation loss: -0.5679 
2022-04-11 22:27:30.274753: Average global foreground Dice: [0.8169, 0.9454, 0.967, 0.3682] 
2022-04-11 22:27:30.275718: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:27:30.911872: lr: 0.006992 
2022-04-11 22:27:30.913210: saving scheduled checkpoint file... 
2022-04-11 22:27:30.961520: saving checkpoint... 
2022-04-11 22:27:35.571745: done, saving took 4.66 seconds 
2022-04-11 22:27:35.591222: done 
2022-04-11 22:27:35.655243: saving checkpoint... 
2022-04-11 22:27:40.293594: done, saving took 4.70 seconds 
2022-04-11 22:27:40.311021: This epoch took 129.623423 s
 
2022-04-11 22:27:40.312521: svmem(total=99932160000, available=79585431552, percent=20.4, used=16339959808, free=51101532160, active=40601849856, inactive=6395834368, buffers=20094976, cached=32470573056, shared=2190729216, slab=757481472) 
2022-04-11 22:27:40.313636: 
epoch:  164 
2022-04-11 22:29:31.658018: train loss : -0.5751 
2022-04-11 22:29:43.611442: validation loss: -0.5745 
2022-04-11 22:29:43.622701: Average global foreground Dice: [0.8332, 0.9426, 0.971, 0.3643] 
2022-04-11 22:29:43.623995: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:29:44.570604: lr: 0.006974 
2022-04-11 22:29:44.571747: saving scheduled checkpoint file... 
2022-04-11 22:29:44.685369: saving checkpoint... 
2022-04-11 22:29:49.293375: done, saving took 4.72 seconds 
2022-04-11 22:29:49.312799: done 
2022-04-11 22:29:49.395255: saving checkpoint... 
2022-04-11 22:29:53.950311: done, saving took 4.64 seconds 
2022-04-11 22:29:53.967930: This epoch took 133.653532 s
 
2022-04-11 22:29:53.969499: svmem(total=99932160000, available=79873986560, percent=20.1, used=16051228672, free=51385012224, active=40186576896, inactive=6525267968, buffers=20127744, cached=32475791360, shared=2190807040, slab=757567488) 
2022-04-11 22:29:53.970365: 
epoch:  165 
2022-04-11 22:31:43.082651: train loss : -0.5817 
2022-04-11 22:31:54.754317: validation loss: -0.5731 
2022-04-11 22:31:54.766212: Average global foreground Dice: [0.8295, 0.9487, 0.9733, 0.3684] 
2022-04-11 22:31:54.767199: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:31:55.541223: lr: 0.006955 
2022-04-11 22:31:55.542616: saving scheduled checkpoint file... 
2022-04-11 22:31:55.652633: saving checkpoint... 
2022-04-11 22:32:00.316596: done, saving took 4.77 seconds 
2022-04-11 22:32:00.334591: done 
2022-04-11 22:32:00.412873: saving checkpoint... 
2022-04-11 22:32:04.920712: done, saving took 4.58 seconds 
2022-04-11 22:32:04.939019: This epoch took 130.967811 s
 
2022-04-11 22:32:04.940996: svmem(total=99932160000, available=79793979392, percent=20.2, used=16131346432, free=51287543808, active=40343400448, inactive=6467604480, buffers=20168704, cached=32493101056, shared=2190729216, slab=757784576) 
2022-04-11 22:32:04.942007: 
epoch:  166 
2022-04-11 22:33:53.563893: train loss : -0.5858 
2022-04-11 22:34:05.047827: validation loss: -0.5819 
2022-04-11 22:34:05.050972: Average global foreground Dice: [0.8248, 0.9471, 0.9721, 0.3733] 
2022-04-11 22:34:05.051953: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:34:05.908710: lr: 0.006936 
2022-04-11 22:34:05.910201: saving scheduled checkpoint file... 
2022-04-11 22:34:06.043923: saving checkpoint... 
2022-04-11 22:34:10.531709: done, saving took 4.62 seconds 
2022-04-11 22:34:10.564046: done 
2022-04-11 22:34:10.674344: saving checkpoint... 
2022-04-11 22:34:15.216959: done, saving took 4.65 seconds 
2022-04-11 22:34:15.303514: This epoch took 130.360604 s
 
2022-04-11 22:34:15.305053: svmem(total=99932160000, available=79835250688, percent=20.1, used=16089853952, free=51311251456, active=40339484672, inactive=6444187648, buffers=20217856, cached=32510836736, shared=2191036416, slab=758468608) 
2022-04-11 22:34:15.306174: 
epoch:  167 
2022-04-11 22:36:05.921435: train loss : -0.5701 
2022-04-11 22:36:18.891958: validation loss: -0.5545 
2022-04-11 22:36:18.896946: Average global foreground Dice: [0.8226, 0.9404, 0.9708, 0.3575] 
2022-04-11 22:36:18.898246: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:36:19.901509: lr: 0.006918 
2022-04-11 22:36:19.902800: saving scheduled checkpoint file... 
2022-04-11 22:36:19.982382: saving checkpoint... 
2022-04-11 22:36:24.447388: done, saving took 4.54 seconds 
2022-04-11 22:36:24.464293: done 
2022-04-11 22:36:24.531586: saving checkpoint... 
2022-04-11 22:36:29.010910: done, saving took 4.55 seconds 
2022-04-11 22:36:29.023585: This epoch took 133.716740 s
 
2022-04-11 22:36:29.025190: svmem(total=99932160000, available=79843450880, percent=20.1, used=16071450624, free=51302023168, active=40318496768, inactive=6470946816, buffers=20336640, cached=32538349568, shared=2201137152, slab=758689792) 
2022-04-11 22:36:29.026036: 
epoch:  168 
2022-04-11 22:38:18.420911: train loss : -0.5414 
2022-04-11 22:38:29.568912: validation loss: -0.5476 
2022-04-11 22:38:29.571261: Average global foreground Dice: [0.813, 0.9465, 0.9681, 0.3401] 
2022-04-11 22:38:29.572443: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:38:30.532898: lr: 0.006899 
2022-04-11 22:38:30.534491: saving scheduled checkpoint file... 
2022-04-11 22:38:30.638347: saving checkpoint... 
2022-04-11 22:38:35.215866: done, saving took 4.68 seconds 
2022-04-11 22:38:35.232360: done 
2022-04-11 22:38:35.233728: This epoch took 126.206765 s
 
2022-04-11 22:38:35.235046: svmem(total=99932160000, available=79891054592, percent=20.1, used=16034840576, free=51341418496, active=40290447360, inactive=6460588032, buffers=20377600, cached=32535523328, shared=2190618624, slab=759406592) 
2022-04-11 22:38:35.236299: 
epoch:  169 
2022-04-11 22:40:24.706061: train loss : -0.5398 
2022-04-11 22:40:36.850244: validation loss: -0.5378 
2022-04-11 22:40:36.856812: Average global foreground Dice: [0.7461, 0.9391, 0.9747, 0.4017] 
2022-04-11 22:40:36.857835: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:40:37.742360: lr: 0.00688 
2022-04-11 22:40:37.744125: saving scheduled checkpoint file... 
2022-04-11 22:40:37.865037: saving checkpoint... 
2022-04-11 22:40:42.431711: done, saving took 4.69 seconds 
2022-04-11 22:40:42.456873: done 
2022-04-11 22:40:42.458243: This epoch took 127.220768 s
 
2022-04-11 22:40:42.459609: svmem(total=99932160000, available=79954989056, percent=20.0, used=15959678976, free=51388268544, active=40215855104, inactive=6488670208, buffers=20418560, cached=32563793920, shared=2201325568, slab=758824960) 
2022-04-11 22:40:42.460564: 
epoch:  170 
2022-04-11 22:42:32.098945: train loss : -0.5210 
2022-04-11 22:42:44.252450: validation loss: -0.5580 
2022-04-11 22:42:44.254871: Average global foreground Dice: [0.813, 0.9454, 0.971, 0.3706] 
2022-04-11 22:42:44.255785: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:42:45.108002: lr: 0.006861 
2022-04-11 22:42:45.109536: saving scheduled checkpoint file... 
2022-04-11 22:42:45.235554: saving checkpoint... 
2022-04-11 22:42:49.742448: done, saving took 4.63 seconds 
2022-04-11 22:42:49.789857: done 
2022-04-11 22:42:49.790824: This epoch took 127.329450 s
 
2022-04-11 22:42:49.792396: svmem(total=99932160000, available=79968178176, percent=20.0, used=15965581312, free=51384025088, active=40226611200, inactive=6486892544, buffers=20451328, cached=32562102272, shared=2182217728, slab=759578624) 
2022-04-11 22:42:49.793729: 
epoch:  171 
2022-04-11 22:44:38.545061: train loss : -0.5602 
2022-04-11 22:44:50.179420: validation loss: -0.5656 
2022-04-11 22:44:50.181350: Average global foreground Dice: [0.8108, 0.9423, 0.9753, 0.3812] 
2022-04-11 22:44:50.182535: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:44:51.087414: lr: 0.006842 
2022-04-11 22:44:51.089079: saving scheduled checkpoint file... 
2022-04-11 22:44:51.232997: saving checkpoint... 
2022-04-11 22:44:55.647803: done, saving took 4.56 seconds 
2022-04-11 22:44:55.684258: done 
2022-04-11 22:44:55.685282: This epoch took 125.890785 s
 
2022-04-11 22:44:55.687201: svmem(total=99932160000, available=79856861184, percent=20.1, used=16068210688, free=51265941504, active=40326184960, inactive=6502408192, buffers=20484096, cached=32577523712, shared=2190925824, slab=759701504) 
2022-04-11 22:44:55.688145: 
epoch:  172 
2022-04-11 22:46:46.123308: train loss : -0.5709 
2022-04-11 22:46:59.027131: validation loss: -0.5886 
2022-04-11 22:46:59.041874: Average global foreground Dice: [0.8278, 0.9434, 0.9683, 0.3802] 
2022-04-11 22:46:59.043172: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:46:59.812705: lr: 0.006824 
2022-04-11 22:46:59.814095: saving scheduled checkpoint file... 
2022-04-11 22:46:59.938413: saving checkpoint... 
2022-04-11 22:47:04.383800: done, saving took 4.57 seconds 
2022-04-11 22:47:04.437437: done 
2022-04-11 22:47:04.585348: saving checkpoint... 
2022-04-11 22:47:09.051410: done, saving took 4.61 seconds 
2022-04-11 22:47:09.076134: This epoch took 133.387194 s
 
2022-04-11 22:47:09.077392: svmem(total=99932160000, available=79749222400, percent=20.2, used=16175951872, free=51139624960, active=40435335168, inactive=6520745984, buffers=20664320, cached=32595918848, shared=2190839808, slab=757551104) 
2022-04-11 22:47:09.078893: 
epoch:  173 
2022-04-11 22:48:58.128397: train loss : -0.5952 
2022-04-11 22:49:09.747228: validation loss: -0.6158 
2022-04-11 22:49:09.758005: Average global foreground Dice: [0.8399, 0.9588, 0.9764, 0.3776] 
2022-04-11 22:49:09.759326: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:49:10.874864: lr: 0.006805 
2022-04-11 22:49:10.876367: saving scheduled checkpoint file... 
2022-04-11 22:49:10.959740: saving checkpoint... 
2022-04-11 22:49:15.564549: done, saving took 4.69 seconds 
2022-04-11 22:49:15.579941: done 
2022-04-11 22:49:15.628139: saving checkpoint... 
2022-04-11 22:49:20.522845: done, saving took 4.94 seconds 
2022-04-11 22:49:20.538498: This epoch took 131.458250 s
 
2022-04-11 22:49:20.539711: svmem(total=99932160000, available=79766126592, percent=20.2, used=16153350144, free=51139010560, active=40413528064, inactive=6543872000, buffers=20713472, cached=32619085824, shared=2196484096, slab=758128640) 
2022-04-11 22:49:20.541086: 
epoch:  174 
2022-04-11 22:51:08.662921: train loss : -0.5956 
2022-04-11 22:51:20.623443: validation loss: -0.6091 
2022-04-11 22:51:20.629925: Average global foreground Dice: [0.8485, 0.9475, 0.9782, 0.3676] 
2022-04-11 22:51:20.631495: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:51:21.648610: lr: 0.006786 
2022-04-11 22:51:21.650671: saving scheduled checkpoint file... 
2022-04-11 22:51:21.729663: saving checkpoint... 
2022-04-11 22:51:26.444286: done, saving took 4.79 seconds 
2022-04-11 22:51:26.462580: done 
2022-04-11 22:51:26.529721: saving checkpoint... 
2022-04-11 22:51:31.065037: done, saving took 4.60 seconds 
2022-04-11 22:51:31.129429: This epoch took 130.587292 s
 
2022-04-11 22:51:31.133139: svmem(total=99932160000, available=79675289600, percent=20.3, used=16250032128, free=51040428032, active=40509222912, inactive=6545620992, buffers=20754432, cached=32620945408, shared=2190618624, slab=757465088) 
2022-04-11 22:51:31.142701: 
epoch:  175 
2022-04-11 22:53:19.930079: train loss : -0.5914 
2022-04-11 22:53:31.631160: validation loss: -0.5768 
2022-04-11 22:53:31.633316: Average global foreground Dice: [0.8446, 0.9434, 0.9711, 0.3983] 
2022-04-11 22:53:31.634590: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:53:32.505407: lr: 0.006767 
2022-04-11 22:53:32.506832: saving scheduled checkpoint file... 
2022-04-11 22:53:32.600129: saving checkpoint... 
2022-04-11 22:53:37.111646: done, saving took 4.60 seconds 
2022-04-11 22:53:37.180195: done 
2022-04-11 22:53:37.260556: saving checkpoint... 
2022-04-11 22:53:41.955265: done, saving took 4.77 seconds 
2022-04-11 22:53:42.017157: This epoch took 130.873059 s
 
2022-04-11 22:53:42.018217: svmem(total=99932160000, available=79801569280, percent=20.1, used=16133332992, free=51149295616, active=40394772480, inactive=6553350144, buffers=20787200, cached=32628744192, shared=2181005312, slab=759193600) 
2022-04-11 22:53:42.019024: 
epoch:  176 
2022-04-11 22:55:31.293035: train loss : -0.5991 
2022-04-11 22:55:43.452085: validation loss: -0.6131 
2022-04-11 22:55:43.454427: Average global foreground Dice: [0.8377, 0.952, 0.9735, 0.3942] 
2022-04-11 22:55:43.456042: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:55:44.094260: lr: 0.006749 
2022-04-11 22:55:44.095920: saving scheduled checkpoint file... 
2022-04-11 22:55:44.206180: saving checkpoint... 
2022-04-11 22:55:48.598253: done, saving took 4.50 seconds 
2022-04-11 22:55:48.615305: done 
2022-04-11 22:55:48.710682: saving checkpoint... 
2022-04-11 22:55:53.287531: done, saving took 4.67 seconds 
2022-04-11 22:55:53.307950: This epoch took 131.287908 s
 
2022-04-11 22:55:53.309542: svmem(total=99932160000, available=79864475648, percent=20.1, used=16062910464, free=51194806272, active=40325251072, inactive=6578016256, buffers=20836352, cached=32653606912, shared=2188361728, slab=759386112) 
2022-04-11 22:55:53.310378: 
epoch:  177 
2022-04-11 22:57:43.996433: train loss : -0.5946 
2022-04-11 22:57:56.296021: validation loss: -0.5838 
2022-04-11 22:57:56.298343: Average global foreground Dice: [0.8389, 0.9415, 0.9551, 0.4017] 
2022-04-11 22:57:56.299673: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 22:57:57.141580: lr: 0.00673 
2022-04-11 22:57:57.143008: saving scheduled checkpoint file... 
2022-04-11 22:57:57.247350: saving checkpoint... 
2022-04-11 22:58:02.167660: done, saving took 5.02 seconds 
2022-04-11 22:58:02.190382: done 
2022-04-11 22:58:02.309238: saving checkpoint... 
2022-04-11 22:58:06.930831: done, saving took 4.74 seconds 
2022-04-11 22:58:06.952436: This epoch took 133.641324 s
 
2022-04-11 22:58:06.954049: svmem(total=99932160000, available=79964405760, percent=20.0, used=15960633344, free=51277201408, active=40227844096, inactive=6597947392, buffers=20869120, cached=32673456128, shared=2190807040, slab=759087104) 
2022-04-11 22:58:06.955372: 
epoch:  178 
2022-04-11 22:59:55.002808: train loss : -0.5891 
2022-04-11 23:00:07.296663: validation loss: -0.6144 
2022-04-11 23:00:07.303736: Average global foreground Dice: [0.8523, 0.948, 0.9725, 0.3993] 
2022-04-11 23:00:07.305125: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:00:08.191982: lr: 0.006711 
2022-04-11 23:00:08.193569: saving scheduled checkpoint file... 
2022-04-11 23:00:08.346885: saving checkpoint... 
2022-04-11 23:00:12.824762: done, saving took 4.63 seconds 
2022-04-11 23:00:12.847964: done 
2022-04-11 23:00:12.976907: saving checkpoint... 
2022-04-11 23:00:17.444900: done, saving took 4.60 seconds 
2022-04-11 23:00:17.463736: This epoch took 130.507593 s
 
2022-04-11 23:00:17.465079: svmem(total=99932160000, available=79838011392, percent=20.1, used=16086986752, free=51142922240, active=40354148352, inactive=6605590528, buffers=20926464, cached=32681324544, shared=2190729216, slab=759275520) 
2022-04-11 23:00:17.465760: 
epoch:  179 
2022-04-11 23:02:07.932637: train loss : -0.5852 
2022-04-11 23:02:20.684069: validation loss: -0.6223 
2022-04-11 23:02:20.686062: Average global foreground Dice: [0.8493, 0.958, 0.9757, 0.4109] 
2022-04-11 23:02:20.687513: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:02:21.344760: lr: 0.006692 
2022-04-11 23:02:21.347400: saving scheduled checkpoint file... 
2022-04-11 23:02:21.433899: saving checkpoint... 
2022-04-11 23:02:27.239336: done, saving took 5.89 seconds 
2022-04-11 23:02:27.298839: done 
2022-04-11 23:02:27.369163: saving checkpoint... 
2022-04-11 23:02:33.297094: done, saving took 5.99 seconds 
2022-04-11 23:02:33.379999: This epoch took 135.913090 s
 
2022-04-11 23:02:33.381358: svmem(total=99932160000, available=80283987968, percent=19.7, used=15727632384, free=51570245632, active=39992537088, inactive=6536392704, buffers=21688320, cached=32612593664, shared=2103926784, slab=759365632) 
2022-04-11 23:02:33.382070: 
epoch:  180 
2022-04-11 23:04:23.656921: train loss : -0.6020 
2022-04-11 23:04:34.866463: validation loss: -0.5901 
2022-04-11 23:04:34.868896: Average global foreground Dice: [0.8391, 0.9464, 0.9778, 0.382] 
2022-04-11 23:04:34.870533: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:04:35.817445: lr: 0.006673 
2022-04-11 23:04:35.819035: saving scheduled checkpoint file... 
2022-04-11 23:04:35.924738: saving checkpoint... 
2022-04-11 23:04:40.703433: done, saving took 4.88 seconds 
2022-04-11 23:04:40.726191: done 
2022-04-11 23:04:40.804530: saving checkpoint... 
2022-04-11 23:04:45.573230: done, saving took 4.85 seconds 
2022-04-11 23:04:45.594944: This epoch took 132.212201 s
 
2022-04-11 23:04:45.597053: svmem(total=99932160000, available=80117460992, percent=19.8, used=15821312000, free=51386314752, active=40088563712, inactive=6626570240, buffers=21721088, cached=32702812160, shared=2176741376, slab=760512512) 
2022-04-11 23:04:45.598436: 
epoch:  181 
2022-04-11 23:06:33.702749: train loss : -0.5972 
2022-04-11 23:06:46.262912: validation loss: -0.6015 
2022-04-11 23:06:46.269816: Average global foreground Dice: [0.8566, 0.9506, 0.9688, 0.3534] 
2022-04-11 23:06:46.271267: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:06:47.154359: lr: 0.006654 
2022-04-11 23:06:47.155478: saving scheduled checkpoint file... 
2022-04-11 23:06:47.261892: saving checkpoint... 
2022-04-11 23:06:52.175759: done, saving took 5.02 seconds 
2022-04-11 23:06:52.220801: done 
2022-04-11 23:06:52.222054: This epoch took 126.622738 s
 
2022-04-11 23:06:52.223093: svmem(total=99932160000, available=80013643776, percent=19.9, used=15925121024, free=51274604544, active=40192503808, inactive=6634360832, buffers=21774336, cached=32710660096, shared=2176741376, slab=760168448) 
2022-04-11 23:06:52.223952: 
epoch:  182 
2022-04-11 23:08:40.991208: train loss : -0.5895 
2022-04-11 23:08:52.290184: validation loss: -0.6264 
2022-04-11 23:08:52.292328: Average global foreground Dice: [0.8588, 0.9551, 0.9749, 0.3973] 
2022-04-11 23:08:52.293143: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:08:53.144070: lr: 0.006636 
2022-04-11 23:08:53.145626: saving scheduled checkpoint file... 
2022-04-11 23:08:53.257817: saving checkpoint... 
2022-04-11 23:08:57.716074: done, saving took 4.57 seconds 
2022-04-11 23:08:57.737710: done 
2022-04-11 23:08:57.849022: saving checkpoint... 
2022-04-11 23:09:04.770605: done, saving took 7.03 seconds 
2022-04-11 23:09:05.208392: This epoch took 132.983626 s
 
2022-04-11 23:09:05.471603: svmem(total=99932160000, available=79816593408, percent=20.1, used=16122032128, free=51060088832, active=40360026112, inactive=6651699200, buffers=21811200, cached=32728227840, shared=2176741376, slab=774520832) 
2022-04-11 23:09:05.623075: 
epoch:  183 
2022-04-11 23:10:52.715427: train loss : -0.6096 
2022-04-11 23:11:03.196646: validation loss: -0.5883 
2022-04-11 23:11:03.202596: Average global foreground Dice: [0.8359, 0.952, 0.9707, 0.3896] 
2022-04-11 23:11:03.204396: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:11:04.039034: lr: 0.006617 
2022-04-11 23:11:04.040576: saving scheduled checkpoint file... 
2022-04-11 23:11:04.158921: saving checkpoint... 
2022-04-11 23:11:08.577749: done, saving took 4.54 seconds 
2022-04-11 23:11:08.602775: done 
2022-04-11 23:11:08.740146: saving checkpoint... 
2022-04-11 23:11:16.443123: done, saving took 7.84 seconds 
2022-04-11 23:11:16.766095: This epoch took 131.009314 s
 
2022-04-11 23:11:16.855270: svmem(total=99932160000, available=79936290816, percent=20.0, used=16002297856, free=51162460160, active=40247689216, inactive=6668967936, buffers=21860352, cached=32745541632, shared=2176741376, slab=774422528) 
2022-04-11 23:11:16.918476: 
epoch:  184 
2022-04-11 23:13:06.735709: train loss : -0.5968 
2022-04-11 23:13:19.118067: validation loss: -0.5798 
2022-04-11 23:13:19.275539: Average global foreground Dice: [0.8422, 0.9463, 0.9689, 0.3858] 
2022-04-11 23:13:19.408662: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:13:20.415036: lr: 0.006598 
2022-04-11 23:13:20.539846: saving scheduled checkpoint file... 
2022-04-11 23:13:20.815356: saving checkpoint... 
2022-04-11 23:13:25.909361: done, saving took 5.22 seconds 
2022-04-11 23:13:25.936913: done 
2022-04-11 23:13:26.071933: saving checkpoint... 
2022-04-11 23:13:30.593521: done, saving took 4.66 seconds 
2022-04-11 23:13:30.617893: This epoch took 133.692499 s
 
2022-04-11 23:13:30.619317: svmem(total=99932160000, available=79849295872, percent=20.1, used=16075419648, free=51067682816, active=40338345984, inactive=6690738176, buffers=21893120, cached=32767164416, shared=2190729216, slab=761294848) 
2022-04-11 23:13:30.620147: 
epoch:  185 
2022-04-11 23:15:20.306358: train loss : -0.5727 
2022-04-11 23:15:32.257874: validation loss: -0.5660 
2022-04-11 23:15:32.261801: Average global foreground Dice: [0.8384, 0.9447, 0.9665, 0.3671] 
2022-04-11 23:15:32.262980: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:15:33.255615: lr: 0.006579 
2022-04-11 23:15:33.257142: saving scheduled checkpoint file... 
2022-04-11 23:15:33.354829: saving checkpoint... 
2022-04-11 23:15:38.418336: done, saving took 5.16 seconds 
2022-04-11 23:15:38.476930: done 
2022-04-11 23:15:38.478446: This epoch took 127.857579 s
 
2022-04-11 23:15:38.479754: svmem(total=99932160000, available=79889563648, percent=20.1, used=16035475456, free=51090436096, active=40298000384, inactive=6708101120, buffers=21942272, cached=32784306176, shared=2190278656, slab=761704448) 
2022-04-11 23:15:38.480525: 
epoch:  186 
2022-04-11 23:17:27.680986: train loss : -0.5354 
2022-04-11 23:17:39.558127: validation loss: -0.5515 
2022-04-11 23:17:39.564100: Average global foreground Dice: [0.8236, 0.938, 0.9648, 0.3702] 
2022-04-11 23:17:39.565167: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:17:40.219757: lr: 0.00656 
2022-04-11 23:17:40.220983: saving scheduled checkpoint file... 
2022-04-11 23:17:40.300882: saving checkpoint... 
2022-04-11 23:17:45.257381: done, saving took 5.04 seconds 
2022-04-11 23:17:45.274655: done 
2022-04-11 23:17:45.276091: This epoch took 126.794657 s
 
2022-04-11 23:17:45.277234: svmem(total=99932160000, available=79842533376, percent=20.1, used=16082067456, free=51025965056, active=40344690688, inactive=6725488640, buffers=21983232, cached=32802144256, shared=2190729216, slab=761061376) 
2022-04-11 23:17:45.278203: 
epoch:  187 
2022-04-11 23:19:34.328397: train loss : -0.4923 
2022-04-11 23:19:46.698898: validation loss: -0.4999 
2022-04-11 23:19:46.711259: Average global foreground Dice: [0.7874, 0.9345, 0.9603, 0.3539] 
2022-04-11 23:19:46.712857: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:19:47.663561: lr: 0.006541 
2022-04-11 23:19:47.665255: saving scheduled checkpoint file... 
2022-04-11 23:19:47.743623: saving checkpoint... 
2022-04-11 23:19:52.641235: done, saving took 4.98 seconds 
2022-04-11 23:19:52.734541: done 
2022-04-11 23:19:52.735942: This epoch took 127.456606 s
 
2022-04-11 23:19:52.736845: svmem(total=99932160000, available=79924985856, percent=20.0, used=16008540160, free=51090907136, active=40274788352, inactive=6733950976, buffers=22016000, cached=32810696704, shared=2181808128, slab=759771136) 
2022-04-11 23:19:52.737654: 
epoch:  188 
2022-04-11 23:21:40.858785: train loss : -0.4554 
2022-04-11 23:21:52.301132: validation loss: -0.4767 
2022-04-11 23:21:52.330389: Average global foreground Dice: [0.745, 0.931, 0.9594, 0.3522] 
2022-04-11 23:21:52.335810: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:21:53.188995: lr: 0.006522 
2022-04-11 23:21:53.190978: saving scheduled checkpoint file... 
2022-04-11 23:21:53.317426: saving checkpoint... 
2022-04-11 23:21:58.231164: done, saving took 5.04 seconds 
2022-04-11 23:21:58.256962: done 
2022-04-11 23:21:58.258168: This epoch took 125.519712 s
 
2022-04-11 23:21:58.259601: svmem(total=99932160000, available=79795695616, percent=20.2, used=16128851968, free=50953777152, active=40387088384, inactive=6750601216, buffers=22073344, cached=32827457536, shared=2190729216, slab=760504320) 
2022-04-11 23:21:58.260918: 
epoch:  189 
2022-04-11 23:23:47.041129: train loss : -0.4368 
2022-04-11 23:23:58.775552: validation loss: -0.4875 
2022-04-11 23:23:58.778331: Average global foreground Dice: [0.7639, 0.9227, 0.9455, 0.3492] 
2022-04-11 23:23:58.779537: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:23:59.723619: lr: 0.006504 
2022-04-11 23:23:59.725396: saving scheduled checkpoint file... 
2022-04-11 23:23:59.875114: saving checkpoint... 
2022-04-11 23:24:04.432990: done, saving took 4.71 seconds 
2022-04-11 23:24:04.476658: done 
2022-04-11 23:24:04.480145: This epoch took 126.218519 s
 
2022-04-11 23:24:04.482011: svmem(total=99932160000, available=79917346816, percent=20.0, used=16007376896, free=51058999296, active=40266682368, inactive=6766772224, buffers=22106112, cached=32843677696, shared=2190540800, slab=760172544) 
2022-04-11 23:24:04.482975: 
epoch:  190 
2022-04-11 23:25:54.232673: train loss : -0.4463 
2022-04-11 23:26:05.519305: validation loss: -0.5402 
2022-04-11 23:26:05.529952: Average global foreground Dice: [0.7963, 0.9358, 0.966, 0.3976] 
2022-04-11 23:26:05.531336: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:26:06.449072: lr: 0.006485 
2022-04-11 23:26:06.465798: saving scheduled checkpoint file... 
2022-04-11 23:26:06.610456: saving checkpoint... 
2022-04-11 23:26:11.164933: done, saving took 4.70 seconds 
2022-04-11 23:26:11.220752: done 
2022-04-11 23:26:11.222020: This epoch took 126.737789 s
 
2022-04-11 23:26:11.223572: svmem(total=99932160000, available=79890960384, percent=20.1, used=16023101440, free=51015028736, active=40284225536, inactive=6794731520, buffers=22183936, cached=32871845888, shared=2201104384, slab=759947264) 
2022-04-11 23:26:11.224695: 
epoch:  191 
2022-04-11 23:28:01.112154: train loss : -0.5202 
2022-04-11 23:28:14.346628: validation loss: -0.5767 
2022-04-11 23:28:14.352335: Average global foreground Dice: [0.8161, 0.9371, 0.9702, 0.4017] 
2022-04-11 23:28:14.353405: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:28:15.322803: lr: 0.006466 
2022-04-11 23:28:15.324708: saving scheduled checkpoint file... 
2022-04-11 23:28:15.419011: saving checkpoint... 
2022-04-11 23:28:19.837852: done, saving took 4.51 seconds 
2022-04-11 23:28:19.857146: done 
2022-04-11 23:28:19.858686: This epoch took 128.633100 s
 
2022-04-11 23:28:19.859723: svmem(total=99932160000, available=79984631808, percent=20.0, used=15942602752, free=51100958720, active=40203259904, inactive=6789304320, buffers=22216704, cached=32866381824, shared=2187964416, slab=761090048) 
2022-04-11 23:28:19.861036: 
epoch:  192 
2022-04-11 23:30:08.287617: train loss : -0.5222 
2022-04-11 23:30:20.913932: validation loss: -0.5641 
2022-04-11 23:30:20.916136: Average global foreground Dice: [0.8126, 0.9421, 0.9651, 0.3599] 
2022-04-11 23:30:20.917113: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:30:21.621507: lr: 0.006447 
2022-04-11 23:30:21.622452: saving scheduled checkpoint file... 
2022-04-11 23:30:21.699116: saving checkpoint... 
2022-04-11 23:30:27.023731: done, saving took 5.40 seconds 
2022-04-11 23:30:27.107636: done 
2022-04-11 23:30:27.108632: This epoch took 127.243969 s
 
2022-04-11 23:30:27.109730: svmem(total=99932160000, available=79787028480, percent=20.2, used=16137355264, free=50885894144, active=40398548992, inactive=6809427968, buffers=22257664, cached=32886652928, shared=2190729216, slab=761069568) 
2022-04-11 23:30:27.111034: 
epoch:  193 
2022-04-11 23:32:16.173201: train loss : -0.5439 
2022-04-11 23:32:27.890886: validation loss: -0.5499 
2022-04-11 23:32:27.894004: Average global foreground Dice: [0.8432, 0.9476, 0.9655, 0.3391] 
2022-04-11 23:32:27.895687: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:32:28.633007: lr: 0.006428 
2022-04-11 23:32:28.634072: saving scheduled checkpoint file... 
2022-04-11 23:32:28.747773: saving checkpoint... 
2022-04-11 23:32:33.148256: done, saving took 4.51 seconds 
2022-04-11 23:32:33.175284: done 
2022-04-11 23:32:33.176703: This epoch took 126.064204 s
 
2022-04-11 23:32:33.177770: svmem(total=99932160000, available=79786385408, percent=20.2, used=16142483456, free=50867896320, active=40402157568, inactive=6822154240, buffers=22290432, cached=32899489792, shared=2186158080, slab=760410112) 
2022-04-11 23:32:33.178960: 
epoch:  194 
2022-04-11 23:34:22.589205: train loss : -0.5744 
2022-04-11 23:34:33.967233: validation loss: -0.5854 
2022-04-11 23:34:34.106102: Average global foreground Dice: [0.8645, 0.9468, 0.9715, 0.382] 
2022-04-11 23:34:34.230480: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:34:35.144540: lr: 0.006409 
2022-04-11 23:34:35.146128: saving scheduled checkpoint file... 
2022-04-11 23:34:35.271396: saving checkpoint... 
2022-04-11 23:34:39.956277: done, saving took 4.81 seconds 
2022-04-11 23:34:40.038889: done 
2022-04-11 23:34:40.040224: This epoch took 126.860286 s
 
2022-04-11 23:34:40.044290: svmem(total=99932160000, available=79761481728, percent=20.2, used=16162840576, free=50835111936, active=40423198720, inactive=6834483200, buffers=22323200, cached=32911884288, shared=2190729216, slab=760115200) 
2022-04-11 23:34:40.046091: 
epoch:  195 
2022-04-11 23:36:29.795094: train loss : -0.5619 
2022-04-11 23:36:41.377947: validation loss: -0.6096 
2022-04-11 23:36:41.518919: Average global foreground Dice: [0.8364, 0.9541, 0.9783, 0.3824] 
2022-04-11 23:36:41.740433: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:36:42.852017: lr: 0.00639 
2022-04-11 23:36:42.958296: saving scheduled checkpoint file... 
2022-04-11 23:36:43.246357: saving checkpoint... 
2022-04-11 23:36:47.747685: done, saving took 4.64 seconds 
2022-04-11 23:36:47.767087: done 
2022-04-11 23:36:47.768067: This epoch took 127.720740 s
 
2022-04-11 23:36:47.769120: svmem(total=99932160000, available=79687028736, percent=20.3, used=16237322240, free=50743250944, active=40504291328, inactive=6851837952, buffers=22380544, cached=32929206272, shared=2190807040, slab=760373248) 
2022-04-11 23:36:47.770055: 
epoch:  196 
2022-04-11 23:38:38.537756: train loss : -0.5941 
2022-04-11 23:38:51.290588: validation loss: -0.6221 
2022-04-11 23:38:51.418567: Average global foreground Dice: [0.8358, 0.9534, 0.9813, 0.41] 
2022-04-11 23:38:51.655977: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:38:52.653730: lr: 0.006371 
2022-04-11 23:38:52.813258: saving scheduled checkpoint file... 
2022-04-11 23:38:53.127025: saving checkpoint... 
2022-04-11 23:38:58.057476: done, saving took 5.06 seconds 
2022-04-11 23:38:58.134965: done 
2022-04-11 23:38:58.136370: This epoch took 130.365429 s
 
2022-04-11 23:38:58.137621: svmem(total=99932160000, available=79966961664, percent=20.0, used=15957262336, free=51005693952, active=40225222656, inactive=6869229568, buffers=22417408, cached=32946786304, shared=2190729216, slab=759373824) 
2022-04-11 23:38:58.138610: 
epoch:  197 
2022-04-11 23:40:49.814526: train loss : -0.5788 
2022-04-11 23:41:02.542315: validation loss: -0.6111 
2022-04-11 23:41:02.729391: Average global foreground Dice: [0.8697, 0.9499, 0.9734, 0.3713] 
2022-04-11 23:41:02.898805: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:41:03.787500: lr: 0.006352 
2022-04-11 23:41:03.789389: saving scheduled checkpoint file... 
2022-04-11 23:41:03.889694: saving checkpoint... 
2022-04-11 23:41:09.846952: done, saving took 6.06 seconds 
2022-04-11 23:41:09.885363: done 
2022-04-11 23:41:09.889549: This epoch took 131.750078 s
 
2022-04-11 23:41:09.890857: svmem(total=99932160000, available=79783927808, percent=20.2, used=16139943936, free=50814832640, active=40398393344, inactive=6877282304, buffers=22458368, cached=32954925056, shared=2191069184, slab=760180736) 
2022-04-11 23:41:09.891896: 
epoch:  198 
2022-04-11 23:42:59.367694: train loss : -0.5998 
2022-04-11 23:43:11.659643: validation loss: -0.6197 
2022-04-11 23:43:11.808589: Average global foreground Dice: [0.85, 0.9561, 0.9789, 0.3955] 
2022-04-11 23:43:11.959434: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:43:12.991538: lr: 0.006333 
2022-04-11 23:43:13.151040: saving scheduled checkpoint file... 
2022-04-11 23:43:13.440884: saving checkpoint... 
2022-04-11 23:43:20.087056: done, saving took 6.71 seconds 
2022-04-11 23:43:20.117535: done 
2022-04-11 23:43:20.124864: This epoch took 130.232195 s
 
2022-04-11 23:43:20.141538: svmem(total=99932160000, available=79550111744, percent=20.4, used=16366915584, free=50563674112, active=40636317696, inactive=6901194752, buffers=22507520, cached=32979062784, shared=2197745664, slab=761823232) 
2022-04-11 23:43:20.144855: 
epoch:  199 
2022-04-11 23:45:08.656923: train loss : -0.6078 
2022-04-11 23:45:21.159616: validation loss: -0.6020 
2022-04-11 23:45:21.365077: Average global foreground Dice: [0.8552, 0.9537, 0.9686, 0.4103] 
2022-04-11 23:45:21.595996: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:45:22.518090: lr: 0.006314 
2022-04-11 23:45:22.528170: saving scheduled checkpoint file... 
2022-04-11 23:45:22.642233: saving checkpoint... 
2022-04-11 23:45:31.576771: done, saving took 9.02 seconds 
2022-04-11 23:45:31.595795: done 
2022-04-11 23:45:31.597231: This epoch took 131.445199 s
 
2022-04-11 23:45:31.598917: svmem(total=99932160000, available=80032870400, percent=19.9, used=15905361920, free=51028910080, active=40166510592, inactive=6897762304, buffers=22548480, cached=32975339520, shared=2176741376, slab=763097088) 
2022-04-11 23:45:31.599680: 
epoch:  200 
2022-04-11 23:47:20.330709: train loss : -0.5874 
2022-04-11 23:47:31.388945: validation loss: -0.6032 
2022-04-11 23:47:31.398285: Average global foreground Dice: [0.8305, 0.9556, 0.9725, 0.3898] 
2022-04-11 23:47:31.399433: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:47:32.390140: lr: 0.006296 
2022-04-11 23:47:32.391907: saving scheduled checkpoint file... 
2022-04-11 23:47:32.499808: saving checkpoint... 
2022-04-11 23:47:41.261040: done, saving took 8.87 seconds 
2022-04-11 23:47:41.527975: done 
2022-04-11 23:47:41.529543: This epoch took 129.929164 s
 
2022-04-11 23:47:41.531562: svmem(total=99932160000, available=80006344704, percent=19.9, used=15931727872, free=50984988672, active=40187781120, inactive=6915072000, buffers=22585344, cached=32992858112, shared=2176741376, slab=774799360) 
2022-04-11 23:47:41.532435: 
epoch:  201 
2022-04-11 23:49:29.598964: train loss : -0.5877 
2022-04-11 23:49:41.602515: validation loss: -0.6271 
2022-04-11 23:49:41.608120: Average global foreground Dice: [0.8511, 0.9514, 0.9755, 0.404] 
2022-04-11 23:49:41.609501: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:49:42.483473: lr: 0.006277 
2022-04-11 23:49:42.485292: saving scheduled checkpoint file... 
2022-04-11 23:49:42.621281: saving checkpoint... 
2022-04-11 23:49:47.393307: done, saving took 4.91 seconds 
2022-04-11 23:49:47.415012: done 
2022-04-11 23:49:47.526998: saving checkpoint... 
2022-04-11 23:49:53.403706: done, saving took 5.99 seconds 
2022-04-11 23:49:53.779410: This epoch took 132.246168 s
 
2022-04-11 23:49:53.904595: svmem(total=99932160000, available=79833247744, percent=20.1, used=16104841216, free=50804011008, active=40355917824, inactive=6922821632, buffers=22618112, cached=33000689664, shared=2176741376, slab=776404992) 
2022-04-11 23:49:54.063810: 
epoch:  202 
2022-04-11 23:51:44.965688: train loss : -0.6184 
2022-04-11 23:51:56.827879: validation loss: -0.6290 
2022-04-11 23:51:56.833737: Average global foreground Dice: [0.8219, 0.9577, 0.9774, 0.4042] 
2022-04-11 23:51:56.834610: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:51:57.612228: lr: 0.006258 
2022-04-11 23:51:57.613545: saving scheduled checkpoint file... 
2022-04-11 23:51:57.731418: saving checkpoint... 
2022-04-11 23:52:02.246665: done, saving took 4.63 seconds 
2022-04-11 23:52:02.270444: done 
2022-04-11 23:52:02.394024: saving checkpoint... 
2022-04-11 23:52:11.064763: done, saving took 8.79 seconds 
2022-04-11 23:52:11.133784: This epoch took 136.927114 s
 
2022-04-11 23:52:11.135239: svmem(total=99932160000, available=80047685632, percent=19.9, used=15890464768, free=51029266432, active=40146968576, inactive=6911991808, buffers=22659072, cached=32989769728, shared=2176741376, slab=762204160) 
2022-04-11 23:52:11.136453: 
epoch:  203 
2022-04-11 23:54:01.029325: train loss : -0.5402 
2022-04-11 23:54:12.941364: validation loss: -0.5803 
2022-04-11 23:54:12.943699: Average global foreground Dice: [0.8282, 0.9444, 0.9737, 0.3924] 
2022-04-11 23:54:12.944521: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:54:13.714034: lr: 0.006239 
2022-04-11 23:54:13.715759: saving scheduled checkpoint file... 
2022-04-11 23:54:13.809054: saving checkpoint... 
2022-04-11 23:54:19.407745: done, saving took 5.69 seconds 
2022-04-11 23:54:19.729564: done 
2022-04-11 23:54:19.855228: This epoch took 128.717531 s
 
2022-04-11 23:54:19.993537: svmem(total=99932160000, available=79798837248, percent=20.1, used=16139231232, free=50734669824, active=40389476352, inactive=6957625344, buffers=22708224, cached=33035550720, shared=2176741376, slab=775569408) 
2022-04-11 23:54:20.127182: 
epoch:  204 
2022-04-11 23:56:08.231267: train loss : -0.5307 
2022-04-11 23:56:19.511257: validation loss: -0.5799 
2022-04-11 23:56:19.513544: Average global foreground Dice: [0.832, 0.9483, 0.9678, 0.3673] 
2022-04-11 23:56:19.514682: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:56:20.465966: lr: 0.00622 
2022-04-11 23:56:20.467584: saving scheduled checkpoint file... 
2022-04-11 23:56:20.545197: saving checkpoint... 
2022-04-11 23:56:25.212894: done, saving took 4.74 seconds 
2022-04-11 23:56:25.227874: done 
2022-04-11 23:56:25.229301: This epoch took 124.976864 s
 
2022-04-11 23:56:25.230791: svmem(total=99932160000, available=80060944384, percent=19.9, used=15936188416, free=50988937216, active=40195989504, inactive=6906343424, buffers=22757376, cached=32984276992, shared=2117685248, slab=762322944) 
2022-04-11 23:56:25.231932: 
epoch:  205 
2022-04-11 23:58:14.686617: train loss : -0.5226 
2022-04-11 23:58:27.695373: validation loss: -0.5629 
2022-04-11 23:58:27.700514: Average global foreground Dice: [0.843, 0.9448, 0.9647, 0.3871] 
2022-04-11 23:58:27.701758: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-11 23:58:28.504521: lr: 0.006201 
2022-04-11 23:58:28.510297: saving scheduled checkpoint file... 
2022-04-11 23:58:28.558416: saving checkpoint... 
2022-04-11 23:58:33.088456: done, saving took 4.58 seconds 
2022-04-11 23:58:33.101225: done 
2022-04-11 23:58:33.102056: This epoch took 127.869312 s
 
2022-04-11 23:58:33.103555: svmem(total=99932160000, available=79921537024, percent=20.0, used=16001875968, free=50844450816, active=40249106432, inactive=6999961600, buffers=22790144, cached=33063043072, shared=2191335424, slab=761671680) 
2022-04-11 23:58:33.104432: 
epoch:  206 
2022-04-12 00:00:22.473679: train loss : -0.5729 
2022-04-12 00:00:34.448169: validation loss: -0.6113 
2022-04-12 00:00:34.455019: Average global foreground Dice: [0.8348, 0.9604, 0.9765, 0.4052] 
2022-04-12 00:00:34.456186: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:00:35.079598: lr: 0.006182 
2022-04-12 00:00:35.080984: saving scheduled checkpoint file... 
2022-04-12 00:00:35.172496: saving checkpoint... 
2022-04-12 00:00:39.942881: done, saving took 4.86 seconds 
2022-04-12 00:00:39.962405: done 
2022-04-12 00:00:40.038601: saving checkpoint... 
2022-04-12 00:00:44.927498: done, saving took 4.96 seconds 
2022-04-12 00:00:44.946306: This epoch took 131.841134 s
 
2022-04-12 00:00:44.947837: svmem(total=99932160000, available=80014929920, percent=19.9, used=15927377920, free=50895339520, active=40187768832, inactive=6998880256, buffers=29790208, cached=33079652352, shared=2166521856, slab=773681152) 
2022-04-12 00:00:44.949365: 
epoch:  207 
2022-04-12 00:02:33.660146: train loss : -0.5800 
2022-04-12 00:02:45.494305: validation loss: -0.5805 
2022-04-12 00:02:45.499398: Average global foreground Dice: [0.8108, 0.9476, 0.9751, 0.3669] 
2022-04-12 00:02:45.500497: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:02:46.410258: lr: 0.006163 
2022-04-12 00:02:46.411954: saving scheduled checkpoint file... 
2022-04-12 00:02:46.517788: saving checkpoint... 
2022-04-12 00:02:51.435925: done, saving took 5.02 seconds 
2022-04-12 00:02:51.453892: done 
2022-04-12 00:02:51.454832: This epoch took 126.504688 s
 
2022-04-12 00:02:51.455722: svmem(total=99932160000, available=79878201344, percent=20.1, used=16039161856, free=50750812160, active=40315420672, inactive=7020253184, buffers=29822976, cached=33112363008, shared=2191609856, slab=773963776) 
2022-04-12 00:02:51.456561: 
epoch:  208 
2022-04-12 00:04:40.657637: train loss : -0.5895 
2022-04-12 00:04:52.191649: validation loss: -0.5871 
2022-04-12 00:04:52.198178: Average global foreground Dice: [0.817, 0.9484, 0.9707, 0.3886] 
2022-04-12 00:04:52.199630: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:04:52.978055: lr: 0.006144 
2022-04-12 00:04:52.979984: saving scheduled checkpoint file... 
2022-04-12 00:04:53.088733: saving checkpoint... 
2022-04-12 00:04:57.795014: done, saving took 4.81 seconds 
2022-04-12 00:04:57.814223: done 
2022-04-12 00:04:57.815509: This epoch took 126.358131 s
 
2022-04-12 00:04:57.816648: svmem(total=99932160000, available=79937851392, percent=20.0, used=15991676928, free=50794147840, active=40266272768, inactive=7024357376, buffers=29855744, cached=33116479488, shared=2179506176, slab=774963200) 
2022-04-12 00:04:57.817582: 
epoch:  209 
2022-04-12 00:06:47.492990: train loss : -0.5971 
2022-04-12 00:06:59.962013: validation loss: -0.6112 
2022-04-12 00:06:59.964567: Average global foreground Dice: [0.846, 0.9501, 0.9783, 0.375] 
2022-04-12 00:06:59.966170: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:07:00.727161: lr: 0.006125 
2022-04-12 00:07:00.728224: saving scheduled checkpoint file... 
2022-04-12 00:07:00.815839: saving checkpoint... 
2022-04-12 00:07:05.382598: done, saving took 4.65 seconds 
2022-04-12 00:07:05.401264: done 
2022-04-12 00:07:05.402477: This epoch took 127.584085 s
 
2022-04-12 00:07:05.403795: svmem(total=99932160000, available=79997206528, percent=19.9, used=15969591296, free=50835939328, active=40245731328, inactive=7004389376, buffers=29917184, cached=33096712192, shared=2142048256, slab=774168576) 
2022-04-12 00:07:05.404673: 
epoch:  210 
2022-04-12 00:08:54.383032: train loss : -0.5982 
2022-04-12 00:09:06.159109: validation loss: -0.6075 
2022-04-12 00:09:06.161388: Average global foreground Dice: [0.8393, 0.9528, 0.9788, 0.3998] 
2022-04-12 00:09:06.162323: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:09:07.128152: lr: 0.006106 
2022-04-12 00:09:07.129667: saving scheduled checkpoint file... 
2022-04-12 00:09:07.208327: saving checkpoint... 
2022-04-12 00:09:11.789299: done, saving took 4.66 seconds 
2022-04-12 00:09:11.808996: done 
2022-04-12 00:09:11.810498: This epoch took 126.405115 s
 
2022-04-12 00:09:11.811728: svmem(total=99932160000, available=79959461888, percent=20.0, used=16007659520, free=50790342656, active=40282697728, inactive=7012044800, buffers=29966336, cached=33104191488, shared=2141859840, slab=773636096) 
2022-04-12 00:09:11.812528: 
epoch:  211 
2022-04-12 00:11:02.355492: train loss : -0.5919 
2022-04-12 00:11:14.280948: validation loss: -0.6074 
2022-04-12 00:11:14.283357: Average global foreground Dice: [0.8468, 0.9581, 0.9798, 0.3857] 
2022-04-12 00:11:14.284402: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:11:14.866632: lr: 0.006087 
2022-04-12 00:11:14.868498: saving scheduled checkpoint file... 
2022-04-12 00:11:14.980545: saving checkpoint... 
2022-04-12 00:11:19.659693: done, saving took 4.79 seconds 
2022-04-12 00:11:19.746680: done 
2022-04-12 00:11:19.842094: saving checkpoint... 
2022-04-12 00:11:25.795681: done, saving took 6.05 seconds 
2022-04-12 00:11:26.315988: This epoch took 134.502493 s
 
2022-04-12 00:11:26.474972: svmem(total=99932160000, available=79980044288, percent=20.0, used=15952019456, free=50793435136, active=40209096704, inactive=7064207360, buffers=29999104, cached=33156706304, shared=2176741376, slab=786948096) 
2022-04-12 00:11:26.621042: 
epoch:  212 
2022-04-12 00:13:17.770991: train loss : -0.6117 
2022-04-12 00:13:29.491739: validation loss: -0.6401 
2022-04-12 00:13:29.509973: Average global foreground Dice: [0.8687, 0.9612, 0.9724, 0.4047] 
2022-04-12 00:13:29.510941: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:13:30.459363: lr: 0.006068 
2022-04-12 00:13:30.600629: saving scheduled checkpoint file... 
2022-04-12 00:13:30.891202: saving checkpoint... 
2022-04-12 00:13:38.046731: done, saving took 7.28 seconds 
2022-04-12 00:13:38.069094: done 
2022-04-12 00:13:38.177109: saving checkpoint... 
2022-04-12 00:13:42.645241: done, saving took 4.57 seconds 
2022-04-12 00:13:42.811292: This epoch took 136.014879 s
 
2022-04-12 00:13:42.814923: svmem(total=99932160000, available=79746326528, percent=20.2, used=16171810816, free=50541277184, active=40447434752, inactive=7096782848, buffers=30031872, cached=33189040128, shared=2190962688, slab=772386816) 
2022-04-12 00:13:42.817554: 
epoch:  213 
2022-04-12 00:15:32.486585: train loss : -0.6191 
2022-04-12 00:15:43.702499: validation loss: -0.6272 
2022-04-12 00:15:43.716515: Average global foreground Dice: [0.8796, 0.9604, 0.9765, 0.3861] 
2022-04-12 00:15:43.718429: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:15:44.467818: lr: 0.006049 
2022-04-12 00:15:44.472775: saving scheduled checkpoint file... 
2022-04-12 00:15:44.598840: saving checkpoint... 
2022-04-12 00:15:49.253382: done, saving took 4.78 seconds 
2022-04-12 00:15:49.285184: done 
2022-04-12 00:15:49.410329: saving checkpoint... 
2022-04-12 00:15:54.165146: done, saving took 4.88 seconds 
2022-04-12 00:15:54.213281: This epoch took 131.394760 s
 
2022-04-12 00:15:54.236106: svmem(total=99932160000, available=79799181312, percent=20.1, used=16107954176, free=50576683008, active=40382078976, inactive=7124353024, buffers=30081024, cached=33217441792, shared=2201600000, slab=773476352) 
2022-04-12 00:15:54.242313: 
epoch:  214 
2022-04-12 00:17:44.034964: train loss : -0.6240 
2022-04-12 00:17:56.788191: validation loss: -0.6592 
2022-04-12 00:17:56.813468: Average global foreground Dice: [0.8877, 0.9612, 0.9811, 0.4037] 
2022-04-12 00:17:56.815079: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:17:57.620603: lr: 0.00603 
2022-04-12 00:17:57.622472: saving scheduled checkpoint file... 
2022-04-12 00:17:57.747221: saving checkpoint... 
2022-04-12 00:18:02.446394: done, saving took 4.82 seconds 
2022-04-12 00:18:02.499074: done 
2022-04-12 00:18:02.611169: saving checkpoint... 
2022-04-12 00:18:08.068727: done, saving took 5.57 seconds 
2022-04-12 00:18:08.176288: This epoch took 133.922585 s
 
2022-04-12 00:18:08.177998: svmem(total=99932160000, available=79630807040, percent=20.3, used=16286691328, free=50400350208, active=40554119168, inactive=7122268160, buffers=30117888, cached=33215000576, shared=2191224832, slab=773885952) 
2022-04-12 00:18:08.179504: 
epoch:  215 
2022-04-12 00:19:57.863822: train loss : -0.6130 
2022-04-12 00:20:09.799704: validation loss: -0.4682 
2022-04-12 00:20:09.808112: Average global foreground Dice: [0.827, 0.9189, 0.9263, 0.3474] 
2022-04-12 00:20:09.835950: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:20:10.656503: lr: 0.006011 
2022-04-12 00:20:10.657907: saving scheduled checkpoint file... 
2022-04-12 00:20:10.732764: saving checkpoint... 
2022-04-12 00:20:15.702536: done, saving took 5.04 seconds 
2022-04-12 00:20:15.716665: done 
2022-04-12 00:20:15.717983: This epoch took 127.537623 s
 
2022-04-12 00:20:15.719286: svmem(total=99932160000, available=79650983936, percent=20.3, used=16267268096, free=50403012608, active=40533123072, inactive=7138779136, buffers=30175232, cached=33231704064, shared=2190401536, slab=776101888) 
2022-04-12 00:20:15.720131: 
epoch:  216 
2022-04-12 00:22:04.930821: train loss : -0.4411 
2022-04-12 00:22:16.920810: validation loss: -0.5507 
2022-04-12 00:22:16.930235: Average global foreground Dice: [0.799, 0.9446, 0.9697, 0.3763] 
2022-04-12 00:22:16.931352: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:22:17.785326: lr: 0.005991 
2022-04-12 00:22:17.786557: saving scheduled checkpoint file... 
2022-04-12 00:22:17.850916: saving checkpoint... 
2022-04-12 00:22:22.626913: done, saving took 4.84 seconds 
2022-04-12 00:22:22.676960: done 
2022-04-12 00:22:22.677808: This epoch took 126.956695 s
 
2022-04-12 00:22:22.678954: svmem(total=99932160000, available=79926792192, percent=20.0, used=16005144576, free=50661371904, active=40275628032, inactive=7142416384, buffers=30224384, cached=33235419136, shared=2176663552, slab=775884800) 
2022-04-12 00:22:22.680151: 
epoch:  217 
2022-04-12 00:24:10.467458: train loss : -0.5315 
2022-04-12 00:24:22.127995: validation loss: -0.5909 
2022-04-12 00:24:22.130428: Average global foreground Dice: [0.8593, 0.9496, 0.9758, 0.3588] 
2022-04-12 00:24:22.132198: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:24:22.998866: lr: 0.005972 
2022-04-12 00:24:23.000282: saving scheduled checkpoint file... 
2022-04-12 00:24:23.049499: saving checkpoint... 
2022-04-12 00:24:27.685483: done, saving took 4.68 seconds 
2022-04-12 00:24:27.705985: done 
2022-04-12 00:24:27.707141: This epoch took 125.022716 s
 
2022-04-12 00:24:27.708782: svmem(total=99932160000, available=79646965760, percent=20.3, used=16271007744, free=50374803456, active=40545976320, inactive=7163023360, buffers=30257152, cached=33256091648, shared=2190618624, slab=776028160) 
2022-04-12 00:24:27.759045: 
epoch:  218 
2022-04-12 00:26:17.119509: train loss : -0.5752 
2022-04-12 00:26:29.168825: validation loss: -0.5625 
2022-04-12 00:26:29.171007: Average global foreground Dice: [0.8317, 0.9509, 0.9709, 0.3748] 
2022-04-12 00:26:29.172669: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:26:30.035414: lr: 0.005953 
2022-04-12 00:26:30.036418: saving scheduled checkpoint file... 
2022-04-12 00:26:30.143957: saving checkpoint... 
2022-04-12 00:26:34.963021: done, saving took 4.93 seconds 
2022-04-12 00:26:34.981886: done 
2022-04-12 00:26:34.983102: This epoch took 127.222669 s
 
2022-04-12 00:26:34.983950: svmem(total=99932160000, available=79854723072, percent=20.1, used=16063119360, free=50580905984, active=40321380352, inactive=7180525568, buffers=30306304, cached=33257828352, shared=2190729216, slab=773660672) 
2022-04-12 00:26:34.985100: 
epoch:  219 
2022-04-12 00:28:23.258730: train loss : -0.5913 
2022-04-12 00:28:35.374702: validation loss: -0.5939 
2022-04-12 00:28:35.405356: Average global foreground Dice: [0.8529, 0.9468, 0.9685, 0.4009] 
2022-04-12 00:28:35.406734: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:28:36.278638: lr: 0.005934 
2022-04-12 00:28:36.280279: saving scheduled checkpoint file... 
2022-04-12 00:28:36.370768: saving checkpoint... 
2022-04-12 00:28:40.969154: done, saving took 4.69 seconds 
2022-04-12 00:28:40.992757: done 
2022-04-12 00:28:40.993710: This epoch took 126.007893 s
 
2022-04-12 00:28:40.994599: svmem(total=99932160000, available=79731130368, percent=20.2, used=16186724352, free=50440028160, active=40458788864, inactive=7181996032, buffers=30339072, cached=33275068416, shared=2190729216, slab=774553600) 
2022-04-12 00:28:40.995672: 
epoch:  220 
2022-04-12 00:30:31.049436: train loss : -0.5715 
2022-04-12 00:30:42.980918: validation loss: -0.6193 
2022-04-12 00:30:42.983959: Average global foreground Dice: [0.8749, 0.9508, 0.9723, 0.3918] 
2022-04-12 00:30:42.985163: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:30:43.895728: lr: 0.005915 
2022-04-12 00:30:43.897322: saving scheduled checkpoint file... 
2022-04-12 00:30:44.024305: saving checkpoint... 
2022-04-12 00:30:48.747613: done, saving took 4.85 seconds 
2022-04-12 00:30:48.775658: done 
2022-04-12 00:30:48.777021: This epoch took 127.779974 s
 
2022-04-12 00:30:48.778232: svmem(total=99932160000, available=79776964608, percent=20.2, used=16140816384, free=50481188864, active=40409464832, inactive=7195193344, buffers=30408704, cached=33279746048, shared=2190594048, slab=773414912) 
2022-04-12 00:30:48.779102: 
epoch:  221 
2022-04-12 00:32:38.259129: train loss : -0.6107 
2022-04-12 00:32:50.379895: validation loss: -0.5879 
2022-04-12 00:32:50.391778: Average global foreground Dice: [0.8601, 0.9573, 0.9709, 0.3614] 
2022-04-12 00:32:50.392790: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:32:51.155764: lr: 0.005896 
2022-04-12 00:32:51.157544: saving scheduled checkpoint file... 
2022-04-12 00:32:51.234514: saving checkpoint... 
2022-04-12 00:32:56.554213: done, saving took 5.40 seconds 
2022-04-12 00:32:56.571376: done 
2022-04-12 00:32:56.572597: This epoch took 127.792653 s
 
2022-04-12 00:32:56.573561: svmem(total=99932160000, available=79824609280, percent=20.1, used=16093196288, free=50544979968, active=40201777152, inactive=7334768640, buffers=30441472, cached=33263542272, shared=2190696448, slab=773427200) 
2022-04-12 00:32:56.574646: 
epoch:  222 
2022-04-12 00:34:46.887834: train loss : -0.5969 
2022-04-12 00:34:59.249942: validation loss: -0.6049 
2022-04-12 00:34:59.362217: Average global foreground Dice: [0.8454, 0.9577, 0.972, 0.378] 
2022-04-12 00:34:59.409916: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:35:00.247775: lr: 0.005877 
2022-04-12 00:35:00.249707: saving scheduled checkpoint file... 
2022-04-12 00:35:00.363726: saving checkpoint... 
2022-04-12 00:35:09.303797: done, saving took 9.05 seconds 
2022-04-12 00:35:09.348130: done 
2022-04-12 00:35:09.356131: This epoch took 132.780464 s
 
2022-04-12 00:35:09.375513: svmem(total=99932160000, available=79787241472, percent=20.2, used=16130908160, free=50488954880, active=40356741120, inactive=7228801024, buffers=30498816, cached=33281798144, shared=2190245888, slab=774205440) 
2022-04-12 00:35:09.389775: 
epoch:  223 
2022-04-12 00:36:58.961181: train loss : -0.6013 
2022-04-12 00:37:10.783113: validation loss: -0.5349 
2022-04-12 00:37:10.785496: Average global foreground Dice: [0.8261, 0.9338, 0.9407, 0.4032] 
2022-04-12 00:37:10.786295: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:37:11.601077: lr: 0.005858 
2022-04-12 00:37:11.602455: saving scheduled checkpoint file... 
2022-04-12 00:37:11.706882: saving checkpoint... 
2022-04-12 00:37:19.387218: done, saving took 7.78 seconds 
2022-04-12 00:37:19.415038: done 
2022-04-12 00:37:19.416283: This epoch took 130.012240 s
 
2022-04-12 00:37:19.417186: svmem(total=99932160000, available=79843385344, percent=20.1, used=16074350592, free=50527739904, active=40348209152, inactive=7213998080, buffers=30547968, cached=33299521536, shared=2190729216, slab=774451200) 
2022-04-12 00:37:19.418021: 
epoch:  224 
2022-04-12 00:39:09.507696: train loss : -0.6058 
2022-04-12 00:39:22.119410: validation loss: -0.6000 
2022-04-12 00:39:22.125032: Average global foreground Dice: [0.8443, 0.9565, 0.975, 0.3938] 
2022-04-12 00:39:22.126156: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:39:22.765474: lr: 0.005839 
2022-04-12 00:39:22.766765: saving scheduled checkpoint file... 
2022-04-12 00:39:22.872488: saving checkpoint... 
2022-04-12 00:39:27.611303: done, saving took 4.84 seconds 
2022-04-12 00:39:27.637414: done 
2022-04-12 00:39:27.638559: This epoch took 128.219754 s
 
2022-04-12 00:39:27.640423: svmem(total=99932160000, available=79912820736, percent=20.0, used=16004485120, free=50590486528, active=40271966208, inactive=7219544064, buffers=30580736, cached=33306607616, shared=2191224832, slab=774168576) 
2022-04-12 00:39:27.641461: 
epoch:  225 
2022-04-12 00:41:18.139353: train loss : -0.5559 
2022-04-12 00:41:30.354752: validation loss: -0.5285 
2022-04-12 00:41:30.364391: Average global foreground Dice: [0.8408, 0.9497, 0.9516, 0.3721] 
2022-04-12 00:41:30.366875: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:41:31.062944: lr: 0.00582 
2022-04-12 00:41:31.064103: saving scheduled checkpoint file... 
2022-04-12 00:41:31.183604: saving checkpoint... 
2022-04-12 00:41:36.037801: done, saving took 4.97 seconds 
2022-04-12 00:41:36.068964: done 
2022-04-12 00:41:36.070405: This epoch took 128.427758 s
 
2022-04-12 00:41:36.071566: svmem(total=99932160000, available=79836000256, percent=20.1, used=16081584128, free=50496393216, active=40351744000, inactive=7236292608, buffers=30621696, cached=33323560960, shared=2190807040, slab=774168576) 
2022-04-12 00:41:36.072482: 
epoch:  226 
2022-04-12 00:43:25.904408: train loss : -0.4962 
2022-04-12 00:43:38.379190: validation loss: -0.4775 
2022-04-12 00:43:38.382494: Average global foreground Dice: [0.74, 0.9428, 0.9526, 0.3689] 
2022-04-12 00:43:38.393745: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:43:39.227898: lr: 0.005801 
2022-04-12 00:43:39.229634: saving scheduled checkpoint file... 
2022-04-12 00:43:39.384675: saving checkpoint... 
2022-04-12 00:43:44.041748: done, saving took 4.81 seconds 
2022-04-12 00:43:44.069611: done 
2022-04-12 00:43:44.071332: This epoch took 127.998028 s
 
2022-04-12 00:43:44.073007: svmem(total=99932160000, available=79936315392, percent=20.0, used=15981387776, free=50579030016, active=40252805120, inactive=7253778432, buffers=30654464, cached=33341087744, shared=2190729216, slab=773529600) 
2022-04-12 00:43:44.074183: 
epoch:  227 
2022-04-12 00:45:34.919142: train loss : -0.4969 
2022-04-12 00:45:46.866294: validation loss: -0.5444 
2022-04-12 00:45:46.902351: Average global foreground Dice: [0.8116, 0.945, 0.968, 0.3778] 
2022-04-12 00:45:46.993793: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:45:47.571021: lr: 0.005781 
2022-04-12 00:45:47.572506: saving scheduled checkpoint file... 
2022-04-12 00:45:47.667058: saving checkpoint... 
2022-04-12 00:45:52.207349: done, saving took 4.63 seconds 
2022-04-12 00:45:52.223193: done 
2022-04-12 00:45:52.224507: This epoch took 128.149689 s
 
2022-04-12 00:45:52.225776: svmem(total=99932160000, available=79744909312, percent=20.2, used=16173305856, free=50379759616, active=40448544768, inactive=7254532096, buffers=30720000, cached=33348374528, shared=2190233600, slab=774893568) 
2022-04-12 00:45:52.226759: 
epoch:  228 
2022-04-12 00:47:43.396205: train loss : -0.5553 
2022-04-12 00:47:54.818048: validation loss: -0.5909 
2022-04-12 00:47:54.839110: Average global foreground Dice: [0.811, 0.9523, 0.9697, 0.4072] 
2022-04-12 00:47:54.840535: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:47:55.753686: lr: 0.005762 
2022-04-12 00:47:55.755167: saving scheduled checkpoint file... 
2022-04-12 00:47:55.808552: saving checkpoint... 
2022-04-12 00:48:00.730644: done, saving took 4.97 seconds 
2022-04-12 00:48:00.788304: done 
2022-04-12 00:48:00.789858: This epoch took 128.561805 s
 
2022-04-12 00:48:00.791281: svmem(total=99932160000, available=79943127040, percent=20.0, used=15974535168, free=50560544768, active=40249532416, inactive=7272472576, buffers=30752768, cached=33366327296, shared=2190729216, slab=775380992) 
2022-04-12 00:48:00.792169: 
epoch:  229 
2022-04-12 00:49:49.969136: train loss : -0.5490 
2022-04-12 00:50:01.942178: validation loss: -0.5650 
2022-04-12 00:50:01.955073: Average global foreground Dice: [0.8093, 0.9585, 0.972, 0.3759] 
2022-04-12 00:50:01.956103: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:50:02.770889: lr: 0.005743 
2022-04-12 00:50:02.772720: saving scheduled checkpoint file... 
2022-04-12 00:50:02.869298: saving checkpoint... 
2022-04-12 00:50:07.756688: done, saving took 4.98 seconds 
2022-04-12 00:50:07.775195: done 
2022-04-12 00:50:07.776468: This epoch took 126.983423 s
 
2022-04-12 00:50:07.777915: svmem(total=99932160000, available=79852728320, percent=20.1, used=16078630912, free=50452770816, active=40350769152, inactive=7275929600, buffers=30793728, cached=33369964544, shared=2176962560, slab=775299072) 
2022-04-12 00:50:07.778718: 
epoch:  230 
2022-04-12 00:51:56.935655: train loss : -0.5955 
2022-04-12 00:52:08.529233: validation loss: -0.5988 
2022-04-12 00:52:08.540813: Average global foreground Dice: [0.8592, 0.955, 0.9783, 0.3631] 
2022-04-12 00:52:08.542144: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:52:09.413158: lr: 0.005724 
2022-04-12 00:52:09.415128: saving scheduled checkpoint file... 
2022-04-12 00:52:09.526048: saving checkpoint... 
2022-04-12 00:52:18.015920: done, saving took 8.60 seconds 
2022-04-12 00:52:18.042608: done 
2022-04-12 00:52:18.058621: This epoch took 130.278986 s
 
2022-04-12 00:52:18.077122: svmem(total=99932160000, available=79848816640, percent=20.1, used=16068448256, free=50440966144, active=40342192128, inactive=7297581056, buffers=30842880, cached=33391902720, shared=2190917632, slab=775020544) 
2022-04-12 00:52:18.097560: 
epoch:  231 
2022-04-12 00:54:06.498425: train loss : -0.6237 
2022-04-12 00:54:18.999587: validation loss: -0.6491 
2022-04-12 00:54:19.010742: Average global foreground Dice: [0.8692, 0.9612, 0.9809, 0.3818] 
2022-04-12 00:54:19.011880: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:54:19.881529: lr: 0.005705 
2022-04-12 00:54:19.883034: saving scheduled checkpoint file... 
2022-04-12 00:54:20.021630: saving checkpoint... 
2022-04-12 00:54:25.553305: done, saving took 5.67 seconds 
2022-04-12 00:54:25.733752: done 
2022-04-12 00:54:25.739783: This epoch took 127.632902 s
 
2022-04-12 00:54:25.824579: svmem(total=99932160000, available=79731318784, percent=20.2, used=16186548224, free=50305859584, active=40461320192, inactive=7314550784, buffers=30875648, cached=33408876544, shared=2190344192, slab=774410240) 
2022-04-12 00:54:25.901739: 
epoch:  232 
2022-04-12 00:56:16.541354: train loss : -0.5884 
2022-04-12 00:56:28.595202: validation loss: -0.6174 
2022-04-12 00:56:28.644030: Average global foreground Dice: [0.8531, 0.9535, 0.9749, 0.4068] 
2022-04-12 00:56:28.649526: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:56:29.203166: lr: 0.005686 
2022-04-12 00:56:29.204976: saving scheduled checkpoint file... 
2022-04-12 00:56:29.324981: saving checkpoint... 
2022-04-12 00:56:34.731773: done, saving took 5.53 seconds 
2022-04-12 00:56:34.866775: done 
2022-04-12 00:56:34.878527: This epoch took 128.973025 s
 
2022-04-12 00:56:34.880928: svmem(total=99932160000, available=79721705472, percent=20.2, used=16185737216, free=50278916096, active=40460169216, inactive=7342084096, buffers=30924800, cached=33436581888, shared=2200608768, slab=774410240) 
2022-04-12 00:56:34.882220: 
epoch:  233 
2022-04-12 00:58:24.343427: train loss : -0.5637 
2022-04-12 00:58:36.460506: validation loss: -0.5923 
2022-04-12 00:58:36.466920: Average global foreground Dice: [0.8424, 0.9435, 0.9718, 0.3837] 
2022-04-12 00:58:36.468002: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 00:58:37.372447: lr: 0.005667 
2022-04-12 00:58:37.374038: saving scheduled checkpoint file... 
2022-04-12 00:58:37.466653: saving checkpoint... 
2022-04-12 00:58:42.610345: done, saving took 5.23 seconds 
2022-04-12 00:58:42.622735: done 
2022-04-12 00:58:42.624552: This epoch took 127.740718 s
 
2022-04-12 00:58:42.626254: svmem(total=99932160000, available=79812313088, percent=20.1, used=16105656320, free=50362740736, active=40382918656, inactive=7338635264, buffers=30957568, cached=33432805376, shared=2190311424, slab=774647808) 
2022-04-12 00:58:42.627443: 
epoch:  234 
2022-04-12 01:00:32.969215: train loss : -0.5761 
2022-04-12 01:00:45.194879: validation loss: -0.5964 
2022-04-12 01:00:45.212350: Average global foreground Dice: [0.8453, 0.9574, 0.972, 0.369] 
2022-04-12 01:00:45.219209: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:00:45.940629: lr: 0.005647 
2022-04-12 01:00:46.018948: saving scheduled checkpoint file... 
2022-04-12 01:00:46.079439: saving checkpoint... 
2022-04-12 01:00:52.026712: done, saving took 6.01 seconds 
2022-04-12 01:00:52.322943: done 
2022-04-12 01:00:52.342309: This epoch took 129.713913 s
 
2022-04-12 01:00:52.347667: svmem(total=99932160000, available=79516954624, percent=20.4, used=16403984384, free=50049900544, active=40683741184, inactive=7352766464, buffers=30998528, cached=33447276544, shared=2187149312, slab=774246400) 
2022-04-12 01:00:52.351290: 
epoch:  235 
2022-04-12 01:02:41.829138: train loss : -0.5928 
2022-04-12 01:02:54.112102: validation loss: -0.6050 
2022-04-12 01:02:54.119371: Average global foreground Dice: [0.8465, 0.9552, 0.9768, 0.3984] 
2022-04-12 01:02:54.120878: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:02:54.981872: lr: 0.005628 
2022-04-12 01:02:54.983729: saving scheduled checkpoint file... 
2022-04-12 01:02:55.030756: saving checkpoint... 
2022-04-12 01:03:00.125246: done, saving took 5.14 seconds 
2022-04-12 01:03:00.140464: done 
2022-04-12 01:03:00.142123: This epoch took 127.788062 s
 
2022-04-12 01:03:00.143362: svmem(total=99932160000, available=79808217088, percent=20.1, used=16109064192, free=50323730432, active=40383983616, inactive=7373783040, buffers=31031296, cached=33468334080, shared=2190807040, slab=774897664) 
2022-04-12 01:03:00.144286: 
epoch:  236 
2022-04-12 01:04:48.159874: train loss : -0.6296 
2022-04-12 01:04:59.518012: validation loss: -0.6574 
2022-04-12 01:04:59.532118: Average global foreground Dice: [0.8699, 0.959, 0.9783, 0.4187] 
2022-04-12 01:04:59.532918: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:05:00.597831: lr: 0.005609 
2022-04-12 01:05:00.599643: saving scheduled checkpoint file... 
2022-04-12 01:05:00.687396: saving checkpoint... 
2022-04-12 01:05:05.845977: done, saving took 5.25 seconds 
2022-04-12 01:05:05.921927: done 
2022-04-12 01:05:05.954204: This epoch took 125.809124 s
 
2022-04-12 01:05:05.971858: svmem(total=99932160000, available=79789764608, percent=20.2, used=16127381504, free=50297499648, active=40400678912, inactive=7381581824, buffers=31072256, cached=33476206592, shared=2190917632, slab=775204864) 
2022-04-12 01:05:05.976134: 
epoch:  237 
2022-04-12 01:06:55.225364: train loss : -0.6323 
2022-04-12 01:07:06.511217: validation loss: -0.6411 
2022-04-12 01:07:06.518179: Average global foreground Dice: [0.8622, 0.9538, 0.9797, 0.429] 
2022-04-12 01:07:06.519910: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:07:07.599634: lr: 0.00559 
2022-04-12 01:07:07.601402: saving scheduled checkpoint file... 
2022-04-12 01:07:07.705906: saving checkpoint... 
2022-04-12 01:07:12.932654: done, saving took 5.33 seconds 
2022-04-12 01:07:12.980564: done 
2022-04-12 01:07:12.983166: This epoch took 126.991878 s
 
2022-04-12 01:07:12.985138: svmem(total=99932160000, available=79635599360, percent=20.3, used=16281747456, free=50125721600, active=40561225728, inactive=7398936576, buffers=31141888, cached=33493549056, shared=2190729216, slab=774672384) 
2022-04-12 01:07:12.990388: 
epoch:  238 
2022-04-12 01:09:02.005864: train loss : -0.5952 
2022-04-12 01:09:13.503501: validation loss: -0.5841 
2022-04-12 01:09:13.529292: Average global foreground Dice: [0.8425, 0.9496, 0.9692, 0.3885] 
2022-04-12 01:09:13.564045: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:09:14.450088: lr: 0.005571 
2022-04-12 01:09:14.456687: saving scheduled checkpoint file... 
2022-04-12 01:09:14.573418: saving checkpoint... 
2022-04-12 01:09:19.784387: done, saving took 5.32 seconds 
2022-04-12 01:09:19.818127: done 
2022-04-12 01:09:19.822274: This epoch took 126.826669 s
 
2022-04-12 01:09:19.831019: svmem(total=99932160000, available=79767187456, percent=20.2, used=16156139520, free=50271784960, active=40152129536, inactive=7658549248, buffers=31174656, cached=33473060864, shared=2184708096, slab=775086080) 
2022-04-12 01:09:19.840246: 
epoch:  239 
2022-04-12 01:11:10.323697: train loss : -0.5860 
2022-04-12 01:11:22.411708: validation loss: -0.6001 
2022-04-12 01:11:22.433670: Average global foreground Dice: [0.8221, 0.9515, 0.9773, 0.3711] 
2022-04-12 01:11:22.435992: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:11:23.321238: lr: 0.005551 
2022-04-12 01:11:23.324349: saving scheduled checkpoint file... 
2022-04-12 01:11:23.403495: saving checkpoint... 
2022-04-12 01:11:28.020458: done, saving took 4.69 seconds 
2022-04-12 01:11:28.046679: done 
2022-04-12 01:11:28.048057: This epoch took 128.200893 s
 
2022-04-12 01:11:28.048982: svmem(total=99932160000, available=79773372416, percent=20.2, used=16154730496, free=50260561920, active=40322191360, inactive=7500988416, buffers=31215616, cached=33485651968, shared=2179825664, slab=774529024) 
2022-04-12 01:11:28.050217: 
epoch:  240 
2022-04-12 01:13:17.091784: train loss : -0.6028 
2022-04-12 01:13:29.358613: validation loss: -0.5999 
2022-04-12 01:13:29.361300: Average global foreground Dice: [0.8294, 0.9591, 0.9775, 0.4016] 
2022-04-12 01:13:29.362826: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:13:30.250006: lr: 0.005532 
2022-04-12 01:13:30.251123: saving scheduled checkpoint file... 
2022-04-12 01:13:30.361952: saving checkpoint... 
2022-04-12 01:13:34.818412: done, saving took 4.57 seconds 
2022-04-12 01:13:34.841222: done 
2022-04-12 01:13:34.842430: This epoch took 126.790987 s
 
2022-04-12 01:13:34.844686: svmem(total=99932160000, available=79792197632, percent=20.2, used=16124964864, free=50272657408, active=40340021248, inactive=7470903296, buffers=31248384, cached=33503289344, shared=2190807040, slab=774778880) 
2022-04-12 01:13:34.846142: 
epoch:  241 
2022-04-12 01:15:24.450714: train loss : -0.6049 
2022-04-12 01:15:36.128111: validation loss: -0.6305 
2022-04-12 01:15:36.132505: Average global foreground Dice: [0.8757, 0.9559, 0.9753, 0.4175] 
2022-04-12 01:15:36.133460: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:15:36.898063: lr: 0.005513 
2022-04-12 01:15:36.899446: saving scheduled checkpoint file... 
2022-04-12 01:15:37.004979: saving checkpoint... 
2022-04-12 01:15:41.598511: done, saving took 4.70 seconds 
2022-04-12 01:15:41.625188: done 
2022-04-12 01:15:41.718811: saving checkpoint... 
2022-04-12 01:15:46.889099: done, saving took 5.26 seconds 
2022-04-12 01:15:46.909639: This epoch took 132.062145 s
 
2022-04-12 01:15:46.910932: svmem(total=99932160000, available=79868743680, percent=20.1, used=16038211584, free=50249097216, active=40268951552, inactive=7560122368, buffers=31313920, cached=33613537280, shared=2200936448, slab=774623232) 
2022-04-12 01:15:46.912048: 
epoch:  242 
2022-04-12 01:17:35.816489: train loss : -0.6128 
2022-04-12 01:17:47.483900: validation loss: -0.5979 
2022-04-12 01:17:47.510130: Average global foreground Dice: [0.8518, 0.9536, 0.9682, 0.3917] 
2022-04-12 01:17:47.563308: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:17:48.541006: lr: 0.005494 
2022-04-12 01:17:48.549543: saving scheduled checkpoint file... 
2022-04-12 01:17:48.695800: saving checkpoint... 
2022-04-12 01:17:56.568989: done, saving took 8.02 seconds 
2022-04-12 01:17:56.617167: done 
2022-04-12 01:17:56.728576: saving checkpoint... 
2022-04-12 01:18:01.488415: done, saving took 4.87 seconds 
2022-04-12 01:18:01.506774: This epoch took 134.593818 s
 
2022-04-12 01:18:01.508592: svmem(total=99932160000, available=79678083072, percent=20.3, used=16238796800, free=50039984128, active=40485154816, inactive=7556849664, buffers=31350784, cached=33622028288, shared=2190839808, slab=775421952) 
2022-04-12 01:18:01.509496: 
epoch:  243 
2022-04-12 01:19:50.827459: train loss : -0.5822 
2022-04-12 01:20:03.350868: validation loss: -0.6219 
2022-04-12 01:20:03.358222: Average global foreground Dice: [0.8584, 0.9562, 0.9729, 0.3842] 
2022-04-12 01:20:03.359029: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:20:04.179950: lr: 0.005474 
2022-04-12 01:20:04.181056: saving scheduled checkpoint file... 
2022-04-12 01:20:04.334445: saving checkpoint... 
2022-04-12 01:20:09.671065: done, saving took 5.49 seconds 
2022-04-12 01:20:09.805264: done 
2022-04-12 01:20:09.936739: saving checkpoint... 
2022-04-12 01:20:14.880433: done, saving took 5.07 seconds 
2022-04-12 01:20:14.902610: This epoch took 133.392324 s
 
2022-04-12 01:20:14.904766: svmem(total=99932160000, available=79891476480, percent=20.1, used=16025546752, free=50245357568, active=40281800704, inactive=7551025152, buffers=31408128, cached=33629847552, shared=2190729216, slab=775356416) 
2022-04-12 01:20:14.906416: 
epoch:  244 
2022-04-12 01:22:06.181190: train loss : -0.6124 
2022-04-12 01:22:18.150070: validation loss: -0.6005 
2022-04-12 01:22:18.160012: Average global foreground Dice: [0.8193, 0.9466, 0.9723, 0.3967] 
2022-04-12 01:22:18.162261: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:22:18.762141: lr: 0.005455 
2022-04-12 01:22:18.763840: saving scheduled checkpoint file... 
2022-04-12 01:22:18.899427: saving checkpoint... 
2022-04-12 01:22:23.326678: done, saving took 4.56 seconds 
2022-04-12 01:22:23.350947: done 
2022-04-12 01:22:23.352361: This epoch took 128.444743 s
 
2022-04-12 01:22:23.353513: svmem(total=99932160000, available=79613181952, percent=20.3, used=16304418816, free=49949581312, active=40575188992, inactive=7552933888, buffers=31457280, cached=33646702592, shared=2190233600, slab=776548352) 
2022-04-12 01:22:23.354302: 
epoch:  245 
2022-04-12 01:24:12.371951: train loss : -0.5807 
2022-04-12 01:24:23.591736: validation loss: -0.6196 
2022-04-12 01:24:23.600302: Average global foreground Dice: [0.8652, 0.9516, 0.9794, 0.3745] 
2022-04-12 01:24:23.601482: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:24:24.731534: lr: 0.005436 
2022-04-12 01:24:24.737856: saving scheduled checkpoint file... 
2022-04-12 01:24:24.846949: saving checkpoint... 
2022-04-12 01:24:29.650817: done, saving took 4.91 seconds 
2022-04-12 01:24:29.728760: done 
2022-04-12 01:24:29.730239: This epoch took 126.375233 s
 
2022-04-12 01:24:29.731657: svmem(total=99932160000, available=79757451264, percent=20.2, used=16159555584, free=50077655040, active=40438034432, inactive=7569608704, buffers=31490048, cached=33663459328, shared=2190729216, slab=775782400) 
2022-04-12 01:24:29.732506: 
epoch:  246 
2022-04-12 01:26:17.510701: train loss : -0.5973 
2022-04-12 01:26:29.924386: validation loss: -0.6211 
2022-04-12 01:26:29.965558: Average global foreground Dice: [0.869, 0.9606, 0.9735, 0.3564] 
2022-04-12 01:26:29.971614: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:26:30.654333: lr: 0.005417 
2022-04-12 01:26:30.664991: saving scheduled checkpoint file... 
2022-04-12 01:26:30.744030: saving checkpoint... 
2022-04-12 01:26:36.279414: done, saving took 5.61 seconds 
2022-04-12 01:26:36.292878: done 
2022-04-12 01:26:36.294415: This epoch took 126.561093 s
 
2022-04-12 01:26:36.295458: svmem(total=99932160000, available=79759437824, percent=20.2, used=16157868032, free=50071678976, active=40432881664, inactive=7577182208, buffers=31539200, cached=33671073792, shared=2190495744, slab=776011776) 
2022-04-12 01:26:36.296918: 
epoch:  247 
2022-04-12 01:28:24.832261: train loss : -0.5353 
2022-04-12 01:28:36.016913: validation loss: -0.5630 
2022-04-12 01:28:36.023759: Average global foreground Dice: [0.8037, 0.9474, 0.9694, 0.3602] 
2022-04-12 01:28:36.025110: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:28:36.933268: lr: 0.005397 
2022-04-12 01:28:36.934926: saving scheduled checkpoint file... 
2022-04-12 01:28:37.024331: saving checkpoint... 
2022-04-12 01:28:42.436479: done, saving took 5.50 seconds 
2022-04-12 01:28:42.602564: done 
2022-04-12 01:28:42.606570: This epoch took 126.308663 s
 
2022-04-12 01:28:42.674470: svmem(total=99932160000, available=79637659648, percent=20.3, used=16269565952, free=49932427264, active=40549482496, inactive=7603400704, buffers=31571968, cached=33698594816, shared=2200641536, slab=776704000) 
2022-04-12 01:28:42.681152: 
epoch:  248 
2022-04-12 01:30:31.866140: train loss : -0.5822 
2022-04-12 01:30:44.091527: validation loss: -0.6145 
2022-04-12 01:30:44.116459: Average global foreground Dice: [0.8519, 0.954, 0.9804, 0.3818] 
2022-04-12 01:30:44.121634: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:30:44.748308: lr: 0.005378 
2022-04-12 01:30:44.750481: saving scheduled checkpoint file... 
2022-04-12 01:30:44.866457: saving checkpoint... 
2022-04-12 01:30:50.553854: done, saving took 5.80 seconds 
2022-04-12 01:30:50.575366: done 
2022-04-12 01:30:50.577029: This epoch took 127.893592 s
 
2022-04-12 01:30:50.578246: svmem(total=99932160000, available=79863242752, percent=20.1, used=16043290624, free=50140516352, active=40314810368, inactive=7621120000, buffers=31612928, cached=33716740096, shared=2201137152, slab=777568256) 
2022-04-12 01:30:50.579250: 
epoch:  249 
2022-04-12 01:32:40.138060: train loss : -0.5048 
2022-04-12 01:32:51.918256: validation loss: -0.5681 
2022-04-12 01:32:51.921298: Average global foreground Dice: [0.8389, 0.9543, 0.9681, 0.3988] 
2022-04-12 01:32:51.922624: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:32:52.728363: lr: 0.005359 
2022-04-12 01:32:52.730042: saving scheduled checkpoint file... 
2022-04-12 01:32:52.858012: saving checkpoint... 
2022-04-12 01:32:57.636836: done, saving took 4.91 seconds 
2022-04-12 01:32:57.659708: done 
2022-04-12 01:32:57.661072: This epoch took 127.080897 s
 
2022-04-12 01:32:57.662075: svmem(total=99932160000, available=79768354816, percent=20.2, used=16138645504, free=50038059008, active=40413507584, inactive=7628005376, buffers=31645696, cached=33723809792, shared=2200489984, slab=777531392) 
2022-04-12 01:32:57.662866: 
epoch:  250 
2022-04-12 01:34:49.292032: train loss : -0.5514 
2022-04-12 01:35:01.767526: validation loss: -0.5737 
2022-04-12 01:35:01.769743: Average global foreground Dice: [0.8456, 0.9486, 0.9667, 0.373] 
2022-04-12 01:35:01.770494: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:35:02.706776: lr: 0.00534 
2022-04-12 01:35:02.708724: saving scheduled checkpoint file... 
2022-04-12 01:35:02.862249: saving checkpoint... 
2022-04-12 01:35:07.493856: done, saving took 4.78 seconds 
2022-04-12 01:35:07.516807: done 
2022-04-12 01:35:07.518508: This epoch took 129.854879 s
 
2022-04-12 01:35:07.519448: svmem(total=99932160000, available=79677284352, percent=20.3, used=16239050752, free=49929342976, active=40508518400, inactive=7635841024, buffers=31703040, cached=33732063232, shared=2191224832, slab=776232960) 
2022-04-12 01:35:07.520230: 
epoch:  251 
2022-04-12 01:36:58.127720: train loss : -0.5721 
2022-04-12 01:37:10.206401: validation loss: -0.5847 
2022-04-12 01:37:10.208602: Average global foreground Dice: [0.8509, 0.9546, 0.9621, 0.3931] 
2022-04-12 01:37:10.216672: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:37:11.210687: lr: 0.00532 
2022-04-12 01:37:11.212446: saving scheduled checkpoint file... 
2022-04-12 01:37:11.307501: saving checkpoint... 
2022-04-12 01:37:15.903166: done, saving took 4.69 seconds 
2022-04-12 01:37:15.927948: done 
2022-04-12 01:37:15.929455: This epoch took 128.408382 s
 
2022-04-12 01:37:15.930484: svmem(total=99932160000, available=79673139200, percent=20.3, used=16236523520, free=49916153856, active=40322654208, inactive=7840190464, buffers=31752192, cached=33747730432, shared=2197798912, slab=776617984) 
2022-04-12 01:37:15.931267: 
epoch:  252 
2022-04-12 01:39:04.696932: train loss : -0.6083 
2022-04-12 01:39:16.071344: validation loss: -0.6116 
2022-04-12 01:39:16.073706: Average global foreground Dice: [0.8451, 0.9581, 0.9785, 0.4015] 
2022-04-12 01:39:16.074689: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:39:17.109266: lr: 0.005301 
2022-04-12 01:39:17.115115: saving scheduled checkpoint file... 
2022-04-12 01:39:17.167917: saving checkpoint... 
2022-04-12 01:39:22.954190: done, saving took 5.84 seconds 
2022-04-12 01:39:22.972678: done 
2022-04-12 01:39:22.974075: This epoch took 127.042062 s
 
2022-04-12 01:39:22.975049: svmem(total=99932160000, available=79501975552, percent=20.4, used=16414793728, free=49751871488, active=40508227584, inactive=7823929344, buffers=31784960, cached=33733709824, shared=2190807040, slab=775245824) 
2022-04-12 01:39:22.975830: 
epoch:  253 
2022-04-12 01:41:11.485822: train loss : -0.6189 
2022-04-12 01:41:23.407916: validation loss: -0.6416 
2022-04-12 01:41:23.414269: Average global foreground Dice: [0.8648, 0.9635, 0.9791, 0.4196] 
2022-04-12 01:41:23.416002: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:41:24.270857: lr: 0.005282 
2022-04-12 01:41:24.273531: saving scheduled checkpoint file... 
2022-04-12 01:41:24.337477: saving checkpoint... 
2022-04-12 01:41:29.021248: done, saving took 4.75 seconds 
2022-04-12 01:41:29.036189: done 
2022-04-12 01:41:29.037501: This epoch took 126.060454 s
 
2022-04-12 01:41:29.038836: svmem(total=99932160000, available=79846084608, percent=20.1, used=16070696960, free=50078269440, active=40272814080, inactive=7727079424, buffers=31825920, cached=33751367680, shared=2190729216, slab=776704000) 
2022-04-12 01:41:29.039498: 
epoch:  254 
2022-04-12 01:43:18.317311: train loss : -0.6213 
2022-04-12 01:43:31.027655: validation loss: -0.5714 
2022-04-12 01:43:31.033365: Average global foreground Dice: [0.8492, 0.9438, 0.9599, 0.4061] 
2022-04-12 01:43:31.034418: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:43:31.991824: lr: 0.005262 
2022-04-12 01:43:31.993516: saving scheduled checkpoint file... 
2022-04-12 01:43:32.044688: saving checkpoint... 
2022-04-12 01:43:37.395142: done, saving took 5.40 seconds 
2022-04-12 01:43:37.552361: done 
2022-04-12 01:43:37.559002: This epoch took 128.518714 s
 
2022-04-12 01:43:37.563058: svmem(total=99932160000, available=79666978816, percent=20.3, used=16249286656, free=49896054784, active=40413536256, inactive=7770124288, buffers=31858688, cached=33754959872, shared=2191224832, slab=776245248) 
2022-04-12 01:43:37.584338: 
epoch:  255 
2022-04-12 01:45:27.262380: train loss : -0.5921 
2022-04-12 01:45:38.964108: validation loss: -0.6305 
2022-04-12 01:45:39.356090: Average global foreground Dice: [0.8731, 0.9574, 0.976, 0.441] 
2022-04-12 01:45:39.583753: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:45:40.680147: lr: 0.005243 
2022-04-12 01:45:40.795061: saving scheduled checkpoint file... 
2022-04-12 01:45:40.998085: saving checkpoint... 
2022-04-12 01:45:45.518798: done, saving took 4.62 seconds 
2022-04-12 01:45:45.539876: done 
2022-04-12 01:45:45.627896: saving checkpoint... 
2022-04-12 01:45:50.521048: done, saving took 4.98 seconds 
2022-04-12 01:45:50.538679: This epoch took 132.926238 s
 
2022-04-12 01:45:50.539922: svmem(total=99932160000, available=79726145536, percent=20.2, used=16180183040, free=49912086528, active=40396230656, inactive=7770288128, buffers=31907840, cached=33807982592, shared=2201092096, slab=775405568) 
2022-04-12 01:45:50.540828: 
epoch:  256 
2022-04-12 01:47:39.524345: train loss : -0.6132 
2022-04-12 01:47:51.344250: validation loss: -0.6259 
2022-04-12 01:47:51.346845: Average global foreground Dice: [0.8678, 0.9581, 0.9805, 0.3791] 
2022-04-12 01:47:51.348020: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:47:52.204975: lr: 0.005224 
2022-04-12 01:47:52.207028: saving scheduled checkpoint file... 
2022-04-12 01:47:52.294942: saving checkpoint... 
2022-04-12 01:47:57.061268: done, saving took 4.85 seconds 
2022-04-12 01:47:57.089300: done 
2022-04-12 01:47:57.193479: saving checkpoint... 
2022-04-12 01:48:01.768115: done, saving took 4.68 seconds 
2022-04-12 01:48:01.788157: This epoch took 131.245928 s
 
2022-04-12 01:48:01.789871: svmem(total=99932160000, available=79693860864, percent=20.3, used=16222871552, free=49896673280, active=40384045056, inactive=7802527744, buffers=31940608, cached=33780674560, shared=2190618624, slab=774320128) 
2022-04-12 01:48:01.791029: 
epoch:  257 
2022-04-12 01:49:51.473508: train loss : -0.6127 
2022-04-12 01:50:03.682858: validation loss: -0.5919 
2022-04-12 01:50:03.693377: Average global foreground Dice: [0.8541, 0.9486, 0.9468, 0.3965] 
2022-04-12 01:50:03.707790: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:50:04.529868: lr: 0.005204 
2022-04-12 01:50:04.587134: saving scheduled checkpoint file... 
2022-04-12 01:50:04.726352: saving checkpoint... 
2022-04-12 01:50:13.916123: done, saving took 9.29 seconds 
2022-04-12 01:50:13.949939: done 
2022-04-12 01:50:13.956234: This epoch took 132.164465 s
 
2022-04-12 01:50:13.957996: svmem(total=99932160000, available=79781244928, percent=20.2, used=16125153280, free=49958305792, active=40340971520, inactive=7779696640, buffers=31997952, cached=33816702976, shared=2201137152, slab=774922240) 
2022-04-12 01:50:14.012996: 
epoch:  258 
2022-04-12 01:52:03.561238: train loss : -0.6051 
2022-04-12 01:52:15.095488: validation loss: -0.6320 
2022-04-12 01:52:15.134221: Average global foreground Dice: [0.8501, 0.9591, 0.9772, 0.4061] 
2022-04-12 01:52:15.135992: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:52:15.984680: lr: 0.005185 
2022-04-12 01:52:15.996855: saving scheduled checkpoint file... 
2022-04-12 01:52:16.082250: saving checkpoint... 
2022-04-12 01:52:22.716220: done, saving took 6.68 seconds 
2022-04-12 01:52:22.732171: done 
2022-04-12 01:52:22.733608: This epoch took 128.717736 s
 
2022-04-12 01:52:22.734751: svmem(total=99932160000, available=79778836480, percent=20.2, used=16138424320, free=49951363072, active=40230551552, inactive=7901696000, buffers=32030720, cached=33810341888, shared=2190123008, slab=774746112) 
2022-04-12 01:52:22.735567: 
epoch:  259 
2022-04-12 01:54:11.798763: train loss : -0.6316 
2022-04-12 01:54:23.663469: validation loss: -0.6529 
2022-04-12 01:54:23.697008: Average global foreground Dice: [0.8803, 0.9618, 0.9812, 0.4146] 
2022-04-12 01:54:23.741777: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:54:24.643553: lr: 0.005166 
2022-04-12 01:54:24.651225: saving scheduled checkpoint file... 
2022-04-12 01:54:24.748670: saving checkpoint... 
2022-04-12 01:54:29.969280: done, saving took 5.29 seconds 
2022-04-12 01:54:30.016057: done 
2022-04-12 01:54:30.064067: saving checkpoint... 
2022-04-12 01:54:40.765445: done, saving took 10.75 seconds 
2022-04-12 01:54:40.953927: This epoch took 138.217510 s
 
2022-04-12 01:54:41.025318: svmem(total=99932160000, available=79772037120, percent=20.2, used=16144961536, free=49925906432, active=40238600192, inactive=7917592576, buffers=32063488, cached=33829228544, shared=2190233600, slab=775319552) 
2022-04-12 01:54:41.093582: 
epoch:  260 
2022-04-12 01:56:28.966258: train loss : -0.6341 
2022-04-12 01:56:40.410562: validation loss: -0.6605 
2022-04-12 01:56:40.476388: Average global foreground Dice: [0.8755, 0.9588, 0.9822, 0.4287] 
2022-04-12 01:56:40.552161: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:56:41.394353: lr: 0.005146 
2022-04-12 01:56:41.443254: saving scheduled checkpoint file... 
2022-04-12 01:56:41.513674: saving checkpoint... 
2022-04-12 01:56:50.327391: done, saving took 8.87 seconds 
2022-04-12 01:56:50.340908: done 
2022-04-12 01:56:50.388754: saving checkpoint... 
2022-04-12 01:56:56.355948: done, saving took 6.01 seconds 
2022-04-12 01:56:56.555448: This epoch took 135.455899 s
 
2022-04-12 01:56:56.562197: svmem(total=99932160000, available=79773270016, percent=20.2, used=16133230592, free=49930285056, active=40039841792, inactive=8108789760, buffers=32243712, cached=33836400640, shared=2200719360, slab=776409088) 
2022-04-12 01:56:56.572273: 
epoch:  261 
2022-04-12 01:58:46.187660: train loss : -0.6342 
2022-04-12 01:58:58.683266: validation loss: -0.6453 
2022-04-12 01:58:58.747065: Average global foreground Dice: [0.8613, 0.9633, 0.9786, 0.4065] 
2022-04-12 01:58:58.752592: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 01:58:59.519832: lr: 0.005127 
2022-04-12 01:58:59.539960: saving scheduled checkpoint file... 
2022-04-12 01:58:59.635959: saving checkpoint... 
2022-04-12 01:59:08.749046: done, saving took 9.19 seconds 
2022-04-12 01:59:08.821056: done 
2022-04-12 01:59:08.905333: saving checkpoint... 
2022-04-12 01:59:14.497047: done, saving took 5.66 seconds 
2022-04-12 01:59:14.661170: This epoch took 138.064631 s
 
2022-04-12 01:59:14.685950: svmem(total=99932160000, available=79731687424, percent=20.2, used=16184725504, free=49852260352, active=40169639936, inactive=8053862400, buffers=32292864, cached=33862881280, shared=2190729216, slab=776687616) 
2022-04-12 01:59:14.701919: 
epoch:  262 
2022-04-12 02:01:04.826926: train loss : -0.6404 
2022-04-12 02:01:15.739706: validation loss: -0.6496 
2022-04-12 02:01:15.780708: Average global foreground Dice: [0.8633, 0.9608, 0.9792, 0.3927] 
2022-04-12 02:01:15.784405: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:01:16.644256: lr: 0.005107 
2022-04-12 02:01:16.645267: saving scheduled checkpoint file... 
2022-04-12 02:01:16.747262: saving checkpoint... 
2022-04-12 02:01:22.516807: done, saving took 5.87 seconds 
2022-04-12 02:01:22.565939: done 
2022-04-12 02:01:22.653687: saving checkpoint... 
2022-04-12 02:01:28.118075: done, saving took 5.55 seconds 
2022-04-12 02:01:28.139676: This epoch took 133.431830 s
 
2022-04-12 02:01:28.141077: svmem(total=99932160000, available=79773204480, percent=20.2, used=16157057024, free=49862676480, active=40311914496, inactive=7902363648, buffers=32333824, cached=33880092672, shared=2176897024, slab=776781824) 
2022-04-12 02:01:28.142596: 
epoch:  263 
2022-04-12 02:03:16.272011: train loss : -0.6455 
2022-04-12 02:03:27.681040: validation loss: -0.6634 
2022-04-12 02:03:27.688435: Average global foreground Dice: [0.8863, 0.9629, 0.982, 0.3849] 
2022-04-12 02:03:27.746425: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:03:28.969498: lr: 0.005088 
2022-04-12 02:03:28.978216: saving scheduled checkpoint file... 
2022-04-12 02:03:29.105415: saving checkpoint... 
2022-04-12 02:03:37.677785: done, saving took 8.69 seconds 
2022-04-12 02:03:37.759252: done 
2022-04-12 02:03:37.822729: saving checkpoint... 
2022-04-12 02:03:43.100415: done, saving took 5.34 seconds 
2022-04-12 02:03:43.118738: This epoch took 134.975045 s
 
2022-04-12 02:03:43.120753: svmem(total=99932160000, available=79940173824, percent=20.0, used=15975882752, free=50021961728, active=40229425152, inactive=7828041728, buffers=32366592, cached=33901948928, shared=2191114240, slab=775626752) 
2022-04-12 02:03:43.122333: 
epoch:  264 
2022-04-12 02:05:33.079222: train loss : -0.6501 
2022-04-12 02:05:45.082648: validation loss: -0.6738 
2022-04-12 02:05:45.090062: Average global foreground Dice: [0.886, 0.9651, 0.9818, 0.426] 
2022-04-12 02:05:45.091434: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:05:46.057066: lr: 0.005069 
2022-04-12 02:05:46.058863: saving scheduled checkpoint file... 
2022-04-12 02:05:46.145998: saving checkpoint... 
2022-04-12 02:05:53.047880: done, saving took 6.99 seconds 
2022-04-12 02:05:53.181890: done 
2022-04-12 02:05:53.312797: saving checkpoint... 
2022-04-12 02:06:00.430460: done, saving took 7.20 seconds 
2022-04-12 02:06:00.553863: This epoch took 137.430127 s
 
2022-04-12 02:06:00.602429: svmem(total=99932160000, available=79626801152, percent=20.3, used=16290267136, free=49689395200, active=40545333248, inactive=7845072896, buffers=32428032, cached=33920069632, shared=2190168064, slab=776159232) 
2022-04-12 02:06:00.697186: 
epoch:  265 
2022-04-12 02:07:50.246358: train loss : -0.6150 
2022-04-12 02:08:02.040777: validation loss: -0.6203 
2022-04-12 02:08:02.092400: Average global foreground Dice: [0.8522, 0.9576, 0.976, 0.3948] 
2022-04-12 02:08:02.145840: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:08:03.041342: lr: 0.005049 
2022-04-12 02:08:03.077909: saving scheduled checkpoint file... 
2022-04-12 02:08:03.246105: saving checkpoint... 
2022-04-12 02:08:08.782257: done, saving took 5.65 seconds 
2022-04-12 02:08:08.843463: done 
2022-04-12 02:08:08.847815: This epoch took 128.145135 s
 
2022-04-12 02:08:08.852441: svmem(total=99932160000, available=79778582528, percent=20.2, used=16137801728, free=49819353088, active=40401408000, inactive=7854440448, buffers=32473088, cached=33942532096, shared=2190774272, slab=776495104) 
2022-04-12 02:08:08.859323: 
epoch:  266 
2022-04-12 02:09:59.481434: train loss : -0.6136 
2022-04-12 02:10:12.315607: validation loss: -0.6147 
2022-04-12 02:10:12.347754: Average global foreground Dice: [0.8679, 0.9505, 0.9736, 0.3891] 
2022-04-12 02:10:12.351890: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:10:13.136515: lr: 0.00503 
2022-04-12 02:10:13.151669: saving scheduled checkpoint file... 
2022-04-12 02:10:13.281375: saving checkpoint... 
2022-04-12 02:10:20.565644: done, saving took 7.39 seconds 
2022-04-12 02:10:20.682039: done 
2022-04-12 02:10:20.694630: This epoch took 131.827052 s
 
2022-04-12 02:10:20.698262: svmem(total=99932160000, available=79772790784, percent=20.2, used=16143704064, free=49795362816, active=40413917184, inactive=7872548864, buffers=32530432, cached=33960562688, shared=2190729216, slab=777908224) 
2022-04-12 02:10:20.719653: 
epoch:  267 
2022-04-12 02:12:09.731910: train loss : -0.6375 
2022-04-12 02:12:20.736253: validation loss: -0.6566 
2022-04-12 02:12:20.754227: Average global foreground Dice: [0.8725, 0.965, 0.9776, 0.4109] 
2022-04-12 02:12:20.778334: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:12:21.430740: lr: 0.00501 
2022-04-12 02:12:21.435401: saving scheduled checkpoint file... 
2022-04-12 02:12:21.662699: saving checkpoint... 
2022-04-12 02:12:28.668535: done, saving took 7.16 seconds 
2022-04-12 02:12:28.715048: done 
2022-04-12 02:12:28.742282: This epoch took 128.005935 s
 
2022-04-12 02:12:28.759256: svmem(total=99932160000, available=79863111680, percent=20.1, used=16053272576, free=49878622208, active=40326430720, inactive=7874121728, buffers=32563200, cached=33967702016, shared=2190729216, slab=778100736) 
2022-04-12 02:12:28.762438: 
epoch:  268 
2022-04-12 02:14:18.338332: train loss : -0.6255 
2022-04-12 02:14:30.746479: validation loss: -0.6463 
2022-04-12 02:14:30.760465: Average global foreground Dice: [0.8604, 0.9651, 0.9817, 0.4171] 
2022-04-12 02:14:30.762922: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:14:31.685284: lr: 0.004991 
2022-04-12 02:14:31.715373: saving scheduled checkpoint file... 
2022-04-12 02:14:31.865166: saving checkpoint... 
2022-04-12 02:14:37.295326: done, saving took 5.56 seconds 
2022-04-12 02:14:37.336371: done 
2022-04-12 02:14:37.510888: saving checkpoint... 
2022-04-12 02:14:42.845252: done, saving took 5.48 seconds 
2022-04-12 02:14:42.891285: This epoch took 134.125458 s
 
2022-04-12 02:14:42.910581: svmem(total=99932160000, available=79860867072, percent=20.1, used=16055373824, free=49857966080, active=40334372864, inactive=7890067456, buffers=32595968, cached=33986224128, shared=2190729216, slab=775516160) 
2022-04-12 02:14:42.938179: 
epoch:  269 
2022-04-12 02:16:32.496437: train loss : -0.6448 
2022-04-12 02:16:44.887673: validation loss: -0.6632 
2022-04-12 02:16:44.892937: Average global foreground Dice: [0.8933, 0.9606, 0.9808, 0.4179] 
2022-04-12 02:16:44.895638: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:16:45.796905: lr: 0.004971 
2022-04-12 02:16:45.800458: saving scheduled checkpoint file... 
2022-04-12 02:16:45.913581: saving checkpoint... 
2022-04-12 02:16:52.031145: done, saving took 6.21 seconds 
2022-04-12 02:16:52.080551: done 
2022-04-12 02:16:52.148371: saving checkpoint... 
2022-04-12 02:16:59.300417: done, saving took 7.20 seconds 
2022-04-12 02:16:59.643563: This epoch took 136.686132 s
 
2022-04-12 02:16:59.667369: svmem(total=99932160000, available=79732998144, percent=20.2, used=16183189504, free=49713115136, active=40456863744, inactive=7907020800, buffers=32645120, cached=34003210240, shared=2190729216, slab=777895936) 
2022-04-12 02:16:59.694724: 
epoch:  270 
2022-04-12 02:18:48.604028: train loss : -0.6481 
2022-04-12 02:19:01.447526: validation loss: -0.6538 
2022-04-12 02:19:01.454777: Average global foreground Dice: [0.8925, 0.9656, 0.9806, 0.3753] 
2022-04-12 02:19:01.455913: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:19:02.260237: lr: 0.004952 
2022-04-12 02:19:02.261680: saving scheduled checkpoint file... 
2022-04-12 02:19:02.313316: saving checkpoint... 
2022-04-12 02:19:08.819917: done, saving took 6.56 seconds 
2022-04-12 02:19:08.992938: done 
2022-04-12 02:19:09.084215: saving checkpoint... 
2022-04-12 02:19:19.676930: done, saving took 10.66 seconds 
2022-04-12 02:19:19.807361: This epoch took 140.101660 s
 
2022-04-12 02:19:19.829779: svmem(total=99932160000, available=79794827264, percent=20.2, used=16111058944, free=49766936576, active=40384966656, inactive=7925268480, buffers=32714752, cached=34021449728, shared=2201104384, slab=776536064) 
2022-04-12 02:19:19.904610: 
epoch:  271 
2022-04-12 02:21:08.034101: train loss : -0.6477 
2022-04-12 02:21:19.920647: validation loss: -0.6126 
2022-04-12 02:21:20.060369: Average global foreground Dice: [0.8202, 0.9614, 0.9704, 0.4072] 
2022-04-12 02:21:20.103907: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:21:21.086390: lr: 0.004933 
2022-04-12 02:21:21.116689: saving scheduled checkpoint file... 
2022-04-12 02:21:21.273102: saving checkpoint... 
2022-04-12 02:21:26.448717: done, saving took 5.27 seconds 
2022-04-12 02:21:26.469019: done 
2022-04-12 02:21:26.470369: This epoch took 126.552235 s
 
2022-04-12 02:21:26.471943: svmem(total=99932160000, available=79818973184, percent=20.1, used=16087232512, free=49773576192, active=40366809088, inactive=7939395584, buffers=32755712, cached=34038595584, shared=2200829952, slab=775921664) 
2022-04-12 02:21:26.473317: 
epoch:  272 
2022-04-12 02:23:15.589627: train loss : -0.6418 
2022-04-12 02:23:28.227576: validation loss: -0.6536 
2022-04-12 02:23:28.245120: Average global foreground Dice: [0.8835, 0.966, 0.9811, 0.3788] 
2022-04-12 02:23:28.248006: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:23:29.219838: lr: 0.004913 
2022-04-12 02:23:29.231227: saving scheduled checkpoint file... 
2022-04-12 02:23:29.343632: saving checkpoint... 
2022-04-12 02:23:35.826520: done, saving took 6.59 seconds 
2022-04-12 02:23:35.944729: done 
2022-04-12 02:23:35.963613: This epoch took 129.488881 s
 
2022-04-12 02:23:35.981696: svmem(total=99932160000, available=79861248000, percent=20.1, used=16054992896, free=49798340608, active=40333979648, inactive=7949676544, buffers=32804864, cached=34046021632, shared=2190729216, slab=776994816) 
2022-04-12 02:23:36.045514: 
epoch:  273 
2022-04-12 02:25:25.691838: train loss : -0.6480 
2022-04-12 02:25:37.859207: validation loss: -0.6345 
2022-04-12 02:25:37.915478: Average global foreground Dice: [0.8578, 0.9615, 0.9811, 0.4045] 
2022-04-12 02:25:37.925107: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:25:38.590976: lr: 0.004894 
2022-04-12 02:25:38.658406: saving scheduled checkpoint file... 
2022-04-12 02:25:38.795561: saving checkpoint... 
2022-04-12 02:25:45.826910: done, saving took 7.17 seconds 
2022-04-12 02:25:45.939614: done 
2022-04-12 02:25:45.943899: This epoch took 129.893649 s
 
2022-04-12 02:25:45.963482: svmem(total=99932160000, available=79917588480, percent=20.0, used=15988076544, free=49846812672, active=40263520256, inactive=7968043008, buffers=32845824, cached=34064424960, shared=2201292800, slab=777515008) 
2022-04-12 02:25:45.972233: 
epoch:  274 
2022-04-12 02:27:33.828706: train loss : -0.6462 
2022-04-12 02:27:44.291855: validation loss: -0.6531 
2022-04-12 02:27:44.298648: Average global foreground Dice: [0.8881, 0.9608, 0.9788, 0.3748] 
2022-04-12 02:27:44.300122: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:27:45.437431: lr: 0.004874 
2022-04-12 02:27:45.443334: saving scheduled checkpoint file... 
2022-04-12 02:27:45.585484: saving checkpoint... 
2022-04-12 02:27:51.856193: done, saving took 6.40 seconds 
2022-04-12 02:27:51.938770: done 
2022-04-12 02:27:51.940551: This epoch took 125.944719 s
 
2022-04-12 02:27:51.944867: svmem(total=99932160000, available=79775268864, percent=20.2, used=16140750848, free=49687040000, active=40417968128, inactive=7974891520, buffers=32878592, cached=34071490560, shared=2190774272, slab=775794688) 
2022-04-12 02:27:51.947131: 
epoch:  275 
2022-04-12 02:29:43.046746: train loss : -0.6453 
2022-04-12 02:29:55.790641: validation loss: -0.6283 
2022-04-12 02:29:55.805408: Average global foreground Dice: [0.8744, 0.9648, 0.9796, 0.3609] 
2022-04-12 02:29:55.807131: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:29:56.755506: lr: 0.004855 
2022-04-12 02:29:56.759603: saving scheduled checkpoint file... 
2022-04-12 02:29:56.856245: saving checkpoint... 
2022-04-12 02:30:03.203559: done, saving took 6.44 seconds 
2022-04-12 02:30:03.217516: done 
2022-04-12 02:30:03.219121: This epoch took 131.269972 s
 
2022-04-12 02:30:03.220444: svmem(total=99932160000, available=79809884160, percent=20.1, used=16106323968, free=49704140800, active=40384000000, inactive=7992172544, buffers=32919552, cached=34088775680, shared=2190618624, slab=776499200) 
2022-04-12 02:30:03.221465: 
epoch:  276 
2022-04-12 02:31:52.353873: train loss : -0.6517 
2022-04-12 02:32:04.862664: validation loss: -0.6419 
2022-04-12 02:32:04.869978: Average global foreground Dice: [0.8765, 0.9586, 0.9758, 0.4138] 
2022-04-12 02:32:04.870971: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:32:05.731224: lr: 0.004835 
2022-04-12 02:32:05.734187: saving scheduled checkpoint file... 
2022-04-12 02:32:05.858631: saving checkpoint... 
2022-04-12 02:32:16.167436: done, saving took 10.43 seconds 
2022-04-12 02:32:16.348420: done 
2022-04-12 02:32:16.384347: This epoch took 133.161984 s
 
2022-04-12 02:32:16.440554: svmem(total=99932160000, available=79832731648, percent=20.1, used=16086978560, free=49719349248, active=40249954304, inactive=8111702016, buffers=32968704, cached=34092863488, shared=2187149312, slab=776900608) 
2022-04-12 02:32:16.444355: 
epoch:  277 
2022-04-12 02:34:06.065000: train loss : -0.6586 
2022-04-12 02:34:17.663585: validation loss: -0.6782 
2022-04-12 02:34:17.682284: Average global foreground Dice: [0.881, 0.9668, 0.9822, 0.4052] 
2022-04-12 02:34:17.719196: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:34:18.624182: lr: 0.004816 
2022-04-12 02:34:18.630215: saving scheduled checkpoint file... 
2022-04-12 02:34:18.777541: saving checkpoint... 
2022-04-12 02:34:25.756269: done, saving took 7.07 seconds 
2022-04-12 02:34:25.852346: done 
2022-04-12 02:34:25.885230: This epoch took 129.427354 s
 
2022-04-12 02:34:25.897058: svmem(total=99932160000, available=79657992192, percent=20.3, used=16258494464, free=49536811008, active=40508485632, inactive=8035078144, buffers=33001472, cached=34103853056, shared=2190323712, slab=776904704) 
2022-04-12 02:34:25.959465: 
epoch:  278 
2022-04-12 02:36:15.474338: train loss : -0.6601 
2022-04-12 02:36:27.316026: validation loss: -0.6639 
2022-04-12 02:36:27.366797: Average global foreground Dice: [0.8765, 0.9645, 0.9825, 0.4153] 
2022-04-12 02:36:27.369010: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:36:28.327002: lr: 0.004796 
2022-04-12 02:36:28.382462: saving scheduled checkpoint file... 
2022-04-12 02:36:28.521411: saving checkpoint... 
2022-04-12 02:36:40.591663: done, saving took 12.20 seconds 
2022-04-12 02:36:40.787402: done 
2022-04-12 02:36:40.906632: saving checkpoint... 
2022-04-12 02:36:50.837975: done, saving took 10.04 seconds 
2022-04-12 02:36:50.973913: This epoch took 145.012902 s
 
2022-04-12 02:36:50.993820: svmem(total=99932160000, available=79821598720, percent=20.1, used=16098201600, free=49715527680, active=40276623360, inactive=8087400448, buffers=33058816, cached=34085371904, shared=2187063296, slab=777310208) 
2022-04-12 02:36:51.060427: 
epoch:  279 
2022-04-12 02:38:40.161494: train loss : -0.6579 
2022-04-12 02:38:52.066719: validation loss: -0.6540 
2022-04-12 02:38:52.101679: Average global foreground Dice: [0.8837, 0.9646, 0.9783, 0.3955] 
2022-04-12 02:38:52.108969: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:38:52.997270: lr: 0.004776 
2022-04-12 02:38:53.029072: saving scheduled checkpoint file... 
2022-04-12 02:38:53.218880: saving checkpoint... 
2022-04-12 02:39:01.564775: done, saving took 8.49 seconds 
2022-04-12 02:39:01.632983: done 
2022-04-12 02:39:01.768695: saving checkpoint... 
2022-04-12 02:39:07.615205: done, saving took 5.98 seconds 
2022-04-12 02:39:07.690757: This epoch took 136.572522 s
 
2022-04-12 02:39:07.701127: svmem(total=99932160000, available=79897190400, percent=20.0, used=16018345984, free=49773699072, active=40271855616, inactive=8022548480, buffers=33099776, cached=34107015168, shared=2191310848, slab=777437184) 
2022-04-12 02:39:07.703188: 
epoch:  280 
2022-04-12 02:40:59.050365: train loss : -0.6530 
2022-04-12 02:41:11.620127: validation loss: -0.6472 
2022-04-12 02:41:11.711994: Average global foreground Dice: [0.8874, 0.9657, 0.9781, 0.3773] 
2022-04-12 02:41:11.793347: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:41:12.650055: lr: 0.004757 
2022-04-12 02:41:12.727166: saving scheduled checkpoint file... 
2022-04-12 02:41:12.932948: saving checkpoint... 
2022-04-12 02:41:19.282369: done, saving took 6.49 seconds 
2022-04-12 02:41:19.387445: done 
2022-04-12 02:41:19.407975: This epoch took 131.701927 s
 
2022-04-12 02:41:19.429211: svmem(total=99932160000, available=79801602048, percent=20.1, used=16118067200, free=49670230016, active=40390217728, inactive=8014172160, buffers=33157120, cached=34110705664, shared=2187259904, slab=777572352) 
2022-04-12 02:41:19.441855: 
epoch:  281 
2022-04-12 02:43:09.796137: train loss : -0.6409 
2022-04-12 02:43:22.708600: validation loss: -0.6393 
2022-04-12 02:43:22.751511: Average global foreground Dice: [0.8645, 0.9549, 0.9811, 0.392] 
2022-04-12 02:43:22.800788: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:43:23.847938: lr: 0.004737 
2022-04-12 02:43:23.892712: saving scheduled checkpoint file... 
2022-04-12 02:43:23.992460: saving checkpoint... 
2022-04-12 02:43:34.402886: done, saving took 10.51 seconds 
2022-04-12 02:43:34.436695: done 
2022-04-12 02:43:34.450750: This epoch took 134.956529 s
 
2022-04-12 02:43:34.493912: svmem(total=99932160000, available=79792148480, percent=20.2, used=16117010432, free=49643388928, active=40397357056, inactive=8042000384, buffers=33189888, cached=34138570752, shared=2197745664, slab=777433088) 
2022-04-12 02:43:34.501845: 
epoch:  282 
2022-04-12 02:45:23.366862: train loss : -0.6266 
2022-04-12 02:45:34.891447: validation loss: -0.6325 
2022-04-12 02:45:34.961690: Average global foreground Dice: [0.8726, 0.9613, 0.9679, 0.3902] 
2022-04-12 02:45:35.037126: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:45:35.974038: lr: 0.004718 
2022-04-12 02:45:35.980260: saving scheduled checkpoint file... 
2022-04-12 02:45:36.056428: saving checkpoint... 
2022-04-12 02:45:43.029693: done, saving took 7.05 seconds 
2022-04-12 02:45:43.164420: done 
2022-04-12 02:45:43.200060: This epoch took 128.694816 s
 
2022-04-12 02:45:43.249042: svmem(total=99932160000, available=79765311488, percent=20.2, used=16150327296, free=49599057920, active=40430084096, inactive=8052903936, buffers=33234944, cached=34149539840, shared=2191224832, slab=776704000) 
2022-04-12 02:45:43.300706: 
epoch:  283 
2022-04-12 02:47:31.762228: train loss : -0.6123 
2022-04-12 02:47:43.903473: validation loss: -0.5846 
2022-04-12 02:47:43.932184: Average global foreground Dice: [0.8713, 0.9504, 0.966, 0.3786] 
2022-04-12 02:47:43.947606: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:47:44.849073: lr: 0.004698 
2022-04-12 02:47:44.853997: saving scheduled checkpoint file... 
2022-04-12 02:47:44.959663: saving checkpoint... 
2022-04-12 02:47:55.773105: done, saving took 10.91 seconds 
2022-04-12 02:47:55.890614: done 
2022-04-12 02:47:55.896138: This epoch took 132.578054 s
 
2022-04-12 02:47:55.959855: svmem(total=99932160000, available=79590043648, percent=20.4, used=16325935104, free=49405349888, active=40601804800, inactive=8071004160, buffers=33267712, cached=34167607296, shared=2190917632, slab=777809920) 
2022-04-12 02:47:56.031313: 
epoch:  284 
2022-04-12 02:49:45.483937: train loss : -0.5761 
2022-04-12 02:49:57.295651: validation loss: -0.6022 
2022-04-12 02:49:57.326320: Average global foreground Dice: [0.8437, 0.9574, 0.9741, 0.415] 
2022-04-12 02:49:57.369158: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:49:58.504544: lr: 0.004679 
2022-04-12 02:49:58.563388: saving scheduled checkpoint file... 
2022-04-12 02:49:58.699531: saving checkpoint... 
2022-04-12 02:50:07.487945: done, saving took 8.90 seconds 
2022-04-12 02:50:07.618964: done 
2022-04-12 02:50:07.622574: This epoch took 131.583033 s
 
2022-04-12 02:50:07.624896: svmem(total=99932160000, available=79743766528, percent=20.2, used=16172224512, free=49551208448, active=40445730816, inactive=8078987264, buffers=33308672, cached=34175418368, shared=2190729216, slab=777134080) 
2022-04-12 02:50:07.653257: 
epoch:  285 
2022-04-12 02:51:57.166421: train loss : -0.6037 
2022-04-12 02:52:09.462575: validation loss: -0.6419 
2022-04-12 02:52:09.583585: Average global foreground Dice: [0.8869, 0.9609, 0.9776, 0.4023] 
2022-04-12 02:52:09.594107: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:52:10.151423: lr: 0.004659 
2022-04-12 02:52:10.154461: saving scheduled checkpoint file... 
2022-04-12 02:52:10.265970: saving checkpoint... 
2022-04-12 02:52:21.056971: done, saving took 10.90 seconds 
2022-04-12 02:52:21.141940: done 
2022-04-12 02:52:21.143475: This epoch took 133.440524 s
 
2022-04-12 02:52:21.144522: svmem(total=99932160000, available=79883440128, percent=20.1, used=16028110848, free=49630601216, active=40302673920, inactive=8143167488, buffers=33357824, cached=34240090112, shared=2195189760, slab=776957952) 
2022-04-12 02:52:21.145402: 
epoch:  286 
2022-04-12 02:54:11.964672: train loss : -0.6284 
2022-04-12 02:54:24.409240: validation loss: -0.6378 
2022-04-12 02:54:24.414644: Average global foreground Dice: [0.8863, 0.9613, 0.9789, 0.3936] 
2022-04-12 02:54:24.418161: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:54:25.040203: lr: 0.004639 
2022-04-12 02:54:25.042946: saving scheduled checkpoint file... 
2022-04-12 02:54:25.173066: saving checkpoint... 
2022-04-12 02:54:33.322067: done, saving took 8.28 seconds 
2022-04-12 02:54:33.391563: done 
2022-04-12 02:54:33.395113: This epoch took 132.248856 s
 
2022-04-12 02:54:33.397543: svmem(total=99932160000, available=79742021632, percent=20.2, used=16163516416, free=49471770624, active=40439881728, inactive=8166555648, buffers=33390592, cached=34263482368, shared=2201214976, slab=777682944) 
2022-04-12 02:54:33.400761: 
epoch:  287 
2022-04-12 02:56:24.598748: train loss : -0.6465 
2022-04-12 02:56:36.591494: validation loss: -0.6541 
2022-04-12 02:56:36.605931: Average global foreground Dice: [0.8732, 0.9607, 0.982, 0.4054] 
2022-04-12 02:56:36.621556: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:56:37.851404: lr: 0.00462 
2022-04-12 02:56:37.855372: saving scheduled checkpoint file... 
2022-04-12 02:56:37.953492: saving checkpoint... 
2022-04-12 02:56:45.764162: done, saving took 7.89 seconds 
2022-04-12 02:56:45.924801: done 
2022-04-12 02:56:45.939171: This epoch took 132.527118 s
 
2022-04-12 02:56:45.943171: svmem(total=99932160000, available=79745863680, percent=20.2, used=16170053632, free=49467740160, active=40448876544, inactive=8163848192, buffers=33456128, cached=34260910080, shared=2190794752, slab=777183232) 
2022-04-12 02:56:45.948425: 
epoch:  288 
2022-04-12 02:58:34.953120: train loss : -0.6532 
2022-04-12 02:58:47.271122: validation loss: -0.6563 
2022-04-12 02:58:47.303888: Average global foreground Dice: [0.8822, 0.9612, 0.9806, 0.4308] 
2022-04-12 02:58:47.361911: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 02:58:48.176671: lr: 0.0046 
2022-04-12 02:58:48.200927: saving scheduled checkpoint file... 
2022-04-12 02:58:48.310328: saving checkpoint... 
2022-04-12 02:58:57.301657: done, saving took 9.06 seconds 
2022-04-12 02:58:57.382387: done 
2022-04-12 02:58:57.435519: saving checkpoint... 
2022-04-12 02:59:05.151144: done, saving took 7.76 seconds 
2022-04-12 02:59:05.303026: This epoch took 139.291766 s
 
2022-04-12 02:59:05.360816: svmem(total=99932160000, available=79733415936, percent=20.2, used=16182616064, free=49436819456, active=40460865536, inactive=8182321152, buffers=33488896, cached=34279235584, shared=2190729216, slab=777551872) 
2022-04-12 02:59:05.444687: 
epoch:  289 
2022-04-12 03:00:53.767967: train loss : -0.6393 
2022-04-12 03:01:05.655106: validation loss: -0.6428 
2022-04-12 03:01:05.720325: Average global foreground Dice: [0.8759, 0.9585, 0.975, 0.4324] 
2022-04-12 03:01:05.795801: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:01:06.413118: lr: 0.004581 
2022-04-12 03:01:06.538644: saving scheduled checkpoint file... 
2022-04-12 03:01:06.594262: saving checkpoint... 
2022-04-12 03:01:18.246072: done, saving took 11.70 seconds 
2022-04-12 03:01:18.359302: done 
2022-04-12 03:01:18.456415: saving checkpoint... 
2022-04-12 03:01:29.959507: done, saving took 11.57 seconds 
2022-04-12 03:01:30.101216: This epoch took 144.650824 s
 
2022-04-12 03:01:30.175406: svmem(total=99932160000, available=79851855872, percent=20.1, used=16060067840, free=49577369600, active=40247357440, inactive=8254828544, buffers=33546240, cached=34261176320, shared=2194653184, slab=778313728) 
2022-04-12 03:01:30.182943: 
epoch:  290 
2022-04-12 03:03:18.764353: train loss : -0.6589 
2022-04-12 03:03:30.190150: validation loss: -0.6735 
2022-04-12 03:03:30.348577: Average global foreground Dice: [0.8926, 0.9639, 0.9844, 0.4233] 
2022-04-12 03:03:30.402425: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:03:31.302549: lr: 0.004561 
2022-04-12 03:03:31.361356: saving scheduled checkpoint file... 
2022-04-12 03:03:31.455644: saving checkpoint... 
2022-04-12 03:03:38.895825: done, saving took 7.53 seconds 
2022-04-12 03:03:38.952355: done 
2022-04-12 03:03:39.013074: saving checkpoint... 
2022-04-12 03:03:45.076031: done, saving took 6.11 seconds 
2022-04-12 03:03:45.117004: This epoch took 134.904427 s
 
2022-04-12 03:03:45.142514: svmem(total=99932160000, available=79594176512, percent=20.4, used=16322256896, free=49301258240, active=40549961728, inactive=8226881536, buffers=33579008, cached=34275065856, shared=2190127104, slab=777826304) 
2022-04-12 03:03:45.145374: 
epoch:  291 
2022-04-12 03:05:35.229917: train loss : -0.6544 
2022-04-12 03:05:46.625267: validation loss: -0.6770 
2022-04-12 03:05:46.636293: Average global foreground Dice: [0.8981, 0.9664, 0.9808, 0.4352] 
2022-04-12 03:05:46.648728: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:05:47.213417: lr: 0.004541 
2022-04-12 03:05:47.219108: saving scheduled checkpoint file... 
2022-04-12 03:05:47.357030: saving checkpoint... 
2022-04-12 03:05:53.634234: done, saving took 6.37 seconds 
2022-04-12 03:05:53.767826: done 
2022-04-12 03:05:53.818064: saving checkpoint... 
2022-04-12 03:06:01.107059: done, saving took 7.34 seconds 
2022-04-12 03:06:01.210817: This epoch took 136.061731 s
 
2022-04-12 03:06:01.238875: svmem(total=99932160000, available=79657684992, percent=20.3, used=16258068480, free=49357053952, active=40518238208, inactive=8208695296, buffers=33624064, cached=34283413504, shared=2190729216, slab=776769536) 
2022-04-12 03:06:01.250485: 
epoch:  292 
2022-04-12 03:07:49.256126: train loss : -0.6328 
2022-04-12 03:08:00.380041: validation loss: -0.6628 
2022-04-12 03:08:00.432458: Average global foreground Dice: [0.8795, 0.9607, 0.9808, 0.3991] 
2022-04-12 03:08:00.447773: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:08:01.323414: lr: 0.004522 
2022-04-12 03:08:01.361763: saving scheduled checkpoint file... 
2022-04-12 03:08:01.510411: saving checkpoint... 
2022-04-12 03:08:10.375126: done, saving took 8.96 seconds 
2022-04-12 03:08:10.473854: done 
2022-04-12 03:08:10.480884: This epoch took 129.225892 s
 
2022-04-12 03:08:10.495463: svmem(total=99932160000, available=79627718656, percent=20.3, used=16291598336, free=49309372416, active=40564715520, inactive=8201162752, buffers=33669120, cached=34297520128, shared=2187259904, slab=777662464) 
2022-04-12 03:08:10.524564: 
epoch:  293 
2022-04-12 03:09:57.878815: train loss : -0.6357 
2022-04-12 03:10:09.506387: validation loss: -0.6369 
2022-04-12 03:10:09.565110: Average global foreground Dice: [0.876, 0.9617, 0.98, 0.4003] 
2022-04-12 03:10:09.631417: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:10:10.691266: lr: 0.004502 
2022-04-12 03:10:10.710559: saving scheduled checkpoint file... 
2022-04-12 03:10:10.866951: saving checkpoint... 
2022-04-12 03:10:19.822772: done, saving took 9.07 seconds 
2022-04-12 03:10:19.912348: done 
2022-04-12 03:10:19.950155: This epoch took 129.423792 s
 
2022-04-12 03:10:20.001721: svmem(total=99932160000, available=79817854976, percent=20.1, used=16097796096, free=49482080256, active=40379645952, inactive=8221171712, buffers=33718272, cached=34318565376, shared=2190807040, slab=776708096) 
2022-04-12 03:10:20.032974: 
epoch:  294 
2022-04-12 03:12:08.972052: train loss : -0.6283 
2022-04-12 03:12:22.057989: validation loss: -0.6418 
2022-04-12 03:12:22.117653: Average global foreground Dice: [0.8576, 0.964, 0.9812, 0.3987] 
2022-04-12 03:12:22.121073: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:12:22.909544: lr: 0.004482 
2022-04-12 03:12:22.933516: saving scheduled checkpoint file... 
2022-04-12 03:12:23.046674: saving checkpoint... 
2022-04-12 03:12:36.103438: done, saving took 13.13 seconds 
2022-04-12 03:12:36.295024: done 
2022-04-12 03:12:36.308698: This epoch took 136.243823 s
 
2022-04-12 03:12:36.407423: svmem(total=99932160000, available=79824097280, percent=20.1, used=16091459584, free=49470840832, active=40368291840, inactive=8238682112, buffers=33759232, cached=34336100352, shared=2190917632, slab=777850880) 
2022-04-12 03:12:36.472480: 
epoch:  295 
2022-04-12 03:14:25.884942: train loss : -0.6361 
2022-04-12 03:14:37.902851: validation loss: -0.6616 
2022-04-12 03:14:37.936629: Average global foreground Dice: [0.9017, 0.9612, 0.9834, 0.4247] 
2022-04-12 03:14:38.024131: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:14:38.707529: lr: 0.004463 
2022-04-12 03:14:38.741523: saving scheduled checkpoint file... 
2022-04-12 03:14:38.990572: saving checkpoint... 
2022-04-12 03:14:52.719754: done, saving took 13.84 seconds 
2022-04-12 03:14:52.848005: done 
2022-04-12 03:14:52.968454: saving checkpoint... 
2022-04-12 03:15:00.448149: done, saving took 7.58 seconds 
2022-04-12 03:15:00.558173: This epoch took 144.080923 s
 
2022-04-12 03:15:00.605886: svmem(total=99932160000, available=80034754560, percent=19.9, used=15873499136, free=49672732672, active=40150204416, inactive=8254476288, buffers=33792000, cached=34352136192, shared=2198163456, slab=779132928) 
2022-04-12 03:15:00.608407: 
epoch:  296 
2022-04-12 03:16:49.622604: train loss : -0.6478 
2022-04-12 03:17:01.447137: validation loss: -0.6518 
2022-04-12 03:17:01.517564: Average global foreground Dice: [0.8891, 0.9568, 0.9817, 0.4067] 
2022-04-12 03:17:01.524870: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:17:02.304021: lr: 0.004443 
2022-04-12 03:17:02.307335: saving scheduled checkpoint file... 
2022-04-12 03:17:02.436667: saving checkpoint... 
2022-04-12 03:17:07.871840: done, saving took 5.56 seconds 
2022-04-12 03:17:07.914599: done 
2022-04-12 03:17:08.023558: saving checkpoint... 
2022-04-12 03:17:17.870277: done, saving took 9.95 seconds 
2022-04-12 03:17:18.014295: This epoch took 137.397262 s
 
2022-04-12 03:17:18.018981: svmem(total=99932160000, available=80309026816, percent=19.6, used=15784673280, free=49929486336, active=40063787008, inactive=8082608128, buffers=33845248, cached=34184155136, shared=2012598272, slab=778358784) 
2022-04-12 03:17:18.021279: 
epoch:  297 
2022-04-12 03:19:06.999699: train loss : -0.6601 
2022-04-12 03:19:19.387586: validation loss: -0.6619 
2022-04-12 03:19:19.431110: Average global foreground Dice: [0.8888, 0.9611, 0.9816, 0.4197] 
2022-04-12 03:19:19.491226: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:19:20.410344: lr: 0.004423 
2022-04-12 03:19:20.476248: saving scheduled checkpoint file... 
2022-04-12 03:19:20.701213: saving checkpoint... 
2022-04-12 03:19:28.203020: done, saving took 7.65 seconds 
2022-04-12 03:19:28.269209: done 
2022-04-12 03:19:28.397140: saving checkpoint... 
2022-04-12 03:19:50.136301: done, saving took 21.86 seconds 
2022-04-12 03:19:51.917754: This epoch took 153.895174 s
 
2022-04-12 03:19:52.487497: svmem(total=99932160000, available=79838318592, percent=20.1, used=16091197440, free=49439162368, active=40350650368, inactive=8270004224, buffers=33886208, cached=34367913984, shared=2176741376, slab=792489984) 
2022-04-12 03:19:53.094809: 
epoch:  298 
2022-04-12 03:21:43.393273: train loss : -0.5945 
2022-04-12 03:21:56.105087: validation loss: -0.5329 
2022-04-12 03:21:56.893515: Average global foreground Dice: [0.8057, 0.9373, 0.9481, 0.3848] 
2022-04-12 03:21:57.953690: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:22:00.120680: lr: 0.004404 
2022-04-12 03:22:00.683182: saving scheduled checkpoint file... 
2022-04-12 03:22:01.232136: saving checkpoint... 
2022-04-12 03:22:12.721290: done, saving took 11.62 seconds 
2022-04-12 03:22:12.796188: done 
2022-04-12 03:22:12.799889: This epoch took 139.362112 s
 
2022-04-12 03:22:12.875084: svmem(total=99932160000, available=79623397376, percent=20.3, used=16291950592, free=49206632448, active=40560451584, inactive=8301547520, buffers=33943552, cached=34399633408, shared=2190884864, slab=779436032) 
2022-04-12 03:22:12.876936: 
epoch:  299 
2022-04-12 03:24:04.366387: train loss : -0.5917 
2022-04-12 03:24:16.754122: validation loss: -0.6152 
2022-04-12 03:24:16.756498: Average global foreground Dice: [0.8448, 0.9546, 0.9737, 0.3754] 
2022-04-12 03:24:16.757454: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:24:17.783012: lr: 0.004384 
2022-04-12 03:24:17.787093: saving scheduled checkpoint file... 
2022-04-12 03:24:17.882349: saving checkpoint... 
2022-04-12 03:24:29.996921: done, saving took 12.21 seconds 
2022-04-12 03:24:30.075358: done 
2022-04-12 03:24:30.143267: This epoch took 137.263468 s
 
2022-04-12 03:24:30.200721: svmem(total=99932160000, available=79692840960, percent=20.3, used=16222494720, free=49267240960, active=40504315904, inactive=8310247424, buffers=33984512, cached=34408439808, shared=2190807040, slab=777822208) 
2022-04-12 03:24:30.206922: 
epoch:  300 
2022-04-12 03:26:20.408869: train loss : -0.6372 
2022-04-12 03:26:31.715400: validation loss: -0.6321 
2022-04-12 03:26:31.866934: Average global foreground Dice: [0.8554, 0.96, 0.9728, 0.3945] 
2022-04-12 03:26:31.919199: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:26:32.841872: lr: 0.004364 
2022-04-12 03:26:32.928798: saving scheduled checkpoint file... 
2022-04-12 03:26:33.174182: saving checkpoint... 
2022-04-12 03:26:40.012464: done, saving took 6.93 seconds 
2022-04-12 03:26:40.209500: done 
2022-04-12 03:26:40.213041: This epoch took 129.913207 s
 
2022-04-12 03:26:40.217690: svmem(total=99932160000, available=79948070912, percent=20.0, used=15967674368, free=49505058816, active=40245960704, inactive=8327135232, buffers=34033664, cached=34425393152, shared=2190422016, slab=778354688) 
2022-04-12 03:26:40.298480: 
epoch:  301 
2022-04-12 03:28:30.011046: train loss : -0.6423 
2022-04-12 03:28:42.010596: validation loss: -0.6546 
2022-04-12 03:28:42.034483: Average global foreground Dice: [0.8799, 0.9619, 0.98, 0.4123] 
2022-04-12 03:28:42.047162: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:28:43.133646: lr: 0.004344 
2022-04-12 03:28:43.140141: saving scheduled checkpoint file... 
2022-04-12 03:28:43.248036: saving checkpoint... 
2022-04-12 03:28:55.261374: done, saving took 12.12 seconds 
2022-04-12 03:28:55.397735: done 
2022-04-12 03:28:55.400881: This epoch took 135.100544 s
 
2022-04-12 03:28:55.404096: svmem(total=99932160000, available=79814713344, percent=20.1, used=16100720640, free=49354100736, active=40380207104, inactive=8344883200, buffers=34066432, cached=34443272192, shared=2190618624, slab=777977856) 
2022-04-12 03:28:55.489860: 
epoch:  302 
2022-04-12 03:30:44.888895: train loss : -0.6568 
2022-04-12 03:30:57.209956: validation loss: -0.6688 
2022-04-12 03:30:57.329182: Average global foreground Dice: [0.8894, 0.9637, 0.9794, 0.4261] 
2022-04-12 03:30:57.336889: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:30:58.311890: lr: 0.004325 
2022-04-12 03:30:58.322591: saving scheduled checkpoint file... 
2022-04-12 03:30:58.484123: saving checkpoint... 
2022-04-12 03:31:11.442170: done, saving took 13.05 seconds 
2022-04-12 03:31:11.710466: done 
2022-04-12 03:31:11.716555: This epoch took 136.205184 s
 
2022-04-12 03:31:11.728544: svmem(total=99932160000, available=79913263104, percent=20.0, used=16002691072, free=49435299840, active=40273080320, inactive=8361611264, buffers=34123776, cached=34460045312, shared=2190155776, slab=779833344) 
2022-04-12 03:31:11.771560: 
epoch:  303 
2022-04-12 03:33:00.936223: train loss : -0.6363 
2022-04-12 03:33:13.306117: validation loss: -0.6606 
2022-04-12 03:33:13.355929: Average global foreground Dice: [0.8898, 0.9629, 0.9793, 0.397] 
2022-04-12 03:33:13.368232: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:33:14.180886: lr: 0.004305 
2022-04-12 03:33:14.217256: saving scheduled checkpoint file... 
2022-04-12 03:33:14.374343: saving checkpoint... 
2022-04-12 03:33:22.048352: done, saving took 7.80 seconds 
2022-04-12 03:33:22.109088: done 
2022-04-12 03:33:22.116474: This epoch took 130.273369 s
 
2022-04-12 03:33:22.121456: svmem(total=99932160000, available=79751356416, percent=20.2, used=16164151296, free=49265459200, active=40442060800, inactive=8370139136, buffers=34156544, cached=34468392960, shared=2190729216, slab=778002432) 
2022-04-12 03:33:22.123440: 
epoch:  304 
2022-04-12 03:35:11.940582: train loss : -0.6473 
2022-04-12 03:35:24.877517: validation loss: -0.6414 
2022-04-12 03:35:24.887695: Average global foreground Dice: [0.8685, 0.9543, 0.9741, 0.4163] 
2022-04-12 03:35:24.890570: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:35:25.776625: lr: 0.004285 
2022-04-12 03:35:25.780021: saving scheduled checkpoint file... 
2022-04-12 03:35:25.900250: saving checkpoint... 
2022-04-12 03:35:32.854534: done, saving took 7.07 seconds 
2022-04-12 03:35:32.922265: done 
2022-04-12 03:35:32.966612: This epoch took 130.838832 s
 
2022-04-12 03:35:32.969942: svmem(total=99932160000, available=79599312896, percent=20.3, used=16323592192, free=49095999488, active=40601780224, inactive=8380006400, buffers=34197504, cached=34478370816, shared=2183294976, slab=778035200) 
2022-04-12 03:35:32.974190: 
epoch:  305 
2022-04-12 03:37:23.073956: train loss : -0.6409 
2022-04-12 03:37:34.888656: validation loss: -0.6704 
2022-04-12 03:37:34.910724: Average global foreground Dice: [0.8877, 0.9678, 0.9773, 0.4128] 
2022-04-12 03:37:34.914562: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:37:35.967039: lr: 0.004265 
2022-04-12 03:37:35.969881: saving scheduled checkpoint file... 
2022-04-12 03:37:36.060205: saving checkpoint... 
2022-04-12 03:37:47.556310: done, saving took 11.58 seconds 
2022-04-12 03:37:47.642019: done 
2022-04-12 03:37:47.643546: This epoch took 134.665850 s
 
2022-04-12 03:37:47.646071: svmem(total=99932160000, available=79687831552, percent=20.3, used=16227782656, free=49167159296, active=40509919232, inactive=8404561920, buffers=34246656, cached=34502971392, shared=2190540800, slab=777748480) 
2022-04-12 03:37:47.647897: 
epoch:  306 
2022-04-12 03:39:35.851421: train loss : -0.6546 
2022-04-12 03:39:48.120065: validation loss: -0.6292 
2022-04-12 03:39:48.155028: Average global foreground Dice: [0.8779, 0.9548, 0.9555, 0.3838] 
2022-04-12 03:39:48.198731: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:39:49.328627: lr: 0.004245 
2022-04-12 03:39:49.379452: saving scheduled checkpoint file... 
2022-04-12 03:39:49.521698: saving checkpoint... 
2022-04-12 03:39:57.673985: done, saving took 8.22 seconds 
2022-04-12 03:39:57.758496: done 
2022-04-12 03:39:57.765744: This epoch took 130.101760 s
 
2022-04-12 03:39:57.794067: svmem(total=99932160000, available=79747641344, percent=20.2, used=16167964672, free=49219018752, active=40446279680, inactive=8412545024, buffers=34279424, cached=34510897152, shared=2190618624, slab=778309632) 
2022-04-12 03:39:57.814045: 
epoch:  307 
2022-04-12 03:41:47.460781: train loss : -0.5769 
2022-04-12 03:41:59.609999: validation loss: -0.5953 
2022-04-12 03:41:59.675116: Average global foreground Dice: [0.8277, 0.9559, 0.966, 0.3911] 
2022-04-12 03:41:59.679597: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:42:00.474880: lr: 0.004226 
2022-04-12 03:42:00.487550: saving scheduled checkpoint file... 
2022-04-12 03:42:00.573495: saving checkpoint... 
2022-04-12 03:42:09.106254: done, saving took 8.59 seconds 
2022-04-12 03:42:09.661126: done 
2022-04-12 03:42:09.669538: This epoch took 131.805819 s
 
2022-04-12 03:42:09.699043: svmem(total=99932160000, available=79637340160, percent=20.3, used=16278036480, free=49090273280, active=40553304064, inactive=8431030272, buffers=34353152, cached=34529497088, shared=2190729216, slab=779304960) 
2022-04-12 03:42:09.740809: 
epoch:  308 
2022-04-12 03:43:58.687770: train loss : -0.6401 
2022-04-12 03:44:09.949094: validation loss: -0.6612 
2022-04-12 03:44:10.040336: Average global foreground Dice: [0.8852, 0.962, 0.9806, 0.3935] 
2022-04-12 03:44:10.117460: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:44:11.060074: lr: 0.004206 
2022-04-12 03:44:11.064778: saving scheduled checkpoint file... 
2022-04-12 03:44:11.153481: saving checkpoint... 
2022-04-12 03:44:22.719110: done, saving took 11.61 seconds 
2022-04-12 03:44:22.803269: done 
2022-04-12 03:44:22.844929: This epoch took 133.084886 s
 
2022-04-12 03:44:22.864272: svmem(total=99932160000, available=79637700608, percent=20.3, used=16277499904, free=49073098752, active=40560177152, inactive=8448712704, buffers=34390016, cached=34547171328, shared=2190917632, slab=778731520) 
2022-04-12 03:44:22.889644: 
epoch:  309 
2022-04-12 03:46:12.783743: train loss : -0.6603 
2022-04-12 03:46:25.808405: validation loss: -0.6350 
2022-04-12 03:46:25.844495: Average global foreground Dice: [0.881, 0.9653, 0.9795, 0.3913] 
2022-04-12 03:46:25.891260: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:46:26.781049: lr: 0.004186 
2022-04-12 03:46:26.845681: saving scheduled checkpoint file... 
2022-04-12 03:46:26.943914: saving checkpoint... 
2022-04-12 03:46:38.033913: done, saving took 11.19 seconds 
2022-04-12 03:46:38.279736: done 
2022-04-12 03:46:38.356123: This epoch took 135.459830 s
 
2022-04-12 03:46:38.363525: svmem(total=99932160000, available=79566622720, percent=20.4, used=16348860416, free=48984408064, active=40620879872, inactive=8465747968, buffers=34439168, cached=34564452352, shared=2190540800, slab=779448320) 
2022-04-12 03:46:38.446602: 
epoch:  310 
2022-04-12 03:48:27.834375: train loss : -0.6728 
2022-04-12 03:48:41.011206: validation loss: -0.6662 
2022-04-12 03:48:41.050342: Average global foreground Dice: [0.8867, 0.9653, 0.9811, 0.4054] 
2022-04-12 03:48:41.077404: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:48:41.942664: lr: 0.004166 
2022-04-12 03:48:41.946398: saving scheduled checkpoint file... 
2022-04-12 03:48:42.061013: saving checkpoint... 
2022-04-12 03:48:48.812755: done, saving took 6.86 seconds 
2022-04-12 03:48:48.876652: done 
2022-04-12 03:48:48.902916: This epoch took 130.452806 s
 
2022-04-12 03:48:48.918419: svmem(total=99932160000, available=79556644864, percent=20.4, used=16358408192, free=48966938624, active=40641695744, inactive=8473378816, buffers=34471936, cached=34572341248, shared=2190729216, slab=780062720) 
2022-04-12 03:48:48.921964: 
epoch:  311 
2022-04-12 03:50:39.117297: train loss : -0.6544 
2022-04-12 03:50:51.242500: validation loss: -0.6855 
2022-04-12 03:50:51.286783: Average global foreground Dice: [0.9015, 0.9675, 0.9811, 0.4148] 
2022-04-12 03:50:51.328739: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:50:52.266092: lr: 0.004146 
2022-04-12 03:50:52.329277: saving scheduled checkpoint file... 
2022-04-12 03:50:52.452811: saving checkpoint... 
2022-04-12 03:51:02.714337: done, saving took 10.38 seconds 
2022-04-12 03:51:02.986624: done 
2022-04-12 03:51:02.991854: This epoch took 134.066614 s
 
2022-04-12 03:51:03.064074: svmem(total=99932160000, available=79941267456, percent=20.0, used=15973888000, free=49333805056, active=40253210624, inactive=8491028480, buffers=34504704, cached=34589962240, shared=2190807040, slab=779399168) 
2022-04-12 03:51:03.068499: 
epoch:  312 
2022-04-12 03:52:52.487692: train loss : -0.6617 
2022-04-12 03:53:04.174256: validation loss: -0.6764 
2022-04-12 03:53:04.259983: Average global foreground Dice: [0.8928, 0.9706, 0.9811, 0.433] 
2022-04-12 03:53:04.266210: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:53:04.772171: lr: 0.004127 
2022-04-12 03:53:04.874698: saving scheduled checkpoint file... 
2022-04-12 03:53:05.045636: saving checkpoint... 
2022-04-12 03:53:13.545656: done, saving took 8.59 seconds 
2022-04-12 03:53:13.648674: done 
2022-04-12 03:53:13.668705: This epoch took 130.535647 s
 
2022-04-12 03:53:13.712692: svmem(total=99932160000, available=79838044160, percent=20.1, used=16077107200, free=49213124608, active=40356573184, inactive=8508350464, buffers=34553856, cached=34607374336, shared=2190729216, slab=779624448) 
2022-04-12 03:53:13.716467: 
epoch:  313 
2022-04-12 03:55:03.124764: train loss : -0.6632 
2022-04-12 03:55:15.516824: validation loss: -0.6495 
2022-04-12 03:55:15.573993: Average global foreground Dice: [0.8728, 0.9624, 0.9823, 0.4245] 
2022-04-12 03:55:15.628994: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:55:16.470959: lr: 0.004107 
2022-04-12 03:55:16.496310: saving scheduled checkpoint file... 
2022-04-12 03:55:16.659751: saving checkpoint... 
2022-04-12 03:55:28.296097: done, saving took 11.76 seconds 
2022-04-12 03:55:28.629897: done 
2022-04-12 03:55:28.701965: This epoch took 134.982636 s
 
2022-04-12 03:55:28.768734: svmem(total=99932160000, available=79873339392, percent=20.1, used=16031477760, free=49240662016, active=40310321152, inactive=8526381056, buffers=34594816, cached=34625425408, shared=2201026560, slab=779444224) 
2022-04-12 03:55:28.780125: 
epoch:  314 
2022-04-12 03:57:18.167513: train loss : -0.6482 
2022-04-12 03:57:30.407283: validation loss: -0.6632 
2022-04-12 03:57:30.551584: Average global foreground Dice: [0.8771, 0.9648, 0.9822, 0.4278] 
2022-04-12 03:57:30.746587: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:57:31.665748: lr: 0.004087 
2022-04-12 03:57:31.730538: saving scheduled checkpoint file... 
2022-04-12 03:57:31.921460: saving checkpoint... 
2022-04-12 03:57:42.081518: done, saving took 10.26 seconds 
2022-04-12 03:57:42.196861: done 
2022-04-12 03:57:42.206818: This epoch took 133.251850 s
 
2022-04-12 03:57:42.274462: svmem(total=99932160000, available=79936061440, percent=20.0, used=15978934272, free=49284939776, active=40261144576, inactive=8534544384, buffers=34635776, cached=34633650176, shared=2190807040, slab=779718656) 
2022-04-12 03:57:42.367644: 
epoch:  315 
2022-04-12 03:59:32.216874: train loss : -0.6598 
2022-04-12 03:59:44.637335: validation loss: -0.6549 
2022-04-12 03:59:44.703092: Average global foreground Dice: [0.8763, 0.9627, 0.9777, 0.4175] 
2022-04-12 03:59:44.761262: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 03:59:45.687340: lr: 0.004067 
2022-04-12 03:59:45.694539: saving scheduled checkpoint file... 
2022-04-12 03:59:45.859719: saving checkpoint... 
2022-04-12 03:59:57.519196: done, saving took 11.78 seconds 
2022-04-12 03:59:57.628864: done 
2022-04-12 03:59:57.631960: This epoch took 135.257383 s
 
2022-04-12 03:59:57.636445: svmem(total=99932160000, available=79872847872, percent=20.1, used=16042024960, free=49204236288, active=40317382656, inactive=8551845888, buffers=34668544, cached=34651230208, shared=2190774272, slab=780386304) 
2022-04-12 03:59:57.648439: 
epoch:  316 
2022-04-12 04:01:47.434275: train loss : -0.6648 
2022-04-12 04:02:00.895768: validation loss: -0.6897 
2022-04-12 04:02:02.559560: Average global foreground Dice: [0.8975, 0.9701, 0.9843, 0.425] 
2022-04-12 04:02:02.571132: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:02:03.080688: lr: 0.004047 
2022-04-12 04:02:03.082022: saving scheduled checkpoint file... 
2022-04-12 04:02:03.206494: saving checkpoint... 
2022-04-12 04:02:09.357901: done, saving took 6.28 seconds 
2022-04-12 04:02:09.450457: done 
2022-04-12 04:02:09.590320: saving checkpoint... 
2022-04-12 04:02:18.872327: done, saving took 9.42 seconds 
2022-04-12 04:02:18.957841: This epoch took 141.259104 s
 
2022-04-12 04:02:19.029230: svmem(total=99932160000, available=79669936128, percent=20.3, used=16245092352, free=48983920640, active=40523730944, inactive=8569085952, buffers=34725888, cached=34668421120, shared=2190618624, slab=780791808) 
2022-04-12 04:02:19.037323: 
epoch:  317 
2022-04-12 04:04:09.192309: train loss : -0.6667 
2022-04-12 04:04:22.327057: validation loss: -0.6674 
2022-04-12 04:04:22.374151: Average global foreground Dice: [0.9003, 0.9661, 0.9815, 0.4051] 
2022-04-12 04:04:22.403332: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:04:23.245676: lr: 0.004027 
2022-04-12 04:04:23.307251: saving scheduled checkpoint file... 
2022-04-12 04:04:23.402088: saving checkpoint... 
2022-04-12 04:04:31.454436: done, saving took 8.14 seconds 
2022-04-12 04:04:31.571970: done 
2022-04-12 04:04:31.619765: saving checkpoint... 
2022-04-12 04:04:45.395113: done, saving took 13.82 seconds 
2022-04-12 04:04:45.578804: This epoch took 146.462715 s
 
2022-04-12 04:04:45.621586: svmem(total=99932160000, available=79774724096, percent=20.2, used=16140767232, free=49104334848, active=40333520896, inactive=8640176128, buffers=34758656, cached=34652299264, shared=2190233600, slab=779632640) 
2022-04-12 04:04:45.652610: 
epoch:  318 
2022-04-12 04:06:35.699454: train loss : -0.6659 
2022-04-12 04:06:48.267911: validation loss: -0.6829 
2022-04-12 04:06:48.516504: Average global foreground Dice: [0.8969, 0.9631, 0.9815, 0.4155] 
2022-04-12 04:06:48.639209: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:06:49.544411: lr: 0.004007 
2022-04-12 04:06:49.604774: saving scheduled checkpoint file... 
2022-04-12 04:06:49.708925: saving checkpoint... 
2022-04-12 04:07:03.580239: done, saving took 13.94 seconds 
2022-04-12 04:07:03.716094: done 
2022-04-12 04:07:03.904226: saving checkpoint... 
2022-04-12 04:07:16.627960: done, saving took 12.79 seconds 
2022-04-12 04:07:16.753442: This epoch took 151.048703 s
 
2022-04-12 04:07:16.760760: svmem(total=99932160000, available=79714390016, percent=20.2, used=16200724480, free=49025425408, active=40437727232, inactive=8607076352, buffers=34836480, cached=34671173632, shared=2190663680, slab=780505088) 
2022-04-12 04:07:16.771377: 
epoch:  319 
2022-04-12 04:09:06.009530: train loss : -0.6557 
2022-04-12 04:09:18.124619: validation loss: -0.6720 
2022-04-12 04:09:18.191317: Average global foreground Dice: [0.8921, 0.9624, 0.9807, 0.4057] 
2022-04-12 04:09:18.199202: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:09:19.030686: lr: 0.003987 
2022-04-12 04:09:19.098889: saving scheduled checkpoint file... 
2022-04-12 04:09:19.224807: saving checkpoint... 
2022-04-12 04:09:30.608675: done, saving took 11.49 seconds 
2022-04-12 04:09:31.099494: done 
2022-04-12 04:09:31.187181: saving checkpoint... 
2022-04-12 04:09:40.296237: done, saving took 9.19 seconds 
2022-04-12 04:09:40.453212: This epoch took 143.647364 s
 
2022-04-12 04:09:40.459437: svmem(total=99932160000, available=79946293248, percent=20.0, used=15975776256, free=49238843392, active=40232304640, inactive=8605749248, buffers=34869248, cached=34682671104, shared=2183647232, slab=780677120) 
2022-04-12 04:09:40.551644: 
epoch:  320 
2022-04-12 04:11:29.939626: train loss : -0.6735 
2022-04-12 04:11:43.129240: validation loss: -0.6813 
2022-04-12 04:11:43.155581: Average global foreground Dice: [0.8999, 0.9686, 0.9808, 0.4358] 
2022-04-12 04:11:43.189955: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:11:44.121640: lr: 0.003967 
2022-04-12 04:11:44.166801: saving scheduled checkpoint file... 
2022-04-12 04:11:44.339512: saving checkpoint... 
2022-04-12 04:11:56.196755: done, saving took 11.99 seconds 
2022-04-12 04:11:56.345906: done 
2022-04-12 04:11:56.518686: saving checkpoint... 
2022-04-12 04:12:08.571334: done, saving took 12.15 seconds 
2022-04-12 04:12:08.725983: This epoch took 148.153737 s
 
2022-04-12 04:12:08.736681: svmem(total=99932160000, available=79992696832, percent=20.0, used=15936258048, free=49686937600, active=40206102528, inactive=8177348608, buffers=34910208, cached=34274054144, shared=2176741376, slab=781242368) 
2022-04-12 04:12:08.833807: 
epoch:  321 
2022-04-12 04:13:58.063912: train loss : -0.6126 
2022-04-12 04:14:08.987774: validation loss: -0.6278 
2022-04-12 04:14:09.065797: Average global foreground Dice: [0.8784, 0.9573, 0.9688, 0.3876] 
2022-04-12 04:14:09.110823: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:14:09.832694: lr: 0.003947 
2022-04-12 04:14:09.837033: saving scheduled checkpoint file... 
2022-04-12 04:14:09.960245: saving checkpoint... 
2022-04-12 04:14:18.069412: done, saving took 8.22 seconds 
2022-04-12 04:14:18.175594: done 
2022-04-12 04:14:18.182306: This epoch took 129.284458 s
 
2022-04-12 04:14:18.214877: svmem(total=99932160000, available=80102088704, percent=19.8, used=15862308864, free=49368281088, active=40141234176, inactive=8567308288, buffers=34959360, cached=34666610688, shared=2141294592, slab=780460032) 
2022-04-12 04:14:18.232560: 
epoch:  322 
2022-04-12 04:16:06.911814: train loss : -0.6258 
2022-04-12 04:16:18.675385: validation loss: -0.6605 
2022-04-12 04:16:18.743446: Average global foreground Dice: [0.878, 0.9644, 0.98, 0.4084] 
2022-04-12 04:16:18.758812: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:16:19.835779: lr: 0.003927 
2022-04-12 04:16:19.837373: saving scheduled checkpoint file... 
2022-04-12 04:16:19.979253: saving checkpoint... 
2022-04-12 04:16:25.352959: done, saving took 5.51 seconds 
2022-04-12 04:16:25.475238: done 
2022-04-12 04:16:25.489745: This epoch took 127.253766 s
 
2022-04-12 04:16:25.535784: svmem(total=99932160000, available=79991869440, percent=20.0, used=15947010048, free=49240375296, active=40227094528, inactive=8610107392, buffers=35008512, cached=34709766144, shared=2166706176, slab=780685312) 
2022-04-12 04:16:25.542821: 
epoch:  323 
2022-04-12 04:18:16.084643: train loss : -0.6450 
2022-04-12 04:18:28.840909: validation loss: -0.6247 
2022-04-12 04:18:28.893685: Average global foreground Dice: [0.8805, 0.9516, 0.9642, 0.3974] 
2022-04-12 04:18:28.900756: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:18:29.713937: lr: 0.003907 
2022-04-12 04:18:29.715758: saving scheduled checkpoint file... 
2022-04-12 04:18:29.813417: saving checkpoint... 
2022-04-12 04:18:35.321534: done, saving took 5.60 seconds 
2022-04-12 04:18:35.340986: done 
2022-04-12 04:18:35.342247: This epoch took 129.795042 s
 
2022-04-12 04:18:35.343224: svmem(total=99932160000, available=80022106112, percent=19.9, used=15892836352, free=49253351424, active=40174665728, inactive=8651112448, buffers=35045376, cached=34750926848, shared=2190540800, slab=782299136) 
2022-04-12 04:18:35.344069: 
epoch:  324 
2022-04-12 04:20:24.722871: train loss : -0.6177 
2022-04-12 04:20:37.216323: validation loss: -0.6421 
2022-04-12 04:20:37.279897: Average global foreground Dice: [0.8769, 0.9641, 0.9785, 0.4195] 
2022-04-12 04:20:37.338572: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:20:38.321618: lr: 0.003887 
2022-04-12 04:20:38.353877: saving scheduled checkpoint file... 
2022-04-12 04:20:38.487890: saving checkpoint... 
2022-04-12 04:20:48.883852: done, saving took 10.50 seconds 
2022-04-12 04:20:48.907494: done 
2022-04-12 04:20:48.908968: This epoch took 133.563554 s
 
2022-04-12 04:20:48.910103: svmem(total=99932160000, available=80038449152, percent=19.9, used=15949819904, free=49188671488, active=40226164736, inactive=8658833408, buffers=35086336, cached=34758582272, shared=2117246976, slab=782094336) 
2022-04-12 04:20:48.911321: 
epoch:  325 
2022-04-12 04:22:37.667247: train loss : -0.6529 
2022-04-12 04:22:50.814687: validation loss: -0.6708 
2022-04-12 04:22:50.826360: Average global foreground Dice: [0.8916, 0.9615, 0.9815, 0.4261] 
2022-04-12 04:22:50.828148: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:22:51.766651: lr: 0.003867 
2022-04-12 04:22:51.773277: saving scheduled checkpoint file... 
2022-04-12 04:22:51.867433: saving checkpoint... 
2022-04-12 04:23:02.291420: done, saving took 10.52 seconds 
2022-04-12 04:23:02.459422: done 
2022-04-12 04:23:02.462418: This epoch took 133.550127 s
 
2022-04-12 04:23:02.465754: svmem(total=99932160000, available=79745343488, percent=20.2, used=16183173120, free=48887857152, active=40466804736, inactive=8725934080, buffers=35135488, cached=34825994240, shared=2176868352, slab=781553664) 
2022-04-12 04:23:02.489494: 
epoch:  326 
2022-04-12 04:24:51.242293: train loss : -0.6462 
2022-04-12 04:25:03.281503: validation loss: -0.6362 
2022-04-12 04:25:03.341686: Average global foreground Dice: [0.869, 0.9616, 0.9782, 0.3737] 
2022-04-12 04:25:03.410064: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:25:04.054517: lr: 0.003847 
2022-04-12 04:25:04.182785: saving scheduled checkpoint file... 
2022-04-12 04:25:04.357382: saving checkpoint... 
2022-04-12 04:25:19.885227: done, saving took 15.64 seconds 
2022-04-12 04:25:20.081136: done 
2022-04-12 04:25:20.147277: This epoch took 137.590460 s
 
2022-04-12 04:25:20.203644: svmem(total=99932160000, available=79578034176, percent=20.4, used=16336777216, free=48702021632, active=40611373056, inactive=8758194176, buffers=35192832, cached=34858168320, shared=2190618624, slab=781160448) 
2022-04-12 04:25:20.295994: 
epoch:  327 
2022-04-12 04:27:08.368491: train loss : -0.6531 
2022-04-12 04:27:18.773716: validation loss: -0.6709 
2022-04-12 04:27:18.907461: Average global foreground Dice: [0.8862, 0.9648, 0.9846, 0.4137] 
2022-04-12 04:27:18.927783: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:27:19.927844: lr: 0.003827 
2022-04-12 04:27:19.959496: saving scheduled checkpoint file... 
2022-04-12 04:27:20.141155: saving checkpoint... 
2022-04-12 04:27:32.302776: done, saving took 12.30 seconds 
2022-04-12 04:27:32.533922: done 
2022-04-12 04:27:32.566557: This epoch took 132.209183 s
 
2022-04-12 04:27:32.613174: svmem(total=99932160000, available=79813062656, percent=20.1, used=16102072320, free=48919621632, active=40381083648, inactive=8775213056, buffers=35233792, cached=34875232256, shared=2190311424, slab=780898304) 
2022-04-12 04:27:32.683438: 
epoch:  328 
2022-04-12 04:29:22.139321: train loss : -0.6610 
2022-04-12 04:29:34.488190: validation loss: -0.6640 
2022-04-12 04:29:34.581877: Average global foreground Dice: [0.8782, 0.9647, 0.9802, 0.3796] 
2022-04-12 04:29:34.669049: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:29:35.590221: lr: 0.003807 
2022-04-12 04:29:35.676221: saving scheduled checkpoint file... 
2022-04-12 04:29:35.924463: saving checkpoint... 
2022-04-12 04:29:48.751399: done, saving took 12.95 seconds 
2022-04-12 04:29:49.406857: done 
2022-04-12 04:29:49.491575: This epoch took 136.797459 s
 
2022-04-12 04:29:49.581372: svmem(total=99932160000, available=79634681856, percent=20.3, used=16280309760, free=48733065216, active=40561336320, inactive=8783155200, buffers=35557376, cached=34883227648, shared=2190430208, slab=781103104) 
2022-04-12 04:29:49.584829: 
epoch:  329 
2022-04-12 04:31:40.927367: train loss : -0.6513 
2022-04-12 04:31:53.578856: validation loss: -0.6554 
2022-04-12 04:31:53.695369: Average global foreground Dice: [0.8823, 0.9663, 0.9804, 0.3968] 
2022-04-12 04:31:53.761801: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:31:54.446910: lr: 0.003787 
2022-04-12 04:31:54.448405: saving scheduled checkpoint file... 
2022-04-12 04:31:54.530568: saving checkpoint... 
2022-04-12 04:32:03.185580: done, saving took 8.74 seconds 
2022-04-12 04:32:03.251524: done 
2022-04-12 04:32:03.252977: This epoch took 133.581633 s
 
2022-04-12 04:32:03.253917: svmem(total=99932160000, available=79714484224, percent=20.2, used=16213114880, free=48798556160, active=40336355328, inactive=8947589120, buffers=35598336, cached=34884890624, shared=2177654784, slab=781193216) 
2022-04-12 04:32:03.255117: 
epoch:  330 
2022-04-12 04:33:51.686132: train loss : -0.6633 
2022-04-12 04:34:03.656410: validation loss: -0.6752 
2022-04-12 04:34:03.690461: Average global foreground Dice: [0.9037, 0.9699, 0.975, 0.4113] 
2022-04-12 04:34:03.695643: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:34:04.639779: lr: 0.003767 
2022-04-12 04:34:04.776739: saving scheduled checkpoint file... 
2022-04-12 04:34:04.864647: saving checkpoint... 
2022-04-12 04:34:10.779987: done, saving took 5.99 seconds 
2022-04-12 04:34:10.846727: done 
2022-04-12 04:34:10.856360: This epoch took 127.600501 s
 
2022-04-12 04:34:10.901052: svmem(total=99932160000, available=79816945664, percent=20.1, used=16097705984, free=48883425280, active=40363368448, inactive=8827703296, buffers=35647488, cached=34915381248, shared=2190729216, slab=781418496) 
2022-04-12 04:34:10.906056: 
epoch:  331 
2022-04-12 04:36:00.372562: train loss : -0.6650 
2022-04-12 04:36:12.293271: validation loss: -0.6955 
2022-04-12 04:36:12.302279: Average global foreground Dice: [0.9058, 0.9663, 0.9827, 0.4297] 
2022-04-12 04:36:12.312814: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:36:13.266885: lr: 0.003747 
2022-04-12 04:36:13.271793: saving scheduled checkpoint file... 
2022-04-12 04:36:13.368183: saving checkpoint... 
2022-04-12 04:36:20.589087: done, saving took 7.31 seconds 
2022-04-12 04:36:20.859196: done 
2022-04-12 04:36:20.886358: This epoch took 129.977942 s
 
2022-04-12 04:36:20.952869: svmem(total=99932160000, available=79839043584, percent=20.1, used=16075792384, free=48887980032, active=40350588928, inactive=8834293760, buffers=35713024, cached=34932674560, shared=2190540800, slab=783167488) 
2022-04-12 04:36:20.964733: 
epoch:  332 
2022-04-12 04:38:09.773359: train loss : -0.6841 
2022-04-12 04:38:20.959408: validation loss: -0.6912 
2022-04-12 04:38:21.050020: Average global foreground Dice: [0.8957, 0.9664, 0.9819, 0.4394] 
2022-04-12 04:38:21.058669: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:38:21.931880: lr: 0.003727 
2022-04-12 04:38:21.996685: saving scheduled checkpoint file... 
2022-04-12 04:38:22.108681: saving checkpoint... 
2022-04-12 04:38:33.961825: done, saving took 11.95 seconds 
2022-04-12 04:38:34.029222: done 
2022-04-12 04:38:34.035717: This epoch took 132.988750 s
 
2022-04-12 04:38:34.108725: svmem(total=99932160000, available=79753420800, percent=20.2, used=16150777856, free=48794599424, active=40425570304, inactive=8850657280, buffers=35758080, cached=34951024640, shared=2201137152, slab=782094336) 
2022-04-12 04:38:34.111829: 
epoch:  333 
2022-04-12 04:40:23.940789: train loss : -0.6829 
2022-04-12 04:40:34.688773: validation loss: -0.6953 
2022-04-12 04:40:34.733130: Average global foreground Dice: [0.8946, 0.9694, 0.9837, 0.4075] 
2022-04-12 04:40:34.764430: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:40:35.637864: lr: 0.003707 
2022-04-12 04:40:35.671265: saving scheduled checkpoint file... 
2022-04-12 04:40:35.879107: saving checkpoint... 
2022-04-12 04:40:46.194581: done, saving took 10.44 seconds 
2022-04-12 04:40:46.378900: done 
2022-04-12 04:40:46.573749: saving checkpoint... 
2022-04-12 04:40:58.675425: done, saving took 12.22 seconds 
2022-04-12 04:40:58.755671: This epoch took 144.565928 s
 
2022-04-12 04:40:58.782826: svmem(total=99932160000, available=79641387008, percent=20.3, used=16273256448, free=48663977984, active=40553353216, inactive=8858615808, buffers=35799040, cached=34959126528, shared=2190684160, slab=782225408) 
2022-04-12 04:40:58.800597: 
epoch:  334 
2022-04-12 04:42:48.496460: train loss : -0.6804 
2022-04-12 04:43:00.539294: validation loss: -0.7058 
2022-04-12 04:43:00.567991: Average global foreground Dice: [0.9058, 0.9683, 0.9858, 0.4567] 
2022-04-12 04:43:00.571097: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:43:01.401228: lr: 0.003687 
2022-04-12 04:43:01.403598: saving scheduled checkpoint file... 
2022-04-12 04:43:01.523289: saving checkpoint... 
2022-04-12 04:43:08.441968: done, saving took 7.04 seconds 
2022-04-12 04:43:08.675065: done 
2022-04-12 04:43:08.852463: saving checkpoint... 
2022-04-12 04:43:17.247911: done, saving took 8.51 seconds 
2022-04-12 04:43:17.300079: This epoch took 138.490200 s
 
2022-04-12 04:43:17.301960: svmem(total=99932160000, available=79631106048, percent=20.3, used=16283385856, free=48636170240, active=40563113984, inactive=8876109824, buffers=35848192, cached=34976755712, shared=2190729216, slab=779812864) 
2022-04-12 04:43:17.303568: 
epoch:  335 
2022-04-12 04:45:08.237666: train loss : -0.6730 
2022-04-12 04:45:21.238384: validation loss: -0.6879 
2022-04-12 04:45:21.252025: Average global foreground Dice: [0.892, 0.9666, 0.9791, 0.4431] 
2022-04-12 04:45:21.254653: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:45:22.205221: lr: 0.003667 
2022-04-12 04:45:22.207951: saving scheduled checkpoint file... 
2022-04-12 04:45:22.283357: saving checkpoint... 
2022-04-12 04:45:27.942667: done, saving took 5.73 seconds 
2022-04-12 04:45:28.016714: done 
2022-04-12 04:45:28.074611: saving checkpoint... 
2022-04-12 04:45:40.371284: done, saving took 12.34 seconds 
2022-04-12 04:45:40.424138: This epoch took 143.119497 s
 
2022-04-12 04:45:40.425743: svmem(total=99932160000, available=79666163712, percent=20.3, used=16248123392, free=48656334848, active=40484016128, inactive=8936042496, buffers=35917824, cached=34991783936, shared=2190917632, slab=780398592) 
2022-04-12 04:45:40.426881: 
epoch:  336 
2022-04-12 04:47:29.468850: train loss : -0.6647 
2022-04-12 04:47:42.041930: validation loss: -0.6817 
2022-04-12 04:47:42.076474: Average global foreground Dice: [0.8992, 0.9655, 0.9824, 0.4131] 
2022-04-12 04:47:42.128464: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:47:42.880978: lr: 0.003647 
2022-04-12 04:47:42.894989: saving scheduled checkpoint file... 
2022-04-12 04:47:43.008248: saving checkpoint... 
2022-04-12 04:47:53.171283: done, saving took 10.24 seconds 
2022-04-12 04:47:53.297865: done 
2022-04-12 04:47:53.369892: saving checkpoint... 
2022-04-12 04:48:06.689642: done, saving took 13.37 seconds 
2022-04-12 04:48:06.891259: This epoch took 146.463110 s
 
2022-04-12 04:48:06.901433: svmem(total=99932160000, available=80082952192, percent=19.9, used=15898288128, free=49064198144, active=40168718336, inactive=8839036928, buffers=35950592, cached=34933723136, shared=2124025856, slab=780980224) 
2022-04-12 04:48:06.953884: 
epoch:  337 
2022-04-12 04:49:56.834144: train loss : -0.6763 
2022-04-12 04:50:08.543261: validation loss: -0.6847 
2022-04-12 04:50:08.555289: Average global foreground Dice: [0.9017, 0.9688, 0.9822, 0.4234] 
2022-04-12 04:50:08.606770: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:50:09.757841: lr: 0.003627 
2022-04-12 04:50:09.820597: saving scheduled checkpoint file... 
2022-04-12 04:50:09.898693: saving checkpoint... 
2022-04-12 04:50:16.845864: done, saving took 7.00 seconds 
2022-04-12 04:50:17.307934: done 
2022-04-12 04:50:17.495260: saving checkpoint... 
2022-04-12 04:50:28.617006: done, saving took 11.17 seconds 
2022-04-12 04:50:29.594482: This epoch took 142.638237 s
 
2022-04-12 04:50:29.882125: svmem(total=99932160000, available=79780323328, percent=20.2, used=16147988480, free=48743034880, active=40410210304, inactive=8908353536, buffers=36007936, cached=35005128704, shared=2176741376, slab=794411008) 
2022-04-12 04:50:30.419631: 
epoch:  338 
2022-04-12 04:52:19.077236: train loss : -0.6767 
2022-04-12 04:52:31.550026: validation loss: -0.6726 
2022-04-12 04:52:31.602726: Average global foreground Dice: [0.8961, 0.9668, 0.9822, 0.4012] 
2022-04-12 04:52:31.661053: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:52:32.474082: lr: 0.003606 
2022-04-12 04:52:32.507739: saving scheduled checkpoint file... 
2022-04-12 04:52:32.632989: saving checkpoint... 
2022-04-12 04:52:47.488767: done, saving took 14.94 seconds 
2022-04-12 04:52:49.091420: done 
2022-04-12 04:52:49.837506: This epoch took 138.933730 s
 
2022-04-12 04:52:50.231485: svmem(total=99932160000, available=79808315392, percent=20.1, used=16120053760, free=48753664000, active=40380166144, inactive=8925700096, buffers=36040704, cached=35022401536, shared=2176741376, slab=795250688) 
2022-04-12 04:52:50.702268: 
epoch:  339 
2022-04-12 04:54:41.031202: train loss : -0.6701 
2022-04-12 04:54:52.687479: validation loss: -0.6848 
2022-04-12 04:54:52.784331: Average global foreground Dice: [0.8951, 0.9677, 0.9844, 0.4147] 
2022-04-12 04:54:52.807961: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:54:53.864063: lr: 0.003586 
2022-04-12 04:54:54.477735: saving scheduled checkpoint file... 
2022-04-12 04:54:55.031888: saving checkpoint... 
2022-04-12 04:55:16.232383: done, saving took 21.28 seconds 
2022-04-12 04:55:16.376751: done 
2022-04-12 04:55:16.386476: This epoch took 145.465681 s
 
2022-04-12 04:55:16.439636: svmem(total=99932160000, available=79764520960, percent=20.2, used=16149446656, free=48691154944, active=40417918976, inactive=8958644224, buffers=36098048, cached=35055460352, shared=2191114240, slab=782356480) 
2022-04-12 04:55:16.479454: 
epoch:  340 
2022-04-12 04:57:06.166647: train loss : -0.6771 
2022-04-12 04:57:18.273538: validation loss: -0.6623 
2022-04-12 04:57:18.617663: Average global foreground Dice: [0.8879, 0.9675, 0.9791, 0.3857] 
2022-04-12 04:57:19.130624: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:57:20.480812: lr: 0.003566 
2022-04-12 04:57:20.743712: saving scheduled checkpoint file... 
2022-04-12 04:57:21.229819: saving checkpoint... 
2022-04-12 04:57:32.026792: done, saving took 10.90 seconds 
2022-04-12 04:57:32.212112: done 
2022-04-12 04:57:32.214774: This epoch took 135.689267 s
 
2022-04-12 04:57:32.251427: svmem(total=99932160000, available=79659528192, percent=20.3, used=16254787584, free=48578342912, active=40531787776, inactive=8965959680, buffers=36139008, cached=35062890496, shared=2190729216, slab=781631488) 
2022-04-12 04:57:32.288315: 
epoch:  341 
2022-04-12 04:59:21.654769: train loss : -0.6825 
2022-04-12 04:59:33.106592: validation loss: -0.6810 
2022-04-12 04:59:33.576989: Average global foreground Dice: [0.8954, 0.9659, 0.9812, 0.4364] 
2022-04-12 04:59:34.209189: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 04:59:36.279562: lr: 0.003546 
2022-04-12 04:59:37.113218: saving scheduled checkpoint file... 
2022-04-12 04:59:37.669608: saving checkpoint... 
2022-04-12 04:59:48.784214: done, saving took 11.20 seconds 
2022-04-12 04:59:49.007912: done 
2022-04-12 04:59:49.041950: This epoch took 136.709885 s
 
2022-04-12 04:59:49.089768: svmem(total=99932160000, available=79823613952, percent=20.1, used=16090701824, free=48723836928, active=40373731328, inactive=8977858560, buffers=36171776, cached=35081449472, shared=2190729216, slab=781074432) 
2022-04-12 04:59:49.092261: 
epoch:  342 
2022-04-12 05:01:37.462415: train loss : -0.6793 
2022-04-12 05:01:49.379936: validation loss: -0.6837 
2022-04-12 05:01:49.610417: Average global foreground Dice: [0.8952, 0.9677, 0.9834, 0.3978] 
2022-04-12 05:01:49.848588: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:01:51.125867: lr: 0.003526 
2022-04-12 05:01:51.413417: saving scheduled checkpoint file... 
2022-04-12 05:01:51.830489: saving checkpoint... 
2022-04-12 05:02:02.115552: done, saving took 10.37 seconds 
2022-04-12 05:02:02.567245: done 
2022-04-12 05:02:02.575732: This epoch took 133.455219 s
 
2022-04-12 05:02:02.595066: svmem(total=99932160000, available=79990861824, percent=20.0, used=15924781056, free=48873684992, active=40210108416, inactive=8993656832, buffers=36212736, cached=35097481216, shared=2189242368, slab=780705792) 
2022-04-12 05:02:02.627030: 
epoch:  343 
2022-04-12 05:03:51.310477: train loss : -0.6913 
2022-04-12 05:04:02.839961: validation loss: -0.6942 
2022-04-12 05:04:03.433269: Average global foreground Dice: [0.9162, 0.9726, 0.9843, 0.4051] 
2022-04-12 05:04:03.917885: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:04:05.870193: lr: 0.003505 
2022-04-12 05:04:06.815392: saving scheduled checkpoint file... 
2022-04-12 05:04:07.636780: saving checkpoint... 
2022-04-12 05:04:18.171002: done, saving took 10.64 seconds 
2022-04-12 05:04:18.228912: done 
2022-04-12 05:04:18.241871: This epoch took 135.578144 s
 
2022-04-12 05:04:18.292732: svmem(total=99932160000, available=79800872960, percent=20.1, used=16112390144, free=48666275840, active=40396464128, inactive=9013452800, buffers=36261888, cached=35117232128, shared=2191609856, slab=779771904) 
2022-04-12 05:04:18.296098: 
epoch:  344 
2022-04-12 05:06:07.506992: train loss : -0.7001 
2022-04-12 05:06:19.590883: validation loss: -0.6889 
2022-04-12 05:06:19.929119: Average global foreground Dice: [0.9068, 0.9734, 0.9852, 0.4332] 
2022-04-12 05:06:20.368654: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:06:21.565890: lr: 0.003485 
2022-04-12 05:06:21.600828: saving scheduled checkpoint file... 
2022-04-12 05:06:21.756658: saving checkpoint... 
2022-04-12 05:06:32.141784: done, saving took 10.49 seconds 
2022-04-12 05:06:32.265210: done 
2022-04-12 05:06:32.438070: saving checkpoint... 
2022-04-12 05:06:40.909323: done, saving took 8.58 seconds 
2022-04-12 05:06:41.310363: This epoch took 143.004810 s
 
2022-04-12 05:06:41.388067: svmem(total=99932160000, available=79701766144, percent=20.2, used=16212439040, free=48558243840, active=40489226240, inactive=9021431808, buffers=36315136, cached=35125161984, shared=2190729216, slab=780857344) 
2022-04-12 05:06:41.394635: 
epoch:  345 
2022-04-12 05:08:30.914840: train loss : -0.6921 
2022-04-12 05:08:43.441952: validation loss: -0.6619 
2022-04-12 05:08:43.509836: Average global foreground Dice: [0.9098, 0.9683, 0.9819, 0.3914] 
2022-04-12 05:08:43.570476: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:08:44.632482: lr: 0.003465 
2022-04-12 05:08:44.692687: saving scheduled checkpoint file... 
2022-04-12 05:08:44.826728: saving checkpoint... 
2022-04-12 05:08:52.788162: done, saving took 8.09 seconds 
2022-04-12 05:08:52.879942: done 
2022-04-12 05:08:52.881732: This epoch took 131.420229 s
 
2022-04-12 05:08:52.883101: svmem(total=99932160000, available=79628959744, percent=20.3, used=16284958720, free=48500449280, active=40407359488, inactive=9165922304, buffers=36352000, cached=35110400000, shared=2190839808, slab=782430208) 
2022-04-12 05:08:52.885779: 
epoch:  346 
2022-04-12 05:10:42.513223: train loss : -0.6972 
2022-04-12 05:10:55.579443: validation loss: -0.6730 
2022-04-12 05:10:55.692568: Average global foreground Dice: [0.8777, 0.9654, 0.9797, 0.4201] 
2022-04-12 05:10:55.699423: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:10:56.440281: lr: 0.003445 
2022-04-12 05:10:56.493772: saving scheduled checkpoint file... 
2022-04-12 05:10:56.625382: saving checkpoint... 
2022-04-12 05:11:15.684424: done, saving took 19.19 seconds 
2022-04-12 05:11:16.173211: done 
2022-04-12 05:11:16.383744: This epoch took 143.493608 s
 
2022-04-12 05:11:18.451467: svmem(total=99932160000, available=79767855104, percent=20.2, used=16146456576, free=48620728320, active=40368840704, inactive=9076412416, buffers=36409344, cached=35128565760, shared=2190540800, slab=782671872) 
2022-04-12 05:11:18.460989: 
epoch:  347 
2022-04-12 05:13:08.976255: train loss : -0.6808 
2022-04-12 05:13:21.102259: validation loss: -0.6587 
2022-04-12 05:13:21.234440: Average global foreground Dice: [0.8872, 0.9586, 0.9727, 0.3955] 
2022-04-12 05:13:21.291630: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:13:22.063722: lr: 0.003424 
2022-04-12 05:13:22.121969: saving scheduled checkpoint file... 
2022-04-12 05:13:22.279715: saving checkpoint... 
2022-04-12 05:13:28.609966: done, saving took 6.42 seconds 
2022-04-12 05:13:28.627700: done 
2022-04-12 05:13:28.630245: This epoch took 130.119263 s
 
2022-04-12 05:13:28.632147: svmem(total=99932160000, available=79863930880, percent=20.1, used=16049725440, free=48699293696, active=40319643648, inactive=9058492416, buffers=36442112, cached=35146698752, shared=2191224832, slab=781926400) 
2022-04-12 05:13:28.635454: 
epoch:  348 
2022-04-12 05:15:17.090031: train loss : -0.6486 
2022-04-12 05:15:28.981299: validation loss: -0.6531 
2022-04-12 05:15:29.080380: Average global foreground Dice: [0.8671, 0.9604, 0.9811, 0.4013] 
2022-04-12 05:15:29.135388: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:15:30.300998: lr: 0.003404 
2022-04-12 05:15:30.353261: saving scheduled checkpoint file... 
2022-04-12 05:15:30.491765: saving checkpoint... 
2022-04-12 05:15:42.531376: done, saving took 12.13 seconds 
2022-04-12 05:15:42.656300: done 
2022-04-12 05:15:42.675876: This epoch took 134.038824 s
 
2022-04-12 05:15:42.692647: svmem(total=99932160000, available=79622717440, percent=20.3, used=16290791424, free=48450347008, active=40570773504, inactive=9057734656, buffers=36491264, cached=35154530304, shared=2191224832, slab=781942784) 
2022-04-12 05:15:42.695704: 
epoch:  349 
2022-04-12 05:17:31.205438: train loss : -0.6897 
2022-04-12 05:17:43.872893: validation loss: -0.6882 
2022-04-12 05:17:44.016374: Average global foreground Dice: [0.8988, 0.9686, 0.984, 0.4108] 
2022-04-12 05:17:44.961586: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:17:45.710607: lr: 0.003384 
2022-04-12 05:17:45.744077: saving scheduled checkpoint file... 
2022-04-12 05:17:45.850259: saving checkpoint... 
2022-04-12 05:17:51.478512: done, saving took 5.72 seconds 
2022-04-12 05:17:51.592912: done 
2022-04-12 05:17:51.596784: This epoch took 128.899073 s
 
2022-04-12 05:17:51.599651: svmem(total=99932160000, available=79703486464, percent=20.2, used=16199843840, free=48477605888, active=40476876800, inactive=9120530432, buffers=36532224, cached=35218178048, shared=2201399296, slab=782159872) 
2022-04-12 05:17:51.601892: 
epoch:  350 
2022-04-12 05:19:39.964028: train loss : -0.6788 
2022-04-12 05:19:52.632784: validation loss: -0.6964 
2022-04-12 05:19:52.643281: Average global foreground Dice: [0.8967, 0.9703, 0.9831, 0.4331] 
2022-04-12 05:19:52.645343: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:19:53.251440: lr: 0.003364 
2022-04-12 05:19:53.269117: saving scheduled checkpoint file... 
2022-04-12 05:19:53.519539: saving checkpoint... 
2022-04-12 05:20:05.341196: done, saving took 12.00 seconds 
2022-04-12 05:20:05.499900: done 
2022-04-12 05:20:05.512029: This epoch took 133.907556 s
 
2022-04-12 05:20:05.593158: svmem(total=99932160000, available=79729672192, percent=20.2, used=16184143872, free=48486268928, active=40453959680, inactive=9127739392, buffers=36573184, cached=35225174016, shared=2190884864, slab=781774848) 
2022-04-12 05:20:05.639838: 
epoch:  351 
2022-04-12 05:21:54.569355: train loss : -0.6738 
2022-04-12 05:22:07.391031: validation loss: -0.6249 
2022-04-12 05:22:07.940765: Average global foreground Dice: [0.9136, 0.9602, 0.9612, 0.3999] 
2022-04-12 05:22:08.760717: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:22:10.933235: lr: 0.003343 
2022-04-12 05:22:11.851778: saving scheduled checkpoint file... 
2022-04-12 05:22:12.421790: saving checkpoint... 
2022-04-12 05:22:20.484953: done, saving took 8.18 seconds 
2022-04-12 05:22:20.563198: done 
2022-04-12 05:22:20.590044: This epoch took 134.946929 s
 
2022-04-12 05:22:20.655332: svmem(total=99932160000, available=79841439744, percent=20.1, used=16072478720, free=48590262272, active=40355258368, inactive=9132412928, buffers=36630528, cached=35232788480, shared=2190508032, slab=782893056) 
2022-04-12 05:22:20.724964: 
epoch:  352 
2022-04-12 05:24:10.142599: train loss : -0.6820 
2022-04-12 05:24:22.829994: validation loss: -0.6907 
2022-04-12 05:24:23.299888: Average global foreground Dice: [0.911, 0.9738, 0.9823, 0.4188] 
2022-04-12 05:24:23.696651: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:24:24.695204: lr: 0.003323 
2022-04-12 05:24:24.698994: saving scheduled checkpoint file... 
2022-04-12 05:24:24.828793: saving checkpoint... 
2022-04-12 05:24:45.381890: done, saving took 20.67 seconds 
2022-04-12 05:24:45.448106: done 
2022-04-12 05:24:45.451085: This epoch took 144.724316 s
 
2022-04-12 05:24:45.452981: svmem(total=99932160000, available=79926009856, percent=20.0, used=15988047872, free=48656195584, active=40263254016, inactive=9151197184, buffers=36663296, cached=35251253248, shared=2190618624, slab=781615104) 
2022-04-12 05:24:45.455189: 
epoch:  353 
2022-04-12 05:26:36.660871: train loss : -0.6966 
2022-04-12 05:26:49.292439: validation loss: -0.6913 
2022-04-12 05:26:49.681473: Average global foreground Dice: [0.9155, 0.9688, 0.9836, 0.4019] 
2022-04-12 05:26:50.425283: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:26:52.627114: lr: 0.003303 
2022-04-12 05:26:53.542001: saving scheduled checkpoint file... 
2022-04-12 05:26:54.517160: saving checkpoint... 
2022-04-12 05:27:15.402942: done, saving took 20.98 seconds 
2022-04-12 05:27:15.542194: done 
2022-04-12 05:27:15.564308: This epoch took 150.107048 s
 
2022-04-12 05:27:15.568550: svmem(total=99932160000, available=79604879360, percent=20.3, used=16309264384, free=48316596224, active=40585523200, inactive=9169457152, buffers=36724736, cached=35269574656, shared=2190462976, slab=781479936) 
2022-04-12 05:27:15.624813: 
epoch:  354 
2022-04-12 05:29:05.559003: train loss : -0.6783 
2022-04-12 05:29:17.263186: validation loss: -0.6649 
2022-04-12 05:29:17.586854: Average global foreground Dice: [0.8913, 0.9629, 0.9792, 0.4075] 
2022-04-12 05:29:18.184143: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:29:19.320390: lr: 0.003282 
2022-04-12 05:29:19.609723: saving scheduled checkpoint file... 
2022-04-12 05:29:19.909488: saving checkpoint... 
2022-04-12 05:29:31.542619: done, saving took 11.68 seconds 
2022-04-12 05:29:31.656690: done 
2022-04-12 05:29:31.674563: This epoch took 136.043659 s
 
2022-04-12 05:29:31.730246: svmem(total=99932160000, available=79782981632, percent=20.2, used=16131022848, free=48477241344, active=40409878528, inactive=9186971648, buffers=36757504, cached=35287138304, shared=2190618624, slab=780918784) 
2022-04-12 05:29:31.740174: 
epoch:  355 
2022-04-12 05:31:22.176808: train loss : -0.6743 
2022-04-12 05:31:34.526174: validation loss: -0.7081 
2022-04-12 05:31:35.292552: Average global foreground Dice: [0.9149, 0.9682, 0.9861, 0.4145] 
2022-04-12 05:31:35.688198: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:31:37.296397: lr: 0.003262 
2022-04-12 05:31:37.643101: saving scheduled checkpoint file... 
2022-04-12 05:31:38.015882: saving checkpoint... 
2022-04-12 05:31:51.689534: done, saving took 13.74 seconds 
2022-04-12 05:31:51.863194: done 
2022-04-12 05:31:51.932099: This epoch took 140.126978 s
 
2022-04-12 05:31:51.987878: svmem(total=99932160000, available=79678177280, percent=20.3, used=16235565056, free=48363638784, active=40515383296, inactive=9195261952, buffers=36798464, cached=35296157696, shared=2190729216, slab=781934592) 
2022-04-12 05:31:52.032130: 
epoch:  356 
2022-04-12 05:33:40.419340: train loss : -0.6902 
2022-04-12 05:33:52.110834: validation loss: -0.6824 
2022-04-12 05:33:52.748083: Average global foreground Dice: [0.9016, 0.9642, 0.9778, 0.4312] 
2022-04-12 05:33:53.325816: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:33:55.223847: lr: 0.003241 
2022-04-12 05:33:55.633877: saving scheduled checkpoint file... 
2022-04-12 05:33:56.270407: saving checkpoint... 
2022-04-12 05:34:08.281326: done, saving took 12.11 seconds 
2022-04-12 05:34:08.318447: done 
2022-04-12 05:34:08.364157: This epoch took 136.328441 s
 
2022-04-12 05:34:08.366849: svmem(total=99932160000, available=79606140928, percent=20.3, used=16307646464, free=48274079744, active=40579796992, inactive=9212801024, buffers=36831232, cached=35313602560, shared=2190807040, slab=782405632) 
2022-04-12 05:34:08.368753: 
epoch:  357 
2022-04-12 05:35:58.711472: train loss : -0.6877 
2022-04-12 05:36:10.282634: validation loss: -0.7008 
2022-04-12 05:36:10.314585: Average global foreground Dice: [0.8916, 0.9714, 0.9831, 0.4593] 
2022-04-12 05:36:10.316156: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:36:11.229381: lr: 0.003221 
2022-04-12 05:36:11.230815: saving scheduled checkpoint file... 
2022-04-12 05:36:11.340809: saving checkpoint... 
2022-04-12 05:36:18.161020: done, saving took 6.93 seconds 
2022-04-12 05:36:18.238913: done 
2022-04-12 05:36:18.336105: saving checkpoint... 
2022-04-12 05:36:24.952855: done, saving took 6.71 seconds 
2022-04-12 05:36:25.057750: This epoch took 136.687909 s
 
2022-04-12 05:36:25.064533: svmem(total=99932160000, available=79798251520, percent=20.1, used=16115310592, free=48461201408, active=40305844224, inactive=9310711808, buffers=36896768, cached=35318751232, shared=2190917632, slab=782110720) 
2022-04-12 05:36:25.090956: 
epoch:  358 
2022-04-12 05:38:13.795406: train loss : -0.6797 
2022-04-12 05:38:25.450226: validation loss: -0.7007 
2022-04-12 05:38:25.456228: Average global foreground Dice: [0.9064, 0.9666, 0.9818, 0.4094] 
2022-04-12 05:38:25.459233: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:38:26.559207: lr: 0.003201 
2022-04-12 05:38:26.573821: saving scheduled checkpoint file... 
2022-04-12 05:38:26.715603: saving checkpoint... 
2022-04-12 05:38:36.416098: done, saving took 9.84 seconds 
2022-04-12 05:38:36.505324: done 
2022-04-12 05:38:36.534971: This epoch took 131.391958 s
 
2022-04-12 05:38:36.569208: svmem(total=99932160000, available=79782068224, percent=20.2, used=16131620864, free=48425238528, active=40389308416, inactive=9260826624, buffers=36941824, cached=35338358784, shared=2190807040, slab=782061568) 
2022-04-12 05:38:36.596164: 
epoch:  359 
2022-04-12 05:40:25.376097: train loss : -0.6832 
2022-04-12 05:40:37.807019: validation loss: -0.6978 
2022-04-12 05:40:37.842163: Average global foreground Dice: [0.906, 0.9705, 0.9828, 0.4058] 
2022-04-12 05:40:37.911075: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:40:39.130218: lr: 0.00318 
2022-04-12 05:40:39.252879: saving scheduled checkpoint file... 
2022-04-12 05:40:39.392179: saving checkpoint... 
2022-04-12 05:40:46.753782: done, saving took 7.44 seconds 
2022-04-12 05:40:46.953106: done 
2022-04-12 05:40:47.023885: saving checkpoint... 
2022-04-12 05:40:57.727365: done, saving took 10.77 seconds 
2022-04-12 05:40:57.886867: This epoch took 141.249120 s
 
2022-04-12 05:40:57.891995: svmem(total=99932160000, available=79882477568, percent=20.1, used=16020631552, free=48517844992, active=40300109824, inactive=9256161280, buffers=36982784, cached=35356700672, shared=2201341952, slab=782901248) 
2022-04-12 05:40:57.903120: 
epoch:  360 
2022-04-12 05:42:48.294159: train loss : -0.6726 
2022-04-12 05:43:00.789252: validation loss: -0.6640 
2022-04-12 05:43:00.813567: Average global foreground Dice: [0.9002, 0.9682, 0.9797, 0.3842] 
2022-04-12 05:43:00.872524: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:43:01.812408: lr: 0.00316 
2022-04-12 05:43:01.859363: saving scheduled checkpoint file... 
2022-04-12 05:43:01.974572: saving checkpoint... 
2022-04-12 05:43:10.155852: done, saving took 8.29 seconds 
2022-04-12 05:43:10.216966: done 
2022-04-12 05:43:10.221418: This epoch took 132.237821 s
 
2022-04-12 05:43:10.224802: svmem(total=99932160000, available=79894630400, percent=20.1, used=16018821120, free=48512335872, active=40294207488, inactive=9263046656, buffers=37015552, cached=35363987456, shared=2190917632, slab=781455360) 
2022-04-12 05:43:10.227068: 
epoch:  361 
2022-04-12 05:45:01.147053: train loss : -0.6522 
2022-04-12 05:45:14.125842: validation loss: -0.6441 
2022-04-12 05:45:14.232522: Average global foreground Dice: [0.8932, 0.9616, 0.9737, 0.4187] 
2022-04-12 05:45:14.247736: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:45:15.088776: lr: 0.003139 
2022-04-12 05:45:15.210591: saving scheduled checkpoint file... 
2022-04-12 05:45:15.357044: saving checkpoint... 
2022-04-12 05:45:27.970123: done, saving took 12.72 seconds 
2022-04-12 05:45:28.014857: done 
2022-04-12 05:45:28.018805: This epoch took 137.790236 s
 
2022-04-12 05:45:28.020431: svmem(total=99932160000, available=79760855040, percent=20.2, used=16152739840, free=48359907328, active=40434118656, inactive=9281400832, buffers=37089280, cached=35382423552, shared=2190729216, slab=782602240) 
2022-04-12 05:45:28.021573: 
epoch:  362 
2022-04-12 05:47:17.903994: train loss : -0.6453 
2022-04-12 05:47:30.345271: validation loss: -0.6578 
2022-04-12 05:47:30.393363: Average global foreground Dice: [0.8872, 0.9689, 0.9791, 0.397] 
2022-04-12 05:47:30.445235: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:47:31.319987: lr: 0.003119 
2022-04-12 05:47:31.348273: saving scheduled checkpoint file... 
2022-04-12 05:47:31.502045: saving checkpoint... 
2022-04-12 05:47:43.571591: done, saving took 12.21 seconds 
2022-04-12 05:47:44.221260: done 
2022-04-12 05:47:44.224652: This epoch took 136.201526 s
 
2022-04-12 05:47:44.227764: svmem(total=99932160000, available=79669440512, percent=20.3, used=16243675136, free=48251068416, active=40531873792, inactive=9296465920, buffers=37130240, cached=35400286208, shared=2191114240, slab=781414400) 
2022-04-12 05:47:44.232224: 
epoch:  363 
2022-04-12 05:49:35.702186: train loss : -0.6531 
2022-04-12 05:49:48.207967: validation loss: -0.6473 
2022-04-12 05:49:48.265920: Average global foreground Dice: [0.8895, 0.9643, 0.9703, 0.4058] 
2022-04-12 05:49:48.267502: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:49:49.149127: lr: 0.003098 
2022-04-12 05:49:49.180434: saving scheduled checkpoint file... 
2022-04-12 05:49:49.360343: saving checkpoint... 
2022-04-12 05:49:59.671981: done, saving took 10.44 seconds 
2022-04-12 05:49:59.820552: done 
2022-04-12 05:49:59.870212: This epoch took 135.633236 s
 
2022-04-12 05:49:59.872982: svmem(total=99932160000, available=79971295232, percent=20.0, used=15942172672, free=48545230848, active=40221700096, inactive=9303535616, buffers=37163008, cached=35407593472, shared=2190696448, slab=783208448) 
2022-04-12 05:49:59.889319: 
epoch:  364 
2022-04-12 05:51:49.261514: train loss : -0.6525 
2022-04-12 05:52:02.282239: validation loss: -0.6939 
2022-04-12 05:52:02.379586: Average global foreground Dice: [0.9085, 0.9693, 0.9844, 0.424] 
2022-04-12 05:52:02.385100: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:52:02.971883: lr: 0.003078 
2022-04-12 05:52:03.048144: saving scheduled checkpoint file... 
2022-04-12 05:52:03.203183: saving checkpoint... 
2022-04-12 05:52:18.136994: done, saving took 15.07 seconds 
2022-04-12 05:52:18.233735: done 
2022-04-12 05:52:18.238768: This epoch took 138.333961 s
 
2022-04-12 05:52:18.269399: svmem(total=99932160000, available=79779180544, percent=20.2, used=16145944576, free=48332951552, active=40424546304, inactive=9310683136, buffers=37220352, cached=35416043520, shared=2179141632, slab=783368192) 
2022-04-12 05:52:18.282994: 
epoch:  365 
2022-04-12 05:54:08.798338: train loss : -0.6618 
2022-04-12 05:54:20.840960: validation loss: -0.6580 
2022-04-12 05:54:20.943357: Average global foreground Dice: [0.9066, 0.9623, 0.9743, 0.4155] 
2022-04-12 05:54:21.014219: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:54:22.091483: lr: 0.003057 
2022-04-12 05:54:22.112066: saving scheduled checkpoint file... 
2022-04-12 05:54:22.320366: saving checkpoint... 
2022-04-12 05:54:38.237044: done, saving took 16.00 seconds 
2022-04-12 05:54:38.364405: done 
2022-04-12 05:54:38.428184: This epoch took 140.126376 s
 
2022-04-12 05:54:38.432219: svmem(total=99932160000, available=79642755072, percent=20.3, used=16269623296, free=48170934272, active=40556281856, inactive=9348669440, buffers=37253120, cached=35454349312, shared=2191720448, slab=784461824) 
2022-04-12 05:54:38.487793: 
epoch:  366 
2022-04-12 05:56:26.749103: train loss : -0.6458 
2022-04-12 05:56:38.571824: validation loss: -0.6790 
2022-04-12 05:56:38.599590: Average global foreground Dice: [0.8918, 0.9603, 0.9813, 0.4307] 
2022-04-12 05:56:38.613589: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:56:39.421352: lr: 0.003037 
2022-04-12 05:56:39.450744: saving scheduled checkpoint file... 
2022-04-12 05:56:39.573770: saving checkpoint... 
2022-04-12 05:56:47.770743: done, saving took 8.28 seconds 
2022-04-12 05:56:47.906225: done 
2022-04-12 05:56:48.017523: This epoch took 129.524548 s
 
2022-04-12 05:56:48.062926: svmem(total=99932160000, available=79740096512, percent=20.2, used=16173359104, free=48260521984, active=40457441280, inactive=9355456512, buffers=37302272, cached=35460976640, shared=2190729216, slab=782934016) 
2022-04-12 05:56:48.173785: 
epoch:  367 
2022-04-12 05:58:38.042530: train loss : -0.6591 
2022-04-12 05:58:49.202978: validation loss: -0.6691 
2022-04-12 05:58:49.367213: Average global foreground Dice: [0.8998, 0.9668, 0.983, 0.4165] 
2022-04-12 05:58:49.543193: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 05:58:50.754185: lr: 0.003016 
2022-04-12 05:58:50.809668: saving scheduled checkpoint file... 
2022-04-12 05:58:50.955870: saving checkpoint... 
2022-04-12 05:59:00.526222: done, saving took 9.67 seconds 
2022-04-12 05:59:00.577893: done 
2022-04-12 05:59:00.607386: This epoch took 132.355638 s
 
2022-04-12 05:59:00.609869: svmem(total=99932160000, available=79661051904, percent=20.3, used=16252461056, free=48173129728, active=40522936320, inactive=9372602368, buffers=37335040, cached=35469234176, shared=2190540800, slab=784199680) 
2022-04-12 05:59:00.612076: 
epoch:  368 
2022-04-12 06:00:50.797160: train loss : -0.6856 
2022-04-12 06:01:02.670917: validation loss: -0.6996 
2022-04-12 06:01:02.709129: Average global foreground Dice: [0.9149, 0.9676, 0.9849, 0.4321] 
2022-04-12 06:01:02.724936: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:01:03.688365: lr: 0.002996 
2022-04-12 06:01:03.730331: saving scheduled checkpoint file... 
2022-04-12 06:01:03.845239: saving checkpoint... 
2022-04-12 06:01:19.389154: done, saving took 15.65 seconds 
2022-04-12 06:01:19.628399: done 
2022-04-12 06:01:19.650288: This epoch took 139.030527 s
 
2022-04-12 06:01:19.678623: svmem(total=99932160000, available=79799275520, percent=20.1, used=16115077120, free=48284692480, active=40391516160, inactive=9389223936, buffers=37392384, cached=35494998016, shared=2189737984, slab=784482304) 
2022-04-12 06:01:19.737709: 
epoch:  369 
2022-04-12 06:03:09.867956: train loss : -0.6777 
2022-04-12 06:03:21.265460: validation loss: -0.6603 
2022-04-12 06:03:21.374008: Average global foreground Dice: [0.8906, 0.967, 0.9747, 0.41] 
2022-04-12 06:03:21.425069: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:03:22.507518: lr: 0.002975 
2022-04-12 06:03:22.561340: saving scheduled checkpoint file... 
2022-04-12 06:03:22.721359: saving checkpoint... 
2022-04-12 06:03:38.791748: done, saving took 16.20 seconds 
2022-04-12 06:03:39.042026: done 
2022-04-12 06:03:39.107521: This epoch took 139.363278 s
 
2022-04-12 06:03:39.120241: svmem(total=99932160000, available=79783845888, percent=20.2, used=16129044480, free=48251822080, active=40415182848, inactive=9407463424, buffers=37425152, cached=35513868288, shared=2191114240, slab=784461824) 
2022-04-12 06:03:39.171564: 
epoch:  370 
2022-04-12 06:05:29.617651: train loss : -0.6774 
2022-04-12 06:05:40.721079: validation loss: -0.6894 
2022-04-12 06:05:40.743193: Average global foreground Dice: [0.8907, 0.9712, 0.9815, 0.4293] 
2022-04-12 06:05:40.768939: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:05:41.842326: lr: 0.002954 
2022-04-12 06:05:41.859989: saving scheduled checkpoint file... 
2022-04-12 06:05:42.094912: saving checkpoint... 
2022-04-12 06:05:54.117092: done, saving took 12.17 seconds 
2022-04-12 06:05:54.207159: done 
2022-04-12 06:05:54.232550: This epoch took 134.993793 s
 
2022-04-12 06:05:54.267414: svmem(total=99932160000, available=79742218240, percent=20.2, used=16171155456, free=48223989760, active=40216403968, inactive=9631309824, buffers=37474304, cached=35499540480, shared=2190696448, slab=784080896) 
2022-04-12 06:05:54.308265: 
epoch:  371 
2022-04-12 06:07:45.295396: train loss : -0.6947 
2022-04-12 06:07:57.887173: validation loss: -0.6849 
2022-04-12 06:07:58.523392: Average global foreground Dice: [0.904, 0.9695, 0.9752, 0.4265] 
2022-04-12 06:07:58.760716: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:07:59.896379: lr: 0.002934 
2022-04-12 06:07:59.912063: saving scheduled checkpoint file... 
2022-04-12 06:08:00.009812: saving checkpoint... 
2022-04-12 06:08:16.911353: done, saving took 17.00 seconds 
2022-04-12 06:08:17.174168: done 
2022-04-12 06:08:17.193536: This epoch took 142.835560 s
 
2022-04-12 06:08:17.196777: svmem(total=99932160000, available=79901466624, percent=20.0, used=16011808768, free=48343175168, active=40154910720, inactive=9575211008, buffers=37535744, cached=35539640320, shared=2190700544, slab=784195584) 
2022-04-12 06:08:17.246990: 
epoch:  372 
2022-04-12 06:10:05.120797: train loss : -0.6814 
2022-04-12 06:10:17.613697: validation loss: -0.7049 
2022-04-12 06:10:18.333013: Average global foreground Dice: [0.9083, 0.9732, 0.985, 0.4225] 
2022-04-12 06:10:19.052101: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:10:21.058228: lr: 0.002913 
2022-04-12 06:10:21.621059: saving scheduled checkpoint file... 
2022-04-12 06:10:22.672378: saving checkpoint... 
2022-04-12 06:10:36.095819: done, saving took 13.51 seconds 
2022-04-12 06:10:39.295443: done 
2022-04-12 06:10:39.422529: saving checkpoint... 
2022-04-12 06:11:00.673992: done, saving took 21.34 seconds 
2022-04-12 06:11:00.922769: This epoch took 163.645336 s
 
2022-04-12 06:11:00.935856: svmem(total=99932160000, available=79804956672, percent=20.1, used=16108257280, free=48227106816, active=40325525504, inactive=9524342784, buffers=37576704, cached=35559219200, shared=2190729216, slab=784384000) 
2022-04-12 06:11:00.977480: 
epoch:  373 
2022-04-12 06:12:49.419834: train loss : -0.7003 
2022-04-12 06:13:01.503634: validation loss: -0.7150 
2022-04-12 06:13:01.565829: Average global foreground Dice: [0.922, 0.9689, 0.9844, 0.4112] 
2022-04-12 06:13:01.610739: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:13:02.603577: lr: 0.002892 
2022-04-12 06:13:02.853471: saving scheduled checkpoint file... 
2022-04-12 06:13:03.032897: saving checkpoint... 
2022-04-12 06:13:15.009852: done, saving took 12.08 seconds 
2022-04-12 06:13:15.339315: done 
2022-04-12 06:13:15.508051: saving checkpoint... 
2022-04-12 06:13:26.649570: done, saving took 11.24 seconds 
2022-04-12 06:13:26.836277: This epoch took 145.805237 s
 
2022-04-12 06:13:26.905735: svmem(total=99932160000, available=79736274944, percent=20.2, used=16176848896, free=48139808768, active=40416653312, inactive=9513578496, buffers=37625856, cached=35577876480, shared=2190807040, slab=784592896) 
2022-04-12 06:13:26.925905: 
epoch:  374 
2022-04-12 06:15:16.285060: train loss : -0.6949 
2022-04-12 06:15:27.947296: validation loss: -0.6958 
2022-04-12 06:15:28.012269: Average global foreground Dice: [0.9199, 0.9712, 0.985, 0.4221] 
2022-04-12 06:15:28.064092: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:15:29.081155: lr: 0.002872 
2022-04-12 06:15:29.125472: saving scheduled checkpoint file... 
2022-04-12 06:15:29.319473: saving checkpoint... 
2022-04-12 06:15:39.857312: done, saving took 10.66 seconds 
2022-04-12 06:15:40.162882: done 
2022-04-12 06:15:40.301427: saving checkpoint... 
2022-04-12 06:15:51.029744: done, saving took 10.84 seconds 
2022-04-12 06:15:51.230228: This epoch took 144.257663 s
 
2022-04-12 06:15:51.298046: svmem(total=99932160000, available=79621943296, percent=20.3, used=16291319808, free=48006815744, active=40558239744, inactive=9506983936, buffers=37670912, cached=35596353536, shared=2190618624, slab=784506880) 
2022-04-12 06:15:51.315233: 
epoch:  375 
2022-04-12 06:17:40.018247: train loss : -0.6921 
2022-04-12 06:17:51.610048: validation loss: -0.7145 
2022-04-12 06:17:51.772074: Average global foreground Dice: [0.9238, 0.9751, 0.9837, 0.4304] 
2022-04-12 06:17:51.851084: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:17:52.934362: lr: 0.002851 
2022-04-12 06:17:52.996124: saving scheduled checkpoint file... 
2022-04-12 06:17:53.142246: saving checkpoint... 
2022-04-12 06:18:04.729687: done, saving took 11.73 seconds 
2022-04-12 06:18:04.860699: done 
2022-04-12 06:18:05.022722: saving checkpoint... 
2022-04-12 06:18:13.948013: done, saving took 9.05 seconds 
2022-04-12 06:18:14.069438: This epoch took 142.690690 s
 
2022-04-12 06:18:14.095348: svmem(total=99932160000, available=79687901184, percent=20.3, used=16225607680, free=48065101824, active=40496271360, inactive=9509261312, buffers=37728256, cached=35603722240, shared=2190450688, slab=784424960) 
2022-04-12 06:18:14.098609: 
epoch:  376 
2022-04-12 06:20:04.067664: train loss : -0.6905 
2022-04-12 06:20:16.379965: validation loss: -0.7063 
2022-04-12 06:20:16.439729: Average global foreground Dice: [0.912, 0.9709, 0.9842, 0.4309] 
2022-04-12 06:20:16.455248: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:20:17.295156: lr: 0.00283 
2022-04-12 06:20:17.317826: saving scheduled checkpoint file... 
2022-04-12 06:20:17.482610: saving checkpoint... 
2022-04-12 06:20:29.294884: done, saving took 11.97 seconds 
2022-04-12 06:20:29.508341: done 
2022-04-12 06:20:29.673969: saving checkpoint... 
2022-04-12 06:20:42.360719: done, saving took 12.83 seconds 
2022-04-12 06:20:42.701906: This epoch took 148.595345 s
 
2022-04-12 06:20:42.769908: svmem(total=99932160000, available=79882956800, percent=20.1, used=16030814208, free=48252805120, active=40188477440, inactive=9628475392, buffers=37769216, cached=35610771456, shared=2190045184, slab=784474112) 
2022-04-12 06:20:42.873952: 
epoch:  377 
2022-04-12 06:22:33.606670: train loss : -0.7067 
2022-04-12 06:22:45.610423: validation loss: -0.6985 
2022-04-12 06:22:45.700750: Average global foreground Dice: [0.9099, 0.9691, 0.9842, 0.3972] 
2022-04-12 06:22:45.760249: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:22:46.783277: lr: 0.00281 
2022-04-12 06:22:46.831472: saving scheduled checkpoint file... 
2022-04-12 06:22:46.982660: saving checkpoint... 
2022-04-12 06:23:03.464701: done, saving took 16.58 seconds 
2022-04-12 06:23:03.651289: done 
2022-04-12 06:23:03.699304: This epoch took 140.749500 s
 
2022-04-12 06:23:03.788942: svmem(total=99932160000, available=79810203648, percent=20.1, used=16092368896, free=48154931200, active=40326836224, inactive=9593757696, buffers=37810176, cached=35647049728, shared=2201268224, slab=784703488) 
2022-04-12 06:23:03.851425: 
epoch:  378 
2022-04-12 06:24:52.753333: train loss : -0.7071 
2022-04-12 06:25:04.401511: validation loss: -0.7041 
2022-04-12 06:25:04.467454: Average global foreground Dice: [0.913, 0.9711, 0.9841, 0.4262] 
2022-04-12 06:25:04.542546: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:25:05.472242: lr: 0.002789 
2022-04-12 06:25:05.519226: saving scheduled checkpoint file... 
2022-04-12 06:25:05.617458: saving checkpoint... 
2022-04-12 06:25:19.348305: done, saving took 13.78 seconds 
2022-04-12 06:25:19.574285: done 
2022-04-12 06:25:19.749799: saving checkpoint... 
2022-04-12 06:25:27.744065: done, saving took 8.06 seconds 
2022-04-12 06:25:27.991999: This epoch took 144.136642 s
 
2022-04-12 06:25:28.093048: svmem(total=99932160000, available=79786819584, percent=20.2, used=16126119936, free=48083771392, active=40381239296, inactive=9609904128, buffers=38404096, cached=35683864576, shared=2190729216, slab=782626816) 
2022-04-12 06:25:28.161334: 
epoch:  379 
2022-04-12 06:27:18.180974: train loss : -0.7008 
2022-04-12 06:27:29.987562: validation loss: -0.7016 
2022-04-12 06:27:30.163632: Average global foreground Dice: [0.9033, 0.9666, 0.9832, 0.4135] 
2022-04-12 06:27:30.223258: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:27:31.391770: lr: 0.002768 
2022-04-12 06:27:31.460369: saving scheduled checkpoint file... 
2022-04-12 06:27:31.601178: saving checkpoint... 
2022-04-12 06:27:40.816752: done, saving took 9.30 seconds 
2022-04-12 06:27:40.894485: done 
2022-04-12 06:27:40.920375: This epoch took 132.700833 s
 
2022-04-12 06:27:40.959823: svmem(total=99932160000, available=79568588800, percent=20.4, used=16344223744, free=47857799168, active=40621731840, inactive=9594806272, buffers=38445056, cached=35691692032, shared=2190729216, slab=783249408) 
2022-04-12 06:27:40.963499: 
epoch:  380 
2022-04-12 06:29:30.119094: train loss : -0.7063 
2022-04-12 06:29:42.416103: validation loss: -0.6743 
2022-04-12 06:29:42.453417: Average global foreground Dice: [0.9027, 0.9735, 0.983, 0.4161] 
2022-04-12 06:29:42.459196: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:29:43.324665: lr: 0.002747 
2022-04-12 06:29:43.373534: saving scheduled checkpoint file... 
2022-04-12 06:29:43.504213: saving checkpoint... 
2022-04-12 06:29:57.198604: done, saving took 13.82 seconds 
2022-04-12 06:29:57.381865: done 
2022-04-12 06:29:57.414563: This epoch took 136.447944 s
 
2022-04-12 06:29:57.429225: svmem(total=99932160000, available=79768842240, percent=20.2, used=16158474240, free=48040390656, active=40446705664, inactive=9585438720, buffers=38477824, cached=35694817280, shared=2176356352, slab=783851520) 
2022-04-12 06:29:57.462241: 
epoch:  381 
2022-04-12 06:31:45.276604: train loss : -0.6625 
2022-04-12 06:31:57.924842: validation loss: -0.6843 
2022-04-12 06:31:57.988915: Average global foreground Dice: [0.9139, 0.9676, 0.9811, 0.4002] 
2022-04-12 06:31:58.032696: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:31:58.964117: lr: 0.002727 
2022-04-12 06:31:59.040771: saving scheduled checkpoint file... 
2022-04-12 06:31:59.235624: saving checkpoint... 
2022-04-12 06:32:08.074612: done, saving took 8.96 seconds 
2022-04-12 06:32:08.174904: done 
2022-04-12 06:32:08.248259: This epoch took 130.751158 s
 
2022-04-12 06:32:08.259505: svmem(total=99932160000, available=79631130624, percent=20.3, used=16281763840, free=47881007104, active=40581410816, inactive=9615351808, buffers=38551552, cached=35730837504, shared=2190696448, slab=783478784) 
2022-04-12 06:32:08.310792: 
epoch:  382 
2022-04-12 06:33:58.588538: train loss : -0.6788 
2022-04-12 06:34:11.688854: validation loss: -0.7007 
2022-04-12 06:34:11.772102: Average global foreground Dice: [0.9207, 0.9712, 0.9838, 0.4102] 
2022-04-12 06:34:11.838534: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:34:12.489086: lr: 0.002706 
2022-04-12 06:34:12.498389: saving scheduled checkpoint file... 
2022-04-12 06:34:12.747851: saving checkpoint... 
2022-04-12 06:34:23.856992: done, saving took 11.25 seconds 
2022-04-12 06:34:23.976903: done 
2022-04-12 06:34:24.011968: This epoch took 135.656699 s
 
2022-04-12 06:34:24.053653: svmem(total=99932160000, available=79755407360, percent=20.2, used=16157200384, free=47987589120, active=40450686976, inactive=9633296384, buffers=38600704, cached=35748769792, shared=2191011840, slab=783794176) 
2022-04-12 06:34:24.118198: 
epoch:  383 
2022-04-12 06:36:15.641778: train loss : -0.6938 
2022-04-12 06:36:27.516878: validation loss: -0.6747 
2022-04-12 06:36:27.661174: Average global foreground Dice: [0.9144, 0.9626, 0.9801, 0.3831] 
2022-04-12 06:36:27.731817: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:36:28.773009: lr: 0.002685 
2022-04-12 06:36:28.879512: saving scheduled checkpoint file... 
2022-04-12 06:36:29.208592: saving checkpoint... 
2022-04-12 06:36:48.177638: done, saving took 19.06 seconds 
2022-04-12 06:36:48.304039: done 
2022-04-12 06:36:48.366515: This epoch took 144.212842 s
 
2022-04-12 06:36:48.401658: svmem(total=99932160000, available=79715905536, percent=20.2, used=16197124096, free=47939198976, active=40497999872, inactive=9641734144, buffers=38670336, cached=35757166592, shared=2190598144, slab=783769600) 
2022-04-12 06:36:48.512750: 
epoch:  384 
2022-04-12 06:38:37.125638: train loss : -0.6986 
2022-04-12 06:38:48.251976: validation loss: -0.6762 
2022-04-12 06:38:48.366163: Average global foreground Dice: [0.9032, 0.9696, 0.9834, 0.3897] 
2022-04-12 06:38:49.706934: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:38:50.521946: lr: 0.002664 
2022-04-12 06:38:50.637460: saving scheduled checkpoint file... 
2022-04-12 06:38:50.776845: saving checkpoint... 
2022-04-12 06:39:04.845894: done, saving took 14.14 seconds 
2022-04-12 06:39:04.989738: done 
2022-04-12 06:39:05.054762: This epoch took 136.492569 s
 
2022-04-12 06:39:05.057934: svmem(total=99932160000, available=79791759360, percent=20.2, used=16121032704, free=47996481536, active=40415121408, inactive=9660284928, buffers=38707200, cached=35775938560, shared=2190729216, slab=783552512) 
2022-04-12 06:39:05.074496: 
epoch:  385 
2022-04-12 06:40:52.995526: train loss : -0.7020 
2022-04-12 06:41:05.522639: validation loss: -0.7047 
2022-04-12 06:41:05.547968: Average global foreground Dice: [0.917, 0.9691, 0.9835, 0.4216] 
2022-04-12 06:41:05.641323: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:41:06.588824: lr: 0.002643 
2022-04-12 06:41:06.630975: saving scheduled checkpoint file... 
2022-04-12 06:41:06.754463: saving checkpoint... 
2022-04-12 06:41:24.559695: done, saving took 17.85 seconds 
2022-04-12 06:41:24.995236: done 
2022-04-12 06:41:25.085297: This epoch took 139.926616 s
 
2022-04-12 06:41:25.133434: svmem(total=99932160000, available=79711121408, percent=20.2, used=16201670656, free=47898415104, active=40502849536, inactive=9677623296, buffers=38756352, cached=35793317888, shared=2190729216, slab=783478784) 
2022-04-12 06:41:25.191126: 
epoch:  386 
2022-04-12 06:43:14.458931: train loss : -0.6957 
2022-04-12 06:43:25.873305: validation loss: -0.7007 
2022-04-12 06:43:26.146863: Average global foreground Dice: [0.9258, 0.9712, 0.9808, 0.4059] 
2022-04-12 06:43:26.228122: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:43:27.328659: lr: 0.002622 
2022-04-12 06:43:27.588435: saving scheduled checkpoint file... 
2022-04-12 06:43:27.835950: saving checkpoint... 
2022-04-12 06:43:42.125729: done, saving took 14.40 seconds 
2022-04-12 06:43:42.513058: done 
2022-04-12 06:43:42.580545: This epoch took 137.328654 s
 
2022-04-12 06:43:42.708177: svmem(total=99932160000, available=79686856704, percent=20.3, used=16225988608, free=47855718400, active=40524029952, inactive=9696251904, buffers=38789120, cached=35811663872, shared=2190995456, slab=782884864) 
2022-04-12 06:43:42.829835: 
epoch:  387 
2022-04-12 06:45:32.641507: train loss : -0.7007 
2022-04-12 06:45:44.724456: validation loss: -0.6869 
2022-04-12 06:45:44.884546: Average global foreground Dice: [0.8816, 0.9667, 0.9864, 0.4246] 
2022-04-12 06:45:44.939761: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:45:45.907649: lr: 0.002601 
2022-04-12 06:45:45.953911: saving scheduled checkpoint file... 
2022-04-12 06:45:46.111460: saving checkpoint... 
2022-04-12 06:45:52.925867: done, saving took 6.92 seconds 
2022-04-12 06:45:52.973270: done 
2022-04-12 06:45:52.997970: This epoch took 130.053040 s
 
2022-04-12 06:45:53.024528: svmem(total=99932160000, available=79594061824, percent=20.4, used=16318562304, free=47755075584, active=40611614720, inactive=9703804928, buffers=38838272, cached=35819683840, shared=2190807040, slab=783872000) 
2022-04-12 06:45:53.035014: 
epoch:  388 
2022-04-12 06:47:41.890784: train loss : -0.7022 
2022-04-12 06:47:54.337188: validation loss: -0.6605 
2022-04-12 06:47:54.406803: Average global foreground Dice: [0.9138, 0.9724, 0.9801, 0.3461] 
2022-04-12 06:47:54.468171: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:47:55.365734: lr: 0.002581 
2022-04-12 06:47:55.420610: saving scheduled checkpoint file... 
2022-04-12 06:47:55.638288: saving checkpoint... 
2022-04-12 06:48:08.064962: done, saving took 12.56 seconds 
2022-04-12 06:48:09.320165: done 
2022-04-12 06:48:09.614686: This epoch took 136.564460 s
 
2022-04-12 06:48:09.674963: svmem(total=99932160000, available=79601225728, percent=20.3, used=16311611392, free=47744733184, active=40600973312, inactive=9721139200, buffers=38871040, cached=35836944384, shared=2190618624, slab=785330176) 
2022-04-12 06:48:09.737547: 
epoch:  389 
2022-04-12 06:50:01.171066: train loss : -0.7021 
2022-04-12 06:50:13.723592: validation loss: -0.6928 
2022-04-12 06:50:13.797659: Average global foreground Dice: [0.9076, 0.9678, 0.985, 0.3937] 
2022-04-12 06:50:13.877448: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:50:15.055725: lr: 0.00256 
2022-04-12 06:50:15.117324: saving scheduled checkpoint file... 
2022-04-12 06:50:15.268889: saving checkpoint... 
2022-04-12 06:50:28.160580: done, saving took 13.01 seconds 
2022-04-12 06:50:28.337036: done 
2022-04-12 06:50:28.346080: This epoch took 138.590257 s
 
2022-04-12 06:50:28.408858: svmem(total=99932160000, available=79884279808, percent=20.1, used=16028340224, free=48010211328, active=40325844992, inactive=9738674176, buffers=38928384, cached=35854680064, shared=2190839808, slab=785965056) 
2022-04-12 06:50:28.465200: 
epoch:  390 
2022-04-12 06:52:18.053044: train loss : -0.7072 
2022-04-12 06:52:30.033743: validation loss: -0.6944 
2022-04-12 06:52:30.122742: Average global foreground Dice: [0.9089, 0.9731, 0.9829, 0.3812] 
2022-04-12 06:52:30.129061: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:52:31.020326: lr: 0.002539 
2022-04-12 06:52:31.116920: saving scheduled checkpoint file... 
2022-04-12 06:52:31.264222: saving checkpoint... 
2022-04-12 06:52:51.352415: done, saving took 20.18 seconds 
2022-04-12 06:52:51.592562: done 
2022-04-12 06:52:51.680806: This epoch took 143.212494 s
 
2022-04-12 06:52:51.759310: svmem(total=99932160000, available=79780728832, percent=20.2, used=16131923968, free=47935741952, active=40399183872, inactive=9736548352, buffers=38998016, cached=35825496064, shared=2190729216, slab=785539072) 
2022-04-12 06:52:51.929451: 
epoch:  391 
2022-04-12 06:54:42.273480: train loss : -0.7074 
2022-04-12 06:54:54.621754: validation loss: -0.7071 
2022-04-12 06:54:54.781312: Average global foreground Dice: [0.9172, 0.9751, 0.9853, 0.3894] 
2022-04-12 06:54:54.787475: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:54:55.782756: lr: 0.002518 
2022-04-12 06:54:55.922825: saving scheduled checkpoint file... 
2022-04-12 06:54:56.061102: saving checkpoint... 
2022-04-12 06:55:06.815321: done, saving took 10.86 seconds 
2022-04-12 06:55:06.997307: done 
2022-04-12 06:55:07.005533: This epoch took 134.901338 s
 
2022-04-12 06:55:07.047743: svmem(total=99932160000, available=79881687040, percent=20.1, used=16031211520, free=47998423040, active=40319696896, inactive=9750921216, buffers=39047168, cached=35863478272, shared=2190450688, slab=784666624) 
2022-04-12 06:55:07.086896: 
epoch:  392 
2022-04-12 06:56:56.218969: train loss : -0.7189 
2022-04-12 06:57:08.540605: validation loss: -0.6757 
2022-04-12 06:57:08.604618: Average global foreground Dice: [0.8913, 0.9684, 0.9794, 0.4109] 
2022-04-12 06:57:08.635926: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:57:09.697654: lr: 0.002497 
2022-04-12 06:57:09.758866: saving scheduled checkpoint file... 
2022-04-12 06:57:09.975175: saving checkpoint... 
2022-04-12 06:57:26.013931: done, saving took 16.18 seconds 
2022-04-12 06:57:26.385970: done 
2022-04-12 06:57:26.558808: This epoch took 139.429473 s
 
2022-04-12 06:57:26.677118: svmem(total=99932160000, available=79778586624, percent=20.2, used=16134680576, free=47876857856, active=40428118016, inactive=9769103360, buffers=39096320, cached=35881525248, shared=2190123008, slab=785334272) 
2022-04-12 06:57:26.793277: 
epoch:  393 
2022-04-12 06:59:15.929398: train loss : -0.7123 
2022-04-12 06:59:28.050975: validation loss: -0.7051 
2022-04-12 06:59:28.478494: Average global foreground Dice: [0.9215, 0.9704, 0.9852, 0.3991] 
2022-04-12 06:59:28.898761: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 06:59:30.890021: lr: 0.002476 
2022-04-12 06:59:31.452633: saving scheduled checkpoint file... 
2022-04-12 06:59:31.993224: saving checkpoint... 
2022-04-12 06:59:46.740396: done, saving took 14.87 seconds 
2022-04-12 06:59:46.897453: done 
2022-04-12 06:59:46.918474: This epoch took 140.067005 s
 
2022-04-12 06:59:46.974591: svmem(total=99932160000, available=79749816320, percent=20.2, used=16162775040, free=47830536192, active=40450838528, inactive=9785802752, buffers=39129088, cached=35899719680, shared=2190708736, slab=784998400) 
2022-04-12 06:59:46.996486: 
epoch:  394 
2022-04-12 07:01:36.939413: train loss : -0.7083 
2022-04-12 07:01:49.043395: validation loss: -0.6972 
2022-04-12 07:01:49.867721: Average global foreground Dice: [0.9185, 0.9723, 0.9853, 0.4189] 
2022-04-12 07:01:50.663099: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:01:52.631299: lr: 0.002455 
2022-04-12 07:01:53.325688: saving scheduled checkpoint file... 
2022-04-12 07:01:54.039624: saving checkpoint... 
2022-04-12 07:02:02.796904: done, saving took 8.90 seconds 
2022-04-12 07:02:02.836756: done 
2022-04-12 07:02:02.846314: This epoch took 135.792623 s
 
2022-04-12 07:02:02.855664: svmem(total=99932160000, available=79928111104, percent=20.0, used=15984324608, free=47993700352, active=40186597376, inactive=9887604736, buffers=39161856, cached=35914973184, shared=2190807040, slab=786173952) 
2022-04-12 07:02:02.860169: 
epoch:  395 
2022-04-12 07:03:53.316153: train loss : -0.7129 
2022-04-12 07:04:04.536247: validation loss: -0.7024 
2022-04-12 07:04:04.634251: Average global foreground Dice: [0.9264, 0.9728, 0.9855, 0.4251] 
2022-04-12 07:04:04.712353: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:04:06.341878: lr: 0.002434 
2022-04-12 07:04:07.735669: saving scheduled checkpoint file... 
2022-04-12 07:04:08.796172: saving checkpoint... 
2022-04-12 07:04:33.288697: done, saving took 24.59 seconds 
2022-04-12 07:04:33.449086: done 
2022-04-12 07:04:33.496771: This epoch took 150.635424 s
 
2022-04-12 07:04:33.509815: svmem(total=99932160000, available=79905026048, percent=20.0, used=16007385088, free=47959056384, active=40254758912, inactive=9857802240, buffers=39211008, cached=35926507520, shared=2190766080, slab=787685376) 
2022-04-12 07:04:33.555006: 
epoch:  396 
2022-04-12 07:06:22.810462: train loss : -0.7162 
2022-04-12 07:06:33.911831: validation loss: -0.7134 
2022-04-12 07:06:33.973864: Average global foreground Dice: [0.9269, 0.9769, 0.9867, 0.4232] 
2022-04-12 07:06:34.026650: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:06:34.736046: lr: 0.002413 
2022-04-12 07:06:34.770781: saving scheduled checkpoint file... 
2022-04-12 07:06:34.924482: saving checkpoint... 
2022-04-12 07:06:55.979297: done, saving took 21.14 seconds 
2022-04-12 07:06:56.237854: done 
2022-04-12 07:06:56.304210: This epoch took 142.743814 s
 
2022-04-12 07:06:56.441338: svmem(total=99932160000, available=79940333568, percent=20.0, used=15986016256, free=47976706048, active=40253808640, inactive=9841864704, buffers=39268352, cached=35930169344, shared=2176741376, slab=787750912) 
2022-04-12 07:06:56.450204: 
epoch:  397 
2022-04-12 07:08:45.576073: train loss : -0.7159 
2022-04-12 07:08:57.302559: validation loss: -0.6969 
2022-04-12 07:08:57.360534: Average global foreground Dice: [0.911, 0.9745, 0.9839, 0.4264] 
2022-04-12 07:08:57.514172: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:08:58.415743: lr: 0.002391 
2022-04-12 07:08:58.473856: saving scheduled checkpoint file... 
2022-04-12 07:08:58.640064: saving checkpoint... 
2022-04-12 07:09:19.494625: done, saving took 20.95 seconds 
2022-04-12 07:09:20.992298: done 
2022-04-12 07:09:21.549096: This epoch took 144.994236 s
 
2022-04-12 07:09:22.110594: svmem(total=99932160000, available=79863754752, percent=20.1, used=16062537728, free=47881592832, active=40324096000, inactive=9846902784, buffers=39321600, cached=35948707840, shared=2176741376, slab=799776768) 
2022-04-12 07:09:22.515626: 
epoch:  398 
2022-04-12 07:11:12.195704: train loss : -0.7145 
2022-04-12 07:11:24.403628: validation loss: -0.7101 
2022-04-12 07:11:24.693775: Average global foreground Dice: [0.9204, 0.9738, 0.9865, 0.4141] 
2022-04-12 07:11:24.819096: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:11:25.752954: lr: 0.00237 
2022-04-12 07:11:25.810109: saving scheduled checkpoint file... 
2022-04-12 07:11:25.936470: saving checkpoint... 
2022-04-12 07:11:48.495059: done, saving took 22.67 seconds 
2022-04-12 07:11:49.709696: done 
2022-04-12 07:11:50.283195: saving checkpoint... 
2022-04-12 07:12:02.485920: done, saving took 12.31 seconds 
2022-04-12 07:12:02.770993: This epoch took 159.794838 s
 
2022-04-12 07:12:02.848590: svmem(total=99932160000, available=79790821376, percent=20.2, used=16121430016, free=47781347328, active=40408559616, inactive=9880231936, buffers=39362560, cached=35990020096, shared=2190782464, slab=787120128) 
2022-04-12 07:12:02.991569: 
epoch:  399 
2022-04-12 07:13:52.097971: train loss : -0.7159 
2022-04-12 07:14:02.657552: validation loss: -0.7151 
2022-04-12 07:14:02.693408: Average global foreground Dice: [0.9295, 0.9719, 0.9833, 0.4201] 
2022-04-12 07:14:02.698724: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:14:03.617456: lr: 0.002349 
2022-04-12 07:14:03.623058: saving scheduled checkpoint file... 
2022-04-12 07:14:03.899148: saving checkpoint... 
2022-04-12 07:14:16.497787: done, saving took 12.73 seconds 
2022-04-12 07:14:16.737245: done 
2022-04-12 07:14:16.856538: saving checkpoint... 
2022-04-12 07:14:29.861542: done, saving took 13.12 seconds 
2022-04-12 07:14:30.528143: This epoch took 147.463310 s
 
2022-04-12 07:14:30.600138: svmem(total=99932160000, available=79737823232, percent=20.2, used=16174063616, free=47710900224, active=40467177472, inactive=9893744640, buffers=39411712, cached=36007784448, shared=2191036416, slab=786071552) 
2022-04-12 07:14:30.678457: 
epoch:  400 
2022-04-12 07:16:22.431639: train loss : -0.7187 
2022-04-12 07:16:34.395286: validation loss: -0.7207 
2022-04-12 07:16:34.468992: Average global foreground Dice: [0.9279, 0.9758, 0.986, 0.4373] 
2022-04-12 07:16:34.476035: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:16:35.421747: lr: 0.002328 
2022-04-12 07:16:35.434870: saving scheduled checkpoint file... 
2022-04-12 07:16:35.620575: saving checkpoint... 
2022-04-12 07:16:50.670356: done, saving took 15.18 seconds 
2022-04-12 07:16:50.820074: done 
2022-04-12 07:16:51.007639: saving checkpoint... 
2022-04-12 07:17:01.493500: done, saving took 10.61 seconds 
2022-04-12 07:17:01.723162: This epoch took 151.032354 s
 
2022-04-12 07:17:01.786898: svmem(total=99932160000, available=79666225152, percent=20.3, used=16246140928, free=47687094272, active=40206966784, inactive=10174799872, buffers=39456768, cached=35959468032, shared=2190729216, slab=785793024) 
2022-04-12 07:17:01.795408: 
epoch:  401 
2022-04-12 07:18:52.414840: train loss : -0.7018 
2022-04-12 07:19:05.084539: validation loss: -0.7100 
2022-04-12 07:19:05.179080: Average global foreground Dice: [0.926, 0.9689, 0.9823, 0.4174] 
2022-04-12 07:19:05.232452: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:19:06.053038: lr: 0.002307 
2022-04-12 07:19:06.150680: saving scheduled checkpoint file... 
2022-04-12 07:19:06.308480: saving checkpoint... 
2022-04-12 07:19:17.814570: done, saving took 11.60 seconds 
2022-04-12 07:19:17.895788: done 
2022-04-12 07:19:17.997980: saving checkpoint... 
2022-04-12 07:19:27.419327: done, saving took 9.47 seconds 
2022-04-12 07:19:27.560357: This epoch took 145.664984 s
 
2022-04-12 07:19:27.615616: svmem(total=99932160000, available=79718424576, percent=20.2, used=16193662976, free=47720853504, active=40439300096, inactive=9904889856, buffers=39514112, cached=35978129408, shared=2190807040, slab=786681856) 
2022-04-12 07:19:27.621308: 
epoch:  402 
2022-04-12 07:21:15.317337: train loss : -0.7217 
2022-04-12 07:21:28.149652: validation loss: -0.7139 
2022-04-12 07:21:28.184170: Average global foreground Dice: [0.9102, 0.9754, 0.9838, 0.4354] 
2022-04-12 07:21:28.265687: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:21:28.966989: lr: 0.002286 
2022-04-12 07:21:29.060914: saving scheduled checkpoint file... 
2022-04-12 07:21:29.218837: saving checkpoint... 
2022-04-12 07:21:42.979543: done, saving took 13.84 seconds 
2022-04-12 07:21:43.170391: done 
2022-04-12 07:21:43.265267: saving checkpoint... 
2022-04-12 07:21:56.258390: done, saving took 13.04 seconds 
2022-04-12 07:21:56.502016: This epoch took 148.878586 s
 
2022-04-12 07:21:56.584164: svmem(total=99932160000, available=79541260288, percent=20.4, used=16371396608, free=47528476672, active=40651833344, inactive=9891930112, buffers=39563264, cached=35992723456, shared=2190311424, slab=786026496) 
2022-04-12 07:21:56.626822: 
epoch:  403 
2022-04-12 07:23:46.126993: train loss : -0.7101 
2022-04-12 07:23:58.078584: validation loss: -0.7065 
2022-04-12 07:23:58.108880: Average global foreground Dice: [0.9154, 0.9714, 0.9838, 0.4171] 
2022-04-12 07:23:58.130846: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:23:59.038499: lr: 0.002264 
2022-04-12 07:23:59.114466: saving scheduled checkpoint file... 
2022-04-12 07:23:59.259358: saving checkpoint... 
2022-04-12 07:24:09.810021: done, saving took 10.63 seconds 
2022-04-12 07:24:09.971931: done 
2022-04-12 07:24:10.079469: This epoch took 133.386778 s
 
2022-04-12 07:24:10.141929: svmem(total=99932160000, available=79919702016, percent=20.0, used=15992586240, free=47828164608, active=40277086208, inactive=9960775680, buffers=39612416, cached=36071796736, shared=2190729216, slab=787161088) 
2022-04-12 07:24:10.214431: 
epoch:  404 
2022-04-12 07:25:58.563195: train loss : -0.6960 
2022-04-12 07:26:09.454079: validation loss: -0.7034 
2022-04-12 07:26:09.936783: Average global foreground Dice: [0.9224, 0.9713, 0.9841, 0.4027] 
2022-04-12 07:26:10.141001: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:26:11.292790: lr: 0.002243 
2022-04-12 07:26:11.358593: saving scheduled checkpoint file... 
2022-04-12 07:26:11.543064: saving checkpoint... 
2022-04-12 07:26:20.675402: done, saving took 9.21 seconds 
2022-04-12 07:26:21.034556: done 
2022-04-12 07:26:21.099380: This epoch took 130.839453 s
 
2022-04-12 07:26:21.337390: svmem(total=99932160000, available=79931809792, percent=20.0, used=15980462080, free=47832469504, active=40269774848, inactive=9965883392, buffers=39673856, cached=36079554560, shared=2190729216, slab=785960960) 
2022-04-12 07:26:21.518413: 
epoch:  405 
2022-04-12 07:28:10.845783: train loss : -0.7063 
2022-04-12 07:28:24.171669: validation loss: -0.6910 
2022-04-12 07:28:25.187365: Average global foreground Dice: [0.9085, 0.9685, 0.9792, 0.4246] 
2022-04-12 07:28:25.680257: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:28:27.780900: lr: 0.002222 
2022-04-12 07:28:28.690517: saving scheduled checkpoint file... 
2022-04-12 07:28:29.386337: saving checkpoint... 
2022-04-12 07:28:46.013893: done, saving took 16.75 seconds 
2022-04-12 07:28:47.510745: done 
2022-04-12 07:28:47.651891: This epoch took 145.976617 s
 
2022-04-12 07:28:47.667989: svmem(total=99932160000, available=79611748352, percent=20.3, used=16299958272, free=47494086656, active=40600399872, inactive=9981411328, buffers=39706624, cached=36098408448, shared=2191224832, slab=785080320) 
2022-04-12 07:28:47.717533: 
epoch:  406 
2022-04-12 07:30:38.161488: train loss : -0.7116 
2022-04-12 07:30:49.466289: validation loss: -0.7150 
2022-04-12 07:30:50.611089: Average global foreground Dice: [0.9107, 0.9764, 0.9841, 0.4391] 
2022-04-12 07:30:51.372604: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:30:53.881980: lr: 0.002201 
2022-04-12 07:30:54.643499: saving scheduled checkpoint file... 
2022-04-12 07:30:54.945998: saving checkpoint... 
2022-04-12 07:31:14.628439: done, saving took 19.78 seconds 
2022-04-12 07:31:14.950902: done 
2022-04-12 07:31:15.154549: saving checkpoint... 
2022-04-12 07:31:31.484792: done, saving took 16.41 seconds 
2022-04-12 07:31:31.798235: This epoch took 164.077214 s
 
2022-04-12 07:31:31.827451: svmem(total=99932160000, available=79841157120, percent=20.1, used=16071077888, free=47703724032, active=40364175360, inactive=10000437248, buffers=39763968, cached=36117594112, shared=2190675968, slab=785035264) 
2022-04-12 07:31:31.865331: 
epoch:  407 
2022-04-12 07:33:21.132783: train loss : -0.7062 
2022-04-12 07:33:33.435717: validation loss: -0.7105 
2022-04-12 07:33:33.749799: Average global foreground Dice: [0.9107, 0.9715, 0.98, 0.4423] 
2022-04-12 07:33:33.841694: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:33:34.877660: lr: 0.002179 
2022-04-12 07:33:34.985049: saving scheduled checkpoint file... 
2022-04-12 07:33:35.105670: saving checkpoint... 
2022-04-12 07:33:48.724722: done, saving took 13.69 seconds 
2022-04-12 07:33:49.650118: done 
2022-04-12 07:33:49.783685: saving checkpoint... 
2022-04-12 07:34:01.346302: done, saving took 11.63 seconds 
2022-04-12 07:34:01.580811: This epoch took 149.653713 s
 
2022-04-12 07:34:01.685671: svmem(total=99932160000, available=79908962304, percent=20.0, used=16012795904, free=47754137600, active=40150564864, inactive=10168238080, buffers=39796736, cached=36125429760, shared=2181156864, slab=785080320) 
2022-04-12 07:34:01.758687: 
epoch:  408 
2022-04-12 07:35:51.128340: train loss : -0.6997 
2022-04-12 07:36:03.874283: validation loss: -0.7203 
2022-04-12 07:36:03.986555: Average global foreground Dice: [0.9249, 0.9776, 0.9832, 0.4237] 
2022-04-12 07:36:04.038844: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:36:05.197863: lr: 0.002158 
2022-04-12 07:36:05.336076: saving scheduled checkpoint file... 
2022-04-12 07:36:05.493869: saving checkpoint... 
2022-04-12 07:36:19.613549: done, saving took 14.21 seconds 
2022-04-12 07:36:19.809441: done 
2022-04-12 07:36:19.913008: saving checkpoint... 
2022-04-12 07:36:31.756083: done, saving took 11.92 seconds 
2022-04-12 07:36:32.072106: This epoch took 150.293941 s
 
2022-04-12 07:36:32.137351: svmem(total=99932160000, available=79792386048, percent=20.2, used=16119508992, free=47627984896, active=40323694592, inactive=10117255168, buffers=39870464, cached=36144795648, shared=2191114240, slab=785141760) 
2022-04-12 07:36:32.169154: 
epoch:  409 
2022-04-12 07:38:19.246177: train loss : -0.6995 
2022-04-12 07:38:31.130662: validation loss: -0.7048 
2022-04-12 07:38:31.210863: Average global foreground Dice: [0.9116, 0.9734, 0.9845, 0.4116] 
2022-04-12 07:38:31.268734: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:38:32.419905: lr: 0.002137 
2022-04-12 07:38:32.467287: saving scheduled checkpoint file... 
2022-04-12 07:38:32.676308: saving checkpoint... 
2022-04-12 07:38:44.775162: done, saving took 12.21 seconds 
2022-04-12 07:38:44.902052: done 
2022-04-12 07:38:44.911200: This epoch took 132.673280 s
 
2022-04-12 07:38:44.960330: svmem(total=99932160000, available=79841271808, percent=20.1, used=16073961472, free=47668903936, active=40330076160, inactive=10074304512, buffers=39911424, cached=36149383168, shared=2187755520, slab=784216064) 
2022-04-12 07:38:44.980531: 
epoch:  410 
2022-04-12 07:40:34.161970: train loss : -0.7093 
2022-04-12 07:40:47.851121: validation loss: -0.7064 
2022-04-12 07:40:47.881563: Average global foreground Dice: [0.9212, 0.9751, 0.9846, 0.3999] 
2022-04-12 07:40:47.885227: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:40:48.792421: lr: 0.002115 
2022-04-12 07:40:48.850119: saving scheduled checkpoint file... 
2022-04-12 07:40:49.066046: saving checkpoint... 
2022-04-12 07:41:02.772944: done, saving took 13.85 seconds 
2022-04-12 07:41:02.822220: done 
2022-04-12 07:41:02.834704: This epoch took 137.805957 s
 
2022-04-12 07:41:02.861414: svmem(total=99932160000, available=79916892160, percent=20.0, used=15995289600, free=47718105088, active=40281214976, inactive=10073423872, buffers=39952384, cached=36178812928, shared=2190729216, slab=784089088) 
2022-04-12 07:41:02.908898: 
epoch:  411 
2022-04-12 07:42:54.017033: train loss : -0.7101 
2022-04-12 07:43:06.780807: validation loss: -0.7150 
2022-04-12 07:43:06.881374: Average global foreground Dice: [0.9111, 0.9696, 0.9867, 0.4242] 
2022-04-12 07:43:06.979173: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:43:07.982301: lr: 0.002094 
2022-04-12 07:43:08.097578: saving scheduled checkpoint file... 
2022-04-12 07:43:08.325756: saving checkpoint... 
2022-04-12 07:43:29.414668: done, saving took 21.20 seconds 
2022-04-12 07:43:29.693205: done 
2022-04-12 07:43:29.747083: This epoch took 146.828316 s
 
2022-04-12 07:43:29.801602: svmem(total=99932160000, available=79789735936, percent=20.2, used=16123039744, free=47572336640, active=40423948288, inactive=10081648640, buffers=40001536, cached=36196782080, shared=2190155776, slab=784035840) 
2022-04-12 07:43:29.845949: 
epoch:  412 
2022-04-12 07:45:19.690769: train loss : -0.7213 
2022-04-12 07:45:32.452212: validation loss: -0.6939 
2022-04-12 07:45:32.639613: Average global foreground Dice: [0.9111, 0.97, 0.979, 0.3771] 
2022-04-12 07:45:32.755949: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:45:33.438843: lr: 0.002072 
2022-04-12 07:45:33.455264: saving scheduled checkpoint file... 
2022-04-12 07:45:33.551790: saving checkpoint... 
2022-04-12 07:45:49.745712: done, saving took 16.26 seconds 
2022-04-12 07:45:50.231056: done 
2022-04-12 07:45:50.286905: This epoch took 140.426960 s
 
2022-04-12 07:45:50.358575: svmem(total=99932160000, available=79648649216, percent=20.3, used=16263536640, free=47412682752, active=40562491392, inactive=10098798592, buffers=40050688, cached=36215889920, shared=2190729216, slab=785022976) 
2022-04-12 07:45:50.486627: 
epoch:  413 
2022-04-12 07:47:37.958561: train loss : -0.7170 
2022-04-12 07:47:49.914708: validation loss: -0.7205 
2022-04-12 07:47:50.035194: Average global foreground Dice: [0.9236, 0.9743, 0.9859, 0.4141] 
2022-04-12 07:47:50.096149: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:47:50.780577: lr: 0.002051 
2022-04-12 07:47:50.861063: saving scheduled checkpoint file... 
2022-04-12 07:47:51.000794: saving checkpoint... 
2022-04-12 07:48:08.145350: done, saving took 17.22 seconds 
2022-04-12 07:48:08.398288: done 
2022-04-12 07:48:08.498571: This epoch took 137.953929 s
 
2022-04-12 07:48:08.504371: svmem(total=99932160000, available=79712595968, percent=20.2, used=16199536640, free=47468728320, active=40490975232, inactive=10106646528, buffers=40083456, cached=36223811584, shared=2190807040, slab=784863232) 
2022-04-12 07:48:08.555674: 
epoch:  414 
2022-04-12 07:49:58.344952: train loss : -0.7229 
2022-04-12 07:50:09.759754: validation loss: -0.7253 
2022-04-12 07:50:09.836891: Average global foreground Dice: [0.9169, 0.9722, 0.9876, 0.4537] 
2022-04-12 07:50:09.861612: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:50:10.810944: lr: 0.00203 
2022-04-12 07:50:10.865175: saving scheduled checkpoint file... 
2022-04-12 07:50:10.963362: saving checkpoint... 
2022-04-12 07:50:20.964128: done, saving took 10.05 seconds 
2022-04-12 07:50:21.163758: done 
2022-04-12 07:50:21.221466: This epoch took 132.620038 s
 
2022-04-12 07:50:21.271056: svmem(total=99932160000, available=79892463616, percent=20.1, used=16009375744, free=47631122432, active=40307634176, inactive=10134147072, buffers=40140800, cached=36251521024, shared=2200948736, slab=785408000) 
2022-04-12 07:50:21.286769: 
epoch:  415 
2022-04-12 07:52:12.019482: train loss : -0.7056 
2022-04-12 07:52:23.723711: validation loss: -0.7087 
2022-04-12 07:52:23.889569: Average global foreground Dice: [0.9215, 0.9724, 0.9837, 0.4024] 
2022-04-12 07:52:24.067470: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:52:25.565945: lr: 0.002008 
2022-04-12 07:52:25.710639: saving scheduled checkpoint file... 
2022-04-12 07:52:26.030425: saving checkpoint... 
2022-04-12 07:52:41.842292: done, saving took 15.90 seconds 
2022-04-12 07:52:42.106194: done 
2022-04-12 07:52:42.142853: This epoch took 140.799176 s
 
2022-04-12 07:52:42.146411: svmem(total=99932160000, available=79625207808, percent=20.3, used=16300740608, free=47398494208, active=40541704192, inactive=10133016576, buffers=40173568, cached=36192751616, shared=2176851968, slab=785944576) 
2022-04-12 07:52:42.151031: 
epoch:  416 
2022-04-12 07:54:32.788377: train loss : -0.7047 
2022-04-12 07:54:43.727943: validation loss: -0.6864 
2022-04-12 07:54:43.739972: Average global foreground Dice: [0.9064, 0.973, 0.9804, 0.432] 
2022-04-12 07:54:43.785895: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:54:44.820165: lr: 0.001987 
2022-04-12 07:54:44.831513: saving scheduled checkpoint file... 
2022-04-12 07:54:44.965348: saving checkpoint... 
2022-04-12 07:54:59.590399: done, saving took 14.72 seconds 
2022-04-12 07:54:59.797832: done 
2022-04-12 07:54:59.827221: This epoch took 137.670793 s
 
2022-04-12 07:54:59.840025: svmem(total=99932160000, available=79698952192, percent=20.2, used=16212533248, free=47454851072, active=40502272000, inactive=10118025216, buffers=40206336, cached=36224569344, shared=2191224832, slab=786358272) 
2022-04-12 07:54:59.870411: 
epoch:  417 
2022-04-12 07:56:47.885848: train loss : -0.7096 
2022-04-12 07:56:58.937771: validation loss: -0.7223 
2022-04-12 07:56:59.069465: Average global foreground Dice: [0.924, 0.9752, 0.9848, 0.443] 
2022-04-12 07:56:59.072101: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:56:59.957296: lr: 0.001965 
2022-04-12 07:57:00.060326: saving scheduled checkpoint file... 
2022-04-12 07:57:00.196696: saving checkpoint... 
2022-04-12 07:57:10.666095: done, saving took 10.60 seconds 
2022-04-12 07:57:11.072646: done 
2022-04-12 07:57:11.295343: saving checkpoint... 
2022-04-12 07:57:27.681063: done, saving took 16.49 seconds 
2022-04-12 07:57:27.725703: This epoch took 147.834941 s
 
2022-04-12 07:57:27.738962: svmem(total=99932160000, available=79627567104, percent=20.3, used=16284172288, free=47323492352, active=40577777664, inactive=10166558720, buffers=40267776, cached=36284227584, shared=2190917632, slab=786731008) 
2022-04-12 07:57:27.750239: 
epoch:  418 
2022-04-12 07:59:18.102785: train loss : -0.7123 
2022-04-12 07:59:30.595378: validation loss: -0.7238 
2022-04-12 07:59:30.664433: Average global foreground Dice: [0.9176, 0.9728, 0.9853, 0.4377] 
2022-04-12 07:59:30.772123: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 07:59:31.501398: lr: 0.001943 
2022-04-12 07:59:31.550690: saving scheduled checkpoint file... 
2022-04-12 07:59:31.643714: saving checkpoint... 
2022-04-12 07:59:44.084050: done, saving took 12.53 seconds 
2022-04-12 07:59:44.233917: done 
2022-04-12 07:59:44.353690: saving checkpoint... 
2022-04-12 07:59:52.767618: done, saving took 8.48 seconds 
2022-04-12 07:59:52.903437: This epoch took 145.135446 s
 
2022-04-12 07:59:52.912085: svmem(total=99932160000, available=79678312448, percent=20.3, used=16233533440, free=47355695104, active=40528289792, inactive=10184806400, buffers=40300544, cached=36302630912, shared=2190729216, slab=787791872) 
2022-04-12 07:59:52.992818: 
epoch:  419 
2022-04-12 08:01:43.527424: train loss : -0.7196 
2022-04-12 08:01:56.035618: validation loss: -0.6645 
2022-04-12 08:01:56.112365: Average global foreground Dice: [0.919, 0.9707, 0.9729, 0.4022] 
2022-04-12 08:01:56.176160: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:01:57.127458: lr: 0.001922 
2022-04-12 08:01:57.143695: saving scheduled checkpoint file... 
2022-04-12 08:01:57.298050: saving checkpoint... 
2022-04-12 08:02:06.131681: done, saving took 8.94 seconds 
2022-04-12 08:02:06.258951: done 
2022-04-12 08:02:06.323739: This epoch took 133.283890 s
 
2022-04-12 08:02:06.385189: svmem(total=99932160000, available=79792078848, percent=20.2, used=16119861248, free=47451934720, active=40415342592, inactive=10202234880, buffers=40341504, cached=36320022528, shared=2190729216, slab=787513344) 
2022-04-12 08:02:06.435473: 
epoch:  420 
2022-04-12 08:03:55.169189: train loss : -0.7035 
2022-04-12 08:04:06.748736: validation loss: -0.7191 
2022-04-12 08:04:06.779872: Average global foreground Dice: [0.9172, 0.9721, 0.9842, 0.4072] 
2022-04-12 08:04:06.813113: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:04:07.892047: lr: 0.0019 
2022-04-12 08:04:07.896074: saving scheduled checkpoint file... 
2022-04-12 08:04:08.087176: saving checkpoint... 
2022-04-12 08:04:21.239674: done, saving took 13.26 seconds 
2022-04-12 08:04:21.396390: done 
2022-04-12 08:04:21.422721: This epoch took 134.941712 s
 
2022-04-12 08:04:21.427012: svmem(total=99932160000, available=79795089408, percent=20.2, used=16117243904, free=47436464128, active=40417771520, inactive=10220081152, buffers=40390656, cached=36338061312, shared=2190254080, slab=786497536) 
2022-04-12 08:04:21.432788: 
epoch:  421 
2022-04-12 08:06:11.074583: train loss : -0.6917 
2022-04-12 08:06:22.761287: validation loss: -0.6914 
2022-04-12 08:06:22.844716: Average global foreground Dice: [0.9106, 0.9726, 0.9793, 0.4209] 
2022-04-12 08:06:22.901042: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:06:23.829499: lr: 0.001879 
2022-04-12 08:06:24.025629: saving scheduled checkpoint file... 
2022-04-12 08:06:24.204674: saving checkpoint... 
2022-04-12 08:06:40.766865: done, saving took 16.69 seconds 
2022-04-12 08:06:41.088897: done 
2022-04-12 08:06:41.150888: This epoch took 139.714046 s
 
2022-04-12 08:06:41.269772: svmem(total=99932160000, available=79811305472, percent=20.1, used=16100352000, free=47444860928, active=40395870208, inactive=10228305920, buffers=40448000, cached=36346499072, shared=2190852096, slab=787320832) 
2022-04-12 08:06:41.430278: 
epoch:  422 
2022-04-12 08:08:31.888229: train loss : -0.7029 
2022-04-12 08:08:44.138980: validation loss: -0.7318 
2022-04-12 08:08:44.158502: Average global foreground Dice: [0.925, 0.973, 0.9863, 0.4404] 
2022-04-12 08:08:44.213091: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:08:45.170256: lr: 0.001857 
2022-04-12 08:08:45.213556: saving scheduled checkpoint file... 
2022-04-12 08:08:45.397088: saving checkpoint... 
2022-04-12 08:08:57.722400: done, saving took 12.45 seconds 
2022-04-12 08:08:57.858532: done 
2022-04-12 08:08:57.912264: This epoch took 136.428270 s
 
2022-04-12 08:08:57.935964: svmem(total=99932160000, available=79708942336, percent=20.2, used=16192249856, free=47324880896, active=40483901440, inactive=10256429056, buffers=40488960, cached=36374540288, shared=2201436160, slab=788197376) 
2022-04-12 08:08:57.967237: 
epoch:  423 
2022-04-12 08:10:48.567607: train loss : -0.7175 
2022-04-12 08:11:01.148610: validation loss: -0.7113 
2022-04-12 08:11:01.334580: Average global foreground Dice: [0.9165, 0.9737, 0.9839, 0.4285] 
2022-04-12 08:11:01.414437: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:11:02.403980: lr: 0.001835 
2022-04-12 08:11:02.463020: saving scheduled checkpoint file... 
2022-04-12 08:11:02.746465: saving checkpoint... 
2022-04-12 08:11:19.687804: done, saving took 17.05 seconds 
2022-04-12 08:11:19.978785: done 
2022-04-12 08:11:20.053638: This epoch took 142.049031 s
 
2022-04-12 08:11:20.182458: svmem(total=99932160000, available=79784570880, percent=20.2, used=16127266816, free=47382044672, active=40421220352, inactive=10264166400, buffers=40546304, cached=36382302208, shared=2190729216, slab=786210816) 
2022-04-12 08:11:20.228673: 
epoch:  424 
2022-04-12 08:13:10.637990: train loss : -0.7183 
2022-04-12 08:13:24.300807: validation loss: -0.7129 
2022-04-12 08:13:24.387660: Average global foreground Dice: [0.924, 0.9725, 0.9852, 0.4217] 
2022-04-12 08:13:24.409925: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:13:25.416517: lr: 0.001813 
2022-04-12 08:13:25.488354: saving scheduled checkpoint file... 
2022-04-12 08:13:25.659144: saving checkpoint... 
2022-04-12 08:13:38.511906: done, saving took 12.96 seconds 
2022-04-12 08:13:38.860495: done 
2022-04-12 08:13:38.965897: saving checkpoint... 
2022-04-12 08:13:52.792150: done, saving took 13.87 seconds 
2022-04-12 08:13:52.893598: This epoch took 152.250868 s
 
2022-04-12 08:13:52.915296: svmem(total=99932160000, available=79605596160, percent=20.3, used=16306130944, free=47195619328, active=40461053952, inactive=10415968256, buffers=40579072, cached=36389830656, shared=2190729216, slab=785543168) 
2022-04-12 08:13:52.967920: 
epoch:  425 
2022-04-12 08:15:41.291389: train loss : -0.7199 
2022-04-12 08:15:52.490978: validation loss: -0.7136 
2022-04-12 08:15:52.585149: Average global foreground Dice: [0.9196, 0.9758, 0.9862, 0.3941] 
2022-04-12 08:15:52.651943: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:15:53.517769: lr: 0.001792 
2022-04-12 08:15:53.678078: saving scheduled checkpoint file... 
2022-04-12 08:15:53.748471: saving checkpoint... 
2022-04-12 08:16:08.750316: done, saving took 15.06 seconds 
2022-04-12 08:16:08.960726: done 
2022-04-12 08:16:09.027880: This epoch took 136.050440 s
 
2022-04-12 08:16:09.202147: svmem(total=99932160000, available=79853441024, percent=20.1, used=16058290176, free=47438348288, active=40225288192, inactive=10405122048, buffers=40644608, cached=36394876928, shared=2190761984, slab=785989632) 
2022-04-12 08:16:09.372068: 
epoch:  426 
2022-04-12 08:17:58.038779: train loss : -0.7177 
2022-04-12 08:18:10.722613: validation loss: -0.7168 
2022-04-12 08:18:10.793005: Average global foreground Dice: [0.8999, 0.9723, 0.9848, 0.4177] 
2022-04-12 08:18:10.872353: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:18:11.786058: lr: 0.00177 
2022-04-12 08:18:11.841777: saving scheduled checkpoint file... 
2022-04-12 08:18:11.904917: saving checkpoint... 
2022-04-12 08:18:26.074753: done, saving took 14.22 seconds 
2022-04-12 08:18:26.159930: done 
2022-04-12 08:18:26.204438: This epoch took 136.762006 s
 
2022-04-12 08:18:26.209641: svmem(total=99932160000, available=79782817792, percent=20.2, used=16128978944, free=47372300288, active=40217305088, inactive=10484187136, buffers=40701952, cached=36390178816, shared=2190729216, slab=785354752) 
2022-04-12 08:18:26.270786: 
epoch:  427 
2022-04-12 08:20:16.238244: train loss : -0.7155 
2022-04-12 08:20:29.484835: validation loss: -0.7236 
2022-04-12 08:20:31.729320: Average global foreground Dice: [0.9281, 0.9731, 0.9859, 0.4173] 
2022-04-12 08:20:32.507272: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:20:33.431398: lr: 0.001748 
2022-04-12 08:20:33.602775: saving scheduled checkpoint file... 
2022-04-12 08:20:33.735226: saving checkpoint... 
2022-04-12 08:20:48.831190: done, saving took 15.17 seconds 
2022-04-12 08:20:48.958557: done 
2022-04-12 08:20:49.008272: This epoch took 142.723423 s
 
2022-04-12 08:20:49.049868: svmem(total=99932160000, available=79542366208, percent=20.4, used=16369192960, free=47124668416, active=40530644992, inactive=10418237440, buffers=40742912, cached=36397555712, shared=2190807040, slab=784859136) 
2022-04-12 08:20:49.107153: 
epoch:  428 
2022-04-12 08:22:38.660280: train loss : -0.7187 
2022-04-12 08:22:50.116807: validation loss: -0.7157 
2022-04-12 08:22:50.174300: Average global foreground Dice: [0.9139, 0.9784, 0.9853, 0.4357] 
2022-04-12 08:22:50.245027: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:22:51.328597: lr: 0.001726 
2022-04-12 08:22:51.383809: saving scheduled checkpoint file... 
2022-04-12 08:22:51.508059: saving checkpoint... 
2022-04-12 08:23:03.343060: done, saving took 11.92 seconds 
2022-04-12 08:23:03.433993: done 
2022-04-12 08:23:03.459978: This epoch took 134.294674 s
 
2022-04-12 08:23:03.508412: svmem(total=99932160000, available=79776374784, percent=20.2, used=16135090176, free=47341015040, active=40373104640, inactive=10360827904, buffers=40783872, cached=36415270912, shared=2190807040, slab=786055168) 
2022-04-12 08:23:03.532767: 
epoch:  429 
2022-04-12 08:24:53.080371: train loss : -0.7342 
2022-04-12 08:25:05.068088: validation loss: -0.7168 
2022-04-12 08:25:05.215041: Average global foreground Dice: [0.9156, 0.9741, 0.9839, 0.4283] 
2022-04-12 08:25:05.323544: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:25:06.476753: lr: 0.001704 
2022-04-12 08:25:06.592776: saving scheduled checkpoint file... 
2022-04-12 08:25:06.884058: saving checkpoint... 
2022-04-12 08:25:24.569254: done, saving took 17.80 seconds 
2022-04-12 08:25:24.636509: done 
2022-04-12 08:25:24.794726: saving checkpoint... 
2022-04-12 08:25:34.479044: done, saving took 9.80 seconds 
2022-04-12 08:25:34.653597: This epoch took 151.086057 s
 
2022-04-12 08:25:34.695890: svmem(total=99932160000, available=80099176448, percent=19.8, used=15931707392, free=47685361664, active=39732641792, inactive=10653474816, buffers=40841216, cached=36274249728, shared=2071523328, slab=787668992) 
2022-04-12 08:25:34.749263: 
epoch:  430 
2022-04-12 08:27:25.646687: train loss : -0.7161 
2022-04-12 08:27:38.476859: validation loss: -0.7131 
2022-04-12 08:27:38.562359: Average global foreground Dice: [0.9282, 0.9748, 0.9837, 0.4434] 
2022-04-12 08:27:38.568077: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:27:39.596661: lr: 0.001682 
2022-04-12 08:27:39.691918: saving scheduled checkpoint file... 
2022-04-12 08:27:39.785793: saving checkpoint... 
2022-04-12 08:27:56.634859: done, saving took 16.93 seconds 
2022-04-12 08:27:56.854499: done 
2022-04-12 08:27:56.988114: saving checkpoint... 
2022-04-12 08:28:16.761990: done, saving took 19.85 seconds 
2022-04-12 08:28:20.234234: This epoch took 165.443653 s
 
2022-04-12 08:28:20.401137: svmem(total=99932160000, available=79866212352, percent=20.1, used=16059359232, free=47443492864, active=39930105856, inactive=10691743744, buffers=40894464, cached=36388413440, shared=2176741376, slab=786583552) 
2022-04-12 08:28:20.462970: 
epoch:  431 
2022-04-12 08:30:10.749885: train loss : -0.7166 
2022-04-12 08:30:22.703434: validation loss: -0.7188 
2022-04-12 08:30:22.913883: Average global foreground Dice: [0.9192, 0.9757, 0.9845, 0.4245] 
2022-04-12 08:30:23.016308: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:30:24.179719: lr: 0.00166 
2022-04-12 08:30:24.312372: saving scheduled checkpoint file... 
2022-04-12 08:30:24.498530: saving checkpoint... 
2022-04-12 08:30:42.332017: done, saving took 17.92 seconds 
2022-04-12 08:30:42.500390: done 
2022-04-12 08:30:42.625209: saving checkpoint... 
2022-04-12 08:30:52.134527: done, saving took 9.60 seconds 
2022-04-12 08:30:52.225972: This epoch took 151.682799 s
 
2022-04-12 08:30:52.245944: svmem(total=99932160000, available=79740256256, percent=20.2, used=16171388928, free=47295344640, active=40325414912, inactive=10450477056, buffers=40964096, cached=36424462336, shared=2190540800, slab=785285120) 
2022-04-12 08:30:52.302829: 
epoch:  432 
2022-04-12 08:32:41.681872: train loss : -0.7242 
2022-04-12 08:33:04.676383: validation loss: -0.7095 
2022-04-12 08:33:04.736118: Average global foreground Dice: [0.9172, 0.9716, 0.9841, 0.413] 
2022-04-12 08:33:04.785322: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:33:05.796570: lr: 0.001638 
2022-04-12 08:33:05.801738: saving scheduled checkpoint file... 
2022-04-12 08:33:05.959875: saving checkpoint... 
2022-04-12 08:33:16.945635: done, saving took 11.09 seconds 
2022-04-12 08:33:17.107443: done 
2022-04-12 08:33:17.135679: This epoch took 144.780435 s
 
2022-04-12 08:33:17.150676: svmem(total=99932160000, available=79741685760, percent=20.2, used=16182161408, free=47303114752, active=40270614528, inactive=10489647104, buffers=41013248, cached=36405870592, shared=2178338816, slab=785584128) 
2022-04-12 08:33:17.211778: 
epoch:  433 
2022-04-12 08:35:06.579296: train loss : -0.7274 
2022-04-12 08:35:19.562860: validation loss: -0.7166 
2022-04-12 08:35:20.840693: Average global foreground Dice: [0.9102, 0.9755, 0.985, 0.4422] 
2022-04-12 08:35:22.698435: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:35:26.197285: lr: 0.001616 
2022-04-12 08:35:26.374141: saving scheduled checkpoint file... 
2022-04-12 08:35:26.648752: saving checkpoint... 
2022-04-12 08:35:44.824749: done, saving took 18.28 seconds 
2022-04-12 08:35:45.082720: done 
2022-04-12 08:35:45.271435: saving checkpoint... 
2022-04-12 08:35:58.932572: done, saving took 13.78 seconds 
2022-04-12 08:35:59.269271: This epoch took 162.032115 s
 
2022-04-12 08:35:59.356766: svmem(total=99932160000, available=79851716608, percent=20.1, used=16059768832, free=47372435456, active=40288301056, inactive=10407194624, buffers=41066496, cached=36458889216, shared=2190729216, slab=785379328) 
2022-04-12 08:35:59.453059: 
epoch:  434 
2022-04-12 08:37:49.180400: train loss : -0.7170 
2022-04-12 08:38:00.710415: validation loss: -0.7182 
2022-04-12 08:38:00.842255: Average global foreground Dice: [0.9206, 0.9762, 0.9846, 0.3961] 
2022-04-12 08:38:00.848878: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:38:01.523315: lr: 0.001594 
2022-04-12 08:38:01.781064: saving scheduled checkpoint file... 
2022-04-12 08:38:02.070259: saving checkpoint... 
2022-04-12 08:38:13.958923: done, saving took 12.03 seconds 
2022-04-12 08:38:14.086341: done 
2022-04-12 08:38:14.117135: This epoch took 134.594218 s
 
2022-04-12 08:38:14.148583: svmem(total=99932160000, available=79785295872, percent=20.2, used=16126119936, free=47287521280, active=40377171968, inactive=10401722368, buffers=41127936, cached=36477390848, shared=2190729216, slab=785657856) 
2022-04-12 08:38:14.201936: 
epoch:  435 
2022-04-12 08:40:05.348001: train loss : -0.7305 
2022-04-12 08:40:18.738892: validation loss: -0.6950 
2022-04-12 08:40:18.842933: Average global foreground Dice: [0.9183, 0.9755, 0.9803, 0.403] 
2022-04-12 08:40:18.906148: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:40:20.217351: lr: 0.001572 
2022-04-12 08:40:20.278063: saving scheduled checkpoint file... 
2022-04-12 08:40:20.555077: saving checkpoint... 
2022-04-12 08:40:43.260463: done, saving took 22.83 seconds 
2022-04-12 08:40:43.651760: done 
2022-04-12 08:40:43.765507: This epoch took 149.475096 s
 
2022-04-12 08:40:43.899114: svmem(total=99932160000, available=79809425408, percent=20.1, used=16102014976, free=47216091136, active=40391278592, inactive=10465906688, buffers=41168896, cached=36572884992, shared=2190729216, slab=785936384) 
2022-04-12 08:40:43.991203: 
epoch:  436 
2022-04-12 08:42:34.904209: train loss : -0.7180 
2022-04-12 08:42:48.359766: validation loss: -0.7016 
2022-04-12 08:42:48.525981: Average global foreground Dice: [0.924, 0.9756, 0.9844, 0.3966] 
2022-04-12 08:42:48.617746: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:42:49.627931: lr: 0.00155 
2022-04-12 08:42:49.802006: saving scheduled checkpoint file... 
2022-04-12 08:42:49.877996: saving checkpoint... 
2022-04-12 08:43:06.088378: done, saving took 16.28 seconds 
2022-04-12 08:43:06.318899: done 
2022-04-12 08:43:06.396749: This epoch took 142.294184 s
 
2022-04-12 08:43:06.409192: svmem(total=99932160000, available=79824609280, percent=20.1, used=16086306816, free=47241469952, active=40261038080, inactive=10566787072, buffers=41201664, cached=36563181568, shared=2191216640, slab=785653760) 
2022-04-12 08:43:06.561267: 
epoch:  437 
2022-04-12 08:44:56.397395: train loss : -0.7351 
2022-04-12 08:45:11.315459: validation loss: -0.7291 
2022-04-12 08:45:11.380456: Average global foreground Dice: [0.9145, 0.9776, 0.9835, 0.4377] 
2022-04-12 08:45:11.472664: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:45:12.493581: lr: 0.001528 
2022-04-12 08:45:12.573452: saving scheduled checkpoint file... 
2022-04-12 08:45:12.704440: saving checkpoint... 
2022-04-12 08:45:27.206481: done, saving took 14.57 seconds 
2022-04-12 08:45:27.665036: done 
2022-04-12 08:45:27.831282: This epoch took 141.196806 s
 
2022-04-12 08:45:27.999176: svmem(total=99932160000, available=79788650496, percent=20.2, used=16122400768, free=47160995840, active=40365621248, inactive=10537971712, buffers=41263104, cached=36607500288, shared=2190917632, slab=787091456) 
2022-04-12 08:45:28.056284: 
epoch:  438 
2022-04-12 08:47:17.903276: train loss : -0.7225 
2022-04-12 08:47:31.272352: validation loss: -0.7366 
2022-04-12 08:47:31.372445: Average global foreground Dice: [0.9291, 0.975, 0.9861, 0.4575] 
2022-04-12 08:47:31.492760: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:47:34.443421: lr: 0.001506 
2022-04-12 08:47:37.133380: saving scheduled checkpoint file... 
2022-04-12 08:47:37.763063: saving checkpoint... 
2022-04-12 08:47:50.544460: done, saving took 12.87 seconds 
2022-04-12 08:47:50.712507: done 
2022-04-12 08:47:50.842699: saving checkpoint... 
2022-04-12 08:48:01.931312: done, saving took 11.17 seconds 
2022-04-12 08:48:02.269433: This epoch took 154.094206 s
 
2022-04-12 08:48:02.288901: svmem(total=99932160000, available=79776612352, percent=20.2, used=16127995904, free=47129313280, active=40422551552, inactive=10523635712, buffers=41295872, cached=36633554944, shared=2197401600, slab=784723968) 
2022-04-12 08:48:02.454831: 
epoch:  439 
2022-04-12 08:49:52.115458: train loss : -0.7261 
2022-04-12 08:50:04.894996: validation loss: -0.7065 
2022-04-12 08:50:04.927390: Average global foreground Dice: [0.921, 0.976, 0.9859, 0.4212] 
2022-04-12 08:50:04.974765: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:50:05.939930: lr: 0.001483 
2022-04-12 08:50:05.985157: saving scheduled checkpoint file... 
2022-04-12 08:50:06.138295: saving checkpoint... 
2022-04-12 08:50:16.147563: done, saving took 10.13 seconds 
2022-04-12 08:50:16.286339: done 
2022-04-12 08:50:16.453538: saving checkpoint... 
2022-04-12 08:50:31.452860: done, saving took 15.10 seconds 
2022-04-12 08:50:31.676712: This epoch took 149.203326 s
 
2022-04-12 08:50:31.763828: svmem(total=99932160000, available=79459278848, percent=20.5, used=16451985408, free=46811193344, active=40736260096, inactive=10524340224, buffers=41353216, cached=36627628032, shared=2190729216, slab=784900096) 
2022-04-12 08:50:31.816979: 
epoch:  440 
2022-04-12 08:52:21.231637: train loss : -0.7308 
2022-04-12 08:52:32.899650: validation loss: -0.7230 
2022-04-12 08:52:32.979017: Average global foreground Dice: [0.9086, 0.9771, 0.9855, 0.4386] 
2022-04-12 08:52:33.119118: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:52:36.851735: lr: 0.001461 
2022-04-12 08:52:37.293848: saving scheduled checkpoint file... 
2022-04-12 08:52:37.587595: saving checkpoint... 
2022-04-12 08:52:58.954008: done, saving took 21.47 seconds 
2022-04-12 08:52:59.197353: done 
2022-04-12 08:52:59.382192: saving checkpoint... 
2022-04-12 08:53:14.359329: done, saving took 15.10 seconds 
2022-04-12 08:53:14.649692: This epoch took 162.826914 s
 
2022-04-12 08:53:14.711673: svmem(total=99932160000, available=79732121600, percent=20.2, used=16179290112, free=47057317888, active=40476930048, inactive=10534744064, buffers=41402368, cached=36654149632, shared=2190618624, slab=785776640) 
2022-04-12 08:53:14.758124: 
epoch:  441 
2022-04-12 08:55:06.073148: train loss : -0.7322 
2022-04-12 08:55:18.594730: validation loss: -0.6983 
2022-04-12 08:55:18.673069: Average global foreground Dice: [0.926, 0.9715, 0.9828, 0.4009] 
2022-04-12 08:55:18.706903: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:55:19.687307: lr: 0.001439 
2022-04-12 08:55:19.740042: saving scheduled checkpoint file... 
2022-04-12 08:55:19.929101: saving checkpoint... 
2022-04-12 08:55:33.454098: done, saving took 13.66 seconds 
2022-04-12 08:55:33.776157: done 
2022-04-12 08:55:33.822430: This epoch took 139.023663 s
 
2022-04-12 08:55:33.884362: svmem(total=99932160000, available=79517769728, percent=20.4, used=16383012864, free=46825086976, active=40586256384, inactive=10656968704, buffers=41451520, cached=36682608640, shared=2201137152, slab=785797120) 
2022-04-12 08:55:33.940189: 
epoch:  442 
2022-04-12 08:57:25.215070: train loss : -0.7234 
2022-04-12 08:57:37.575325: validation loss: -0.7221 
2022-04-12 08:57:38.345497: Average global foreground Dice: [0.9274, 0.9784, 0.9878, 0.4398] 
2022-04-12 08:57:38.463301: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 08:57:39.220082: lr: 0.001416 
2022-04-12 08:57:39.269278: saving scheduled checkpoint file... 
2022-04-12 08:57:39.368357: saving checkpoint... 
2022-04-12 08:57:53.814310: done, saving took 14.53 seconds 
2022-04-12 08:57:54.015060: done 
2022-04-12 08:57:54.144648: saving checkpoint... 
2022-04-12 08:58:12.372414: done, saving took 18.29 seconds 
2022-04-12 08:58:12.592772: This epoch took 158.623029 s
 
2022-04-12 08:58:12.643957: svmem(total=99932160000, available=80024133632, percent=19.9, used=15887466496, free=47312879616, active=40144773120, inactive=10607132672, buffers=41500672, cached=36690313216, shared=2190319616, slab=785428480) 
2022-04-12 08:58:12.704285: 
epoch:  443 
2022-04-12 09:00:01.799353: train loss : -0.7312 
2022-04-12 09:00:13.110830: validation loss: -0.7220 
2022-04-12 09:00:13.253443: Average global foreground Dice: [0.9231, 0.9764, 0.9869, 0.4314] 
2022-04-12 09:00:13.593290: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:00:14.687634: lr: 0.001394 
2022-04-12 09:00:14.708144: saving scheduled checkpoint file... 
2022-04-12 09:00:14.855519: saving checkpoint... 
2022-04-12 09:00:27.283871: done, saving took 12.52 seconds 
2022-04-12 09:00:27.538295: done 
2022-04-12 09:00:27.688894: saving checkpoint... 
2022-04-12 09:00:51.169933: done, saving took 23.56 seconds 
2022-04-12 09:00:51.544140: This epoch took 158.733140 s
 
2022-04-12 09:00:51.646574: svmem(total=99932160000, available=79651508224, percent=20.3, used=16259579904, free=46920568832, active=40527798272, inactive=10621661184, buffers=41541632, cached=36710469632, shared=2190729216, slab=786309120) 
2022-04-12 09:00:51.815172: 
epoch:  444 
2022-04-12 09:02:40.557527: train loss : -0.7365 
2022-04-12 09:02:55.412213: validation loss: -0.7023 
2022-04-12 09:02:55.551689: Average global foreground Dice: [0.9188, 0.9705, 0.9807, 0.4282] 
2022-04-12 09:02:55.665726: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:02:56.413017: lr: 0.001372 
2022-04-12 09:02:56.483181: saving scheduled checkpoint file... 
2022-04-12 09:02:56.631502: saving checkpoint... 
2022-04-12 09:03:15.828380: done, saving took 19.30 seconds 
2022-04-12 09:03:16.078840: done 
2022-04-12 09:03:16.155561: This epoch took 144.271928 s
 
2022-04-12 09:03:16.223229: svmem(total=99932160000, available=79738851328, percent=20.2, used=16172113920, free=46989418496, active=40462712832, inactive=10614345728, buffers=41590784, cached=36729036800, shared=2190729216, slab=786362368) 
2022-04-12 09:03:16.298915: 
epoch:  445 
2022-04-12 09:05:05.851076: train loss : -0.7354 
2022-04-12 09:05:17.949600: validation loss: -0.7284 
2022-04-12 09:05:18.195395: Average global foreground Dice: [0.9228, 0.9747, 0.985, 0.4442] 
2022-04-12 09:05:18.267761: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:05:19.280904: lr: 0.001349 
2022-04-12 09:05:19.396378: saving scheduled checkpoint file... 
2022-04-12 09:05:19.646578: saving checkpoint... 
2022-04-12 09:05:37.992149: done, saving took 18.48 seconds 
2022-04-12 09:05:38.198118: done 
2022-04-12 09:05:38.411162: saving checkpoint... 
2022-04-12 09:05:56.624728: done, saving took 18.34 seconds 
2022-04-12 09:05:56.985541: This epoch took 160.509261 s
 
2022-04-12 09:05:57.094923: svmem(total=99932160000, available=80041873408, percent=19.9, used=15883108352, free=47283458048, active=40185409536, inactive=10604335104, buffers=41652224, cached=36723941376, shared=2176741376, slab=786317312) 
2022-04-12 09:05:57.214345: 
epoch:  446 
2022-04-12 09:07:46.545322: train loss : -0.7351 
2022-04-12 09:07:58.683018: validation loss: -0.7115 
2022-04-12 09:07:58.758997: Average global foreground Dice: [0.9242, 0.9802, 0.9868, 0.4277] 
2022-04-12 09:07:58.781957: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:07:59.726386: lr: 0.001327 
2022-04-12 09:07:59.774715: saving scheduled checkpoint file... 
2022-04-12 09:07:59.961707: saving checkpoint... 
2022-04-12 09:08:14.565250: done, saving took 14.74 seconds 
2022-04-12 09:08:14.831315: done 
2022-04-12 09:08:15.065758: saving checkpoint... 
2022-04-12 09:08:36.013093: done, saving took 21.08 seconds 
2022-04-12 09:08:38.398408: This epoch took 161.159884 s
 
2022-04-12 09:08:39.150815: svmem(total=99932160000, available=79896588288, percent=20.0, used=16028229632, free=47118508032, active=40313139200, inactive=10623803392, buffers=41713664, cached=36743708672, shared=2176741376, slab=801206272) 
2022-04-12 09:08:39.728831: 
epoch:  447 
2022-04-12 09:10:31.385998: train loss : -0.7334 
2022-04-12 09:10:43.666348: validation loss: -0.7220 
2022-04-12 09:10:43.732225: Average global foreground Dice: [0.9334, 0.9755, 0.9857, 0.4145] 
2022-04-12 09:10:43.873421: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:10:44.774927: lr: 0.001304 
2022-04-12 09:10:44.869123: saving scheduled checkpoint file... 
2022-04-12 09:10:45.058776: saving checkpoint... 
2022-04-12 09:11:11.096313: done, saving took 26.17 seconds 
2022-04-12 09:11:11.693235: done 
2022-04-12 09:11:11.914901: saving checkpoint... 
2022-04-12 09:11:27.262384: done, saving took 15.50 seconds 
2022-04-12 09:11:27.391497: This epoch took 166.923072 s
 
2022-04-12 09:11:27.469185: svmem(total=99932160000, available=79631687680, percent=20.3, used=16279212032, free=46872788992, active=40553889792, inactive=10633814016, buffers=41771008, cached=36738387968, shared=2190729216, slab=788398080) 
2022-04-12 09:11:27.501370: 
epoch:  448 
2022-04-12 09:13:18.217238: train loss : -0.7342 
2022-04-12 09:13:32.132518: validation loss: -0.7068 
2022-04-12 09:13:32.269537: Average global foreground Dice: [0.9197, 0.9761, 0.984, 0.415] 
2022-04-12 09:13:32.372082: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:13:33.614706: lr: 0.001282 
2022-04-12 09:13:33.704182: saving scheduled checkpoint file... 
2022-04-12 09:13:33.804790: saving checkpoint... 
2022-04-12 09:13:47.801159: done, saving took 14.09 seconds 
2022-04-12 09:13:47.905807: done 
2022-04-12 09:13:47.954916: This epoch took 140.422022 s
 
2022-04-12 09:13:48.006594: svmem(total=99932160000, available=79712550912, percent=20.2, used=16198459392, free=46935150592, active=40482996224, inactive=10652004352, buffers=41803776, cached=36756746240, shared=2190618624, slab=788525056) 
2022-04-12 09:13:48.059290: 
epoch:  449 
2022-04-12 09:15:37.575260: train loss : -0.7255 
2022-04-12 09:15:50.207664: validation loss: -0.7213 
2022-04-12 09:15:51.005162: Average global foreground Dice: [0.9324, 0.976, 0.9859, 0.4209] 
2022-04-12 09:15:51.234606: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:15:52.179838: lr: 0.001259 
2022-04-12 09:15:52.232402: saving scheduled checkpoint file... 
2022-04-12 09:15:52.326404: saving checkpoint... 
2022-04-12 09:16:03.714924: done, saving took 11.47 seconds 
2022-04-12 09:16:03.872392: done 
2022-04-12 09:16:03.893492: This epoch took 135.777098 s
 
2022-04-12 09:16:03.970937: svmem(total=99932160000, available=79837560832, percent=20.1, used=16073367552, free=47042646016, active=40358133760, inactive=10669498368, buffers=41852928, cached=36774293504, shared=2190729216, slab=787484672) 
2022-04-12 09:16:04.021066: 
epoch:  450 
2022-04-12 09:17:52.394910: train loss : -0.7286 
2022-04-12 09:18:04.716941: validation loss: -0.7416 
2022-04-12 09:18:04.879293: Average global foreground Dice: [0.9323, 0.9756, 0.9864, 0.4452] 
2022-04-12 09:18:04.915118: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:18:05.962701: lr: 0.001236 
2022-04-12 09:18:06.011942: saving scheduled checkpoint file... 
2022-04-12 09:18:06.173151: saving checkpoint... 
2022-04-12 09:18:19.992833: done, saving took 13.92 seconds 
2022-04-12 09:18:20.187701: done 
2022-04-12 09:18:20.293704: saving checkpoint... 
2022-04-12 09:18:33.322463: done, saving took 13.11 seconds 
2022-04-12 09:18:33.600779: This epoch took 149.556667 s
 
2022-04-12 09:18:33.636938: svmem(total=99932160000, available=79668264960, percent=20.3, used=16242659328, free=46854811648, active=40538939392, inactive=10672439296, buffers=41906176, cached=36792782848, shared=2190696448, slab=786608128) 
2022-04-12 09:18:33.725807: 
epoch:  451 
2022-04-12 09:20:22.992622: train loss : -0.7332 
2022-04-12 09:20:34.746853: validation loss: -0.7257 
2022-04-12 09:20:34.842313: Average global foreground Dice: [0.922, 0.976, 0.986, 0.4449] 
2022-04-12 09:20:34.889413: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:20:36.052565: lr: 0.001214 
2022-04-12 09:20:36.099339: saving scheduled checkpoint file... 
2022-04-12 09:20:36.313303: saving checkpoint... 
2022-04-12 09:20:49.333601: done, saving took 13.13 seconds 
2022-04-12 09:20:49.446897: done 
2022-04-12 09:20:49.624913: saving checkpoint... 
2022-04-12 09:21:04.420463: done, saving took 14.89 seconds 
2022-04-12 09:21:04.668387: This epoch took 150.879939 s
 
2022-04-12 09:21:04.777025: svmem(total=99932160000, available=79574364160, percent=20.4, used=16350580736, free=46751936512, active=40644902912, inactive=10667388928, buffers=41947136, cached=36787695616, shared=2176552960, slab=788553728) 
2022-04-12 09:21:04.842293: 
epoch:  452 
2022-04-12 09:22:54.003200: train loss : -0.7300 
2022-04-12 09:23:05.630527: validation loss: -0.6968 
2022-04-12 09:23:06.094449: Average global foreground Dice: [0.9058, 0.9736, 0.9839, 0.3932] 
2022-04-12 09:23:06.141039: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:23:07.114258: lr: 0.001191 
2022-04-12 09:23:07.192356: saving scheduled checkpoint file... 
2022-04-12 09:23:07.441431: saving checkpoint... 
2022-04-12 09:23:17.482256: done, saving took 10.17 seconds 
2022-04-12 09:23:17.611187: done 
2022-04-12 09:23:17.638098: This epoch took 132.741033 s
 
2022-04-12 09:23:17.695719: svmem(total=99932160000, available=79627431936, percent=20.3, used=16282800128, free=46787465216, active=40578162688, inactive=10699632640, buffers=42008576, cached=36819886080, shared=2191335424, slab=786984960) 
2022-04-12 09:23:17.714644: 
epoch:  453 
2022-04-12 09:25:08.178889: train loss : -0.7367 
2022-04-12 09:25:21.217791: validation loss: -0.7251 
2022-04-12 09:25:21.339268: Average global foreground Dice: [0.9146, 0.9775, 0.9852, 0.4305] 
2022-04-12 09:25:21.408844: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:25:22.209101: lr: 0.001168 
2022-04-12 09:25:22.380215: saving scheduled checkpoint file... 
2022-04-12 09:25:22.565871: saving checkpoint... 
2022-04-12 09:25:37.495189: done, saving took 15.05 seconds 
2022-04-12 09:25:37.705526: done 
2022-04-12 09:25:37.777336: This epoch took 140.027073 s
 
2022-04-12 09:25:37.828113: svmem(total=99932160000, available=79568601088, percent=20.4, used=16342003712, free=46710120448, active=40641458176, inactive=10717659136, buffers=42049536, cached=36837986304, shared=2190917632, slab=786669568) 
2022-04-12 09:25:37.880531: 
epoch:  454 
2022-04-12 09:27:29.028146: train loss : -0.7359 
2022-04-12 09:27:41.983770: validation loss: -0.7088 
2022-04-12 09:27:42.212986: Average global foreground Dice: [0.9177, 0.9772, 0.987, 0.423] 
2022-04-12 09:27:42.306504: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:27:43.283914: lr: 0.001145 
2022-04-12 09:27:43.362182: saving scheduled checkpoint file... 
2022-04-12 09:27:43.497290: saving checkpoint... 
2022-04-12 09:27:52.390049: done, saving took 8.98 seconds 
2022-04-12 09:27:52.680446: done 
2022-04-12 09:27:52.848362: This epoch took 134.897775 s
 
2022-04-12 09:27:52.918814: svmem(total=99932160000, available=79686176768, percent=20.3, used=16226459648, free=46810263552, active=40527216640, inactive=10732462080, buffers=42090496, cached=36853346304, shared=2188746752, slab=787087360) 
2022-04-12 09:27:52.982412: 
epoch:  455 
2022-04-12 09:29:41.774282: train loss : -0.7268 
2022-04-12 09:29:53.923248: validation loss: -0.7192 
2022-04-12 09:29:54.003424: Average global foreground Dice: [0.928, 0.9744, 0.9844, 0.4082] 
2022-04-12 09:29:54.060835: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:29:55.070008: lr: 0.001122 
2022-04-12 09:29:55.169464: saving scheduled checkpoint file... 
2022-04-12 09:29:55.353308: saving checkpoint... 
2022-04-12 09:30:04.703853: done, saving took 9.43 seconds 
2022-04-12 09:30:05.609240: done 
2022-04-12 09:30:05.629295: This epoch took 132.587015 s
 
2022-04-12 09:30:05.670510: svmem(total=99932160000, available=79714197504, percent=20.2, used=16210513920, free=46949191680, active=40306499584, inactive=10807226368, buffers=42131456, cached=36730322944, shared=2176851968, slab=788103168) 
2022-04-12 09:30:05.679702: 
epoch:  456 
2022-04-12 09:31:53.897284: train loss : -0.7327 
2022-04-12 09:32:05.356492: validation loss: -0.7244 
2022-04-12 09:32:07.996505: Average global foreground Dice: [0.9239, 0.9744, 0.984, 0.4259] 
2022-04-12 09:32:09.306310: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:32:11.548108: lr: 0.001099 
2022-04-12 09:32:11.780284: saving scheduled checkpoint file... 
2022-04-12 09:32:12.015701: saving checkpoint... 
2022-04-12 09:32:22.352856: done, saving took 10.39 seconds 
2022-04-12 09:32:22.524748: done 
2022-04-12 09:32:22.551022: This epoch took 136.829166 s
 
2022-04-12 09:32:22.572732: svmem(total=99932160000, available=79634747392, percent=20.3, used=16265502720, free=46851260416, active=40482377728, inactive=10733084672, buffers=42180608, cached=36773216256, shared=2201137152, slab=786935808) 
2022-04-12 09:32:22.614967: 
epoch:  457 
2022-04-12 09:34:12.265741: train loss : -0.7278 
2022-04-12 09:34:27.109507: validation loss: -0.7173 
2022-04-12 09:34:27.728805: Average global foreground Dice: [0.9195, 0.9752, 0.9848, 0.3958] 
2022-04-12 09:34:28.289705: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:34:29.429802: lr: 0.001076 
2022-04-12 09:34:29.451238: saving scheduled checkpoint file... 
2022-04-12 09:34:29.567899: saving checkpoint... 
2022-04-12 09:34:56.319662: done, saving took 26.84 seconds 
2022-04-12 09:34:56.483558: done 
2022-04-12 09:34:56.650631: This epoch took 154.032539 s
 
2022-04-12 09:34:56.778890: svmem(total=99932160000, available=79757881344, percent=20.2, used=16166768640, free=46955786240, active=40414130176, inactive=10700423168, buffers=42213376, cached=36767391744, shared=2176741376, slab=789680128) 
2022-04-12 09:34:56.904866: 
epoch:  458 
2022-04-12 09:36:46.308577: train loss : -0.7395 
2022-04-12 09:36:59.025627: validation loss: -0.7132 
2022-04-12 09:36:59.439737: Average global foreground Dice: [0.9311, 0.9795, 0.986, 0.411] 
2022-04-12 09:36:59.716542: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:37:01.334700: lr: 0.001053 
2022-04-12 09:37:01.633884: saving scheduled checkpoint file... 
2022-04-12 09:37:01.978198: saving checkpoint... 
2022-04-12 09:37:15.417780: done, saving took 13.53 seconds 
2022-04-12 09:37:15.595533: done 
2022-04-12 09:37:15.657679: This epoch took 138.674221 s
 
2022-04-12 09:37:15.716603: svmem(total=99932160000, available=79640707072, percent=20.3, used=16270090240, free=46821138432, active=40554799104, inactive=10687094784, buffers=42295296, cached=36798636032, shared=2190618624, slab=789262336) 
2022-04-12 09:37:15.775241: 
epoch:  459 
2022-04-12 09:39:04.523401: train loss : -0.7444 
2022-04-12 09:39:16.001498: validation loss: -0.7302 
2022-04-12 09:39:16.846261: Average global foreground Dice: [0.9346, 0.9772, 0.9856, 0.4207] 
2022-04-12 09:39:17.245703: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:39:18.583943: lr: 0.00103 
2022-04-12 09:39:18.941386: saving scheduled checkpoint file... 
2022-04-12 09:39:19.329171: saving checkpoint... 
2022-04-12 09:39:32.701953: done, saving took 13.47 seconds 
2022-04-12 09:39:32.803270: done 
2022-04-12 09:39:32.849122: This epoch took 137.052274 s
 
2022-04-12 09:39:32.865457: svmem(total=99932160000, available=79696003072, percent=20.2, used=16214581248, free=46898925568, active=40335032320, inactive=10833723392, buffers=42328064, cached=36776325120, shared=2190729216, slab=787906560) 
2022-04-12 09:39:32.907068: 
epoch:  460 
2022-04-12 09:41:23.735556: train loss : -0.7305 
2022-04-12 09:41:35.430166: validation loss: -0.7260 
2022-04-12 09:41:37.007400: Average global foreground Dice: [0.9231, 0.9747, 0.9879, 0.4331] 
2022-04-12 09:41:38.366696: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:41:42.174878: lr: 0.001007 
2022-04-12 09:41:42.997514: saving scheduled checkpoint file... 
2022-04-12 09:41:43.671504: saving checkpoint... 
2022-04-12 09:42:07.762281: done, saving took 24.20 seconds 
2022-04-12 09:42:10.149456: done 
2022-04-12 09:42:11.274628: This epoch took 158.344586 s
 
2022-04-12 09:42:12.184485: svmem(total=99932160000, available=79809351680, percent=20.1, used=16115179520, free=46963838976, active=40324558848, inactive=10758299648, buffers=42385408, cached=36810756096, shared=2176741376, slab=802299904) 
2022-04-12 09:42:12.742571: 
epoch:  461 
2022-04-12 09:44:02.232686: train loss : -0.7324 
2022-04-12 09:44:14.035773: validation loss: -0.7217 
2022-04-12 09:44:14.088169: Average global foreground Dice: [0.9374, 0.9778, 0.9855, 0.4199] 
2022-04-12 09:44:14.094532: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:44:15.054928: lr: 0.000983 
2022-04-12 09:44:15.140667: saving scheduled checkpoint file... 
2022-04-12 09:44:15.310883: saving checkpoint... 
2022-04-12 09:44:29.622128: done, saving took 14.41 seconds 
2022-04-12 09:44:30.513158: done 
2022-04-12 09:44:30.848180: This epoch took 137.617138 s
 
2022-04-12 09:44:32.599114: svmem(total=99932160000, available=79949836288, percent=20.0, used=15974707200, free=47090192384, active=40161800192, inactive=10798972928, buffers=42418176, cached=36824842240, shared=2176741376, slab=801607680) 
2022-04-12 09:44:34.258657: 
epoch:  462 
2022-04-12 09:46:22.755065: train loss : -0.7418 
2022-04-12 09:46:34.797537: validation loss: -0.7307 
2022-04-12 09:46:34.933860: Average global foreground Dice: [0.9331, 0.9745, 0.9876, 0.4424] 
2022-04-12 09:46:35.052599: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:46:36.156095: lr: 0.00096 
2022-04-12 09:46:36.243617: saving scheduled checkpoint file... 
2022-04-12 09:46:36.449352: saving checkpoint... 
2022-04-12 09:46:54.880199: done, saving took 18.54 seconds 
2022-04-12 09:46:57.270893: done 
2022-04-12 09:46:58.120988: This epoch took 143.206356 s
 
2022-04-12 09:46:58.919029: svmem(total=99932160000, available=79825879040, percent=20.1, used=16098709504, free=46946172928, active=40333922304, inactive=10771107840, buffers=42463232, cached=36844814336, shared=2176741376, slab=802074624) 
2022-04-12 09:46:59.537810: 
epoch:  463 
2022-04-12 09:48:50.472417: train loss : -0.7383 
2022-04-12 09:49:03.822063: validation loss: -0.7298 
2022-04-12 09:49:04.349028: Average global foreground Dice: [0.9282, 0.977, 0.9875, 0.412] 
2022-04-12 09:49:04.771440: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:49:05.998650: lr: 0.000937 
2022-04-12 09:49:06.423942: saving scheduled checkpoint file... 
2022-04-12 09:49:07.210678: saving checkpoint... 
2022-04-12 09:49:26.550778: done, saving took 19.45 seconds 
2022-04-12 09:49:29.288453: done 
2022-04-12 09:49:29.975609: This epoch took 150.062815 s
 
2022-04-12 09:49:30.514710: svmem(total=99932160000, available=79847378944, percent=20.1, used=16077090816, free=46946361344, active=40333172736, inactive=10765217792, buffers=42512384, cached=36866195456, shared=2176741376, slab=802213888) 
2022-04-12 09:49:30.942562: 
epoch:  464 
2022-04-12 09:51:20.954444: train loss : -0.7397 
2022-04-12 09:51:31.866781: validation loss: -0.7295 
2022-04-12 09:51:31.941176: Average global foreground Dice: [0.929, 0.9767, 0.986, 0.4112] 
2022-04-12 09:51:31.999617: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:51:33.053864: lr: 0.000913 
2022-04-12 09:51:33.103058: saving scheduled checkpoint file... 
2022-04-12 09:51:33.295212: saving checkpoint... 
2022-04-12 09:51:48.625341: done, saving took 15.45 seconds 
2022-04-12 09:51:49.525465: done 
2022-04-12 09:51:49.785614: This epoch took 138.374992 s
 
2022-04-12 09:51:50.069945: svmem(total=99932160000, available=79918477312, percent=20.0, used=16005931008, free=47009660928, active=40269680640, inactive=10770227200, buffers=42553344, cached=36874014720, shared=2176741376, slab=802488320) 
2022-04-12 09:51:50.340696: 
epoch:  465 
2022-04-12 09:53:41.261530: train loss : -0.7436 
2022-04-12 09:53:53.977232: validation loss: -0.7228 
2022-04-12 09:53:54.033378: Average global foreground Dice: [0.9185, 0.9722, 0.9858, 0.4254] 
2022-04-12 09:53:54.082665: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:53:54.701937: lr: 0.00089 
2022-04-12 09:53:54.763657: saving scheduled checkpoint file... 
2022-04-12 09:53:55.014000: saving checkpoint... 
2022-04-12 09:54:08.437397: done, saving took 13.55 seconds 
2022-04-12 09:54:08.593085: done 
2022-04-12 09:54:08.648132: This epoch took 138.092236 s
 
2022-04-12 09:54:08.712366: svmem(total=99932160000, available=80339316736, percent=19.6, used=15735214080, free=47411785728, active=40021032960, inactive=10631598080, buffers=42586112, cached=36742574080, shared=2026553344, slab=789479424) 
2022-04-12 09:54:08.752507: 
epoch:  466 
2022-04-12 09:55:59.973563: train loss : -0.7358 
2022-04-12 09:56:12.582546: validation loss: -0.7269 
2022-04-12 09:56:12.701807: Average global foreground Dice: [0.9248, 0.9745, 0.9853, 0.4224] 
2022-04-12 09:56:12.757354: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:56:13.690255: lr: 0.000866 
2022-04-12 09:56:13.727469: saving scheduled checkpoint file... 
2022-04-12 09:56:13.808392: saving checkpoint... 
2022-04-12 09:56:27.160334: done, saving took 13.42 seconds 
2022-04-12 09:56:27.360480: done 
2022-04-12 09:56:27.377429: This epoch took 138.598849 s
 
2022-04-12 09:56:27.519584: svmem(total=99932160000, available=79782518784, percent=20.2, used=16127721472, free=46837628928, active=40421441536, inactive=10810056704, buffers=42647552, cached=36924162048, shared=2190839808, slab=787857408) 
2022-04-12 09:56:27.601249: 
epoch:  467 
2022-04-12 09:58:17.947318: train loss : -0.7375 
2022-04-12 09:58:29.550118: validation loss: -0.7317 
2022-04-12 09:58:29.652505: Average global foreground Dice: [0.9382, 0.9772, 0.988, 0.4419] 
2022-04-12 09:58:29.788651: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 09:58:33.915864: lr: 0.000842 
2022-04-12 09:58:34.128765: saving scheduled checkpoint file... 
2022-04-12 09:58:34.295683: saving checkpoint... 
2022-04-12 09:58:44.700908: done, saving took 10.48 seconds 
2022-04-12 09:58:44.861190: done 
2022-04-12 09:58:44.914995: This epoch took 137.215907 s
 
2022-04-12 09:58:44.929241: svmem(total=99932160000, available=79747391488, percent=20.2, used=16152543232, free=46783877120, active=40448069632, inactive=10838880256, buffers=42680320, cached=36953059328, shared=2201137152, slab=787779584) 
2022-04-12 09:58:44.983211: 
epoch:  468 
2022-04-12 10:00:35.621172: train loss : -0.7375 
2022-04-12 10:00:49.468795: validation loss: -0.7371 
2022-04-12 10:00:49.581242: Average global foreground Dice: [0.9216, 0.9772, 0.9856, 0.4407] 
2022-04-12 10:00:49.792448: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:00:50.557414: lr: 0.000819 
2022-04-12 10:00:50.616212: saving scheduled checkpoint file... 
2022-04-12 10:00:50.800020: saving checkpoint... 
2022-04-12 10:01:01.916151: done, saving took 11.19 seconds 
2022-04-12 10:01:02.048120: done 
2022-04-12 10:01:02.164443: saving checkpoint... 
2022-04-12 10:01:12.858441: done, saving took 10.78 seconds 
2022-04-12 10:01:13.311243: This epoch took 148.232511 s
 
2022-04-12 10:01:13.472056: svmem(total=99932160000, available=79740440576, percent=20.2, used=16183697408, free=46611443712, active=40475815936, inactive=10980036608, buffers=42737664, cached=37094281216, shared=2176851968, slab=788623360) 
2022-04-12 10:01:13.581959: 
epoch:  469 
2022-04-12 10:03:03.181039: train loss : -0.7354 
2022-04-12 10:03:15.283365: validation loss: -0.7260 
2022-04-12 10:03:15.349442: Average global foreground Dice: [0.9183, 0.9762, 0.9871, 0.4465] 
2022-04-12 10:03:15.384541: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:03:16.239441: lr: 0.000795 
2022-04-12 10:03:16.303023: saving scheduled checkpoint file... 
2022-04-12 10:03:16.431928: saving checkpoint... 
2022-04-12 10:03:25.993778: done, saving took 9.69 seconds 
2022-04-12 10:03:26.115575: done 
2022-04-12 10:03:26.218688: saving checkpoint... 
2022-04-12 10:03:39.237538: done, saving took 13.12 seconds 
2022-04-12 10:03:39.513592: This epoch took 145.818669 s
 
2022-04-12 10:03:39.577087: svmem(total=99932160000, available=79811399680, percent=20.1, used=16098750464, free=46666158080, active=40366899200, inactive=11039830016, buffers=42770432, cached=37124481024, shared=2190729216, slab=789360640) 
2022-04-12 10:03:39.642070: 
epoch:  470 
2022-04-12 10:05:28.596290: train loss : -0.7354 
2022-04-12 10:05:39.665319: validation loss: -0.7064 
2022-04-12 10:05:40.281608: Average global foreground Dice: [0.9317, 0.9772, 0.9846, 0.3918] 
2022-04-12 10:05:40.349151: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:05:41.369781: lr: 0.000771 
2022-04-12 10:05:41.468252: saving scheduled checkpoint file... 
2022-04-12 10:05:41.682480: saving checkpoint... 
2022-04-12 10:05:51.969499: done, saving took 10.41 seconds 
2022-04-12 10:05:52.120764: done 
2022-04-12 10:05:52.128793: This epoch took 132.422318 s
 
2022-04-12 10:05:52.191247: svmem(total=99932160000, available=79754846208, percent=20.2, used=16155451392, free=46637096960, active=40099770368, inactive=11327823872, buffers=42819584, cached=37096792064, shared=2190807040, slab=789028864) 
2022-04-12 10:05:52.207466: 
epoch:  471 
2022-04-12 10:07:42.093796: train loss : -0.7436 
2022-04-12 10:07:55.459826: validation loss: -0.7180 
2022-04-12 10:07:55.660683: Average global foreground Dice: [0.9268, 0.976, 0.9869, 0.4216] 
2022-04-12 10:07:55.807785: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:07:57.687771: lr: 0.000747 
2022-04-12 10:07:57.747293: saving scheduled checkpoint file... 
2022-04-12 10:07:57.904356: saving checkpoint... 
2022-04-12 10:08:14.714296: done, saving took 16.94 seconds 
2022-04-12 10:08:14.860922: done 
2022-04-12 10:08:14.886496: This epoch took 142.675847 s
 
2022-04-12 10:08:15.011831: svmem(total=99932160000, available=79714426880, percent=20.2, used=16195641344, free=46538063872, active=40401637376, inactive=11125481472, buffers=42885120, cached=37155569664, shared=2190729216, slab=788635648) 
2022-04-12 10:08:15.030528: 
epoch:  472 
2022-04-12 10:10:06.346777: train loss : -0.7427 
2022-04-12 10:10:17.605034: validation loss: -0.7287 
2022-04-12 10:10:17.667892: Average global foreground Dice: [0.9282, 0.9766, 0.9849, 0.4339] 
2022-04-12 10:10:17.733310: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:10:18.788011: lr: 0.000723 
2022-04-12 10:10:18.816684: saving scheduled checkpoint file... 
2022-04-12 10:10:18.996810: saving checkpoint... 
2022-04-12 10:10:32.817100: done, saving took 13.90 seconds 
2022-04-12 10:10:32.962238: done 
2022-04-12 10:10:33.017313: This epoch took 137.922867 s
 
2022-04-12 10:10:33.090845: svmem(total=99932160000, available=79518244864, percent=20.4, used=16391786496, free=46434906112, active=40268075008, inactive=11366899712, buffers=42926080, cached=37062541312, shared=2190807040, slab=788615168) 
2022-04-12 10:10:33.143434: 
epoch:  473 
2022-04-12 10:12:21.594577: train loss : -0.7441 
2022-04-12 10:12:33.178473: validation loss: -0.7311 
2022-04-12 10:12:33.313520: Average global foreground Dice: [0.9269, 0.9742, 0.9856, 0.4059] 
2022-04-12 10:12:33.540263: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:12:34.261179: lr: 0.000699 
2022-04-12 10:12:34.361033: saving scheduled checkpoint file... 
2022-04-12 10:12:34.495827: saving checkpoint... 
2022-04-12 10:12:54.412525: done, saving took 19.99 seconds 
2022-04-12 10:12:54.567671: done 
2022-04-12 10:12:54.630384: This epoch took 141.411376 s
 
2022-04-12 10:12:54.691563: svmem(total=99932160000, available=79545413632, percent=20.4, used=16358117376, free=46444605440, active=40421765120, inactive=11198226432, buffers=42958848, cached=37086478336, shared=2197368832, slab=789123072) 
2022-04-12 10:12:54.775820: 
epoch:  474 
2022-04-12 10:14:42.761912: train loss : -0.7435 
2022-04-12 10:14:54.849347: validation loss: -0.7305 
2022-04-12 10:14:54.887216: Average global foreground Dice: [0.9271, 0.9764, 0.9869, 0.4295] 
2022-04-12 10:14:54.944758: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:14:56.028681: lr: 0.000675 
2022-04-12 10:14:56.043859: saving scheduled checkpoint file... 
2022-04-12 10:14:56.126545: saving checkpoint... 
2022-04-12 10:15:06.554050: done, saving took 10.47 seconds 
2022-04-12 10:15:06.685035: done 
2022-04-12 10:15:06.747495: This epoch took 131.959639 s
 
2022-04-12 10:15:06.829036: svmem(total=99932160000, available=79802990592, percent=20.1, used=16107466752, free=46684696576, active=40266293248, inactive=11111612416, buffers=42999808, cached=37096996864, shared=2190389248, slab=788328448) 
2022-04-12 10:15:06.908515: 
epoch:  475 
2022-04-12 10:16:55.543207: train loss : -0.7414 
2022-04-12 10:17:07.964127: validation loss: -0.7174 
2022-04-12 10:17:08.057578: Average global foreground Dice: [0.9244, 0.9755, 0.9847, 0.4084] 
2022-04-12 10:17:08.154675: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:17:09.023500: lr: 0.00065 
2022-04-12 10:17:09.144901: saving scheduled checkpoint file... 
2022-04-12 10:17:09.305729: saving checkpoint... 
2022-04-12 10:17:23.573256: done, saving took 14.35 seconds 
2022-04-12 10:17:23.763497: done 
2022-04-12 10:17:23.877287: This epoch took 136.958561 s
 
2022-04-12 10:17:23.917533: svmem(total=99932160000, available=79680552960, percent=20.3, used=16229429248, free=46543708160, active=40457662464, inactive=11063549952, buffers=43053056, cached=37115969536, shared=2190729216, slab=789237760) 
2022-04-12 10:17:23.962305: 
epoch:  476 
2022-04-12 10:19:13.747037: train loss : -0.7486 
2022-04-12 10:19:25.098494: validation loss: -0.7163 
2022-04-12 10:19:25.271867: Average global foreground Dice: [0.9332, 0.9773, 0.9858, 0.411] 
2022-04-12 10:19:25.321065: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:19:26.408422: lr: 0.000626 
2022-04-12 10:19:26.583218: saving scheduled checkpoint file... 
2022-04-12 10:19:26.768160: saving checkpoint... 
2022-04-12 10:19:38.497419: done, saving took 11.83 seconds 
2022-04-12 10:19:38.725813: done 
2022-04-12 10:19:38.783702: This epoch took 134.716669 s
 
2022-04-12 10:19:38.854498: svmem(total=99932160000, available=79611932672, percent=20.3, used=16301502464, free=46467235840, active=40553570304, inactive=11044749312, buffers=43094016, cached=37120327680, shared=2187259904, slab=789315584) 
2022-04-12 10:19:38.857361: 
epoch:  477 
2022-04-12 10:21:28.297563: train loss : -0.7461 
2022-04-12 10:21:40.106985: validation loss: -0.7279 
2022-04-12 10:21:40.163331: Average global foreground Dice: [0.9406, 0.9776, 0.9858, 0.4168] 
2022-04-12 10:21:40.235867: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:21:41.300575: lr: 0.000601 
2022-04-12 10:21:41.394875: saving scheduled checkpoint file... 
2022-04-12 10:21:41.579336: saving checkpoint... 
2022-04-12 10:22:02.258971: done, saving took 20.81 seconds 
2022-04-12 10:22:02.679883: done 
2022-04-12 10:22:02.822295: This epoch took 143.910058 s
 
2022-04-12 10:22:02.895002: svmem(total=99932160000, available=79491403776, percent=20.5, used=16418476032, free=46338232320, active=40638525440, inactive=11092226048, buffers=43143168, cached=37132308480, shared=2190729216, slab=789594112) 
2022-04-12 10:22:03.021448: 
epoch:  478 
2022-04-12 10:23:52.893013: train loss : -0.7433 
2022-04-12 10:24:07.089214: validation loss: -0.7233 
2022-04-12 10:24:07.181113: Average global foreground Dice: [0.9222, 0.9764, 0.9858, 0.416] 
2022-04-12 10:24:07.235627: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:24:08.575740: lr: 0.000577 
2022-04-12 10:24:08.637625: saving scheduled checkpoint file... 
2022-04-12 10:24:08.795877: saving checkpoint... 
2022-04-12 10:24:19.489637: done, saving took 10.78 seconds 
2022-04-12 10:24:19.683076: done 
2022-04-12 10:24:19.752031: This epoch took 136.663293 s
 
2022-04-12 10:24:19.811549: svmem(total=99932160000, available=79806181376, percent=20.1, used=16093048832, free=46635585536, active=40365490176, inactive=11065303040, buffers=43192320, cached=37160333312, shared=2201358336, slab=790421504) 
2022-04-12 10:24:19.825217: 
epoch:  479 
2022-04-12 10:26:09.011115: train loss : -0.7489 
2022-04-12 10:26:20.955873: validation loss: -0.7281 
2022-04-12 10:26:20.983416: Average global foreground Dice: [0.9243, 0.9786, 0.9853, 0.4124] 
2022-04-12 10:26:21.049814: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:26:21.971201: lr: 0.000552 
2022-04-12 10:26:21.981164: saving scheduled checkpoint file... 
2022-04-12 10:26:22.124341: saving checkpoint... 
2022-04-12 10:26:35.738428: done, saving took 13.72 seconds 
2022-04-12 10:26:35.889570: done 
2022-04-12 10:26:35.980313: This epoch took 136.123008 s
 
2022-04-12 10:26:35.982652: svmem(total=99932160000, available=79753863168, percent=20.2, used=16168464384, free=46639755264, active=40177004544, inactive=11252363264, buffers=43241472, cached=37080698880, shared=2178338816, slab=789368832) 
2022-04-12 10:26:36.033439: 
epoch:  480 
2022-04-12 10:28:24.988206: train loss : -0.7290 
2022-04-12 10:28:35.851696: validation loss: -0.7391 
2022-04-12 10:28:35.943103: Average global foreground Dice: [0.9276, 0.9746, 0.9855, 0.4432] 
2022-04-12 10:28:36.112813: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:28:37.268405: lr: 0.000527 
2022-04-12 10:28:37.385025: saving scheduled checkpoint file... 
2022-04-12 10:28:37.529161: saving checkpoint... 
2022-04-12 10:28:49.444562: done, saving took 12.01 seconds 
2022-04-12 10:28:49.805642: done 
2022-04-12 10:28:49.843924: This epoch took 133.805649 s
 
2022-04-12 10:28:49.932404: svmem(total=99932160000, available=79867764736, percent=20.1, used=16041938944, free=46736306176, active=40179388416, inactive=11153113088, buffers=43274240, cached=37110640640, shared=2190942208, slab=788934656) 
2022-04-12 10:28:49.998298: 
epoch:  481 
2022-04-12 10:30:41.538811: train loss : -0.7371 
2022-04-12 10:30:54.677882: validation loss: -0.7315 
2022-04-12 10:30:55.330936: Average global foreground Dice: [0.9388, 0.9775, 0.9866, 0.4296] 
2022-04-12 10:30:55.578973: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:30:56.654921: lr: 0.000502 
2022-04-12 10:30:56.706681: saving scheduled checkpoint file... 
2022-04-12 10:30:56.888211: saving checkpoint... 
2022-04-12 10:31:13.906090: done, saving took 17.14 seconds 
2022-04-12 10:31:14.154730: done 
2022-04-12 10:31:14.223040: This epoch took 144.174151 s
 
2022-04-12 10:31:14.338403: svmem(total=99932160000, available=79752425472, percent=20.2, used=16165986304, free=46602510336, active=40381165568, inactive=11082223616, buffers=43331584, cached=37120331776, shared=2182217728, slab=789016576) 
2022-04-12 10:31:14.341595: 
epoch:  482 
2022-04-12 10:33:05.771387: train loss : -0.7490 
2022-04-12 10:33:18.342022: validation loss: -0.7400 
2022-04-12 10:33:18.510086: Average global foreground Dice: [0.9361, 0.9765, 0.9874, 0.4361] 
2022-04-12 10:33:18.699469: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:33:19.760073: lr: 0.000477 
2022-04-12 10:33:19.818568: saving scheduled checkpoint file... 
2022-04-12 10:33:19.979548: saving checkpoint... 
2022-04-12 10:33:28.626035: done, saving took 8.77 seconds 
2022-04-12 10:33:28.964240: done 
2022-04-12 10:33:29.219285: saving checkpoint... 
2022-04-12 10:33:41.685130: done, saving took 12.60 seconds 
2022-04-12 10:33:42.089079: This epoch took 147.708322 s
 
2022-04-12 10:33:42.159201: svmem(total=99932160000, available=79810056192, percent=20.1, used=16099762176, free=46559756288, active=40093155328, inactive=11421327360, buffers=43364352, cached=37229277184, shared=2190917632, slab=788590592) 
2022-04-12 10:33:42.280940: 
epoch:  483 
2022-04-12 10:35:32.754763: train loss : -0.7408 
2022-04-12 10:35:45.479110: validation loss: -0.7504 
2022-04-12 10:35:45.652501: Average global foreground Dice: [0.9434, 0.9781, 0.9883, 0.4378] 
2022-04-12 10:35:45.923815: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:35:47.739876: lr: 0.000451 
2022-04-12 10:35:48.681170: saving scheduled checkpoint file... 
2022-04-12 10:35:49.708155: saving checkpoint... 
2022-04-12 10:36:11.097243: done, saving took 21.51 seconds 
2022-04-12 10:36:11.519332: done 
2022-04-12 10:36:11.733814: saving checkpoint... 
2022-04-12 10:36:27.855224: done, saving took 16.27 seconds 
2022-04-12 10:36:28.318120: This epoch took 165.981308 s
 
2022-04-12 10:36:28.380349: svmem(total=99932160000, available=79505367040, percent=20.4, used=16404037632, free=46137671680, active=40558469120, inactive=11370188800, buffers=43425792, cached=37347024896, shared=2190729216, slab=789073920) 
2022-04-12 10:36:28.439461: 
epoch:  484 
2022-04-12 10:38:19.256969: train loss : -0.7512 
2022-04-12 10:38:32.403644: validation loss: -0.7221 
2022-04-12 10:38:32.552268: Average global foreground Dice: [0.9324, 0.9766, 0.9856, 0.4037] 
2022-04-12 10:38:32.572438: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:38:33.644865: lr: 0.000426 
2022-04-12 10:38:33.733453: saving scheduled checkpoint file... 
2022-04-12 10:38:33.888534: saving checkpoint... 
2022-04-12 10:38:47.600361: done, saving took 13.80 seconds 
2022-04-12 10:38:47.967565: done 
2022-04-12 10:38:48.248490: This epoch took 139.691247 s
 
2022-04-12 10:38:48.357236: svmem(total=99932160000, available=79736033280, percent=20.2, used=16173580288, free=46362882048, active=40409837568, inactive=11296849920, buffers=43470848, cached=37352226816, shared=2190917632, slab=788832256) 
2022-04-12 10:38:48.471047: 
epoch:  485 
2022-04-12 10:40:37.210875: train loss : -0.7450 
2022-04-12 10:40:49.443675: validation loss: -0.7382 
2022-04-12 10:40:49.499646: Average global foreground Dice: [0.9355, 0.9759, 0.9858, 0.4589] 
2022-04-12 10:40:49.557786: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:40:50.588960: lr: 0.0004 
2022-04-12 10:40:50.709888: saving scheduled checkpoint file... 
2022-04-12 10:40:50.855801: saving checkpoint... 
2022-04-12 10:41:05.108387: done, saving took 14.34 seconds 
2022-04-12 10:41:06.238636: done 
2022-04-12 10:41:06.343225: saving checkpoint... 
2022-04-12 10:41:19.287564: done, saving took 13.02 seconds 
2022-04-12 10:41:19.476129: This epoch took 150.850238 s
 
2022-04-12 10:41:19.530745: svmem(total=99932160000, available=79483621376, percent=20.5, used=16426360832, free=46091501568, active=40705908736, inactive=11267022848, buffers=43528192, cached=37370769408, shared=2190598144, slab=789676032) 
2022-04-12 10:41:19.580804: 
epoch:  486 
2022-04-12 10:43:08.811584: train loss : -0.7480 
2022-04-12 10:43:21.269743: validation loss: -0.7233 
2022-04-12 10:43:21.553039: Average global foreground Dice: [0.9339, 0.9761, 0.9879, 0.4329] 
2022-04-12 10:43:21.631314: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:43:22.452842: lr: 0.000375 
2022-04-12 10:43:22.511315: saving scheduled checkpoint file... 
2022-04-12 10:43:22.710626: saving checkpoint... 
2022-04-12 10:43:38.198466: done, saving took 15.59 seconds 
2022-04-12 10:43:38.366160: done 
2022-04-12 10:43:38.492020: saving checkpoint... 
2022-04-12 10:43:50.413317: done, saving took 12.01 seconds 
2022-04-12 10:43:50.579878: This epoch took 150.956978 s
 
2022-04-12 10:43:50.668868: svmem(total=99932160000, available=79871488000, percent=20.1, used=16038440960, free=46460899328, active=40332152832, inactive=11272613888, buffers=43560960, cached=37389258752, shared=2190618624, slab=789344256) 
2022-04-12 10:43:50.758960: 
epoch:  487 
2022-04-12 10:45:40.823678: train loss : -0.7500 
2022-04-12 10:45:52.901905: validation loss: -0.7303 
2022-04-12 10:45:52.992280: Average global foreground Dice: [0.9342, 0.9786, 0.9871, 0.423] 
2022-04-12 10:45:53.083570: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:45:54.210286: lr: 0.000348 
2022-04-12 10:45:54.324905: saving scheduled checkpoint file... 
2022-04-12 10:45:54.486037: saving checkpoint... 
2022-04-12 10:46:26.271306: done, saving took 31.92 seconds 
2022-04-12 10:46:26.506813: done 
2022-04-12 10:46:26.711153: saving checkpoint... 
2022-04-12 10:46:41.777797: done, saving took 15.19 seconds 
2022-04-12 10:46:42.399246: This epoch took 171.430690 s
 
2022-04-12 10:46:42.498017: svmem(total=99932160000, available=79759245312, percent=20.2, used=16150528000, free=46328926208, active=40445804544, inactive=11289993216, buffers=43626496, cached=37409079296, shared=2190729216, slab=790712320) 
2022-04-12 10:46:42.540180: 
epoch:  488 
2022-04-12 10:48:32.742444: train loss : -0.7556 
2022-04-12 10:48:43.819398: validation loss: -0.7117 
2022-04-12 10:48:43.951055: Average global foreground Dice: [0.9237, 0.9782, 0.985, 0.4478] 
2022-04-12 10:48:44.069658: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:48:46.879444: lr: 0.000322 
2022-04-12 10:48:47.640789: saving scheduled checkpoint file... 
2022-04-12 10:48:48.346627: saving checkpoint... 
2022-04-12 10:50:14.397403: done, saving took 86.18 seconds 
2022-04-12 10:50:17.915895: done 
2022-04-12 10:50:18.142784: saving checkpoint... 
2022-04-12 10:53:31.993855: done, saving took 193.99 seconds 
2022-04-12 10:53:37.649462: This epoch took 415.060446 s
 
2022-04-12 10:53:37.696685: svmem(total=99932160000, available=79777062912, percent=20.2, used=16146751488, free=46314487808, active=40422301696, inactive=11319345152, buffers=43749376, cached=37427171328, shared=2176741376, slab=791089152) 
2022-04-12 10:53:37.803887: 
epoch:  489 
2022-04-12 10:55:28.278608: train loss : -0.7518 
2022-04-12 10:55:40.109466: validation loss: -0.7308 
2022-04-12 10:55:40.234464: Average global foreground Dice: [0.9314, 0.9788, 0.9871, 0.4441] 
2022-04-12 10:55:40.347791: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:55:41.922305: lr: 0.000296 
2022-04-12 10:55:43.042102: saving scheduled checkpoint file... 
2022-04-12 10:55:44.166051: saving checkpoint... 
2022-04-12 10:56:05.363934: done, saving took 21.34 seconds 
2022-04-12 10:56:07.635598: done 
2022-04-12 10:56:07.902005: saving checkpoint... 
2022-04-12 10:56:22.151201: done, saving took 14.40 seconds 
2022-04-12 10:56:22.317989: This epoch took 164.389865 s
 
2022-04-12 10:56:22.376417: svmem(total=99932160000, available=79702224896, percent=20.2, used=16210927616, free=46221070336, active=40500142080, inactive=11342639104, buffers=43810816, cached=37456351232, shared=2187149312, slab=790196224) 
2022-04-12 10:56:22.401522: 
epoch:  490 
2022-04-12 10:58:12.551810: train loss : -0.7461 
2022-04-12 10:58:26.978886: validation loss: -0.7504 
2022-04-12 10:58:27.401654: Average global foreground Dice: [0.9336, 0.9795, 0.9878, 0.4395] 
2022-04-12 10:58:27.523669: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 10:58:28.709452: lr: 0.000269 
2022-04-12 10:58:28.818364: saving scheduled checkpoint file... 
2022-04-12 10:58:28.963612: saving checkpoint... 
2022-04-12 10:58:43.424821: done, saving took 14.55 seconds 
2022-04-12 10:58:43.543790: done 
2022-04-12 10:58:43.669032: saving checkpoint... 
2022-04-12 10:58:55.626709: done, saving took 12.03 seconds 
2022-04-12 10:58:55.693362: This epoch took 153.258359 s
 
2022-04-12 10:58:55.756435: svmem(total=99932160000, available=79665811456, percent=20.3, used=16244244480, free=46164004864, active=40535498752, inactive=11361558528, buffers=43843584, cached=37480067072, shared=2190233600, slab=789966848) 
2022-04-12 10:58:55.767032: 
epoch:  491 
2022-04-12 11:00:44.819814: train loss : -0.7487 
2022-04-12 11:00:58.357890: validation loss: -0.7305 
2022-04-12 11:00:58.468752: Average global foreground Dice: [0.9458, 0.9783, 0.9886, 0.4066] 
2022-04-12 11:00:58.534861: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:00:59.551413: lr: 0.000242 
2022-04-12 11:00:59.595536: saving scheduled checkpoint file... 
2022-04-12 11:00:59.752351: saving checkpoint... 
2022-04-12 11:01:11.802645: done, saving took 12.14 seconds 
2022-04-12 11:01:12.305025: done 
2022-04-12 11:01:12.314831: This epoch took 136.518878 s
 
2022-04-12 11:01:12.321106: svmem(total=99932160000, available=79669477376, percent=20.3, used=16239714304, free=46150025216, active=40533475328, inactive=11379511296, buffers=43900928, cached=37498519552, shared=2190917632, slab=790827008) 
2022-04-12 11:01:12.380765: 
epoch:  492 
2022-04-12 11:03:02.982398: train loss : -0.7511 
2022-04-12 11:03:15.867574: validation loss: -0.7133 
2022-04-12 11:03:15.961490: Average global foreground Dice: [0.9205, 0.9776, 0.9857, 0.3957] 
2022-04-12 11:03:16.023875: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:03:17.036443: lr: 0.000215 
2022-04-12 11:03:17.097158: saving scheduled checkpoint file... 
2022-04-12 11:03:17.203279: saving checkpoint... 
2022-04-12 11:03:33.050550: done, saving took 15.95 seconds 
2022-04-12 11:03:33.183448: done 
2022-04-12 11:03:33.242411: This epoch took 140.837102 s
 
2022-04-12 11:03:33.316494: svmem(total=99932160000, available=79621447680, percent=20.3, used=16288448512, free=46093004800, active=40590499840, inactive=11387863040, buffers=43966464, cached=37506740224, shared=2190290944, slab=789442560) 
2022-04-12 11:03:33.388449: 
epoch:  493 
2022-04-12 11:05:23.623509: train loss : -0.7429 
2022-04-12 11:05:37.424306: validation loss: -0.7398 
2022-04-12 11:05:37.567203: Average global foreground Dice: [0.9458, 0.976, 0.9881, 0.4283] 
2022-04-12 11:05:37.617593: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:05:38.593274: lr: 0.000187 
2022-04-12 11:05:38.707292: saving scheduled checkpoint file... 
2022-04-12 11:05:38.887983: saving checkpoint... 
2022-04-12 11:05:54.147740: done, saving took 15.38 seconds 
2022-04-12 11:05:54.582239: done 
2022-04-12 11:05:54.658041: This epoch took 141.196310 s
 
2022-04-12 11:05:54.738536: svmem(total=99932160000, available=79478022144, percent=20.5, used=16458899456, free=45995290624, active=40261160960, inactive=11738468352, buffers=44023808, cached=37433946112, shared=2162810880, slab=795111424) 
2022-04-12 11:05:54.872697: 
epoch:  494 
2022-04-12 11:07:43.893434: train loss : -0.7399 
2022-04-12 11:08:07.164034: validation loss: -0.7445 
2022-04-12 11:08:07.755677: Average global foreground Dice: [0.9411, 0.9752, 0.9864, 0.4299] 
2022-04-12 11:08:08.154065: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:08:09.751603: lr: 0.000158 
2022-04-12 11:08:09.996245: saving scheduled checkpoint file... 
2022-04-12 11:08:10.558644: saving checkpoint... 
2022-04-12 11:08:32.598149: done, saving took 22.16 seconds 
2022-04-12 11:08:32.931199: done 
2022-04-12 11:08:33.051218: This epoch took 158.101364 s
 
2022-04-12 11:08:33.166809: svmem(total=99932160000, available=78518161408, percent=21.4, used=17418604544, free=46061916160, active=40228925440, inactive=11701747712, buffers=44093440, cached=36407545856, shared=2162810880, slab=796815360) 
2022-04-12 11:08:33.341197: 
epoch:  495 
2022-04-12 11:10:26.138322: train loss : -0.7493 
2022-04-12 11:10:44.655771: validation loss: -0.7384 
2022-04-12 11:10:44.748450: Average global foreground Dice: [0.9311, 0.9797, 0.9881, 0.4395] 
2022-04-12 11:10:44.864970: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:10:45.716798: lr: 0.00013 
2022-04-12 11:10:45.731440: saving scheduled checkpoint file... 
2022-04-12 11:10:45.911293: saving checkpoint... 
2022-04-12 11:11:08.947454: done, saving took 23.17 seconds 
2022-04-12 11:11:09.233706: done 
2022-04-12 11:11:09.391389: This epoch took 155.890541 s
 
2022-04-12 11:11:09.453069: svmem(total=99932160000, available=78332125184, percent=21.6, used=17604616192, free=46305476608, active=39589871616, inactive=12091469824, buffers=44126208, cached=35977940992, shared=2162810880, slab=798117888) 
2022-04-12 11:11:09.558538: 
epoch:  496 
2022-04-12 11:12:59.083831: train loss : -0.7470 
2022-04-12 11:13:19.300535: validation loss: -0.7043 
2022-04-12 11:13:19.525346: Average global foreground Dice: [0.9359, 0.9769, 0.9839, 0.3827] 
2022-04-12 11:13:19.684702: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:13:20.825734: lr: 0.0001 
2022-04-12 11:13:21.181089: saving scheduled checkpoint file... 
2022-04-12 11:13:22.013598: saving checkpoint... 
2022-04-12 11:13:40.133545: done, saving took 18.19 seconds 
2022-04-12 11:13:40.717962: done 
2022-04-12 11:13:40.880476: This epoch took 151.228867 s
 
2022-04-12 11:13:41.141849: svmem(total=99932160000, available=77968347136, percent=22.0, used=17968259072, free=45947277312, active=39810408448, inactive=12237737984, buffers=44175360, cached=35972448256, shared=2162810880, slab=798203904) 
2022-04-12 11:13:41.210984: 
epoch:  497 
2022-04-12 11:15:30.910351: train loss : -0.7434 
2022-04-12 11:15:51.397234: validation loss: -0.7292 
2022-04-12 11:15:51.507958: Average global foreground Dice: [0.9251, 0.9762, 0.9846, 0.4186] 
2022-04-12 11:15:51.757680: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:15:52.736954: lr: 6.9e-05 
2022-04-12 11:15:52.803125: saving scheduled checkpoint file... 
2022-04-12 11:15:52.943228: saving checkpoint... 
2022-04-12 11:16:14.166414: done, saving took 21.31 seconds 
2022-04-12 11:16:15.012937: done 
2022-04-12 11:16:15.217062: This epoch took 153.881671 s
 
2022-04-12 11:16:15.286026: svmem(total=99932160000, available=78198120448, percent=21.7, used=17738338304, free=46239580160, active=39248814080, inactive=12500672512, buffers=44236800, cached=35910004736, shared=2162810880, slab=797519872) 
2022-04-12 11:16:15.488598: 
epoch:  498 
2022-04-12 11:18:05.223917: train loss : -0.7571 
2022-04-12 11:18:17.694856: validation loss: -0.7109 
2022-04-12 11:18:17.881387: Average global foreground Dice: [0.9213, 0.9782, 0.9865, 0.3975] 
2022-04-12 11:18:18.094887: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:18:19.001129: lr: 3.7e-05 
2022-04-12 11:18:19.065809: saving scheduled checkpoint file... 
2022-04-12 11:18:19.356120: saving checkpoint... 
2022-04-12 11:18:30.343036: done, saving took 11.11 seconds 
2022-04-12 11:18:30.767827: done 
2022-04-12 11:18:30.888628: This epoch took 135.106513 s
 
2022-04-12 11:18:31.010516: svmem(total=99932160000, available=77869801472, percent=22.1, used=18066419712, free=46376665088, active=39506796544, inactive=12105916416, buffers=44277760, cached=35444797440, shared=2162810880, slab=797028352) 
2022-04-12 11:18:31.131937: 
epoch:  499 
2022-04-12 11:20:21.628004: train loss : -0.7427 
2022-04-12 11:20:41.969996: validation loss: -0.7351 
2022-04-12 11:20:42.366725: Average global foreground Dice: [0.9381, 0.9765, 0.9884, 0.445] 
2022-04-12 11:20:42.660851: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-12 11:20:43.935789: lr: 0.0 
2022-04-12 11:20:44.628821: saving scheduled checkpoint file... 
2022-04-12 11:20:45.560447: saving checkpoint... 
2022-04-12 11:21:06.863830: done, saving took 21.40 seconds 
2022-04-12 11:21:07.208100: done 
2022-04-12 11:21:07.333468: This epoch took 156.134458 s
 
2022-04-12 11:21:07.646135: saving checkpoint... 
2022-04-12 11:21:28.383583: done, saving took 20.86 seconds 
2022-04-12 11:33:45.180327: finished prediction 
2022-04-12 11:33:45.205782: evaluation of raw predictions