Starting... 
2022-04-06 14:16:01.561212: Using splits from existing split file: /home/woody/iwi5/iwi5039h/nnUNet_data/nnUNet_preprocessed/Task500_Glacier_zonefronts/splits_final.pkl 
2022-04-06 14:16:01.618421: The split file contains 5 splits. 
2022-04-06 14:16:01.619127: Desired fold for training: 0 
2022-04-06 14:16:01.619669: This split has 447 training and 112 validation cases. 
2022-04-06 14:16:02.356000: TRAINING KEYS:
 odict_keys(['Crane_2002-11-09_ERS_20_2_061', 'Crane_2002-11-18_ERS_20_3_195', 'Crane_2003-01-18_ERS_20_1_061', 'Crane_2006-11-27_ERS_20_1_195', 'Crane_2007-02-05_ERS_20_1_195', 'Crane_2007-03-31_ENVISAT_20_3_467', 'Crane_2007-05-05_ENVISAT_20_1_467', 'Crane_2007-06-09_ENVISAT_20_1_467', 'Crane_2007-07-14_ENVISAT_20_1_467', 'Crane_2007-11-21_PALSAR_17_2_135', 'Crane_2008-01-05_ENVISAT_20_2_467', 'Crane_2008-10-13_TSX_7_3_034', 'Crane_2008-10-24_TSX_7_2_034', 'Crane_2008-11-04_TSX_7_1_034', 'Crane_2008-11-23_PALSAR_17_2_135', 'Crane_2009-09-29_PALSAR_17_1_137', 'Crane_2009-11-02_TSX_7_3_034', 'Crane_2009-11-13_TSX_7_1_034', 'Crane_2010-05-29_ENVISAT_20_3_467', 'Crane_2010-07-03_ENVISAT_20_2_467', 'Crane_2010-09-15_PALSAR_17_4_138', 'Crane_2010-10-14_PALSAR_17_2_135', 'Crane_2010-11-05_TSX_7_2_110', 'Crane_2010-11-16_TSX_7_2_110', 'Crane_2010-12-08_TSX_7_1_110', 'Crane_2010-12-14_TSX_7_1_034', 'Crane_2010-12-19_TSX_7_1_110', 'Crane_2010-12-25_TSX_7_1_034', 'Crane_2011-06-02_TSX_7_2_110', 'Crane_2011-07-16_TSX_7_1_110', 'Crane_2011-08-13_TSX_7_1_034', 'Crane_2011-09-04_TSX_7_1_034', 'Crane_2011-10-18_TSX_7_1_034', 'Crane_2011-12-01_TSX_7_1_034', 'Crane_2011-12-12_TSX_7_2_034', 'Crane_2012-05-14_TSX_7_2_034', 'Crane_2012-06-16_TSX_7_2_034', 'Crane_2012-06-27_TSX_7_2_034', 'Crane_2012-07-03_TSX_7_1_125', 'Crane_2012-07-30_TSX_7_2_034', 'Crane_2012-08-21_TSX_7_3_034', 'Crane_2012-09-01_TSX_7_1_034', 'Crane_2012-10-04_TSX_7_3_034', 'Crane_2013-02-24_TSX_7_2_034', 'Crane_2013-07-17_TSX_7_3_034', 'Crane_2013-07-22_TSX_7_3_110', 'Crane_2013-08-02_TSX_7_3_110', 'Crane_2013-08-19_TSX_7_4_034', 'Crane_2013-11-04_TSX_7_4_034', 'Crane_2013-11-15_TSX_7_3_034', 'Crane_2013-11-26_TSX_7_4_034', 'Crane_2013-12-07_TSX_7_5_034', 'Crane_2013-12-18_TSX_7_4_034', 'Crane_2013-12-29_TSX_7_4_034', 'Crane_2014-01-20_TSX_7_2_034', 'DBE_1995-11-16_ERS_20_1_152', 'DBE_1997-01-29_ERS_20_2_424', 'DBE_1997-03-05_ERS_20_1_424', 'DBE_1998-02-18_ERS_20_1_424', 'DBE_1999-11-10_ERS_20_4_424', 'DBE_2000-10-04_RSAT_20_4_282', 'DBE_2000-10-06_RSAT_12_3_306', 'DBE_2000-10-13_RSAT_20_4_063', 'DBE_2002-02-27_ERS_20_1_424', 'DBE_2002-11-16_ERS_20_3_161', 'DBE_2002-12-21_ERS_20_4_161', 'DBE_2003-01-08_ERS_20_4_424', 'DBE_2003-02-12_ERS_20_3_424', 'DBE_2003-11-19_ERS_20_3_424', 'DBE_2004-03-03_ENVISAT_20_3_424', 'DBE_2004-03-03_ERS_20_3_424', 'DBE_2004-05-12_ENVISAT_20_3_424', 'DBE_2004-08-25_ENVISAT_20_3_424', 'DBE_2004-09-13_RSAT_12_3_282', 'DBE_2004-09-29_ENVISAT_20_3_424', 'DBE_2004-10-07_RSAT_12_4_282', 'DBE_2004-10-16_RSAT_12_3_063', 'DBE_2004-11-03_ENVISAT_20_4_424', 'DBE_2004-11-24_RSAT_12_3_282', 'DBE_2004-12-11_RSAT_12_4_182', 'DBE_2004-12-18_RSAT_12_3_282', 'DBE_2005-01-12_ERS_20_6_424', 'DBE_2005-02-16_ENVISAT_20_1_424', 'DBE_2005-03-23_ENVISAT_20_1_424', 'DBE_2005-05-11_RSAT_12_3_282', 'DBE_2005-06-01_ENVISAT_20_2_424', 'DBE_2005-07-06_ENVISAT_20_3_424', 'DBE_2006-01-06_RSAT_12_4_282', 'DBE_2006-02-23_RSAT_12_3_282', 'DBE_2006-03-08_ENVISAT_20_1_424', 'DBE_2006-03-19_RSAT_12_4_282', 'DBE_2006-04-12_RSAT_12_1_282', 'DBE_2006-05-06_RSAT_12_3_282', 'DBE_2006-05-18_PALSAR_17_2_135', 'DBE_2006-07-03_PALSAR_17_3_135', 'DBE_2006-07-26_ENVISAT_20_3_424', 'DBE_2006-08-06_PALSAR_17_2_137', 'DBE_2006-08-10_RSAT_20_4_282', 'DBE_2006-08-30_ENVISAT_20_2_424', 'DBE_2006-09-03_RSAT_20_5_282', 'DBE_2006-10-04_ENVISAT_20_3_424', 'DBE_2006-11-06_PALSAR_17_3_137', 'DBE_2006-11-08_ENVISAT_20_3_424', 'DBE_2007-01-01_RSAT_12_5_282', 'DBE_2007-01-25_RSAT_12_5_282', 'DBE_2007-02-18_RSAT_12_4_282', 'DBE_2007-03-14_RSAT_12_3_282', 'DBE_2007-05-01_RSAT_12_4_282', 'DBE_2007-11-11_PALSAR_17_4_129', 'DBE_2007-12-03_RSAT_12_4_282', 'DBE_2007-12-27_PALSAR_17_5_129', 'DBE_2007-12-27_RSAT_12_5_282', 'DBE_2008-01-20_RSAT_12_4_282', 'DBE_2008-03-30_TSX_7_3_049', 'DBE_2008-09-28_PALSAR_17_5_129', 'DBE_2008-10-25_TSX_7_2_049', 'DBE_2008-11-05_TSX_7_2_049', 'DBE_2008-11-13_PALSAR_17_5_129', 'DBE_2009-02-13_PALSAR_17_5_129', 'DBE_2009-11-03_TSX_7_2_049', 'DBE_2009-12-03_PALSAR_17_5_132', 'DBE_2010-01-18_PALSAR_17_5_132', 'DBE_2010-03-17_ENVISAT_20_6_424', 'DBE_2010-10-10_TSX_7_2_049', 'DBE_2010-10-21_PALSAR_17_4_132', 'DBE_2010-11-19_PALSAR_17_4_129', 'DBE_2010-11-23_TSX_7_5_049', 'DBE_2010-12-01_PALSAR_17_4_127', 'DBE_2011-01-04_PALSAR_17_4_129', 'DBE_2011-01-16_PALSAR_17_4_127', 'DBE_2011-01-17_TSX_7_4_049', 'DBE_2011-01-28_TSX_7_2_049', 'DBE_2011-07-05_TSX_7_1_110', 'DBE_2011-07-27_TSX_7_1_110', 'DBE_2011-09-09_TSX_7_1_110', 'DBE_2011-09-20_TSX_7_1_110', 'DBE_2011-10-23_TSX_7_1_110', 'DBE_2011-11-14_TSX_7_1_110', 'DBE_2011-12-06_TSX_7_5_110', 'DBE_2011-12-28_TSX_7_2_110', 'DBE_2012-04-05_TSX_7_1_110', 'DBE_2012-05-08_TSX_7_1_110', 'DBE_2012-06-21_TSX_7_1_110', 'DBE_2012-08-04_TSX_7_1_110', 'DBE_2012-10-31_TSX_7_1_110', 'DBE_2012-11-11_TSX_7_1_110', 'DBE_2013-03-12_TSX_7_1_110', 'DBE_2013-03-23_TSX_7_1_110', 'DBE_2013-04-03_TSX_7_1_110', 'DBE_2013-07-11_TSX_7_1_110', 'DBE_2013-09-04_TSX_7_1_110', 'DBE_2013-10-07_TSX_7_1_110', 'DBE_2013-12-01_TSX_7_2_110', 'DBE_2013-12-23_TSX_7_2_110', 'DBE_2014-01-03_TSX_7_5_110', 'DBE_2014-02-05_TSX_7_1_110', 'DBE_2014-03-21_TSX_7_2_110', 'JAC_2009-04-16_TSX_6_1_005', 'JAC_2009-05-08_TSX_6_1_005', 'JAC_2009-05-19_TSX_6_1_005', 'JAC_2009-05-30_TSX_6_1_005', 'JAC_2009-06-21_TSX_6_1_005', 'JAC_2009-07-02_TSX_6_1_005', 'JAC_2009-07-13_TSX_6_1_005', 'JAC_2009-07-24_TSX_6_1_005', 'JAC_2009-08-04_TSX_6_1_005', 'JAC_2009-08-15_TSX_6_1_005', 'JAC_2009-08-26_TSX_6_1_005', 'JAC_2009-09-06_TSX_6_1_005', 'JAC_2009-09-17_TSX_6_1_005', 'JAC_2009-09-28_TSX_6_1_005', 'JAC_2009-10-09_TSX_6_1_005', 'JAC_2009-10-20_TSX_6_1_005', 'JAC_2009-11-11_TSX_6_1_005', 'JAC_2009-11-22_TSX_6_1_005', 'JAC_2009-12-14_TSX_6_1_005', 'JAC_2009-12-25_TSX_6_1_005', 'JAC_2010-01-05_TSX_6_1_005', 'JAC_2010-01-16_TSX_6_1_005', 'JAC_2010-02-07_TSX_6_1_005', 'JAC_2010-02-18_TSX_6_1_005', 'JAC_2010-03-12_TSX_6_1_005', 'JAC_2010-03-23_TSX_6_1_005', 'JAC_2010-04-14_TSX_6_1_005', 'JAC_2010-04-25_TSX_6_1_005', 'JAC_2010-05-06_TSX_6_1_005', 'JAC_2010-05-17_TSX_6_1_005', 'JAC_2010-06-19_TSX_6_1_005', 'JAC_2010-06-30_TSX_6_1_005', 'JAC_2010-07-11_TSX_6_1_005', 'JAC_2010-07-22_TSX_6_1_005', 'JAC_2010-09-04_TSX_6_1_005', 'JAC_2010-10-07_TSX_6_1_005', 'JAC_2010-10-18_TSX_6_1_005', 'JAC_2010-11-09_TSX_6_1_005', 'JAC_2010-11-20_TSX_6_1_005', 'JAC_2011-01-03_TSX_6_1_005', 'JAC_2011-01-25_TSX_6_1_005', 'JAC_2011-02-16_TSX_6_1_005', 'JAC_2011-03-10_TSX_6_1_005', 'JAC_2011-03-21_TSX_6_1_005', 'JAC_2011-04-12_TSX_6_1_005', 'JAC_2011-04-23_TSX_6_1_005', 'JAC_2011-05-26_TSX_6_1_005', 'JAC_2011-07-20_TSX_6_1_005', 'JAC_2011-08-23_TSX_6_1_027', 'JAC_2011-09-03_TSX_6_1_027', 'JAC_2011-09-14_TSX_6_1_027', 'JAC_2011-10-05_TSX_6_1_005', 'JAC_2011-10-16_TSX_6_1_005', 'JAC_2011-11-08_TSX_6_1_027', 'JAC_2011-11-18_TSX_6_1_005', 'JAC_2011-12-10_TSX_6_1_005', 'JAC_2011-12-21_TSX_6_1_005', 'JAC_2012-01-13_TSX_6_1_027', 'JAC_2012-01-23_TSX_6_1_005', 'JAC_2012-02-14_TSX_6_1_005', 'JAC_2012-02-25_TSX_6_1_005', 'JAC_2012-03-29_TSX_6_1_005', 'JAC_2012-06-03_TSX_6_1_005', 'JAC_2012-06-14_TSX_6_1_005', 'JAC_2012-07-07_TSX_6_1_027', 'JAC_2012-07-18_TSX_6_1_027', 'JAC_2012-08-09_TSX_6_1_027', 'JAC_2012-09-21_TSX_6_1_005', 'JAC_2012-10-03_TSX_6_1_027', 'JAC_2012-10-14_TSX_6_1_027', 'JAC_2012-11-04_TSX_6_1_005', 'JAC_2012-11-15_TSX_6_1_005', 'JAC_2012-12-07_TSX_6_1_005', 'JAC_2012-12-19_TSX_6_1_027', 'JAC_2013-01-09_TSX_6_1_005', 'JAC_2013-01-20_TSX_6_1_005', 'JAC_2013-02-11_TSX_6_1_005', 'JAC_2013-02-22_TSX_6_1_005', 'JAC_2013-03-06_TSX_6_1_027', 'JAC_2013-03-17_TSX_6_1_027', 'JAC_2013-04-18_TSX_6_1_005', 'JAC_2013-05-10_TSX_6_1_005', 'JAC_2013-06-24_TSX_6_1_027', 'JAC_2013-07-05_TSX_6_1_027', 'JAC_2013-07-27_TSX_6_1_027', 'JAC_2013-08-17_TSX_6_1_005', 'JAC_2013-08-28_TSX_6_1_005', 'JAC_2013-09-20_TSX_6_1_027', 'JAC_2013-10-12_TSX_6_1_027', 'JAC_2013-10-23_TSX_6_1_027', 'JAC_2013-11-14_TSX_6_1_027', 'JAC_2013-11-25_TSX_6_1_027', 'JAC_2013-12-06_TSX_6_1_027', 'JAC_2014-03-03_TSX_6_1_005', 'JAC_2014-03-14_TSX_6_1_005', 'JAC_2014-03-25_TSX_6_1_005', 'JAC_2014-04-05_TSX_6_1_005', 'JAC_2014-04-27_TSX_6_1_005', 'JAC_2014-06-10_TSX_6_1_005', 'JAC_2014-06-21_TSX_6_1_005', 'JAC_2014-07-02_TSX_6_1_005', 'JAC_2014-07-14_TSX_6_1_027', 'JAC_2014-08-04_TSX_6_1_005', 'JAC_2014-09-18_TSX_6_1_027', 'JAC_2014-09-28_TSX_6_1_005', 'JAC_2014-10-09_TSX_6_1_005', 'JAC_2014-10-20_TSX_6_1_005', 'JAC_2014-11-11_TSX_6_1_005', 'JAC_2014-12-03_TSX_6_1_005', 'JAC_2014-12-14_TSX_6_1_005', 'JAC_2015-01-16_TSX_6_1_005', 'JAC_2015-01-27_TSX_6_1_005', 'JAC_2015-02-07_TSX_6_1_005', 'JAC_2015-02-19_TSX_6_1_027', 'JAC_2015-03-12_TSX_6_1_005', 'JAC_2015-03-23_TSX_6_1_005', 'JAC_2015-04-14_TSX_6_1_005', 'JAC_2015-04-25_TSX_6_1_005', 'JAC_2015-05-17_TSX_6_1_005', 'JAC_2015-06-19_TSX_6_1_005', 'JAC_2015-07-22_TSX_6_1_005', 'JAC_2015-08-02_TSX_6_1_005', 'JAC_2015-08-13_TSX_6_1_005', 'JAC_2015-09-04_TSX_6_1_005', 'JAC_2015-09-26_TSX_6_1_005', 'JAC_2015-10-18_TSX_6_1_005', 'JAC_2015-10-29_TSX_6_1_005', 'JAC_2015-11-09_TSX_6_1_005', 'JAC_2015-12-23_TSX_6_1_005', 'Jorum_2006-11-27_ERS_20_3_195', 'Jorum_2007-02-05_ERS_20_1_195', 'Jorum_2007-03-31_ENVISAT_20_2_467', 'Jorum_2007-07-14_ENVISAT_20_2_467', 'Jorum_2007-09-22_ENVISAT_20_3_467', 'Jorum_2007-10-27_ENVISAT_20_3_467', 'Jorum_2007-11-21_PALSAR_17_2_135', 'Jorum_2008-01-05_ENVISAT_20_2_467', 'Jorum_2009-12-25_PALSAR_17_1_136', 'Jorum_2010-03-20_ENVISAT_20_2_467', 'Jorum_2010-05-29_ENVISAT_20_2_467', 'Jorum_2010-07-03_ENVISAT_20_2_467', 'Jorum_2010-09-15_PALSAR_17_2_138', 'Jorum_2010-10-14_PALSAR_17_2_135', 'Jorum_2010-11-05_TSX_7_1_110', 'Jorum_2010-11-16_TSX_7_2_110', 'Jorum_2010-11-27_TSX_7_2_110', 'Jorum_2010-11-29_PALSAR_17_1_135', 'Jorum_2010-12-14_TSX_7_1_034', 'Jorum_2010-12-30_TSX_7_1_110', 'Jorum_2011-06-02_TSX_7_3_110', 'Jorum_2011-06-24_TSX_7_2_110', 'Jorum_2011-07-16_TSX_7_3_110', 'Jorum_2011-08-13_TSX_7_2_034', 'Jorum_2011-09-04_TSX_7_4_034', 'Jorum_2011-10-18_TSX_7_3_034', 'Jorum_2011-12-01_TSX_7_1_034', 'Jorum_2011-12-12_TSX_7_3_034', 'Jorum_2012-06-27_TSX_7_2_034', 'Jorum_2012-07-03_TSX_7_2_125', 'Jorum_2012-07-25_TSX_7_2_125', 'Jorum_2012-08-10_TSX_7_2_034', 'Jorum_2012-08-21_TSX_7_2_034', 'Jorum_2012-09-01_TSX_7_2_034', 'Jorum_2012-10-04_TSX_7_2_034', 'Jorum_2012-11-06_TSX_7_2_034', 'Jorum_2013-03-07_TSX_7_2_034', 'Jorum_2013-03-29_TSX_7_2_034', 'Jorum_2013-06-30_TSX_7_2_110', 'Jorum_2013-07-17_TSX_7_1_034', 'Jorum_2013-08-19_TSX_7_2_034', 'Jorum_2013-11-04_TSX_7_2_034', 'Jorum_2013-11-15_TSX_7_2_034', 'Jorum_2013-12-07_TSX_7_5_034', 'Jorum_2013-12-18_TSX_7_5_034', 'Jorum_2013-12-29_TSX_7_4_034', 'Jorum_2014-01-20_TSX_7_2_034', 'Jorum_2016-07-03_S1_20_2_009', 'Jorum_2016-09-01_S1_20_2_009', 'Jorum_2016-12-30_S1_20_2_009', 'Jorum_2017-09-08_S1_20_2_009', 'Jorum_2018-01-06_S1_20_3_009', 'Jorum_2018-03-07_S1_20_3_009', 'Jorum_2018-06-11_S1_20_2_009', 'Jorum_2018-09-03_S1_20_3_009', 'Jorum_2019-01-01_S1_20_3_009', 'Jorum_2019-03-02_S1_20_3_009', 'Jorum_2019-09-22_S1_20_3_009', 'Jorum_2020-01-08_S1_20_3_009', 'Jorum_2020-03-08_S1_20_3_009', 'SI_1995-11-13_ERS_20_3_109', 'SI_1995-11-14_ERS_20_4_118', 'SI_1995-11-15_ERS_20_3_118', 'SI_1995-11-16_ERS_20_4_152', 'SI_1996-01-23_ERS_20_5_109', 'SI_1996-01-24_ERS_20_4_118', 'SI_1996-02-12_ERS_20_2_390', 'SI_1996-02-29_ERS_20_2_152', 'SI_1996-03-18_ERS_20_2_390', 'SI_1997-01-27_ERS_20_4_390', 'SI_2000-09-10_RSAT_20_5_282', 'SI_2000-09-19_RSAT_20_5_063', 'SI_2000-10-04_RSAT_20_5_282', 'SI_2000-10-13_RSAT_20_5_063', 'SI_2002-01-21_ERS_20_2_390', 'SI_2002-02-25_ERS_20_3_390', 'SI_2002-11-12_ERS_20_4_109', 'SI_2002-12-21_ERS_20_3_161', 'SI_2003-12-05_ENVISAT_20_4_152', 'SI_2004-07-02_ENVISAT_20_4_152', 'SI_2004-08-06_ENVISAT_20_4_152', 'SI_2004-09-13_RSAT_12_5_282', 'SI_2004-10-15_ERS_20_5_152', 'SI_2004-11-09_RSAT_12_5_063', 'SI_2004-11-19_ERS_20_4_152', 'SI_2004-11-24_RSAT_12_4_282', 'SI_2004-12-18_RSAT_12_4_282', 'SI_2004-12-24_ENVISAT_20_5_152', 'SI_2004-12-24_ERS_20_5_152', 'SI_2005-01-28_ERS_20_5_152', 'SI_2005-02-28_RSAT_12_5_282', 'SI_2005-03-04_ENVISAT_20_3_152', 'SI_2005-04-08_ENVISAT_20_2_152', 'SI_2005-04-17_RSAT_12_3_282', 'SI_2005-05-11_RSAT_12_2_282', 'SI_2005-07-22_ENVISAT_20_2_152', 'SI_2005-08-26_ENVISAT_20_2_152', 'SI_2006-01-06_RSAT_12_2_282', 'SI_2006-02-23_RSAT_12_1_282', 'SI_2006-03-05_ENVISAT_20_2_381', 'SI_2006-03-19_RSAT_12_2_282', 'SI_2006-04-12_RSAT_12_1_282', 'SI_2006-05-06_RSAT_12_3_282', 'SI_2006-07-03_PALSAR_17_3_135', 'SI_2006-07-23_ENVISAT_20_4_381', 'SI_2006-08-10_RSAT_20_3_282', 'SI_2006-09-03_RSAT_12_2_282', 'SI_2006-09-03_RSAT_20_2_282', 'SI_2006-10-18_ERS_20_4_118', 'SI_2006-11-21_ERS_20_3_109', 'SI_2007-01-01_RSAT_12_3_282', 'SI_2007-01-14_ENVISAT_20_3_381', 'SI_2007-03-14_RSAT_12_2_282', 'SI_2007-03-25_ENVISAT_20_3_381', 'SI_2007-05-01_RSAT_12_2_282', 'SI_2007-06-03_ENVISAT_20_3_381', 'SI_2007-07-08_ENVISAT_20_3_381', 'SI_2007-09-16_ENVISAT_20_3_381', 'SI_2007-10-21_ENVISAT_20_3_381', 'SI_2007-12-03_RSAT_12_4_282', 'SI_2007-12-27_RSAT_12_2_282', 'SI_2008-01-20_RSAT_12_2_282', 'SI_2008-04-13_ENVISAT_20_2_381', 'SI_2008-05-18_ENVISAT_20_3_381', 'SI_2008-07-27_ENVISAT_20_4_381', 'SI_2008-08-31_ENVISAT_20_4_381', 'SI_2009-02-22_ENVISAT_20_2_381', 'SI_2009-03-29_ENVISAT_20_4_381', 'SI_2009-07-12_ENVISAT_20_3_381', 'SI_2009-08-16_ENVISAT_20_3_381', 'SI_2010-01-13_PALSAR_17_2_127', 'SI_2010-01-22_ERS_20_3_152', 'SI_2010-02-26_ERS_20_2_152', 'SI_2010-04-02_ERS_20_2_152', 'SI_2010-10-16_PALSAR_17_3_127', 'SI_2010-12-01_PALSAR_17_3_127', 'SI_2011-01-16_PALSAR_17_2_127', 'SI_2011-03-03_PALSAR_17_2_127', 'SI_2011-06-14_TSX_7_1_125', 'SI_2011-07-06_TSX_7_1_125', 'SI_2011-08-19_TSX_7_1_125', 'SI_2011-09-10_TSX_7_1_125', 'SI_2011-09-21_TSX_7_1_125', 'SI_2011-12-29_TSX_7_1_125', 'SI_2012-01-09_TSX_7_1_125', 'SI_2012-03-04_TSX_7_1_125', 'SI_2012-03-15_TSX_7_1_125', 'SI_2012-03-26_TSX_7_1_125', 'SI_2012-05-20_TSX_7_1_125', 'SI_2012-06-22_TSX_7_1_125', 'SI_2012-08-05_TSX_7_1_125', 'SI_2012-08-16_TSX_7_1_125', 'SI_2013-07-12_TSX_7_1_125', 'SI_2013-08-14_TSX_7_1_125', 'SI_2013-09-16_TSX_7_1_125', 'SI_2013-10-19_TSX_7_1_125', 'SI_2013-10-28_TSX_7_1_104', 'SI_2013-11-21_TSX_7_1_125', 'SI_2014-01-04_TSX_7_2_125', 'SI_2014-04-02_TSX_7_1_125', 'SI_2014-12-11_TSX_7_4_125']) 
2022-04-06 14:16:02.357023: VALIDATION KEYS:
 odict_keys(['Crane_2007-09-22_ENVISAT_20_1_467', 'Crane_2007-09-24_PALSAR_17_1_137', 'Crane_2007-10-27_ENVISAT_20_2_467', 'Crane_2010-11-27_TSX_7_1_110', 'Crane_2010-11-29_PALSAR_17_1_135', 'Crane_2010-12-30_TSX_7_1_110', 'Crane_2011-06-24_TSX_7_1_110', 'Crane_2012-07-25_TSX_7_2_125', 'Crane_2012-08-10_TSX_7_3_034', 'Crane_2012-11-06_TSX_7_2_034', 'Crane_2013-03-07_TSX_7_2_034', 'Crane_2013-03-29_TSX_7_2_034', 'Crane_2013-06-30_TSX_7_2_110', 'Crane_2014-02-11_TSX_7_3_034', 'DBE_1996-01-25_ERS_20_1_152', 'DBE_2000-09-10_RSAT_20_3_282', 'DBE_2004-04-07_ENVISAT_20_3_424', 'DBE_2004-06-16_ENVISAT_20_3_424', 'DBE_2004-07-21_ENVISAT_20_3_424', 'DBE_2004-11-09_RSAT_12_3_063', 'DBE_2004-12-08_ENVISAT_20_5_424', 'DBE_2004-12-08_ERS_20_5_424', 'DBE_2005-04-27_ENVISAT_20_2_424', 'DBE_2005-08-10_ENVISAT_20_2_424', 'DBE_2006-04-12_ENVISAT_20_1_424', 'DBE_2006-06-21_ENVISAT_20_2_424', 'DBE_2006-06-21_PALSAR_17_2_137', 'DBE_2007-08-15_ENVISAT_20_5_424', 'DBE_2007-09-19_ENVISAT_20_6_424', 'DBE_2008-10-14_TSX_7_3_049', 'DBE_2008-12-29_PALSAR_17_5_129', 'DBE_2009-10-23_TSX_7_2_049', 'DBE_2010-01-22_ERS_20_6_152', 'DBE_2010-04-21_ENVISAT_20_6_424', 'DBE_2010-10-16_PALSAR_17_4_127', 'DBE_2010-11-12_TSX_7_4_049', 'DBE_2010-12-06_PALSAR_17_3_132', 'DBE_2012-03-14_TSX_7_1_110', 'DBE_2012-06-10_TSX_7_1_110', 'DBE_2012-08-15_TSX_7_1_110', 'DBE_2013-01-16_TSX_7_5_110', 'DBE_2013-05-28_TSX_7_1_110', 'DBE_2013-06-19_TSX_7_1_110', 'DBE_2013-11-09_TSX_7_1_110', 'DBE_2013-12-12_TSX_7_3_110', 'JAC_2009-04-27_TSX_6_1_005', 'JAC_2009-06-10_TSX_6_1_005', 'JAC_2009-10-31_TSX_6_1_005', 'JAC_2009-12-03_TSX_6_1_005', 'JAC_2010-09-15_TSX_6_1_005', 'JAC_2010-12-01_TSX_6_1_005', 'JAC_2010-12-23_TSX_6_1_005', 'JAC_2011-05-15_TSX_6_1_005', 'JAC_2011-06-17_TSX_6_1_005', 'JAC_2012-03-07_TSX_6_1_005', 'JAC_2012-04-20_TSX_6_1_005', 'JAC_2012-05-24_TSX_6_1_027', 'JAC_2012-08-20_TSX_6_1_027', 'JAC_2012-09-11_TSX_6_1_027', 'JAC_2013-04-07_TSX_6_1_005', 'JAC_2013-05-22_TSX_6_1_027', 'JAC_2013-06-13_TSX_6_1_027', 'JAC_2013-09-08_TSX_6_1_005', 'JAC_2013-12-17_TSX_6_1_027', 'JAC_2014-01-29_TSX_6_1_005', 'JAC_2014-04-16_TSX_6_1_005', 'JAC_2014-07-03_TSX_6_1_027', 'JAC_2014-07-25_TSX_6_1_027', 'JAC_2014-12-25_TSX_6_1_005', 'JAC_2015-01-05_TSX_6_1_005', 'JAC_2015-05-06_TSX_6_1_005', 'JAC_2015-06-08_TSX_6_1_005', 'JAC_2015-07-11_TSX_6_1_005', 'JAC_2015-11-20_TSX_6_1_005', 'JAC_2015-12-12_TSX_6_1_005', 'Jorum_2003-01-27_ERS_20_1_195', 'Jorum_2007-05-05_ENVISAT_20_2_467', 'Jorum_2007-06-09_ENVISAT_20_2_467', 'Jorum_2008-11-23_PALSAR_17_2_135', 'Jorum_2010-12-08_TSX_7_2_110', 'Jorum_2010-12-19_TSX_7_1_110', 'Jorum_2010-12-25_TSX_7_1_034', 'Jorum_2012-05-14_TSX_7_2_034', 'Jorum_2012-06-16_TSX_7_2_034', 'Jorum_2012-07-30_TSX_7_2_034', 'Jorum_2013-02-24_TSX_7_3_034', 'Jorum_2013-07-22_TSX_7_1_110', 'Jorum_2013-08-02_TSX_7_2_110', 'Jorum_2013-11-26_TSX_7_5_034', 'Jorum_2014-02-11_TSX_7_2_034', 'Jorum_2017-06-16_S1_20_2_009', 'Jorum_2020-06-12_S1_20_3_009', 'SI_1996-01-22_ERS_20_3_109', 'SI_1997-03-03_ERS_20_4_390', 'SI_1999-11-07_ERS_20_2_381', 'SI_2000-02-20_ERS_20_3_381', 'SI_2003-01-25_ERS_20_3_161', 'SI_2004-02-13_ENVISAT_20_4_152', 'SI_2004-12-03_RSAT_12_5_063', 'SI_2006-01-29_ENVISAT_20_3_381', 'SI_2006-05-18_PALSAR_17_2_135', 'SI_2006-08-27_ENVISAT_20_3_381', 'SI_2007-01-25_RSAT_12_1_282', 'SI_2011-07-28_TSX_7_1_125', 'SI_2011-11-26_TSX_7_2_125', 'SI_2012-04-28_TSX_7_1_125', 'SI_2012-07-03_TSX_7_1_125', 'SI_2013-11-08_TSX_7_2_104', 'SI_2013-12-02_TSX_7_1_125', 'SI_2013-12-13_TSX_7_2_125', 'SI_2014-03-22_TSX_7_1_125', 'SI_2014-12-22_TSX_7_2_125']) 
2022-04-06 14:17:15.007322: lr: 0.01 
2022-04-06 14:17:35.214908: Unable to plot network architecture: 
2022-04-06 14:17:35.217746: No module named 'hiddenlayer' 
2022-04-06 14:17:35.221112: 
printing the network instead:
 
2022-04-06 14:17:35.225236: Generic_UNet(
  (conv_blocks_localization): ModuleList(
    (0): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (1): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (2): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (3): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (4): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(512, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (5): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(256, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (6): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(128, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
    (7): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(64, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
  )
  (conv_blocks_context): ModuleList(
    (0): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(1, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (1): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(32, 64, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (2): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(64, 128, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (3): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(128, 256, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (4): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(256, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (5): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (6): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (7): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
          (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (8): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
            (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
  )
  (td): ModuleList()
  (tu): ModuleList(
    (0): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (1): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (2): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (3): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (4): ConvTranspose2d(480, 256, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (5): ConvTranspose2d(256, 128, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (6): ConvTranspose2d(128, 64, kernel_size=[2, 2], stride=[2, 2], bias=False)
    (7): ConvTranspose2d(64, 32, kernel_size=[2, 2], stride=[2, 2], bias=False)
  )
  (seg_outputs): ModuleList(
    (0): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (1): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (2): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (3): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (4): Conv2d(256, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (5): Conv2d(128, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (6): Conv2d(64, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (7): Conv2d(32, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
  )
) 
2022-04-06 14:17:35.233388: 
 
2022-04-06 14:17:35.237235: svmem(total=99932160000, available=89208614912, percent=10.7, used=8181133312, free=86491205632, active=6190018560, inactive=4833574912, buffers=10117120, cached=5249703936, shared=755785728, slab=740827136) 
2022-04-06 14:17:35.238549: 
epoch:  0 
2022-04-06 14:20:06.985090: train loss : 0.9082 
2022-04-06 14:20:39.127314: validation loss: 0.8050 
2022-04-06 14:20:39.131757: Average global foreground Dice: [0.0011, 0.2521, 0.5427, 0.0044] 
2022-04-06 14:20:39.132651: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:20:40.030569: lr: 0.009982 
2022-04-06 14:20:40.032384: saving scheduled checkpoint file... 
2022-04-06 14:20:40.163269: saving checkpoint... 
2022-04-06 14:20:44.589472: done, saving took 4.56 seconds 
2022-04-06 14:20:44.611350: done 
2022-04-06 14:20:44.612158: This epoch took 189.362744 s
 
2022-04-06 14:20:44.612903: svmem(total=99932160000, available=87692906496, percent=12.2, used=9214038016, free=74312404992, active=11489583104, inactive=11691077632, buffers=10518528, cached=16395198464, shared=1225306112, slab=769114112) 
2022-04-06 14:20:44.613542: 
epoch:  1 
2022-04-06 14:22:30.579828: train loss : 0.6597 
2022-04-06 14:22:47.154359: validation loss: 0.5054 
2022-04-06 14:22:47.156661: Average global foreground Dice: [0.0929, 0.7204, 0.3517, 0.0] 
2022-04-06 14:22:47.158442: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:22:47.708783: lr: 0.009964 
2022-04-06 14:22:47.709661: saving scheduled checkpoint file... 
2022-04-06 14:22:47.881970: saving checkpoint... 
2022-04-06 14:22:52.166296: done, saving took 4.45 seconds 
2022-04-06 14:22:52.175415: done 
2022-04-06 14:22:52.222529: saving checkpoint... 
2022-04-06 14:22:56.492116: done, saving took 4.31 seconds 
2022-04-06 14:22:56.504002: This epoch took 131.889751 s
 
2022-04-06 14:22:56.505086: svmem(total=99932160000, available=87506563072, percent=12.4, used=9492766720, free=73792532480, active=17122783232, inactive=6559969280, buffers=10776576, cached=16636084224, shared=1127432192, slab=782045184) 
2022-04-06 14:22:56.505833: 
epoch:  2 
2022-04-06 14:25:16.211433: train loss : 0.5740 
2022-04-06 14:25:34.454957: validation loss: 0.5364 
2022-04-06 14:25:34.456805: Average global foreground Dice: [0.3368, 0.6112, 0.5751, 0.0] 
2022-04-06 14:25:34.457691: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:25:35.003785: lr: 0.009946 
2022-04-06 14:25:35.004750: saving scheduled checkpoint file... 
2022-04-06 14:25:35.160497: saving checkpoint... 
2022-04-06 14:25:39.542531: done, saving took 4.54 seconds 
2022-04-06 14:25:39.572890: done 
2022-04-06 14:25:39.621426: saving checkpoint... 
2022-04-06 14:25:44.207976: done, saving took 4.63 seconds 
2022-04-06 14:25:44.221770: This epoch took 167.715079 s
 
2022-04-06 14:25:44.222915: svmem(total=99932160000, available=87384719360, percent=12.6, used=9576935424, free=73785647104, active=19967713280, inactive=3712770048, buffers=8404992, cached=16561172480, shared=1162391552, slab=786980864) 
2022-04-06 14:25:44.224379: 
epoch:  3 
2022-04-06 14:27:32.507876: train loss : 0.5022 
2022-04-06 14:27:53.841259: validation loss: 0.5224 
2022-04-06 14:27:53.851475: Average global foreground Dice: [0.2465, 0.675, 0.5231, 0.0005] 
2022-04-06 14:27:53.858592: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:27:54.457201: lr: 0.009928 
2022-04-06 14:27:54.459486: saving scheduled checkpoint file... 
2022-04-06 14:27:54.622034: saving checkpoint... 
2022-04-06 14:27:59.105646: done, saving took 4.64 seconds 
2022-04-06 14:27:59.120145: done 
2022-04-06 14:27:59.167425: saving checkpoint... 
2022-04-06 14:28:03.531894: done, saving took 4.41 seconds 
2022-04-06 14:28:03.545440: This epoch took 139.318917 s
 
2022-04-06 14:28:03.546432: svmem(total=99932160000, available=87240691712, percent=12.7, used=9656655872, free=73787428864, active=20868816896, inactive=2812391424, buffers=8658944, cached=16479416320, shared=1225330688, slab=790265856) 
2022-04-06 14:28:03.547141: 
epoch:  4 
2022-04-06 14:29:54.002135: train loss : 0.4446 
2022-04-06 14:30:16.850233: validation loss: 0.4434 
2022-04-06 14:30:16.852021: Average global foreground Dice: [0.3378, 0.6905, 0.6468, 0.0014] 
2022-04-06 14:30:16.852835: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:30:17.662677: lr: 0.00991 
2022-04-06 14:30:17.663612: saving scheduled checkpoint file... 
2022-04-06 14:30:17.838060: saving checkpoint... 
2022-04-06 14:30:22.265469: done, saving took 4.60 seconds 
2022-04-06 14:30:22.304492: done 
2022-04-06 14:30:22.360774: saving checkpoint... 
2022-04-06 14:30:26.760920: done, saving took 4.45 seconds 
2022-04-06 14:30:26.778772: This epoch took 143.230911 s
 
2022-04-06 14:30:26.779752: svmem(total=99932160000, available=87187746816, percent=12.8, used=9708265472, free=73891237888, active=20718239744, inactive=2853470208, buffers=8957952, cached=16323698688, shared=1225330688, slab=792522752) 
2022-04-06 14:30:26.780362: 
epoch:  5 
2022-04-06 14:32:15.753733: train loss : 0.3893 
2022-04-06 14:32:31.537248: validation loss: 0.3488 
2022-04-06 14:32:31.541079: Average global foreground Dice: [0.4141, 0.6935, 0.6533, 0.0] 
2022-04-06 14:32:31.544196: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:32:32.293885: lr: 0.009892 
2022-04-06 14:32:32.295072: saving scheduled checkpoint file... 
2022-04-06 14:32:32.499648: saving checkpoint... 
2022-04-06 14:32:37.076589: done, saving took 4.78 seconds 
2022-04-06 14:32:37.101868: done 
2022-04-06 14:32:37.248401: saving checkpoint... 
2022-04-06 14:32:41.635960: done, saving took 4.53 seconds 
2022-04-06 14:32:41.651721: This epoch took 134.870739 s
 
2022-04-06 14:32:41.652617: svmem(total=99932160000, available=87128260608, percent=12.8, used=9791488000, free=73993613312, active=20710719488, inactive=2758283264, buffers=9220096, cached=16137838592, shared=1200857088, slab=793530368) 
2022-04-06 14:32:41.653362: 
epoch:  6 
2022-04-06 14:34:31.391388: train loss : 0.3970 
2022-04-06 14:34:42.386258: validation loss: 0.3784 
2022-04-06 14:34:42.388699: Average global foreground Dice: [0.3801, 0.7021, 0.5554, 0.0057] 
2022-04-06 14:34:42.390424: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:34:43.216695: lr: 0.009874 
2022-04-06 14:34:43.221215: saving scheduled checkpoint file... 
2022-04-06 14:34:43.414080: saving checkpoint... 
2022-04-06 14:34:47.840255: done, saving took 4.62 seconds 
2022-04-06 14:34:47.856393: done 
2022-04-06 14:34:47.993274: saving checkpoint... 
2022-04-06 14:34:52.655438: done, saving took 4.80 seconds 
2022-04-06 14:34:52.678898: This epoch took 131.024778 s
 
2022-04-06 14:34:52.682619: svmem(total=99932160000, available=87071813632, percent=12.9, used=9821978624, free=73978413056, active=20552384512, inactive=2929393664, buffers=9461760, cached=16122306560, shared=1225330688, slab=794955776) 
2022-04-06 14:34:52.685438: 
epoch:  7 
2022-04-06 14:36:42.970793: train loss : 0.3236 
2022-04-06 14:36:54.534987: validation loss: 0.3292 
2022-04-06 14:36:54.537145: Average global foreground Dice: [0.465, 0.6487, 0.6878, 0.0004] 
2022-04-06 14:36:54.539792: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:36:55.384016: lr: 0.009856 
2022-04-06 14:36:55.385017: saving scheduled checkpoint file... 
2022-04-06 14:36:55.604074: saving checkpoint... 
2022-04-06 14:37:00.068191: done, saving took 4.68 seconds 
2022-04-06 14:37:00.092433: done 
2022-04-06 14:37:00.240937: saving checkpoint... 
2022-04-06 14:37:05.529263: done, saving took 5.43 seconds 
2022-04-06 14:37:05.546165: This epoch took 132.859016 s
 
2022-04-06 14:37:05.547223: svmem(total=99932160000, available=87044501504, percent=12.9, used=9849987072, free=73943457792, active=20750966784, inactive=2772287488, buffers=9748480, cached=16128966656, shared=1225388032, slab=795709440) 
2022-04-06 14:37:05.547842: 
epoch:  8 
2022-04-06 14:38:55.675710: train loss : 0.2104 
2022-04-06 14:39:07.935857: validation loss: 0.2579 
2022-04-06 14:39:07.941273: Average global foreground Dice: [0.4499, 0.7521, 0.5789, 0.0] 
2022-04-06 14:39:07.946990: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:39:08.687059: lr: 0.009838 
2022-04-06 14:39:08.687982: saving scheduled checkpoint file... 
2022-04-06 14:39:08.847763: saving checkpoint... 
2022-04-06 14:39:13.470090: done, saving took 4.78 seconds 
2022-04-06 14:39:13.507335: done 
2022-04-06 14:39:13.662415: saving checkpoint... 
2022-04-06 14:39:18.095236: done, saving took 4.58 seconds 
2022-04-06 14:39:18.121959: This epoch took 132.573455 s
 
2022-04-06 14:39:18.128046: svmem(total=99932160000, available=87042326528, percent=12.9, used=9851752448, free=73992679424, active=20689915904, inactive=2781454336, buffers=9994240, cached=16077733888, shared=1225388032, slab=795111424) 
2022-04-06 14:39:18.129010: 
epoch:  9 
2022-04-06 14:41:07.657071: train loss : 0.2720 
2022-04-06 14:41:19.493600: validation loss: 0.2486 
2022-04-06 14:41:19.498102: Average global foreground Dice: [0.4656, 0.7343, 0.6489, 0.0] 
2022-04-06 14:41:19.501435: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:41:20.611193: lr: 0.00982 
2022-04-06 14:41:20.614506: saving scheduled checkpoint file... 
2022-04-06 14:41:20.818698: saving checkpoint... 
2022-04-06 14:41:25.244438: done, saving took 4.63 seconds 
2022-04-06 14:41:25.291183: done 
2022-04-06 14:41:25.454413: saving checkpoint... 
2022-04-06 14:41:29.932913: done, saving took 4.63 seconds 
2022-04-06 14:41:29.960282: This epoch took 131.830690 s
 
2022-04-06 14:41:29.964211: svmem(total=99932160000, available=86941532160, percent=13.0, used=9952772096, free=73982275584, active=20726943744, inactive=2751942656, buffers=10248192, cached=15986864128, shared=1225388032, slab=796033024) 
2022-04-06 14:41:29.966593: 
epoch:  10 
2022-04-06 14:43:20.556410: train loss : 0.2559 
2022-04-06 14:43:52.306912: validation loss: 0.1648 
2022-04-06 14:43:52.316000: Average global foreground Dice: [0.5059, 0.719, 0.718, 0.0] 
2022-04-06 14:43:52.316927: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:43:53.127980: lr: 0.009802 
2022-04-06 14:43:53.135975: saving scheduled checkpoint file... 
2022-04-06 14:43:53.391173: saving checkpoint... 
2022-04-06 14:43:57.757692: done, saving took 4.62 seconds 
2022-04-06 14:43:57.776240: done 
2022-04-06 14:43:57.823112: saving checkpoint... 
2022-04-06 14:44:02.191290: done, saving took 4.41 seconds 
2022-04-06 14:44:02.204092: This epoch took 152.233635 s
 
2022-04-06 14:44:02.205140: svmem(total=99932160000, available=86676721664, percent=13.3, used=10217099264, free=73763999744, active=20947226624, inactive=2750529536, buffers=10534912, cached=15940526080, shared=1225388032, slab=797306880) 
2022-04-06 14:44:02.205951: 
epoch:  11 
2022-04-06 14:45:51.793196: train loss : 0.2294 
2022-04-06 14:46:08.365793: validation loss: 0.2181 
2022-04-06 14:46:08.377891: Average global foreground Dice: [0.4326, 0.7536, 0.6623, 0.0] 
2022-04-06 14:46:08.378748: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:46:09.222448: lr: 0.009784 
2022-04-06 14:46:09.225769: saving scheduled checkpoint file... 
2022-04-06 14:46:09.342079: saving checkpoint... 
2022-04-06 14:46:13.755061: done, saving took 4.52 seconds 
2022-04-06 14:46:13.780825: done 
2022-04-06 14:46:13.866776: saving checkpoint... 
2022-04-06 14:46:18.300482: done, saving took 4.51 seconds 
2022-04-06 14:46:18.312509: This epoch took 136.105861 s
 
2022-04-06 14:46:18.315199: svmem(total=99932160000, available=86948577280, percent=13.0, used=9945329664, free=73791840256, active=20900130816, inactive=2768556032, buffers=10821632, cached=16184168448, shared=1225388032, slab=796712960) 
2022-04-06 14:46:18.318213: 
epoch:  12 
2022-04-06 14:48:07.570039: train loss : 0.2197 
2022-04-06 14:48:20.953509: validation loss: 0.2136 
2022-04-06 14:48:20.955222: Average global foreground Dice: [0.547, 0.7469, 0.6235, 0.0] 
2022-04-06 14:48:20.955974: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:48:21.481469: lr: 0.009766 
2022-04-06 14:48:21.482260: saving scheduled checkpoint file... 
2022-04-06 14:48:21.549276: saving checkpoint... 
2022-04-06 14:48:25.944649: done, saving took 4.46 seconds 
2022-04-06 14:48:25.962337: done 
2022-04-06 14:48:26.034066: saving checkpoint... 
2022-04-06 14:48:30.477671: done, saving took 4.51 seconds 
2022-04-06 14:48:30.492624: This epoch took 132.173408 s
 
2022-04-06 14:48:30.493641: svmem(total=99932160000, available=86834929664, percent=13.1, used=10082902016, free=73758314496, active=20893032448, inactive=2810265600, buffers=11067392, cached=16079876096, shared=1200914432, slab=796467200) 
2022-04-06 14:48:30.494351: 
epoch:  13 
2022-04-06 14:50:18.680562: train loss : 0.1789 
2022-04-06 14:50:29.482854: validation loss: 0.1964 
2022-04-06 14:50:29.484363: Average global foreground Dice: [0.4154, 0.7588, 0.7313, 0.0] 
2022-04-06 14:50:29.485093: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:50:30.365420: lr: 0.009748 
2022-04-06 14:50:30.372291: saving scheduled checkpoint file... 
2022-04-06 14:50:30.443156: saving checkpoint... 
2022-04-06 14:50:34.936689: done, saving took 4.56 seconds 
2022-04-06 14:50:34.953199: done 
2022-04-06 14:50:35.032702: saving checkpoint... 
2022-04-06 14:50:39.390836: done, saving took 4.44 seconds 
2022-04-06 14:50:39.426435: This epoch took 128.929933 s
 
2022-04-06 14:50:39.430502: svmem(total=99932160000, available=86763020288, percent=13.2, used=10130591744, free=73840193536, active=20852649984, inactive=2766311424, buffers=11313152, cached=15950061568, shared=1225400320, slab=797270016) 
2022-04-06 14:50:39.436371: 
epoch:  14 
2022-04-06 14:52:28.391820: train loss : 0.1572 
2022-04-06 14:52:43.155300: validation loss: 0.1067 
2022-04-06 14:52:43.161628: Average global foreground Dice: [0.5347, 0.7741, 0.7501, 0.0003] 
2022-04-06 14:52:43.168731: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:52:43.811415: lr: 0.00973 
2022-04-06 14:52:43.814361: saving scheduled checkpoint file... 
2022-04-06 14:52:43.985909: saving checkpoint... 
2022-04-06 14:52:48.570587: done, saving took 4.75 seconds 
2022-04-06 14:52:48.596140: done 
2022-04-06 14:52:48.700146: saving checkpoint... 
2022-04-06 14:52:53.129295: done, saving took 4.53 seconds 
2022-04-06 14:52:53.154525: This epoch took 133.713132 s
 
2022-04-06 14:52:53.159505: svmem(total=99932160000, available=86647246848, percent=13.3, used=10246217728, free=73868115968, active=20769120256, inactive=2822422528, buffers=11579392, cached=15806246912, shared=1225424896, slab=796639232) 
2022-04-06 14:52:53.162966: 
epoch:  15 
2022-04-06 14:54:42.256410: train loss : 0.1418 
2022-04-06 14:54:54.556114: validation loss: 0.1114 
2022-04-06 14:54:54.557920: Average global foreground Dice: [0.3837, 0.7942, 0.7502, 0.0001] 
2022-04-06 14:54:54.558651: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:54:55.542544: lr: 0.009712 
2022-04-06 14:54:55.545155: saving scheduled checkpoint file... 
2022-04-06 14:54:55.859985: saving checkpoint... 
2022-04-06 14:55:01.181903: done, saving took 5.63 seconds 
2022-04-06 14:55:01.204021: done 
2022-04-06 14:55:01.303325: saving checkpoint... 
2022-04-06 14:55:05.739232: done, saving took 4.53 seconds 
2022-04-06 14:55:05.764931: This epoch took 132.598600 s
 
2022-04-06 14:55:05.768528: svmem(total=99932160000, available=86662651904, percent=13.3, used=10229866496, free=73854124032, active=20831309824, inactive=2775093248, buffers=11833344, cached=15836336128, shared=1225441280, slab=796016640) 
2022-04-06 14:55:05.772752: 
epoch:  16 
2022-04-06 14:56:54.115307: train loss : 0.1248 
2022-04-06 14:57:06.106181: validation loss: 0.0996 
2022-04-06 14:57:06.107736: Average global foreground Dice: [0.5241, 0.7576, 0.7594, 0.0] 
2022-04-06 14:57:06.108681: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:57:07.056843: lr: 0.009693 
2022-04-06 14:57:07.062395: saving scheduled checkpoint file... 
2022-04-06 14:57:07.194746: saving checkpoint... 
2022-04-06 14:57:11.781111: done, saving took 4.72 seconds 
2022-04-06 14:57:11.806317: done 
2022-04-06 14:57:11.913470: saving checkpoint... 
2022-04-06 14:57:16.327883: done, saving took 4.51 seconds 
2022-04-06 14:57:16.363653: This epoch took 130.587621 s
 
2022-04-06 14:57:16.364780: svmem(total=99932160000, available=86633943040, percent=13.3, used=10259378176, free=73868705792, active=20801970176, inactive=2790305792, buffers=12099584, cached=15791976448, shared=1225441280, slab=796545024) 
2022-04-06 14:57:16.365531: 
epoch:  17 
2022-04-06 14:59:06.542187: train loss : 0.0673 
2022-04-06 14:59:22.624883: validation loss: 0.0618 
2022-04-06 14:59:22.635418: Average global foreground Dice: [0.5502, 0.779, 0.7592, 0.0002] 
2022-04-06 14:59:22.644782: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 14:59:23.400662: lr: 0.009675 
2022-04-06 14:59:23.411705: saving scheduled checkpoint file... 
2022-04-06 14:59:23.526292: saving checkpoint... 
2022-04-06 14:59:27.967512: done, saving took 4.55 seconds 
2022-04-06 14:59:27.988784: done 
2022-04-06 14:59:28.037204: saving checkpoint... 
2022-04-06 14:59:32.340104: done, saving took 4.35 seconds 
2022-04-06 14:59:32.351872: This epoch took 135.985627 s
 
2022-04-06 14:59:32.353055: svmem(total=99932160000, available=86594457600, percent=13.3, used=10298834944, free=73743974400, active=20940091392, inactive=2777997312, buffers=12353536, cached=15876997120, shared=1225445376, slab=796971008) 
2022-04-06 14:59:32.353751: 
epoch:  18 
2022-04-06 15:01:21.634749: train loss : 0.0973 
2022-04-06 15:01:33.291465: validation loss: 0.0433 
2022-04-06 15:01:33.296604: Average global foreground Dice: [0.5894, 0.8052, 0.7671, 0.0001] 
2022-04-06 15:01:33.301713: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:01:34.229930: lr: 0.009657 
2022-04-06 15:01:34.248092: saving scheduled checkpoint file... 
2022-04-06 15:01:34.362571: saving checkpoint... 
2022-04-06 15:01:38.788244: done, saving took 4.54 seconds 
2022-04-06 15:01:38.815344: done 
2022-04-06 15:01:38.882746: saving checkpoint... 
2022-04-06 15:01:43.189334: done, saving took 4.37 seconds 
2022-04-06 15:01:43.204309: This epoch took 130.849971 s
 
2022-04-06 15:01:43.205668: svmem(total=99932160000, available=86578368512, percent=13.4, used=10314915840, free=73736413184, active=20447109120, inactive=3273441280, buffers=12771328, cached=15868059648, shared=1225445376, slab=797503488) 
2022-04-06 15:01:43.206465: 
epoch:  19 
2022-04-06 15:03:31.431063: train loss : 0.0473 
2022-04-06 15:03:42.737065: validation loss: 0.1026 
2022-04-06 15:03:42.746343: Average global foreground Dice: [0.5247, 0.7853, 0.7595, 0.0] 
2022-04-06 15:03:42.760619: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:03:43.406732: lr: 0.009639 
2022-04-06 15:03:43.480318: saving scheduled checkpoint file... 
2022-04-06 15:03:43.594388: saving checkpoint... 
2022-04-06 15:03:49.971860: done, saving took 6.46 seconds 
2022-04-06 15:03:50.069945: done 
2022-04-06 15:03:50.137378: saving checkpoint... 
2022-04-06 15:03:57.387520: done, saving took 7.30 seconds 
2022-04-06 15:03:57.482587: This epoch took 134.275345 s
 
2022-04-06 15:03:57.549362: svmem(total=99932160000, available=86773096448, percent=13.2, used=10168287232, free=73753157632, active=20526305280, inactive=3183292416, buffers=13025280, cached=15997689856, shared=1176498176, slab=797364224) 
2022-04-06 15:03:57.601738: 
epoch:  20 
2022-04-06 15:05:47.327485: train loss : 0.0579 
2022-04-06 15:05:58.610134: validation loss: 0.0166 
2022-04-06 15:05:58.616675: Average global foreground Dice: [0.6095, 0.8001, 0.8106, 0.0006] 
2022-04-06 15:05:58.630726: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:05:59.541594: lr: 0.009621 
2022-04-06 15:05:59.560081: saving scheduled checkpoint file... 
2022-04-06 15:05:59.678298: saving checkpoint... 
2022-04-06 15:06:04.205594: done, saving took 4.63 seconds 
2022-04-06 15:06:04.230231: done 
2022-04-06 15:06:04.285875: saving checkpoint... 
2022-04-06 15:06:08.700453: done, saving took 4.46 seconds 
2022-04-06 15:06:08.715543: This epoch took 131.090993 s
 
2022-04-06 15:06:08.717498: svmem(total=99932160000, available=86629105664, percent=13.3, used=10263928832, free=73738473472, active=20572364800, inactive=3153768448, buffers=13303808, cached=15916453888, shared=1225445376, slab=797315072) 
2022-04-06 15:06:08.719442: 
epoch:  21 
2022-04-06 15:07:56.515209: train loss : -0.0185 
2022-04-06 15:08:08.273438: validation loss: 0.1058 
2022-04-06 15:08:08.280650: Average global foreground Dice: [0.3782, 0.8024, 0.7104, 0.0011] 
2022-04-06 15:08:08.287001: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:08:09.226921: lr: 0.009603 
2022-04-06 15:08:09.227846: saving scheduled checkpoint file... 
2022-04-06 15:08:09.341207: saving checkpoint... 
2022-04-06 15:08:13.928297: done, saving took 4.70 seconds 
2022-04-06 15:08:13.955927: done 
2022-04-06 15:08:14.028955: saving checkpoint... 
2022-04-06 15:08:18.470198: done, saving took 4.51 seconds 
2022-04-06 15:08:18.487488: This epoch took 129.765443 s
 
2022-04-06 15:08:18.488616: svmem(total=99932160000, available=87535960064, percent=12.4, used=9450385408, free=74123411456, active=20450861056, inactive=3257720832, buffers=13881344, cached=16344481792, shared=1133883392, slab=766599168) 
2022-04-06 15:08:18.489283: 
epoch:  22 
2022-04-06 15:10:05.645003: train loss : 0.0569 
2022-04-06 15:10:18.541293: validation loss: 0.3565 
2022-04-06 15:10:18.543352: Average global foreground Dice: [0.537, 0.7603, 0.5604, 0.0] 
2022-04-06 15:10:18.544239: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:10:19.334642: lr: 0.009585 
2022-04-06 15:10:19.335653: saving scheduled checkpoint file... 
2022-04-06 15:10:19.448321: saving checkpoint... 
2022-04-06 15:10:23.833777: done, saving took 4.50 seconds 
2022-04-06 15:10:23.853414: done 
2022-04-06 15:10:23.935864: saving checkpoint... 
2022-04-06 15:10:28.370416: done, saving took 4.52 seconds 
2022-04-06 15:10:28.387046: This epoch took 129.897027 s
 
2022-04-06 15:10:28.387970: svmem(total=99932160000, available=87435931648, percent=12.5, used=9550278656, free=74162122752, active=20334186496, inactive=3338653696, buffers=14114816, cached=16205643776, shared=1133883392, slab=765841408) 
2022-04-06 15:10:28.388762: 
epoch:  23 
2022-04-06 15:12:17.859757: train loss : 0.1359 
2022-04-06 15:12:29.762707: validation loss: -0.0318 
2022-04-06 15:12:29.771945: Average global foreground Dice: [0.6066, 0.8017, 0.8253, 0.0] 
2022-04-06 15:12:29.772874: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:12:30.627337: lr: 0.009567 
2022-04-06 15:12:30.628345: saving scheduled checkpoint file... 
2022-04-06 15:12:30.736464: saving checkpoint... 
2022-04-06 15:12:35.106084: done, saving took 4.48 seconds 
2022-04-06 15:12:35.122589: done 
2022-04-06 15:12:35.170029: saving checkpoint... 
2022-04-06 15:12:39.435382: done, saving took 4.31 seconds 
2022-04-06 15:12:39.444248: This epoch took 131.054838 s
 
2022-04-06 15:12:39.445195: svmem(total=99932160000, available=87496790016, percent=12.4, used=9489416192, free=74063126528, active=20538433536, inactive=3229544448, buffers=14376960, cached=16365240320, shared=1133883392, slab=765829120) 
2022-04-06 15:12:39.445986: 
epoch:  24 
2022-04-06 15:14:28.529924: train loss : -0.0536 
2022-04-06 15:14:39.239090: validation loss: -0.0716 
2022-04-06 15:14:39.243906: Average global foreground Dice: [0.5904, 0.8245, 0.8498, 0.0009] 
2022-04-06 15:14:39.248944: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:14:40.199308: lr: 0.009549 
2022-04-06 15:14:40.201372: saving scheduled checkpoint file... 
2022-04-06 15:14:40.269091: saving checkpoint... 
2022-04-06 15:14:44.868358: done, saving took 4.66 seconds 
2022-04-06 15:14:44.880988: done 
2022-04-06 15:14:44.928163: saving checkpoint... 
2022-04-06 15:14:49.322922: done, saving took 4.44 seconds 
2022-04-06 15:14:49.336800: This epoch took 129.890175 s
 
2022-04-06 15:14:49.337806: svmem(total=99932160000, available=87595573248, percent=12.3, used=9390546944, free=74092281856, active=20477235200, inactive=3264184320, buffers=14622720, cached=16434708480, shared=1133887488, slab=765304832) 
2022-04-06 15:14:49.338612: 
epoch:  25 
2022-04-06 15:16:37.268696: train loss : -0.0271 
2022-04-06 15:16:48.680750: validation loss: -0.0176 
2022-04-06 15:16:48.682501: Average global foreground Dice: [0.5777, 0.8267, 0.7886, 0.0032] 
2022-04-06 15:16:48.683357: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:16:49.549498: lr: 0.009531 
2022-04-06 15:16:49.556762: saving scheduled checkpoint file... 
2022-04-06 15:16:49.663997: saving checkpoint... 
2022-04-06 15:16:54.134558: done, saving took 4.57 seconds 
2022-04-06 15:16:54.150633: done 
2022-04-06 15:16:54.199514: saving checkpoint... 
2022-04-06 15:16:58.608826: done, saving took 4.46 seconds 
2022-04-06 15:16:58.623224: This epoch took 129.283886 s
 
2022-04-06 15:16:58.624509: svmem(total=99932160000, available=87539970048, percent=12.4, used=9446019072, free=74065530880, active=20454195200, inactive=3313397760, buffers=14888960, cached=16405721088, shared=1133887488, slab=766132224) 
2022-04-06 15:16:58.625300: 
epoch:  26 
2022-04-06 15:18:47.655361: train loss : -0.0121 
2022-04-06 15:18:59.007067: validation loss: -0.0421 
2022-04-06 15:18:59.010815: Average global foreground Dice: [0.6283, 0.8111, 0.8, 0.0728] 
2022-04-06 15:18:59.016184: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:18:59.722628: lr: 0.009513 
2022-04-06 15:18:59.725686: saving scheduled checkpoint file... 
2022-04-06 15:18:59.909204: saving checkpoint... 
2022-04-06 15:19:04.453816: done, saving took 4.72 seconds 
2022-04-06 15:19:04.484860: done 
2022-04-06 15:19:04.547291: saving checkpoint... 
2022-04-06 15:19:09.011129: done, saving took 4.52 seconds 
2022-04-06 15:19:09.029353: This epoch took 130.403113 s
 
2022-04-06 15:19:09.031838: svmem(total=99932160000, available=87506092032, percent=12.4, used=9479761920, free=74120241152, active=20408705024, inactive=3303919616, buffers=15159296, cached=16316997632, shared=1133887488, slab=765624320) 
2022-04-06 15:19:09.033412: 
epoch:  27 
2022-04-06 15:20:56.668925: train loss : -0.0172 
2022-04-06 15:21:08.417027: validation loss: -0.0674 
2022-04-06 15:21:08.418752: Average global foreground Dice: [0.5739, 0.8258, 0.8435, 0.0654] 
2022-04-06 15:21:08.419567: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:21:09.502620: lr: 0.009495 
2022-04-06 15:21:09.510316: saving scheduled checkpoint file... 
2022-04-06 15:21:09.611925: saving checkpoint... 
2022-04-06 15:21:14.029817: done, saving took 4.51 seconds 
2022-04-06 15:21:14.052730: done 
2022-04-06 15:21:14.145559: saving checkpoint... 
2022-04-06 15:21:18.559893: done, saving took 4.50 seconds 
2022-04-06 15:21:18.579088: This epoch took 129.543356 s
 
2022-04-06 15:21:18.583172: svmem(total=99932160000, available=87485960192, percent=12.5, used=9499787264, free=74142806016, active=20214898688, inactive=3473973248, buffers=15405056, cached=16274161664, shared=1133887488, slab=766210048) 
2022-04-06 15:21:18.583966: 
epoch:  28 
2022-04-06 15:23:08.110628: train loss : -0.0905 
2022-04-06 15:23:19.879542: validation loss: -0.1168 
2022-04-06 15:23:19.888243: Average global foreground Dice: [0.5563, 0.861, 0.8664, 0.0971] 
2022-04-06 15:23:19.889046: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:23:20.761429: lr: 0.009476 
2022-04-06 15:23:20.770470: saving scheduled checkpoint file... 
2022-04-06 15:23:20.985729: saving checkpoint... 
2022-04-06 15:23:25.420639: done, saving took 4.64 seconds 
2022-04-06 15:23:25.452471: done 
2022-04-06 15:23:25.552691: saving checkpoint... 
2022-04-06 15:23:29.974557: done, saving took 4.52 seconds 
2022-04-06 15:23:29.993533: This epoch took 131.408862 s
 
2022-04-06 15:23:29.994519: svmem(total=99932160000, available=87515815936, percent=12.4, used=9470386176, free=74175438848, active=20244267008, inactive=3412291584, buffers=15663104, cached=16270671872, shared=1133887488, slab=765874176) 
2022-04-06 15:23:29.995129: 
epoch:  29 
2022-04-06 15:25:18.830500: train loss : 0.0165 
2022-04-06 15:25:30.450658: validation loss: -0.0572 
2022-04-06 15:25:30.471036: Average global foreground Dice: [0.6039, 0.8382, 0.8211, 0.0509] 
2022-04-06 15:25:30.487189: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:25:31.677994: lr: 0.009458 
2022-04-06 15:25:31.683689: saving scheduled checkpoint file... 
2022-04-06 15:25:31.803942: saving checkpoint... 
2022-04-06 15:25:36.141720: done, saving took 4.45 seconds 
2022-04-06 15:25:36.163050: done 
2022-04-06 15:25:36.217263: saving checkpoint... 
2022-04-06 15:25:40.511489: done, saving took 4.34 seconds 
2022-04-06 15:25:40.523965: This epoch took 130.528276 s
 
2022-04-06 15:25:40.525148: svmem(total=99932160000, available=87443308544, percent=12.5, used=9543290880, free=74052337664, active=20270514176, inactive=3504017408, buffers=16089088, cached=16320442368, shared=1133887488, slab=765571072) 
2022-04-06 15:25:40.525936: 
epoch:  30 
2022-04-06 15:27:28.735978: train loss : -0.0549 
2022-04-06 15:27:40.598015: validation loss: -0.1198 
2022-04-06 15:27:40.600395: Average global foreground Dice: [0.5356, 0.8533, 0.8552, 0.0969] 
2022-04-06 15:27:40.601225: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:27:41.547796: lr: 0.00944 
2022-04-06 15:27:41.555821: saving scheduled checkpoint file... 
2022-04-06 15:27:41.613752: saving checkpoint... 
2022-04-06 15:27:46.126518: done, saving took 4.56 seconds 
2022-04-06 15:27:46.150975: done 
2022-04-06 15:27:46.202005: saving checkpoint... 
2022-04-06 15:27:50.487047: done, saving took 4.33 seconds 
2022-04-06 15:27:50.501052: This epoch took 129.974219 s
 
2022-04-06 15:27:50.502103: svmem(total=99932160000, available=87421267968, percent=12.5, used=9564930048, free=74054705152, active=20385124352, inactive=3390230528, buffers=16605184, cached=16295919616, shared=1133887488, slab=766722048) 
2022-04-06 15:27:50.502842: 
epoch:  31 
2022-04-06 15:29:38.407191: train loss : -0.0873 
2022-04-06 15:29:50.622661: validation loss: -0.1587 
2022-04-06 15:29:50.631755: Average global foreground Dice: [0.6072, 0.8779, 0.8349, 0.151] 
2022-04-06 15:29:50.635615: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:29:51.320550: lr: 0.009422 
2022-04-06 15:29:51.326226: saving scheduled checkpoint file... 
2022-04-06 15:29:51.474426: saving checkpoint... 
2022-04-06 15:29:55.924093: done, saving took 4.59 seconds 
2022-04-06 15:29:55.979302: done 
2022-04-06 15:29:56.039997: saving checkpoint... 
2022-04-06 15:30:00.637981: done, saving took 4.65 seconds 
2022-04-06 15:30:00.648593: This epoch took 130.145110 s
 
2022-04-06 15:30:00.649802: svmem(total=99932160000, available=87586291712, percent=12.4, used=9399775232, free=74054709248, active=20092145664, inactive=3684712448, buffers=16842752, cached=16460832768, shared=1133887488, slab=766484480) 
2022-04-06 15:30:00.650598: 
epoch:  32 
2022-04-06 15:31:49.200405: train loss : -0.1512 
2022-04-06 15:32:00.681524: validation loss: -0.1417 
2022-04-06 15:32:00.700305: Average global foreground Dice: [0.6513, 0.8021, 0.8827, 0.1582] 
2022-04-06 15:32:00.738070: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:32:01.515154: lr: 0.009404 
2022-04-06 15:32:01.519407: saving scheduled checkpoint file... 
2022-04-06 15:32:01.601635: saving checkpoint... 
2022-04-06 15:32:06.051074: done, saving took 4.53 seconds 
2022-04-06 15:32:06.080117: done 
2022-04-06 15:32:06.134884: saving checkpoint... 
2022-04-06 15:32:10.565454: done, saving took 4.48 seconds 
2022-04-06 15:32:10.575494: This epoch took 129.924047 s
 
2022-04-06 15:32:10.576585: svmem(total=99932160000, available=87475105792, percent=12.5, used=9510735872, free=74062299136, active=20491235328, inactive=3281190912, buffers=17121280, cached=16342003712, shared=1133899776, slab=766156800) 
2022-04-06 15:32:10.577333: 
epoch:  33 
2022-04-06 15:33:58.612781: train loss : -0.1262 
2022-04-06 15:34:09.842885: validation loss: -0.1399 
2022-04-06 15:34:09.855988: Average global foreground Dice: [0.6091, 0.832, 0.8962, 0.1558] 
2022-04-06 15:34:09.859551: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:34:10.703794: lr: 0.009386 
2022-04-06 15:34:10.711205: saving scheduled checkpoint file... 
2022-04-06 15:34:10.810263: saving checkpoint... 
2022-04-06 15:34:15.333261: done, saving took 4.61 seconds 
2022-04-06 15:34:15.367922: done 
2022-04-06 15:34:15.427445: saving checkpoint... 
2022-04-06 15:34:19.866163: done, saving took 4.49 seconds 
2022-04-06 15:34:19.883073: This epoch took 129.304923 s
 
2022-04-06 15:34:19.884525: svmem(total=99932160000, available=87502725120, percent=12.4, used=9483235328, free=74110488576, active=20421419008, inactive=3298357248, buffers=17367040, cached=16321069056, shared=1133903872, slab=766013440) 
2022-04-06 15:34:19.885432: 
epoch:  34 
2022-04-06 15:36:07.362203: train loss : -0.2038 
2022-04-06 15:36:19.625001: validation loss: -0.2383 
2022-04-06 15:36:19.626920: Average global foreground Dice: [0.6309, 0.8958, 0.9073, 0.1751] 
2022-04-06 15:36:19.627894: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:36:20.523245: lr: 0.009368 
2022-04-06 15:36:20.528732: saving scheduled checkpoint file... 
2022-04-06 15:36:20.684874: saving checkpoint... 
2022-04-06 15:36:25.188313: done, saving took 4.65 seconds 
2022-04-06 15:36:25.206907: done 
2022-04-06 15:36:25.291362: saving checkpoint... 
2022-04-06 15:36:29.706670: done, saving took 4.50 seconds 
2022-04-06 15:36:29.722444: This epoch took 129.836122 s
 
2022-04-06 15:36:29.723456: svmem(total=99932160000, available=87470292992, percent=12.5, used=9515671552, free=74139095040, active=20130291712, inactive=3562131456, buffers=17629184, cached=16259764224, shared=1133903872, slab=766320640) 
2022-04-06 15:36:29.723986: 
epoch:  35 
2022-04-06 15:38:19.468862: train loss : -0.1873 
2022-04-06 15:38:33.051436: validation loss: -0.1917 
2022-04-06 15:38:33.062283: Average global foreground Dice: [0.6338, 0.856, 0.8875, 0.177] 
2022-04-06 15:38:33.072176: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:38:34.262336: lr: 0.00935 
2022-04-06 15:38:34.272126: saving scheduled checkpoint file... 
2022-04-06 15:38:34.383234: saving checkpoint... 
2022-04-06 15:38:38.825832: done, saving took 4.55 seconds 
2022-04-06 15:38:38.847566: done 
2022-04-06 15:38:38.899974: saving checkpoint... 
2022-04-06 15:38:43.315505: done, saving took 4.46 seconds 
2022-04-06 15:38:43.327580: This epoch took 133.603079 s
 
2022-04-06 15:38:43.328674: svmem(total=99932160000, available=87555342336, percent=12.4, used=9430614016, free=74058248192, active=20411494400, inactive=3361644544, buffers=18444288, cached=16424853504, shared=1133903872, slab=765788160) 
2022-04-06 15:38:43.329432: 
epoch:  36 
2022-04-06 15:40:30.788746: train loss : -0.0833 
2022-04-06 15:40:43.428268: validation loss: -0.0743 
2022-04-06 15:40:43.509638: Average global foreground Dice: [0.623, 0.8405, 0.8483, 0.1554] 
2022-04-06 15:40:43.516215: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:40:44.069674: lr: 0.009331 
2022-04-06 15:40:44.086210: saving scheduled checkpoint file... 
2022-04-06 15:40:44.245771: saving checkpoint... 
2022-04-06 15:40:49.272305: done, saving took 5.17 seconds 
2022-04-06 15:40:49.338322: done 
2022-04-06 15:40:49.418431: saving checkpoint... 
2022-04-06 15:40:54.851121: done, saving took 5.48 seconds 
2022-04-06 15:40:54.867651: This epoch took 131.537613 s
 
2022-04-06 15:40:54.869009: svmem(total=99932160000, available=87560540160, percent=12.4, used=9450336256, free=74048020480, active=20233891840, inactive=3548344320, buffers=18681856, cached=16415121408, shared=1109430272, slab=766586880) 
2022-04-06 15:40:54.869935: 
epoch:  37 
2022-04-06 15:42:42.902884: train loss : -0.0892 
2022-04-06 15:42:56.254396: validation loss: -0.1830 
2022-04-06 15:42:56.258992: Average global foreground Dice: [0.5443, 0.8676, 0.8617, 0.1726] 
2022-04-06 15:42:56.260009: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:42:57.026695: lr: 0.009313 
2022-04-06 15:42:57.027582: saving scheduled checkpoint file... 
2022-04-06 15:42:57.159409: saving checkpoint... 
2022-04-06 15:43:01.741665: done, saving took 4.71 seconds 
2022-04-06 15:43:01.764977: done 
2022-04-06 15:43:01.838607: saving checkpoint... 
2022-04-06 15:43:06.250470: done, saving took 4.48 seconds 
2022-04-06 15:43:06.262242: This epoch took 131.391631 s
 
2022-04-06 15:43:06.263303: svmem(total=99932160000, available=87689863168, percent=12.3, used=9296023552, free=74111463424, active=20339195904, inactive=3377209344, buffers=18935808, cached=16505737216, shared=1133903872, slab=766808064) 
2022-04-06 15:43:06.263993: 
epoch:  38 
2022-04-06 15:44:54.589075: train loss : -0.2211 
2022-04-06 15:45:06.356269: validation loss: -0.2015 
2022-04-06 15:45:06.358199: Average global foreground Dice: [0.6404, 0.8543, 0.8694, 0.1909] 
2022-04-06 15:45:06.360055: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:45:07.154062: lr: 0.009295 
2022-04-06 15:45:07.160695: saving scheduled checkpoint file... 
2022-04-06 15:45:07.313551: saving checkpoint... 
2022-04-06 15:45:11.459529: done, saving took 4.30 seconds 
2022-04-06 15:45:11.493068: done 
2022-04-06 15:45:11.586285: saving checkpoint... 
2022-04-06 15:45:16.016713: done, saving took 4.52 seconds 
2022-04-06 15:45:16.044900: This epoch took 129.780206 s
 
2022-04-06 15:45:16.047625: svmem(total=99932160000, available=87629053952, percent=12.3, used=9381056512, free=74134097920, active=20360310784, inactive=3332427776, buffers=19464192, cached=16397541376, shared=1109430272, slab=766910464) 
2022-04-06 15:45:16.051926: 
epoch:  39 
2022-04-06 15:47:06.623165: train loss : -0.2483 
2022-04-06 15:47:17.772788: validation loss: -0.2416 
2022-04-06 15:47:17.791945: Average global foreground Dice: [0.6858, 0.8729, 0.8949, 0.2292] 
2022-04-06 15:47:17.793092: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:47:18.538674: lr: 0.009277 
2022-04-06 15:47:18.547544: saving scheduled checkpoint file... 
2022-04-06 15:47:18.696165: saving checkpoint... 
2022-04-06 15:47:23.309440: done, saving took 4.76 seconds 
2022-04-06 15:47:23.361075: done 
2022-04-06 15:47:23.468535: saving checkpoint... 
2022-04-06 15:47:27.903739: done, saving took 4.54 seconds 
2022-04-06 15:47:27.935635: This epoch took 131.879835 s
 
2022-04-06 15:47:27.937516: svmem(total=99932160000, available=87609274368, percent=12.3, used=9376526336, free=74167189504, active=20387713024, inactive=3275608064, buffers=19709952, cached=16368734208, shared=1133903872, slab=766410752) 
2022-04-06 15:47:27.939520: 
epoch:  40 
2022-04-06 15:49:17.198308: train loss : -0.2732 
2022-04-06 15:49:28.988223: validation loss: -0.2541 
2022-04-06 15:49:28.989936: Average global foreground Dice: [0.6373, 0.8808, 0.9054, 0.2367] 
2022-04-06 15:49:28.990689: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:49:30.132762: lr: 0.009259 
2022-04-06 15:49:30.133773: saving scheduled checkpoint file... 
2022-04-06 15:49:30.279195: saving checkpoint... 
2022-04-06 15:49:34.893802: done, saving took 4.76 seconds 
2022-04-06 15:49:34.923196: done 
2022-04-06 15:49:35.032087: saving checkpoint... 
2022-04-06 15:49:39.480852: done, saving took 4.55 seconds 
2022-04-06 15:49:39.499840: This epoch took 131.558972 s
 
2022-04-06 15:49:39.500841: svmem(total=99932160000, available=87528566784, percent=12.4, used=9457311744, free=74188374016, active=20295188480, inactive=3347410944, buffers=20107264, cached=16266366976, shared=1133903872, slab=765923328) 
2022-04-06 15:49:39.501462: 
epoch:  41 
2022-04-06 15:51:29.201715: train loss : -0.2784 
2022-04-06 15:51:41.876386: validation loss: -0.3449 
2022-04-06 15:51:41.904361: Average global foreground Dice: [0.6457, 0.914, 0.9382, 0.2235] 
2022-04-06 15:51:41.909381: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:51:42.463431: lr: 0.009241 
2022-04-06 15:51:42.473294: saving scheduled checkpoint file... 
2022-04-06 15:51:42.600062: saving checkpoint... 
2022-04-06 15:51:47.547818: done, saving took 5.07 seconds 
2022-04-06 15:51:47.593257: done 
2022-04-06 15:51:47.676536: saving checkpoint... 
2022-04-06 15:51:52.159744: done, saving took 4.56 seconds 
2022-04-06 15:51:52.217205: This epoch took 132.715224 s
 
2022-04-06 15:51:52.242658: svmem(total=99932160000, available=87672766464, percent=12.3, used=9368891392, free=86957494272, active=7640817664, inactive=3297542144, buffers=5599232, cached=3600175104, shared=1109430272, slab=703524864) 
2022-04-06 15:51:52.264593: 
epoch:  42 
2022-04-06 15:53:40.362371: train loss : -0.1922 
2022-04-06 15:53:51.801096: validation loss: -0.1309 
2022-04-06 15:53:51.808953: Average global foreground Dice: [0.6397, 0.8464, 0.8599, 0.2059] 
2022-04-06 15:53:51.811622: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:53:52.560269: lr: 0.009223 
2022-04-06 15:53:52.593275: saving scheduled checkpoint file... 
2022-04-06 15:53:52.686327: saving checkpoint... 
2022-04-06 15:53:57.153586: done, saving took 4.54 seconds 
2022-04-06 15:53:57.195134: done 
2022-04-06 15:53:57.265365: saving checkpoint... 
2022-04-06 15:54:01.811924: done, saving took 4.60 seconds 
2022-04-06 15:54:01.827776: This epoch took 129.537461 s
 
2022-04-06 15:54:01.829749: svmem(total=99932160000, available=87075721216, percent=12.9, used=9924681728, free=75270160384, active=12079407104, inactive=10488184832, buffers=9224192, cached=14728093696, shared=1133907968, slab=741613568) 
2022-04-06 15:54:01.831727: 
epoch:  43 
2022-04-06 15:55:50.438217: train loss : -0.2115 
2022-04-06 15:56:02.347041: validation loss: -0.2736 
2022-04-06 15:56:02.350381: Average global foreground Dice: [0.6163, 0.8855, 0.9174, 0.254] 
2022-04-06 15:56:02.353457: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:56:03.293473: lr: 0.009204 
2022-04-06 15:56:03.295230: saving scheduled checkpoint file... 
2022-04-06 15:56:03.389714: saving checkpoint... 
2022-04-06 15:56:07.818164: done, saving took 4.52 seconds 
2022-04-06 15:56:07.839532: done 
2022-04-06 15:56:07.892113: saving checkpoint... 
2022-04-06 15:56:12.285558: done, saving took 4.44 seconds 
2022-04-06 15:56:12.295975: This epoch took 130.460483 s
 
2022-04-06 15:56:12.296989: svmem(total=99932160000, available=84176302080, percent=15.8, used=12802605056, free=70129061888, active=19384045568, inactive=8211521536, buffers=9637888, cached=16990855168, shared=1148723200, slab=762912768) 
2022-04-06 15:56:12.297678: 
epoch:  44 
2022-04-06 15:57:59.633844: train loss : -0.2434 
2022-04-06 15:58:11.277022: validation loss: -0.2639 
2022-04-06 15:58:11.286393: Average global foreground Dice: [0.6552, 0.8962, 0.9112, 0.2448] 
2022-04-06 15:58:11.290372: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 15:58:12.058690: lr: 0.009186 
2022-04-06 15:58:12.065137: saving scheduled checkpoint file... 
2022-04-06 15:58:12.138282: saving checkpoint... 
2022-04-06 15:58:16.696681: done, saving took 4.62 seconds 
2022-04-06 15:58:16.720409: done 
2022-04-06 15:58:16.797262: saving checkpoint... 
2022-04-06 15:58:21.230582: done, saving took 4.51 seconds 
2022-04-06 15:58:21.249858: This epoch took 128.951491 s
 
2022-04-06 15:58:21.252521: svmem(total=99932160000, available=87555092480, percent=12.4, used=9435045888, free=73437622272, active=19171467264, inactive=5223759872, buffers=9900032, cached=17049591808, shared=1133903872, slab=761659392) 
2022-04-06 15:58:21.254678: 
epoch:  45 
2022-04-06 16:00:08.114596: train loss : -0.2864 
2022-04-06 16:00:30.447572: validation loss: -0.2752 
2022-04-06 16:00:30.452591: Average global foreground Dice: [0.6958, 0.8849, 0.9043, 0.2502] 
2022-04-06 16:00:30.456555: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:00:31.405602: lr: 0.009168 
2022-04-06 16:00:31.424650: saving scheduled checkpoint file... 
2022-04-06 16:00:31.558953: saving checkpoint... 
2022-04-06 16:00:36.127264: done, saving took 4.68 seconds 
2022-04-06 16:00:36.145144: done 
2022-04-06 16:00:36.232006: saving checkpoint... 
2022-04-06 16:00:40.616268: done, saving took 4.47 seconds 
2022-04-06 16:00:40.633489: This epoch took 139.378130 s
 
2022-04-06 16:00:40.634495: svmem(total=99932160000, available=87569076224, percent=12.4, used=9419718656, free=73477615616, active=20564254720, inactive=3789795328, buffers=10170368, cached=17024655360, shared=1133903872, slab=763691008) 
2022-04-06 16:00:40.635222: 
epoch:  46 
2022-04-06 16:02:29.991377: train loss : -0.2845 
2022-04-06 16:02:41.051445: validation loss: -0.3341 
2022-04-06 16:02:41.060916: Average global foreground Dice: [0.6677, 0.9046, 0.9397, 0.254] 
2022-04-06 16:02:41.068288: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:02:41.940175: lr: 0.00915 
2022-04-06 16:02:41.944700: saving scheduled checkpoint file... 
2022-04-06 16:02:42.094454: saving checkpoint... 
2022-04-06 16:02:46.746428: done, saving took 4.79 seconds 
2022-04-06 16:02:46.785037: done 
2022-04-06 16:02:46.910460: saving checkpoint... 
2022-04-06 16:02:51.344024: done, saving took 4.55 seconds 
2022-04-06 16:02:51.362042: This epoch took 130.726145 s
 
2022-04-06 16:02:51.363109: svmem(total=99932160000, available=87572975616, percent=12.4, used=9414586368, free=73514979328, active=20594819072, inactive=3718606848, buffers=10440704, cached=16992153600, shared=1133903872, slab=766758912) 
2022-04-06 16:02:51.363818: 
epoch:  47 
2022-04-06 16:04:40.335414: train loss : -0.1876 
2022-04-06 16:04:54.449038: validation loss: -0.1070 
2022-04-06 16:04:54.540452: Average global foreground Dice: [0.6169, 0.8101, 0.8168, 0.2266] 
2022-04-06 16:04:54.547723: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:04:55.413248: lr: 0.009132 
2022-04-06 16:04:55.420030: saving scheduled checkpoint file... 
2022-04-06 16:04:55.517781: saving checkpoint... 
2022-04-06 16:04:59.891937: done, saving took 4.47 seconds 
2022-04-06 16:04:59.907788: done 
2022-04-06 16:04:59.910055: This epoch took 128.545502 s
 
2022-04-06 16:04:59.913336: svmem(total=99932160000, available=87558172672, percent=12.4, used=9453133824, free=73378807808, active=20766269440, inactive=3680538624, buffers=10678272, cached=17089540096, shared=1109434368, slab=768471040) 
2022-04-06 16:04:59.916432: 
epoch:  48 
2022-04-06 16:06:48.091359: train loss : -0.1827 
2022-04-06 16:06:59.987175: validation loss: -0.2277 
2022-04-06 16:06:59.991188: Average global foreground Dice: [0.4191, 0.8757, 0.9355, 0.2368] 
2022-04-06 16:06:59.992108: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:07:00.858171: lr: 0.009114 
2022-04-06 16:07:00.862668: saving scheduled checkpoint file... 
2022-04-06 16:07:00.940593: saving checkpoint... 
2022-04-06 16:07:05.577504: done, saving took 4.71 seconds 
2022-04-06 16:07:05.608191: done 
2022-04-06 16:07:05.617837: This epoch took 125.700360 s
 
2022-04-06 16:07:05.623296: svmem(total=99932160000, available=87622955008, percent=12.3, used=9387618304, free=73381769216, active=20703748096, inactive=3742818304, buffers=10940416, cached=17151832064, shared=1109434368, slab=768188416) 
2022-04-06 16:07:05.629572: 
epoch:  49 
2022-04-06 16:08:56.151343: train loss : -0.2407 
2022-04-06 16:09:07.507358: validation loss: -0.2428 
2022-04-06 16:09:07.519473: Average global foreground Dice: [0.6446, 0.8923, 0.8872, 0.2154] 
2022-04-06 16:09:07.545160: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:09:08.490144: lr: 0.009095 
2022-04-06 16:09:08.491010: saving scheduled checkpoint file... 
2022-04-06 16:09:08.588650: saving checkpoint... 
2022-04-06 16:09:13.101039: done, saving took 4.61 seconds 
2022-04-06 16:09:13.117008: done 
2022-04-06 16:09:13.118992: This epoch took 127.480517 s
 
2022-04-06 16:09:13.121407: svmem(total=99932160000, available=87485861888, percent=12.5, used=9514020864, free=73377566720, active=20666445824, inactive=3780673536, buffers=11177984, cached=17029394432, shared=1119842304, slab=768864256) 
2022-04-06 16:09:13.123025: 
epoch:  50 
2022-04-06 16:11:00.834083: train loss : -0.2177 
2022-04-06 16:11:13.442841: validation loss: -0.2370 
2022-04-06 16:11:13.451854: Average global foreground Dice: [0.6605, 0.8752, 0.8892, 0.2404] 
2022-04-06 16:11:13.452713: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:11:14.456796: lr: 0.009077 
2022-04-06 16:11:14.468757: saving scheduled checkpoint file... 
2022-04-06 16:11:14.537761: saving checkpoint... 
2022-04-06 16:11:18.960899: done, saving took 4.48 seconds 
2022-04-06 16:11:18.979099: done 
2022-04-06 16:11:19.055273: saving checkpoint... 
2022-04-06 16:11:23.448478: done, saving took 4.47 seconds 
2022-04-06 16:11:23.464178: This epoch took 130.340124 s
 
2022-04-06 16:11:23.465458: svmem(total=99932160000, available=87529967616, percent=12.4, used=9455779840, free=73451016192, active=20633272320, inactive=3739619328, buffers=11431936, cached=17013932032, shared=1133907968, slab=769228800) 
2022-04-06 16:11:23.466250: 
epoch:  51 
2022-04-06 16:13:11.132603: train loss : -0.2740 
2022-04-06 16:13:21.824789: validation loss: -0.3640 
2022-04-06 16:13:21.830060: Average global foreground Dice: [0.7164, 0.8879, 0.9485, 0.2915] 
2022-04-06 16:13:21.833820: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:13:22.376002: lr: 0.009059 
2022-04-06 16:13:22.379186: saving scheduled checkpoint file... 
2022-04-06 16:13:22.495797: saving checkpoint... 
2022-04-06 16:13:26.890283: done, saving took 4.51 seconds 
2022-04-06 16:13:26.943562: done 
2022-04-06 16:13:27.042297: saving checkpoint... 
2022-04-06 16:13:31.598120: done, saving took 4.64 seconds 
2022-04-06 16:13:31.646202: This epoch took 128.179302 s
 
2022-04-06 16:13:31.656816: svmem(total=99932160000, available=87609323520, percent=12.3, used=9375305728, free=73473122304, active=20607283200, inactive=3744579584, buffers=11669504, cached=17072062464, shared=1133907968, slab=769265664) 
2022-04-06 16:13:31.672706: 
epoch:  52 
2022-04-06 16:15:21.763457: train loss : -0.3293 
2022-04-06 16:15:33.133702: validation loss: -0.2993 
2022-04-06 16:15:33.146464: Average global foreground Dice: [0.691, 0.906, 0.9165, 0.2411] 
2022-04-06 16:15:33.149278: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:15:34.040981: lr: 0.009041 
2022-04-06 16:15:34.056039: saving scheduled checkpoint file... 
2022-04-06 16:15:34.211930: saving checkpoint... 
2022-04-06 16:15:38.707332: done, saving took 4.65 seconds 
2022-04-06 16:15:38.737950: done 
2022-04-06 16:15:38.834030: saving checkpoint... 
2022-04-06 16:15:43.271761: done, saving took 4.53 seconds 
2022-04-06 16:15:43.285919: This epoch took 131.601568 s
 
2022-04-06 16:15:43.286993: svmem(total=99932160000, available=87487283200, percent=12.5, used=9498050560, free=73506123776, active=20609507328, inactive=3709276160, buffers=11964416, cached=16916021248, shared=1133916160, slab=769286144) 
2022-04-06 16:15:43.287677: 
epoch:  53 
2022-04-06 16:17:33.742129: train loss : -0.3029 
2022-04-06 16:17:46.400980: validation loss: -0.3391 
2022-04-06 16:17:46.402608: Average global foreground Dice: [0.6956, 0.9079, 0.9279, 0.283] 
2022-04-06 16:17:46.403268: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:17:47.270780: lr: 0.009023 
2022-04-06 16:17:47.275152: saving scheduled checkpoint file... 
2022-04-06 16:17:47.360683: saving checkpoint... 
2022-04-06 16:17:51.732419: done, saving took 4.46 seconds 
2022-04-06 16:17:51.743306: done 
2022-04-06 16:17:51.793031: saving checkpoint... 
2022-04-06 16:17:56.093911: done, saving took 4.35 seconds 
2022-04-06 16:17:56.102236: This epoch took 132.813963 s
 
2022-04-06 16:17:56.103345: svmem(total=99932160000, available=87483686912, percent=12.5, used=9501331456, free=73375686656, active=20746706944, inactive=3703205888, buffers=12234752, cached=17042907136, shared=1133916160, slab=770793472) 
2022-04-06 16:17:56.103989: 
epoch:  54 
2022-04-06 16:19:45.425862: train loss : -0.3243 
2022-04-06 16:19:56.452804: validation loss: -0.2659 
2022-04-06 16:19:56.458051: Average global foreground Dice: [0.6823, 0.8715, 0.9089, 0.2636] 
2022-04-06 16:19:56.458922: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:19:57.529105: lr: 0.009004 
2022-04-06 16:19:57.534662: saving scheduled checkpoint file... 
2022-04-06 16:19:57.631709: saving checkpoint... 
2022-04-06 16:20:01.977131: done, saving took 4.43 seconds 
2022-04-06 16:20:01.988863: done 
2022-04-06 16:20:02.037673: saving checkpoint... 
2022-04-06 16:20:06.347011: done, saving took 4.36 seconds 
2022-04-06 16:20:06.367394: This epoch took 130.262771 s
 
2022-04-06 16:20:06.371314: svmem(total=99932160000, available=87456522240, percent=12.5, used=9528311808, free=73373593600, active=20529467392, inactive=3920809984, buffers=12488704, cached=17017765888, shared=1133916160, slab=770400256) 
2022-04-06 16:20:06.374713: 
epoch:  55 
2022-04-06 16:21:54.467719: train loss : -0.3235 
2022-04-06 16:22:04.926404: validation loss: -0.3264 
2022-04-06 16:22:04.932781: Average global foreground Dice: [0.6806, 0.8904, 0.9378, 0.2896] 
2022-04-06 16:22:04.935120: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:22:05.758967: lr: 0.008986 
2022-04-06 16:22:05.762377: saving scheduled checkpoint file... 
2022-04-06 16:22:05.859663: saving checkpoint... 
2022-04-06 16:22:10.289683: done, saving took 4.52 seconds 
2022-04-06 16:22:10.314597: done 
2022-04-06 16:22:10.368922: saving checkpoint... 
2022-04-06 16:22:14.717097: done, saving took 4.40 seconds 
2022-04-06 16:22:14.730496: This epoch took 128.352378 s
 
2022-04-06 16:22:14.731535: svmem(total=99932160000, available=87404552192, percent=12.5, used=9580392448, free=73364766720, active=20691070976, inactive=3764834304, buffers=14389248, cached=16972611584, shared=1133932544, slab=772165632) 
2022-04-06 16:22:14.732184: 
epoch:  56 
2022-04-06 16:24:03.177855: train loss : -0.2996 
2022-04-06 16:24:15.275115: validation loss: -0.3979 
2022-04-06 16:24:15.285670: Average global foreground Dice: [0.7377, 0.9141, 0.9399, 0.279] 
2022-04-06 16:24:15.303007: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:24:15.865972: lr: 0.008968 
2022-04-06 16:24:15.881619: saving scheduled checkpoint file... 
2022-04-06 16:24:15.981350: saving checkpoint... 
2022-04-06 16:24:20.455195: done, saving took 4.55 seconds 
2022-04-06 16:24:20.500663: done 
2022-04-06 16:24:20.556649: saving checkpoint... 
2022-04-06 16:24:24.978777: done, saving took 4.47 seconds 
2022-04-06 16:24:25.000063: This epoch took 130.267282 s
 
2022-04-06 16:24:25.002655: svmem(total=99932160000, available=87456821248, percent=12.5, used=9527717888, free=73364463616, active=20662439936, inactive=3794419712, buffers=14635008, cached=17025343488, shared=1133940736, slab=772157440) 
2022-04-06 16:24:25.005987: 
epoch:  57 
2022-04-06 16:26:12.645960: train loss : -0.3014 
2022-04-06 16:26:23.953175: validation loss: -0.3944 
2022-04-06 16:26:24.059596: Average global foreground Dice: [0.729, 0.9149, 0.9443, 0.2751] 
2022-04-06 16:26:24.063327: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:26:24.769971: lr: 0.00895 
2022-04-06 16:26:24.772908: saving scheduled checkpoint file... 
2022-04-06 16:26:24.837736: saving checkpoint... 
2022-04-06 16:26:29.390399: done, saving took 4.62 seconds 
2022-04-06 16:26:29.409573: done 
2022-04-06 16:26:29.455195: saving checkpoint... 
2022-04-06 16:26:33.843796: done, saving took 4.43 seconds 
2022-04-06 16:26:33.853214: This epoch took 128.843760 s
 
2022-04-06 16:26:33.854236: svmem(total=99932160000, available=87434137600, percent=12.5, used=9550721024, free=73388789760, active=20699389952, inactive=3732738048, buffers=14897152, cached=16977752064, shared=1133940736, slab=771944448) 
2022-04-06 16:26:33.854857: 
epoch:  58 
2022-04-06 16:28:21.507303: train loss : -0.3463 
2022-04-06 16:28:32.812747: validation loss: -0.3846 
2022-04-06 16:28:32.821205: Average global foreground Dice: [0.718, 0.9077, 0.9435, 0.3183] 
2022-04-06 16:28:32.822279: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:28:33.596898: lr: 0.008931 
2022-04-06 16:28:33.607924: saving scheduled checkpoint file... 
2022-04-06 16:28:33.814075: saving checkpoint... 
2022-04-06 16:28:38.369961: done, saving took 4.75 seconds 
2022-04-06 16:28:38.398415: done 
2022-04-06 16:28:38.481945: saving checkpoint... 
2022-04-06 16:28:42.935231: done, saving took 4.53 seconds 
2022-04-06 16:28:42.950715: This epoch took 129.095243 s
 
2022-04-06 16:28:42.951820: svmem(total=99932160000, available=87357779968, percent=12.6, used=9626853376, free=73436442624, active=20503470080, inactive=3885912064, buffers=15159296, cached=16853704704, shared=1133940736, slab=771862528) 
2022-04-06 16:28:42.952666: 
epoch:  59 
2022-04-06 16:30:32.447990: train loss : -0.3655 
2022-04-06 16:30:43.883627: validation loss: -0.3687 
2022-04-06 16:30:43.890265: Average global foreground Dice: [0.7218, 0.9039, 0.9348, 0.276] 
2022-04-06 16:30:43.894658: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:30:44.970304: lr: 0.008913 
2022-04-06 16:30:44.981509: saving scheduled checkpoint file... 
2022-04-06 16:30:45.080204: saving checkpoint... 
2022-04-06 16:30:49.513282: done, saving took 4.52 seconds 
2022-04-06 16:30:49.527852: done 
2022-04-06 16:30:49.575927: saving checkpoint... 
2022-04-06 16:30:54.028027: done, saving took 4.50 seconds 
2022-04-06 16:30:54.036655: This epoch took 131.083315 s
 
2022-04-06 16:30:54.037780: svmem(total=99932160000, available=87562096640, percent=12.4, used=9422344192, free=73323851776, active=20564783104, inactive=3932422144, buffers=17657856, cached=17168306176, shared=1133944832, slab=771702784) 
2022-04-06 16:30:54.038535: 
epoch:  60 
2022-04-06 16:32:43.432104: train loss : -0.3516 
2022-04-06 16:32:55.533772: validation loss: -0.3906 
2022-04-06 16:32:55.535851: Average global foreground Dice: [0.7251, 0.9189, 0.9324, 0.3031] 
2022-04-06 16:32:55.537279: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:32:56.432175: lr: 0.008895 
2022-04-06 16:32:56.437881: saving scheduled checkpoint file... 
2022-04-06 16:32:56.555403: saving checkpoint... 
2022-04-06 16:33:01.063571: done, saving took 4.62 seconds 
2022-04-06 16:33:01.090500: done 
2022-04-06 16:33:01.144783: saving checkpoint... 
2022-04-06 16:33:05.633521: done, saving took 4.54 seconds 
2022-04-06 16:33:05.645879: This epoch took 131.606463 s
 
2022-04-06 16:33:05.647067: svmem(total=99932160000, available=87448719360, percent=12.5, used=9535709184, free=73356910592, active=20670251008, inactive=3798704128, buffers=17903616, cached=17021636608, shared=1133944832, slab=771796992) 
2022-04-06 16:33:05.647830: 
epoch:  61 
2022-04-06 16:34:54.668488: train loss : -0.3145 
2022-04-06 16:35:06.264695: validation loss: -0.3664 
2022-04-06 16:35:06.270086: Average global foreground Dice: [0.6643, 0.9056, 0.9338, 0.3122] 
2022-04-06 16:35:06.270910: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:35:07.120439: lr: 0.008877 
2022-04-06 16:35:07.125546: saving scheduled checkpoint file... 
2022-04-06 16:35:07.271429: saving checkpoint... 
2022-04-06 16:35:11.777463: done, saving took 4.65 seconds 
2022-04-06 16:35:11.791576: done 
2022-04-06 16:35:11.871176: saving checkpoint... 
2022-04-06 16:35:16.289552: done, saving took 4.49 seconds 
2022-04-06 16:35:16.306653: This epoch took 130.657970 s
 
2022-04-06 16:35:16.307705: svmem(total=99932160000, available=87516516352, percent=12.4, used=9467904000, free=73390055424, active=20540719104, inactive=3891392512, buffers=18182144, cached=17056018432, shared=1133944832, slab=770330624) 
2022-04-06 16:35:16.308547: 
epoch:  62 
2022-04-06 16:37:06.813786: train loss : -0.3343 
2022-04-06 16:37:19.244664: validation loss: -0.1054 
2022-04-06 16:37:19.254842: Average global foreground Dice: [0.7108, 0.7806, 0.7973, 0.3011] 
2022-04-06 16:37:19.261066: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:37:19.895584: lr: 0.008859 
2022-04-06 16:37:19.896978: saving scheduled checkpoint file... 
2022-04-06 16:37:20.116640: saving checkpoint... 
2022-04-06 16:37:24.675664: done, saving took 4.78 seconds 
2022-04-06 16:37:24.693843: done 
2022-04-06 16:37:24.697719: This epoch took 128.388436 s
 
2022-04-06 16:37:24.701262: svmem(total=99932160000, available=87618977792, percent=12.3, used=9365458944, free=73416904704, active=20620795904, inactive=3785691136, buffers=18427904, cached=17131368448, shared=1133944832, slab=770334720) 
2022-04-06 16:37:24.705013: 
epoch:  63 
2022-04-06 16:39:13.926675: train loss : -0.1868 
2022-04-06 16:39:25.577120: validation loss: -0.3254 
2022-04-06 16:39:25.611031: Average global foreground Dice: [0.6923, 0.9114, 0.9208, 0.2921] 
2022-04-06 16:39:25.611905: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:39:26.934325: lr: 0.00884 
2022-04-06 16:39:26.935966: saving scheduled checkpoint file... 
2022-04-06 16:39:27.172736: saving checkpoint... 
2022-04-06 16:39:34.307168: done, saving took 7.37 seconds 
2022-04-06 16:39:34.333824: done 
2022-04-06 16:39:34.341096: This epoch took 129.633795 s
 
2022-04-06 16:39:34.346409: svmem(total=99932160000, available=87505408000, percent=12.4, used=9478733824, free=73415327744, active=20583964672, inactive=3824205824, buffers=18673664, cached=17019424768, shared=1133944832, slab=772009984) 
2022-04-06 16:39:34.351827: 
epoch:  64 
2022-04-06 16:41:22.597228: train loss : -0.3444 
2022-04-06 16:41:34.663823: validation loss: -0.3529 
2022-04-06 16:41:34.665967: Average global foreground Dice: [0.6601, 0.8941, 0.9242, 0.3173] 
2022-04-06 16:41:34.669019: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:41:35.601375: lr: 0.008822 
2022-04-06 16:41:35.602249: saving scheduled checkpoint file... 
2022-04-06 16:41:35.760832: saving checkpoint... 
2022-04-06 16:41:41.467089: done, saving took 5.86 seconds 
2022-04-06 16:41:41.480488: done 
2022-04-06 16:41:41.481369: This epoch took 127.123231 s
 
2022-04-06 16:41:41.482208: svmem(total=99932160000, available=87546466304, percent=12.4, used=9437884416, free=73419710464, active=20646621184, inactive=3756298240, buffers=18919424, cached=17055645696, shared=1133944832, slab=772624384) 
2022-04-06 16:41:41.482857: 
epoch:  65 
2022-04-06 16:43:32.150839: train loss : -0.3876 
2022-04-06 16:43:44.064246: validation loss: -0.4280 
2022-04-06 16:43:44.075298: Average global foreground Dice: [0.7087, 0.9206, 0.95, 0.3111] 
2022-04-06 16:43:44.076586: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:43:44.875776: lr: 0.008804 
2022-04-06 16:43:44.882516: saving scheduled checkpoint file... 
2022-04-06 16:43:45.130797: saving checkpoint... 
2022-04-06 16:43:49.537067: done, saving took 4.65 seconds 
2022-04-06 16:43:49.570202: done 
2022-04-06 16:43:49.631710: saving checkpoint... 
2022-04-06 16:43:53.934619: done, saving took 4.36 seconds 
2022-04-06 16:43:53.945684: This epoch took 132.462189 s
 
2022-04-06 16:43:53.946499: svmem(total=99932160000, available=87547465728, percent=12.4, used=9436835840, free=73319559168, active=20624654336, inactive=3880947712, buffers=19181568, cached=17156583424, shared=1133944832, slab=772505600) 
2022-04-06 16:43:53.947291: 
epoch:  66 
2022-04-06 16:45:42.054360: train loss : -0.4192 
2022-04-06 16:45:53.024009: validation loss: -0.4402 
2022-04-06 16:45:53.025805: Average global foreground Dice: [0.7389, 0.9008, 0.9661, 0.3307] 
2022-04-06 16:45:53.026560: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:45:53.910764: lr: 0.008785 
2022-04-06 16:45:53.912679: saving scheduled checkpoint file... 
2022-04-06 16:45:54.017300: saving checkpoint... 
2022-04-06 16:45:58.610909: done, saving took 4.70 seconds 
2022-04-06 16:45:58.638699: done 
2022-04-06 16:45:58.691378: saving checkpoint... 
2022-04-06 16:46:03.123432: done, saving took 4.48 seconds 
2022-04-06 16:46:03.135905: This epoch took 129.187982 s
 
2022-04-06 16:46:03.140825: svmem(total=99932160000, available=87554818048, percent=12.4, used=9429508096, free=73313193984, active=20642648064, inactive=3864317952, buffers=19443712, cached=17170014208, shared=1133944832, slab=772644864) 
2022-04-06 16:46:03.144248: 
epoch:  67 
2022-04-06 16:47:52.301256: train loss : -0.3790 
2022-04-06 16:48:03.545864: validation loss: -0.3057 
2022-04-06 16:48:03.553876: Average global foreground Dice: [0.6866, 0.8752, 0.9255, 0.2958] 
2022-04-06 16:48:03.559702: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:48:04.240029: lr: 0.008767 
2022-04-06 16:48:04.241239: saving scheduled checkpoint file... 
2022-04-06 16:48:04.327949: saving checkpoint... 
2022-04-06 16:48:08.747277: done, saving took 4.51 seconds 
2022-04-06 16:48:08.774098: done 
2022-04-06 16:48:08.832155: saving checkpoint... 
2022-04-06 16:48:13.266922: done, saving took 4.49 seconds 
2022-04-06 16:48:13.282381: This epoch took 130.134651 s
 
2022-04-06 16:48:13.285936: svmem(total=99932160000, available=87508127744, percent=12.4, used=9476161536, free=73315762176, active=20727685120, inactive=3780579328, buffers=19689472, cached=17120546816, shared=1133944832, slab=772734976) 
2022-04-06 16:48:13.289269: 
epoch:  68 
2022-04-06 16:50:02.688102: train loss : -0.2861 
2022-04-06 16:50:13.898463: validation loss: -0.2701 
2022-04-06 16:50:13.912402: Average global foreground Dice: [0.6897, 0.8817, 0.9055, 0.2918] 
2022-04-06 16:50:13.913461: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:50:14.762484: lr: 0.008749 
2022-04-06 16:50:14.772119: saving scheduled checkpoint file... 
2022-04-06 16:50:14.942725: saving checkpoint... 
2022-04-06 16:50:19.410503: done, saving took 4.63 seconds 
2022-04-06 16:50:19.447159: done 
2022-04-06 16:50:19.457434: This epoch took 126.163386 s
 
2022-04-06 16:50:19.465749: svmem(total=99932160000, available=87602659328, percent=12.3, used=9381588992, free=73376473088, active=20505145344, inactive=3941928960, buffers=19935232, cached=17154162688, shared=1133944832, slab=772104192) 
2022-04-06 16:50:19.477887: 
epoch:  69 
2022-04-06 16:52:07.510570: train loss : -0.3578 
2022-04-06 16:52:18.504997: validation loss: -0.4284 
2022-04-06 16:52:18.522906: Average global foreground Dice: [0.687, 0.914, 0.9429, 0.3159] 
2022-04-06 16:52:18.527824: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:52:19.275469: lr: 0.008731 
2022-04-06 16:52:19.285228: saving scheduled checkpoint file... 
2022-04-06 16:52:19.496255: saving checkpoint... 
2022-04-06 16:52:24.027269: done, saving took 4.74 seconds 
2022-04-06 16:52:24.051303: done 
2022-04-06 16:52:24.155909: saving checkpoint... 
2022-04-06 16:52:28.615406: done, saving took 4.56 seconds 
2022-04-06 16:52:28.633492: This epoch took 129.146888 s
 
2022-04-06 16:52:28.636222: svmem(total=99932160000, available=87526350848, percent=12.4, used=9457975296, free=73427644416, active=20624199680, inactive=3777048576, buffers=20180992, cached=17026359296, shared=1133948928, slab=770023424) 
2022-04-06 16:52:28.638383: 
epoch:  70 
2022-04-06 16:54:19.107900: train loss : -0.4152 
2022-04-06 16:54:30.739961: validation loss: -0.3024 
2022-04-06 16:54:30.741727: Average global foreground Dice: [0.6952, 0.9014, 0.9073, 0.3201] 
2022-04-06 16:54:30.744688: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:54:31.969869: lr: 0.008712 
2022-04-06 16:54:31.970992: saving scheduled checkpoint file... 
2022-04-06 16:54:32.131936: saving checkpoint... 
2022-04-06 16:54:36.571686: done, saving took 4.60 seconds 
2022-04-06 16:54:36.588518: done 
2022-04-06 16:54:36.681284: saving checkpoint... 
2022-04-06 16:54:41.066530: done, saving took 4.48 seconds 
2022-04-06 16:54:41.082968: This epoch took 132.442111 s
 
2022-04-06 16:54:41.084054: svmem(total=99932160000, available=87501324288, percent=12.4, used=9483128832, free=73416171520, active=20575240192, inactive=3829551104, buffers=20443136, cached=17012416512, shared=1133948928, slab=770138112) 
2022-04-06 16:54:41.084903: 
epoch:  71 
2022-04-06 16:56:29.208917: train loss : -0.4070 
2022-04-06 16:56:41.231836: validation loss: -0.4355 
2022-04-06 16:56:41.240515: Average global foreground Dice: [0.7153, 0.9188, 0.9515, 0.3161] 
2022-04-06 16:56:41.241563: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:56:42.477318: lr: 0.008694 
2022-04-06 16:56:42.478315: saving scheduled checkpoint file... 
2022-04-06 16:56:42.563431: saving checkpoint... 
2022-04-06 16:56:46.886121: done, saving took 4.41 seconds 
2022-04-06 16:56:46.925778: done 
2022-04-06 16:56:46.982439: saving checkpoint... 
2022-04-06 16:56:51.306088: done, saving took 4.37 seconds 
2022-04-06 16:56:51.321910: This epoch took 130.236295 s
 
2022-04-06 16:56:51.325753: svmem(total=99932160000, available=87510134784, percent=12.4, used=9474260992, free=73310195712, active=20634517504, inactive=3877904384, buffers=20713472, cached=17126989824, shared=1133948928, slab=770969600) 
2022-04-06 16:56:51.329057: 
epoch:  72 
2022-04-06 16:58:39.852451: train loss : -0.4237 
2022-04-06 16:58:52.657311: validation loss: -0.3685 
2022-04-06 16:58:52.662598: Average global foreground Dice: [0.7264, 0.9207, 0.9445, 0.2999] 
2022-04-06 16:58:52.665541: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 16:58:53.400007: lr: 0.008676 
2022-04-06 16:58:53.401810: saving scheduled checkpoint file... 
2022-04-06 16:58:53.508599: saving checkpoint... 
2022-04-06 16:58:57.918307: done, saving took 4.52 seconds 
2022-04-06 16:58:57.938255: done 
2022-04-06 16:58:57.989900: saving checkpoint... 
2022-04-06 16:59:02.294893: done, saving took 4.35 seconds 
2022-04-06 16:59:02.308975: This epoch took 130.978325 s
 
2022-04-06 16:59:02.310038: svmem(total=99932160000, available=87475392512, percent=12.5, used=9509130240, free=73313591296, active=20673761280, inactive=3836616704, buffers=21098496, cached=17088339968, shared=1133948928, slab=767516672) 
2022-04-06 16:59:02.310829: 
epoch:  73 
2022-04-06 17:00:51.954167: train loss : -0.3701 
2022-04-06 17:01:03.223226: validation loss: -0.4041 
2022-04-06 17:01:03.225037: Average global foreground Dice: [0.7398, 0.9151, 0.9503, 0.3064] 
2022-04-06 17:01:03.225936: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:01:06.489650: lr: 0.008658 
2022-04-06 17:01:06.490497: saving scheduled checkpoint file... 
2022-04-06 17:01:06.571769: saving checkpoint... 
2022-04-06 17:01:10.968363: done, saving took 4.48 seconds 
2022-04-06 17:01:10.982644: done 
2022-04-06 17:01:11.029594: saving checkpoint... 
2022-04-06 17:01:15.436539: done, saving took 4.45 seconds 
2022-04-06 17:01:15.447499: This epoch took 133.135947 s
 
2022-04-06 17:01:15.448840: svmem(total=99932160000, available=87450697728, percent=12.5, used=9533706240, free=73341018112, active=20535537664, inactive=3951992832, buffers=21385216, cached=17036050432, shared=1133948928, slab=767311872) 
2022-04-06 17:01:15.449649: 
epoch:  74 
2022-04-06 17:03:05.572175: train loss : -0.4296 
2022-04-06 17:03:17.530904: validation loss: -0.4631 
2022-04-06 17:03:17.532624: Average global foreground Dice: [0.7369, 0.9268, 0.9661, 0.3339] 
2022-04-06 17:03:17.533238: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:03:18.240891: lr: 0.008639 
2022-04-06 17:03:18.255034: saving scheduled checkpoint file... 
2022-04-06 17:03:18.357444: saving checkpoint... 
2022-04-06 17:03:22.850201: done, saving took 4.58 seconds 
2022-04-06 17:03:22.883129: done 
2022-04-06 17:03:22.935303: saving checkpoint... 
2022-04-06 17:03:27.357489: done, saving took 4.47 seconds 
2022-04-06 17:03:27.368217: This epoch took 131.917867 s
 
2022-04-06 17:03:27.369085: svmem(total=99932160000, available=87459729408, percent=12.5, used=9524768768, free=73314357248, active=20527513600, inactive=3985043456, buffers=21630976, cached=17071403008, shared=1133948928, slab=767868928) 
2022-04-06 17:03:27.369727: 
epoch:  75 
2022-04-06 17:05:16.625008: train loss : -0.4602 
2022-04-06 17:05:29.326388: validation loss: -0.4879 
2022-04-06 17:05:29.331489: Average global foreground Dice: [0.7245, 0.933, 0.9635, 0.3422] 
2022-04-06 17:05:29.336241: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:05:30.186733: lr: 0.008621 
2022-04-06 17:05:30.198997: saving scheduled checkpoint file... 
2022-04-06 17:05:30.288531: saving checkpoint... 
2022-04-06 17:05:34.709394: done, saving took 4.50 seconds 
2022-04-06 17:05:34.728000: done 
2022-04-06 17:05:34.806763: saving checkpoint... 
2022-04-06 17:05:39.204202: done, saving took 4.48 seconds 
2022-04-06 17:05:39.218266: This epoch took 131.847849 s
 
2022-04-06 17:05:39.219292: svmem(total=99932160000, available=87491923968, percent=12.4, used=9492471808, free=73399386112, active=20622708736, inactive=3808256000, buffers=21917696, cached=17018384384, shared=1133948928, slab=767406080) 
2022-04-06 17:05:39.219889: 
epoch:  76 
2022-04-06 17:07:27.496358: train loss : -0.4559 
2022-04-06 17:07:38.689594: validation loss: -0.4744 
2022-04-06 17:07:38.700553: Average global foreground Dice: [0.7265, 0.9321, 0.9594, 0.3367] 
2022-04-06 17:07:38.709433: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:07:39.280990: lr: 0.008603 
2022-04-06 17:07:39.288795: saving scheduled checkpoint file... 
2022-04-06 17:07:39.420932: saving checkpoint... 
2022-04-06 17:07:44.078711: done, saving took 4.78 seconds 
2022-04-06 17:07:44.116742: done 
2022-04-06 17:07:44.224103: saving checkpoint... 
2022-04-06 17:07:48.741208: done, saving took 4.61 seconds 
2022-04-06 17:07:48.766507: This epoch took 129.545924 s
 
2022-04-06 17:07:48.770136: svmem(total=99932160000, available=87520751616, percent=12.4, used=9463844864, free=73406443520, active=20187930624, inactive=4229738496, buffers=22155264, cached=17039716352, shared=1133948928, slab=767238144) 
2022-04-06 17:07:48.773377: 
epoch:  77 
2022-04-06 17:09:38.576608: train loss : -0.3911 
2022-04-06 17:09:50.452995: validation loss: -0.3766 
2022-04-06 17:09:50.455417: Average global foreground Dice: [0.6999, 0.9096, 0.9348, 0.2964] 
2022-04-06 17:09:50.457234: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:09:51.380800: lr: 0.008584 
2022-04-06 17:09:51.396131: saving scheduled checkpoint file... 
2022-04-06 17:09:51.498461: saving checkpoint... 
2022-04-06 17:09:55.830758: done, saving took 4.42 seconds 
2022-04-06 17:09:55.864643: done 
2022-04-06 17:09:55.872599: This epoch took 127.096571 s
 
2022-04-06 17:09:55.884052: svmem(total=99932160000, available=87496163328, percent=12.4, used=9487720448, free=73312014336, active=20561981440, inactive=3953254400, buffers=22401024, cached=17110024192, shared=1133948928, slab=768126976) 
2022-04-06 17:09:55.893786: 
epoch:  78 
2022-04-06 17:11:44.167195: train loss : -0.4157 
2022-04-06 17:11:55.106755: validation loss: -0.4090 
2022-04-06 17:11:55.118593: Average global foreground Dice: [0.7017, 0.909, 0.9295, 0.3336] 
2022-04-06 17:11:55.131910: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:11:55.810374: lr: 0.008566 
2022-04-06 17:11:55.816267: saving scheduled checkpoint file... 
2022-04-06 17:11:55.910320: saving checkpoint... 
2022-04-06 17:12:00.301702: done, saving took 4.48 seconds 
2022-04-06 17:12:00.336694: done 
2022-04-06 17:12:00.394922: saving checkpoint... 
2022-04-06 17:12:04.711553: done, saving took 4.37 seconds 
2022-04-06 17:12:04.736768: This epoch took 128.835128 s
 
2022-04-06 17:12:04.743220: svmem(total=99932160000, available=87502708736, percent=12.4, used=9481506816, free=73312174080, active=20552056832, inactive=3964280832, buffers=22646784, cached=17115832320, shared=1133948928, slab=767221760) 
2022-04-06 17:12:04.749801: 
epoch:  79 
2022-04-06 17:13:53.655004: train loss : -0.4394 
2022-04-06 17:14:06.817436: validation loss: -0.4557 
2022-04-06 17:14:06.831739: Average global foreground Dice: [0.7021, 0.9291, 0.955, 0.3534] 
2022-04-06 17:14:06.832890: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:14:07.703939: lr: 0.008548 
2022-04-06 17:14:07.711788: saving scheduled checkpoint file... 
2022-04-06 17:14:07.801833: saving checkpoint... 
2022-04-06 17:14:12.182827: done, saving took 4.46 seconds 
2022-04-06 17:14:12.204925: done 
2022-04-06 17:14:12.259449: saving checkpoint... 
2022-04-06 17:14:16.527651: done, saving took 4.32 seconds 
2022-04-06 17:14:16.541000: This epoch took 131.788951 s
 
2022-04-06 17:14:16.541941: svmem(total=99932160000, available=87477399552, percent=12.5, used=9507418112, free=73395015680, active=20544757760, inactive=3887259648, buffers=1716224, cached=17028009984, shared=1133948928, slab=768241664) 
2022-04-06 17:14:16.542696: 
epoch:  80 
2022-04-06 17:16:05.027913: train loss : -0.4499 
2022-04-06 17:16:18.936445: validation loss: -0.4846 
2022-04-06 17:16:18.947264: Average global foreground Dice: [0.7555, 0.9351, 0.963, 0.3248] 
2022-04-06 17:16:18.956767: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:16:19.690121: lr: 0.008529 
2022-04-06 17:16:19.711114: saving scheduled checkpoint file... 
2022-04-06 17:16:19.806703: saving checkpoint... 
2022-04-06 17:16:24.304776: done, saving took 4.58 seconds 
2022-04-06 17:16:24.342080: done 
2022-04-06 17:16:24.401854: saving checkpoint... 
2022-04-06 17:16:28.766755: done, saving took 4.41 seconds 
2022-04-06 17:16:28.793451: This epoch took 132.250080 s
 
2022-04-06 17:16:28.799378: svmem(total=99932160000, available=87542853632, percent=12.4, used=9464782848, free=78066900992, active=10598174720, inactive=9214496768, buffers=9007104, cached=12391469056, shared=1133948928, slab=725196800) 
2022-04-06 17:16:28.804027: 
epoch:  81 
2022-04-06 17:18:17.714136: train loss : -0.4633 
2022-04-06 17:18:29.225460: validation loss: -0.4817 
2022-04-06 17:18:29.253866: Average global foreground Dice: [0.7414, 0.9315, 0.964, 0.3414] 
2022-04-06 17:18:29.275126: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:18:29.964854: lr: 0.008511 
2022-04-06 17:18:29.973905: saving scheduled checkpoint file... 
2022-04-06 17:18:30.064443: saving checkpoint... 
2022-04-06 17:18:34.591397: done, saving took 4.61 seconds 
2022-04-06 17:18:34.618503: done 
2022-04-06 17:18:34.670545: saving checkpoint... 
2022-04-06 17:18:39.055816: done, saving took 4.43 seconds 
2022-04-06 17:18:39.068477: This epoch took 130.259851 s
 
2022-04-06 17:18:39.069421: svmem(total=99932160000, available=87409475584, percent=12.5, used=9589547008, free=74564386816, active=15269396480, inactive=8026263552, buffers=9273344, cached=15768952832, shared=1133948928, slab=740831232) 
2022-04-06 17:18:39.070224: 
epoch:  82 
2022-04-06 17:20:27.078236: train loss : -0.4705 
2022-04-06 17:20:38.882488: validation loss: -0.4325 
2022-04-06 17:20:38.893187: Average global foreground Dice: [0.7825, 0.9238, 0.9303, 0.3443] 
2022-04-06 17:20:38.903026: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:20:39.490659: lr: 0.008493 
2022-04-06 17:20:39.504882: saving scheduled checkpoint file... 
2022-04-06 17:20:39.587255: saving checkpoint... 
2022-04-06 17:20:44.047657: done, saving took 4.53 seconds 
2022-04-06 17:20:44.094588: done 
2022-04-06 17:20:44.190431: saving checkpoint... 
2022-04-06 17:20:48.859968: done, saving took 4.75 seconds 
2022-04-06 17:20:48.896102: This epoch took 129.825338 s
 
2022-04-06 17:20:48.908214: svmem(total=99932160000, available=87535529984, percent=12.4, used=9454972928, free=74120736768, active=18774409216, inactive=4943814656, buffers=9990144, cached=16346460160, shared=1133948928, slab=759746560) 
2022-04-06 17:20:48.915619: 
epoch:  83 
2022-04-06 17:22:37.743542: train loss : -0.4260 
2022-04-06 17:22:49.065067: validation loss: -0.3356 
2022-04-06 17:22:49.076439: Average global foreground Dice: [0.6606, 0.8878, 0.9285, 0.328] 
2022-04-06 17:22:49.089360: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:22:50.312589: lr: 0.008474 
2022-04-06 17:22:50.317661: saving scheduled checkpoint file... 
2022-04-06 17:22:50.442255: saving checkpoint... 
2022-04-06 17:22:55.020837: done, saving took 4.70 seconds 
2022-04-06 17:22:55.033041: done 
2022-04-06 17:22:55.033857: This epoch took 126.106634 s
 
2022-04-06 17:22:55.034788: svmem(total=99932160000, available=84210180096, percent=15.7, used=12762361856, free=70011392000, active=23798157312, inactive=3900669952, buffers=10240000, cached=17148166144, shared=1148768256, slab=777117696) 
2022-04-06 17:22:55.035544: 
epoch:  84 
2022-04-06 17:24:45.838858: train loss : -0.3242 
2022-04-06 17:24:58.219316: validation loss: -0.3210 
2022-04-06 17:24:58.226749: Average global foreground Dice: [0.6971, 0.8944, 0.9262, 0.3033] 
2022-04-06 17:24:58.229123: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:24:59.287681: lr: 0.008456 
2022-04-06 17:24:59.291444: saving scheduled checkpoint file... 
2022-04-06 17:24:59.396961: saving checkpoint... 
2022-04-06 17:25:04.040339: done, saving took 4.75 seconds 
2022-04-06 17:25:04.059066: done 
2022-04-06 17:25:04.060112: This epoch took 129.023924 s
 
2022-04-06 17:25:04.061308: svmem(total=99932160000, available=87601971200, percent=12.3, used=9384050688, free=73356161024, active=20808024064, inactive=3661828096, buffers=10493952, cached=17181454336, shared=1133948928, slab=770412544) 
2022-04-06 17:25:04.062083: 
epoch:  85 
2022-04-06 17:26:53.139197: train loss : -0.4146 
2022-04-06 17:27:04.665092: validation loss: -0.4580 
2022-04-06 17:27:04.676557: Average global foreground Dice: [0.7378, 0.9207, 0.9545, 0.3634] 
2022-04-06 17:27:04.684949: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:27:05.497333: lr: 0.008438 
2022-04-06 17:27:05.510298: saving scheduled checkpoint file... 
2022-04-06 17:27:05.654011: saving checkpoint... 
2022-04-06 17:27:10.139789: done, saving took 4.62 seconds 
2022-04-06 17:27:10.162195: done 
2022-04-06 17:27:10.169581: This epoch took 126.106754 s
 
2022-04-06 17:27:10.175349: svmem(total=99932160000, available=87660261376, percent=12.3, used=9348812800, free=73419681792, active=20953169920, inactive=3451133952, buffers=10752000, cached=17152913408, shared=1109475328, slab=771411968) 
2022-04-06 17:27:10.185809: 
epoch:  86 
2022-04-06 17:28:59.559537: train loss : -0.4701 
2022-04-06 17:29:11.098278: validation loss: -0.4573 
2022-04-06 17:29:11.104572: Average global foreground Dice: [0.7374, 0.9164, 0.9637, 0.365] 
2022-04-06 17:29:11.107982: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:29:11.699252: lr: 0.008419 
2022-04-06 17:29:11.699927: saving scheduled checkpoint file... 
2022-04-06 17:29:11.861825: saving checkpoint... 
2022-04-06 17:29:16.559224: done, saving took 4.86 seconds 
2022-04-06 17:29:16.581098: done 
2022-04-06 17:29:16.682487: saving checkpoint... 
2022-04-06 17:29:21.124798: done, saving took 4.54 seconds 
2022-04-06 17:29:21.139184: This epoch took 130.943839 s
 
2022-04-06 17:29:21.140199: svmem(total=99932160000, available=87560769536, percent=12.4, used=9423896576, free=73460768768, active=20862013440, inactive=3501510656, buffers=10997760, cached=17036496896, shared=1133948928, slab=772206592) 
2022-04-06 17:29:21.140821: 
epoch:  87 
2022-04-06 17:31:10.152812: train loss : -0.4623 
2022-04-06 17:31:22.437378: validation loss: -0.4881 
2022-04-06 17:31:22.441048: Average global foreground Dice: [0.7622, 0.9203, 0.9522, 0.3781] 
2022-04-06 17:31:22.444014: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:31:23.334384: lr: 0.008401 
2022-04-06 17:31:23.356900: saving scheduled checkpoint file... 
2022-04-06 17:31:23.488715: saving checkpoint... 
2022-04-06 17:31:27.971221: done, saving took 4.59 seconds 
2022-04-06 17:31:27.988719: done 
2022-04-06 17:31:28.091297: saving checkpoint... 
2022-04-06 17:31:32.489465: done, saving took 4.50 seconds 
2022-04-06 17:31:32.507137: This epoch took 131.365855 s
 
2022-04-06 17:31:32.508089: svmem(total=99932160000, available=87496388608, percent=12.4, used=9487773696, free=73458212864, active=20780814336, inactive=3583750144, buffers=11284480, cached=16974888960, shared=1133948928, slab=772022272) 
2022-04-06 17:31:32.508641: 
epoch:  88 
2022-04-06 17:33:22.838632: train loss : -0.4658 
2022-04-06 17:33:34.067181: validation loss: -0.4905 
2022-04-06 17:33:34.080453: Average global foreground Dice: [0.7619, 0.9296, 0.9639, 0.3614] 
2022-04-06 17:33:34.090981: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:33:34.859163: lr: 0.008383 
2022-04-06 17:33:34.863904: saving scheduled checkpoint file... 
2022-04-06 17:33:35.025775: saving checkpoint... 
2022-04-06 17:33:39.744565: done, saving took 4.88 seconds 
2022-04-06 17:33:39.794198: done 
2022-04-06 17:33:39.897479: saving checkpoint... 
2022-04-06 17:33:44.332154: done, saving took 4.53 seconds 
2022-04-06 17:33:44.353239: This epoch took 131.844038 s
 
2022-04-06 17:33:44.355885: svmem(total=99932160000, available=87559815168, percent=12.4, used=9423495168, free=73457131520, active=20787261440, inactive=3577872384, buffers=11546624, cached=17039986688, shared=1133948928, slab=773099520) 
2022-04-06 17:33:44.357475: 
epoch:  89 
2022-04-06 17:35:32.664807: train loss : -0.4806 
2022-04-06 17:35:44.353801: validation loss: -0.4790 
2022-04-06 17:35:44.357731: Average global foreground Dice: [0.7326, 0.9407, 0.9639, 0.3529] 
2022-04-06 17:35:44.360956: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:35:45.232196: lr: 0.008364 
2022-04-06 17:35:45.244246: saving scheduled checkpoint file... 
2022-04-06 17:35:45.344031: saving checkpoint... 
2022-04-06 17:35:49.758545: done, saving took 4.51 seconds 
2022-04-06 17:35:49.789360: done 
2022-04-06 17:35:49.839290: saving checkpoint... 
2022-04-06 17:35:54.139361: done, saving took 4.35 seconds 
2022-04-06 17:35:54.151400: This epoch took 129.790967 s
 
2022-04-06 17:35:54.152317: svmem(total=99932160000, available=87455109120, percent=12.5, used=9528565760, free=73349091328, active=20795965440, inactive=3675467776, buffers=11816960, cached=17042685952, shared=1133948928, slab=773050368) 
2022-04-06 17:35:54.153057: 
epoch:  90 
2022-04-06 17:37:42.164107: train loss : -0.4816 
2022-04-06 17:37:53.860794: validation loss: -0.4937 
2022-04-06 17:37:53.862525: Average global foreground Dice: [0.733, 0.9403, 0.9657, 0.3479] 
2022-04-06 17:37:53.865569: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:37:54.739930: lr: 0.008346 
2022-04-06 17:37:54.752560: saving scheduled checkpoint file... 
2022-04-06 17:37:54.849843: saving checkpoint... 
2022-04-06 17:37:59.215195: done, saving took 4.45 seconds 
2022-04-06 17:37:59.242098: done 
2022-04-06 17:37:59.297597: saving checkpoint... 
2022-04-06 17:38:03.601308: done, saving took 4.35 seconds 
2022-04-06 17:38:03.614929: This epoch took 129.461194 s
 
2022-04-06 17:38:03.616071: svmem(total=99932160000, available=87485095936, percent=12.5, used=9498542080, free=73350754304, active=21021454336, inactive=3448217600, buffers=12054528, cached=17070809088, shared=1133948928, slab=774451200) 
2022-04-06 17:38:03.616711: 
epoch:  91 
2022-04-06 17:39:53.638959: train loss : -0.4300 
2022-04-06 17:40:06.500838: validation loss: -0.2944 
2022-04-06 17:40:06.515932: Average global foreground Dice: [0.7123, 0.8569, 0.8754, 0.3349] 
2022-04-06 17:40:06.516762: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:40:07.332366: lr: 0.008328 
2022-04-06 17:40:07.339824: saving scheduled checkpoint file... 
2022-04-06 17:40:07.446538: saving checkpoint... 
2022-04-06 17:40:11.890202: done, saving took 4.55 seconds 
2022-04-06 17:40:11.905305: done 
2022-04-06 17:40:11.908929: This epoch took 128.291486 s
 
2022-04-06 17:40:11.911922: svmem(total=99932160000, available=87394852864, percent=12.5, used=9588621312, free=73348706304, active=20903133184, inactive=3569348608, buffers=12300288, cached=16982532096, shared=1133953024, slab=774242304) 
2022-04-06 17:40:11.915950: 
epoch:  92 
2022-04-06 17:42:00.135957: train loss : -0.3631 
2022-04-06 17:42:11.441425: validation loss: -0.4650 
2022-04-06 17:42:11.446644: Average global foreground Dice: [0.7536, 0.9249, 0.9488, 0.3385] 
2022-04-06 17:42:11.454206: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:42:12.387491: lr: 0.008309 
2022-04-06 17:42:12.388449: saving scheduled checkpoint file... 
2022-04-06 17:42:12.481821: saving checkpoint... 
2022-04-06 17:42:16.975975: done, saving took 4.59 seconds 
2022-04-06 17:42:17.002228: done 
2022-04-06 17:42:17.008321: This epoch took 125.089001 s
 
2022-04-06 17:42:17.016053: svmem(total=99932160000, available=87451070464, percent=12.5, used=9532129280, free=73346301952, active=20761415680, inactive=3711971328, buffers=12529664, cached=17041199104, shared=1133953024, slab=774041600) 
2022-04-06 17:42:17.022355: 
epoch:  93 
2022-04-06 17:44:06.578434: train loss : -0.4330 
2022-04-06 17:44:17.280575: validation loss: -0.5117 
2022-04-06 17:44:17.286446: Average global foreground Dice: [0.7824, 0.9374, 0.9585, 0.3356] 
2022-04-06 17:44:17.287587: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:44:18.156024: lr: 0.008291 
2022-04-06 17:44:18.168648: saving scheduled checkpoint file... 
2022-04-06 17:44:18.264709: saving checkpoint... 
2022-04-06 17:44:22.698462: done, saving took 4.52 seconds 
2022-04-06 17:44:22.710842: done 
2022-04-06 17:44:22.712571: This epoch took 125.683620 s
 
2022-04-06 17:44:22.714456: svmem(total=99932160000, available=87484153856, percent=12.5, used=9498640384, free=73346510848, active=20652843008, inactive=3820879872, buffers=12767232, cached=17074241536, shared=1133957120, slab=774569984) 
2022-04-06 17:44:22.716508: 
epoch:  94 
2022-04-06 17:46:11.104115: train loss : -0.4443 
2022-04-06 17:46:21.786784: validation loss: -0.4210 
2022-04-06 17:46:21.788548: Average global foreground Dice: [0.745, 0.912, 0.9479, 0.3628] 
2022-04-06 17:46:21.789536: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:46:22.628227: lr: 0.008272 
2022-04-06 17:46:22.637565: saving scheduled checkpoint file... 
2022-04-06 17:46:22.735352: saving checkpoint... 
2022-04-06 17:46:27.285746: done, saving took 4.63 seconds 
2022-04-06 17:46:27.330776: done 
2022-04-06 17:46:27.423911: saving checkpoint... 
2022-04-06 17:46:31.894049: done, saving took 4.55 seconds 
2022-04-06 17:46:31.912750: This epoch took 129.194960 s
 
2022-04-06 17:46:31.915163: svmem(total=99932160000, available=87453929472, percent=12.5, used=9529020416, free=73420861440, active=20687392768, inactive=3713904640, buffers=13037568, cached=16969240576, shared=1133957120, slab=773939200) 
2022-04-06 17:46:31.919634: 
epoch:  95 
2022-04-06 17:48:20.650048: train loss : -0.4446 
2022-04-06 17:48:33.433383: validation loss: -0.4648 
2022-04-06 17:48:33.441715: Average global foreground Dice: [0.7096, 0.9291, 0.9562, 0.351] 
2022-04-06 17:48:33.442503: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:48:34.454702: lr: 0.008254 
2022-04-06 17:48:34.455680: saving scheduled checkpoint file... 
2022-04-06 17:48:34.569265: saving checkpoint... 
2022-04-06 17:48:39.096452: done, saving took 4.64 seconds 
2022-04-06 17:48:39.110362: done 
2022-04-06 17:48:39.158078: saving checkpoint... 
2022-04-06 17:48:43.562129: done, saving took 4.45 seconds 
2022-04-06 17:48:43.576029: This epoch took 131.654049 s
 
2022-04-06 17:48:43.577012: svmem(total=99932160000, available=87546417152, percent=12.4, used=9436774400, free=73400782848, active=20874678272, inactive=3546255360, buffers=13299712, cached=17081303040, shared=1133957120, slab=773541888) 
2022-04-06 17:48:43.577671: 
epoch:  96 
2022-04-06 17:50:31.789699: train loss : -0.4638 
2022-04-06 17:50:43.135391: validation loss: -0.5158 
2022-04-06 17:50:43.286010: Average global foreground Dice: [0.8106, 0.932, 0.9602, 0.3647] 
2022-04-06 17:50:43.287657: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:50:44.000737: lr: 0.008236 
2022-04-06 17:50:44.007103: saving scheduled checkpoint file... 
2022-04-06 17:50:44.133175: saving checkpoint... 
2022-04-06 17:50:48.644446: done, saving took 4.63 seconds 
2022-04-06 17:50:48.661497: done 
2022-04-06 17:50:48.711821: saving checkpoint... 
2022-04-06 17:50:53.133667: done, saving took 4.47 seconds 
2022-04-06 17:50:53.150665: This epoch took 129.572409 s
 
2022-04-06 17:50:53.153291: svmem(total=99932160000, available=87536009216, percent=12.4, used=9446944768, free=73346572288, active=20949180416, inactive=3527839744, buffers=13545472, cached=17125097472, shared=1133957120, slab=773763072) 
2022-04-06 17:50:53.155410: 
epoch:  97 
2022-04-06 17:52:42.084534: train loss : -0.4656 
2022-04-06 17:52:53.613331: validation loss: -0.5066 
2022-04-06 17:52:53.616187: Average global foreground Dice: [0.7506, 0.9387, 0.9653, 0.3728] 
2022-04-06 17:52:53.618898: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:52:54.631178: lr: 0.008217 
2022-04-06 17:52:54.632068: saving scheduled checkpoint file... 
2022-04-06 17:52:54.716081: saving checkpoint... 
2022-04-06 17:52:59.213139: done, saving took 4.58 seconds 
2022-04-06 17:52:59.229637: done 
2022-04-06 17:52:59.308513: saving checkpoint... 
2022-04-06 17:53:03.717598: done, saving took 4.49 seconds 
2022-04-06 17:53:03.735555: This epoch took 130.577428 s
 
2022-04-06 17:53:03.736551: svmem(total=99932160000, available=87506051072, percent=12.4, used=9477017600, free=73421246464, active=20885729280, inactive=3515420672, buffers=13791232, cached=17020104704, shared=1133957120, slab=774189056) 
2022-04-06 17:53:03.737303: 
epoch:  98 
2022-04-06 17:54:53.459671: train loss : -0.4773 
2022-04-06 17:55:04.269891: validation loss: -0.4495 
2022-04-06 17:55:04.279104: Average global foreground Dice: [0.7132, 0.9325, 0.9525, 0.3748] 
2022-04-06 17:55:04.286876: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:55:04.983964: lr: 0.008199 
2022-04-06 17:55:04.998923: saving scheduled checkpoint file... 
2022-04-06 17:55:05.146574: saving checkpoint... 
2022-04-06 17:55:09.816617: done, saving took 4.81 seconds 
2022-04-06 17:55:09.851640: done 
2022-04-06 17:55:09.956022: saving checkpoint... 
2022-04-06 17:55:14.383035: done, saving took 4.53 seconds 
2022-04-06 17:55:14.405601: This epoch took 130.667660 s
 
2022-04-06 17:55:14.408260: svmem(total=99932160000, available=87593459712, percent=12.3, used=9389187072, free=73447784448, active=20764045312, inactive=3610046464, buffers=14061568, cached=17081126912, shared=1133957120, slab=774701056) 
2022-04-06 17:55:14.410138: 
epoch:  99 
2022-04-06 17:57:02.044188: train loss : -0.4805 
2022-04-06 17:57:13.506365: validation loss: -0.4794 
2022-04-06 17:57:13.514976: Average global foreground Dice: [0.7326, 0.9358, 0.9592, 0.3444] 
2022-04-06 17:57:13.525263: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:57:14.445408: lr: 0.008181 
2022-04-06 17:57:14.465110: saving scheduled checkpoint file... 
2022-04-06 17:57:14.636290: saving checkpoint... 
2022-04-06 17:57:19.128070: done, saving took 4.64 seconds 
2022-04-06 17:57:19.175189: done 
2022-04-06 17:57:19.281600: saving checkpoint... 
2022-04-06 17:57:23.735484: done, saving took 4.55 seconds 
2022-04-06 17:57:23.756896: This epoch took 129.344899 s
 
2022-04-06 17:57:23.758896: svmem(total=99932160000, available=87511707648, percent=12.4, used=9471221760, free=73441914880, active=20832894976, inactive=3544113152, buffers=14299136, cached=17004724224, shared=1133957120, slab=775372800) 
2022-04-06 17:57:23.760929: 
epoch:  100 
2022-04-06 17:59:11.471113: train loss : -0.4614 
2022-04-06 17:59:22.876571: validation loss: -0.4262 
2022-04-06 17:59:22.880934: Average global foreground Dice: [0.7419, 0.917, 0.9332, 0.3643] 
2022-04-06 17:59:22.883754: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 17:59:23.687795: lr: 0.008162 
2022-04-06 17:59:23.699745: saving scheduled checkpoint file... 
2022-04-06 17:59:23.838191: saving checkpoint... 
2022-04-06 17:59:28.410709: done, saving took 4.70 seconds 
2022-04-06 17:59:28.442756: done 
2022-04-06 17:59:28.450634: This epoch took 124.687370 s
 
2022-04-06 17:59:28.457091: svmem(total=99932160000, available=87542312960, percent=12.4, used=9440219136, free=73438818304, active=20862201856, inactive=3515645952, buffers=14536704, cached=17038585856, shared=1133957120, slab=776052736) 
2022-04-06 17:59:28.464616: 
epoch:  101 
2022-04-06 18:01:18.739386: train loss : -0.3893 
2022-04-06 18:01:29.985955: validation loss: -0.3962 
2022-04-06 18:01:29.994720: Average global foreground Dice: [0.7402, 0.9054, 0.939, 0.3271] 
2022-04-06 18:01:30.005891: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:01:30.534479: lr: 0.008144 
2022-04-06 18:01:30.547133: saving scheduled checkpoint file... 
2022-04-06 18:01:30.695773: saving checkpoint... 
2022-04-06 18:01:35.330989: done, saving took 4.77 seconds 
2022-04-06 18:01:35.364688: done 
2022-04-06 18:01:35.370857: This epoch took 126.900261 s
 
2022-04-06 18:01:35.377132: svmem(total=99932160000, available=87609536512, percent=12.3, used=9372741632, free=73511600128, active=20730052608, inactive=3579777024, buffers=14794752, cached=17033023488, shared=1133957120, slab=775098368) 
2022-04-06 18:01:35.383475: 
epoch:  102 
2022-04-06 18:03:26.046511: train loss : -0.4043 
2022-04-06 18:03:37.158294: validation loss: -0.4001 
2022-04-06 18:03:37.174683: Average global foreground Dice: [0.742, 0.9133, 0.9312, 0.3505] 
2022-04-06 18:03:37.181024: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:03:38.011969: lr: 0.008125 
2022-04-06 18:03:38.013070: saving scheduled checkpoint file... 
2022-04-06 18:03:38.206656: saving checkpoint... 
2022-04-06 18:03:42.939076: done, saving took 4.93 seconds 
2022-04-06 18:03:42.953307: done 
2022-04-06 18:03:42.956794: This epoch took 127.568257 s
 
2022-04-06 18:03:42.959860: svmem(total=99932160000, available=87578853376, percent=12.4, used=9403801600, free=73339494400, active=20926869504, inactive=3556261888, buffers=15048704, cached=17173815296, shared=1133957120, slab=775680000) 
2022-04-06 18:03:42.962980: 
epoch:  103 
2022-04-06 18:05:30.986594: train loss : -0.4623 
2022-04-06 18:05:42.586575: validation loss: -0.4504 
2022-04-06 18:05:42.595350: Average global foreground Dice: [0.7812, 0.9238, 0.9503, 0.3366] 
2022-04-06 18:05:42.600827: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:05:43.447384: lr: 0.008107 
2022-04-06 18:05:43.450047: saving scheduled checkpoint file... 
2022-04-06 18:05:43.538162: saving checkpoint... 
2022-04-06 18:05:48.041519: done, saving took 4.59 seconds 
2022-04-06 18:05:48.071049: done 
2022-04-06 18:05:48.076979: This epoch took 125.112809 s
 
2022-04-06 18:05:48.080642: svmem(total=99932160000, available=87483338752, percent=12.5, used=9499332608, free=73335103488, active=20996268032, inactive=3487375360, buffers=15310848, cached=17082413056, shared=1133965312, slab=775643136) 
2022-04-06 18:05:48.084027: 
epoch:  104 
2022-04-06 18:07:36.566859: train loss : -0.4754 
2022-04-06 18:07:48.014383: validation loss: -0.4694 
2022-04-06 18:07:48.023669: Average global foreground Dice: [0.7691, 0.9295, 0.9507, 0.3588] 
2022-04-06 18:07:48.032701: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:07:48.792499: lr: 0.008088 
2022-04-06 18:07:48.798625: saving scheduled checkpoint file... 
2022-04-06 18:07:48.883911: saving checkpoint... 
2022-04-06 18:07:53.336044: done, saving took 4.53 seconds 
2022-04-06 18:07:53.357078: done 
2022-04-06 18:07:53.440488: saving checkpoint... 
2022-04-06 18:07:57.907342: done, saving took 4.55 seconds 
2022-04-06 18:07:57.923053: This epoch took 129.834129 s
 
2022-04-06 18:07:57.924164: svmem(total=99932160000, available=87568515072, percent=12.4, used=9414221824, free=73413427200, active=20779540480, inactive=3626737664, buffers=15548416, cached=17088962560, shared=1133969408, slab=775860224) 
2022-04-06 18:07:57.924829: 
epoch:  105 
2022-04-06 18:09:46.808860: train loss : -0.4785 
2022-04-06 18:09:58.358720: validation loss: -0.5130 
2022-04-06 18:09:58.369726: Average global foreground Dice: [0.7625, 0.9349, 0.9638, 0.3897] 
2022-04-06 18:09:58.373563: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:09:59.255083: lr: 0.00807 
2022-04-06 18:09:59.258150: saving scheduled checkpoint file... 
2022-04-06 18:09:59.389108: saving checkpoint... 
2022-04-06 18:10:03.851314: done, saving took 4.59 seconds 
2022-04-06 18:10:03.900454: done 
2022-04-06 18:10:04.009495: saving checkpoint... 
2022-04-06 18:10:08.444819: done, saving took 4.53 seconds 
2022-04-06 18:10:08.466978: This epoch took 130.541479 s
 
2022-04-06 18:10:08.470645: svmem(total=99932160000, available=87563309056, percent=12.4, used=9419526144, free=73436561408, active=20785504256, inactive=3596201984, buffers=15802368, cached=17060270080, shared=1133993984, slab=775864320) 
2022-04-06 18:10:08.474244: 
epoch:  106 
2022-04-06 18:11:57.747007: train loss : -0.5024 
2022-04-06 18:12:09.538743: validation loss: -0.4971 
2022-04-06 18:12:09.540739: Average global foreground Dice: [0.7796, 0.9371, 0.9628, 0.336] 
2022-04-06 18:12:09.541756: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:12:10.435338: lr: 0.008052 
2022-04-06 18:12:10.444661: saving scheduled checkpoint file... 
2022-04-06 18:12:10.594810: saving checkpoint... 
2022-04-06 18:12:15.065259: done, saving took 4.60 seconds 
2022-04-06 18:12:15.107803: done 
2022-04-06 18:12:15.206795: saving checkpoint... 
2022-04-06 18:12:19.645066: done, saving took 4.53 seconds 
2022-04-06 18:12:19.663073: This epoch took 131.185713 s
 
2022-04-06 18:12:19.664088: svmem(total=99932160000, available=87485394944, percent=12.5, used=9497251840, free=73436123136, active=20825190400, inactive=3554320384, buffers=16048128, cached=16982736896, shared=1133998080, slab=775790592) 
2022-04-06 18:12:19.664746: 
epoch:  107 
2022-04-06 18:14:08.677968: train loss : -0.3257 
2022-04-06 18:14:20.788240: validation loss: -0.3465 
2022-04-06 18:14:20.803448: Average global foreground Dice: [0.6121, 0.9026, 0.9534, 0.3338] 
2022-04-06 18:14:20.820458: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:14:22.329561: lr: 0.008033 
2022-04-06 18:14:22.339818: saving scheduled checkpoint file... 
2022-04-06 18:14:22.432907: saving checkpoint... 
2022-04-06 18:14:26.848963: done, saving took 4.50 seconds 
2022-04-06 18:14:26.857064: done 
2022-04-06 18:14:26.857875: This epoch took 127.192522 s
 
2022-04-06 18:14:26.858602: svmem(total=99932160000, available=87465041920, percent=12.5, used=9518264320, free=73325899776, active=20927741952, inactive=3561205760, buffers=16285696, cached=17071710208, shared=1133998080, slab=775430144) 
2022-04-06 18:14:26.859325: 
epoch:  108 
2022-04-06 18:16:16.043522: train loss : -0.3633 
2022-04-06 18:16:29.001209: validation loss: -0.3863 
2022-04-06 18:16:29.010251: Average global foreground Dice: [0.7379, 0.9161, 0.9334, 0.3079] 
2022-04-06 18:16:29.010914: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:16:29.920010: lr: 0.008015 
2022-04-06 18:16:29.923654: saving scheduled checkpoint file... 
2022-04-06 18:16:29.975846: saving checkpoint... 
2022-04-06 18:16:34.432262: done, saving took 4.51 seconds 
2022-04-06 18:16:34.441504: done 
2022-04-06 18:16:34.442181: This epoch took 127.581788 s
 
2022-04-06 18:16:34.443072: svmem(total=99932160000, available=87446188032, percent=12.5, used=9536450560, free=73329213440, active=20983009280, inactive=3505434624, buffers=16547840, cached=17049948160, shared=1133998080, slab=775364608) 
2022-04-06 18:16:34.443674: 
epoch:  109 
2022-04-06 18:18:22.042702: train loss : -0.4191 
2022-04-06 18:18:33.038021: validation loss: -0.4788 
2022-04-06 18:18:33.069942: Average global foreground Dice: [0.7521, 0.9337, 0.9543, 0.3548] 
2022-04-06 18:18:33.075272: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:18:33.733630: lr: 0.007996 
2022-04-06 18:18:33.736635: saving scheduled checkpoint file... 
2022-04-06 18:18:33.843672: saving checkpoint... 
2022-04-06 18:18:38.495265: done, saving took 4.76 seconds 
2022-04-06 18:18:38.507324: done 
2022-04-06 18:18:38.511953: This epoch took 124.067691 s
 
2022-04-06 18:18:38.513357: svmem(total=99932160000, available=87497695232, percent=12.4, used=9485008896, free=73327255552, active=20973441024, inactive=3516477440, buffers=16801792, cached=17103093760, shared=1133998080, slab=775819264) 
2022-04-06 18:18:38.516565: 
epoch:  110 
2022-04-06 18:20:29.973952: train loss : -0.4640 
2022-04-06 18:20:41.393069: validation loss: -0.4983 
2022-04-06 18:20:41.407708: Average global foreground Dice: [0.7645, 0.93, 0.9527, 0.3613] 
2022-04-06 18:20:41.413642: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:20:42.218933: lr: 0.007978 
2022-04-06 18:20:42.232934: saving scheduled checkpoint file... 
2022-04-06 18:20:42.306971: saving checkpoint... 
2022-04-06 18:20:46.806067: done, saving took 4.56 seconds 
2022-04-06 18:20:46.833071: done 
2022-04-06 18:20:46.836574: This epoch took 128.316675 s
 
2022-04-06 18:20:46.840843: svmem(total=99932160000, available=87434891264, percent=12.5, used=9547653120, free=73323347968, active=20940939264, inactive=3549663232, buffers=17063936, cached=17044094976, shared=1134002176, slab=776331264) 
2022-04-06 18:20:46.842341: 
epoch:  111 
2022-04-06 18:22:36.056746: train loss : -0.4992 
2022-04-06 18:22:47.247416: validation loss: -0.5346 
2022-04-06 18:22:47.264881: Average global foreground Dice: [0.7736, 0.9423, 0.9705, 0.3616] 
2022-04-06 18:22:47.283536: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:22:48.164604: lr: 0.007959 
2022-04-06 18:22:48.167608: saving scheduled checkpoint file... 
2022-04-06 18:22:48.285223: saving checkpoint... 
2022-04-06 18:22:52.776835: done, saving took 4.60 seconds 
2022-04-06 18:22:52.821669: done 
2022-04-06 18:22:52.831164: This epoch took 125.986537 s
 
2022-04-06 18:22:52.844425: svmem(total=99932160000, available=87599702016, percent=12.3, used=9380667392, free=73387012096, active=20863905792, inactive=3565608960, buffers=17301504, cached=17147179008, shared=1134002176, slab=775479296) 
2022-04-06 18:22:52.855371: 
epoch:  112 
2022-04-06 18:24:42.130994: train loss : -0.4811 
2022-04-06 18:24:53.856436: validation loss: -0.5138 
2022-04-06 18:24:53.865012: Average global foreground Dice: [0.7419, 0.936, 0.9613, 0.3717] 
2022-04-06 18:24:53.866661: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:24:54.618014: lr: 0.007941 
2022-04-06 18:24:54.619683: saving scheduled checkpoint file... 
2022-04-06 18:24:54.783193: saving checkpoint... 
2022-04-06 18:24:59.293480: done, saving took 4.67 seconds 
2022-04-06 18:24:59.321801: done 
2022-04-06 18:24:59.326297: This epoch took 126.458728 s
 
2022-04-06 18:24:59.332569: svmem(total=99932160000, available=87526383616, percent=12.4, used=9455771648, free=73435205632, active=20745646080, inactive=3639947264, buffers=17567744, cached=17023614976, shared=1134002176, slab=770707456) 
2022-04-06 18:24:59.335600: 
epoch:  113 
2022-04-06 18:26:47.861630: train loss : -0.5140 
2022-04-06 18:27:00.493321: validation loss: -0.5223 
2022-04-06 18:27:00.508505: Average global foreground Dice: [0.7985, 0.9397, 0.9656, 0.3649] 
2022-04-06 18:27:00.525846: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:27:01.268757: lr: 0.007922 
2022-04-06 18:27:01.282812: saving scheduled checkpoint file... 
2022-04-06 18:27:01.386332: saving checkpoint... 
2022-04-06 18:27:05.718518: done, saving took 4.42 seconds 
2022-04-06 18:27:05.750084: done 
2022-04-06 18:27:05.815887: saving checkpoint... 
2022-04-06 18:27:10.291924: done, saving took 4.54 seconds 
2022-04-06 18:27:10.312799: This epoch took 130.976098 s
 
2022-04-06 18:27:10.314858: svmem(total=99932160000, available=88410537984, percent=11.5, used=8698929152, free=86007619584, active=7669907456, inactive=4585611264, buffers=7200768, cached=5218410496, shared=1042821120, slab=670003200) 
2022-04-06 18:27:10.316812: 
epoch:  114 
2022-04-06 18:28:58.017345: train loss : -0.5312 
2022-04-06 18:29:25.681859: validation loss: -0.5200 
2022-04-06 18:29:25.723335: Average global foreground Dice: [0.7487, 0.943, 0.9646, 0.3601] 
2022-04-06 18:29:25.763387: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:29:26.314933: lr: 0.007904 
2022-04-06 18:29:26.335826: saving scheduled checkpoint file... 
2022-04-06 18:29:26.437226: saving checkpoint... 
2022-04-06 18:29:30.918235: done, saving took 4.55 seconds 
2022-04-06 18:29:31.008142: done 
2022-04-06 18:29:31.072265: saving checkpoint... 
2022-04-06 18:29:35.496208: done, saving took 4.47 seconds 
2022-04-06 18:29:35.585076: This epoch took 145.264112 s
 
2022-04-06 18:29:35.636718: svmem(total=99932160000, available=85482086400, percent=14.5, used=11593887744, free=77107888128, active=10988732416, inactive=10146652160, buffers=8867840, cached=11221516288, shared=1067294720, slab=689909760) 
2022-04-06 18:29:35.696269: 
epoch:  115 
2022-04-06 18:31:24.984145: train loss : -0.5218 
2022-04-06 18:31:36.677991: validation loss: -0.5278 
2022-04-06 18:31:36.689266: Average global foreground Dice: [0.7689, 0.9382, 0.9622, 0.371] 
2022-04-06 18:31:36.700798: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:31:37.744084: lr: 0.007885 
2022-04-06 18:31:37.750354: saving scheduled checkpoint file... 
2022-04-06 18:31:37.915818: saving checkpoint... 
2022-04-06 18:31:42.415536: done, saving took 4.66 seconds 
2022-04-06 18:31:42.454225: done 
2022-04-06 18:31:42.510991: saving checkpoint... 
2022-04-06 18:31:46.908286: done, saving took 4.44 seconds 
2022-04-06 18:31:46.922952: This epoch took 131.181128 s
 
2022-04-06 18:31:46.923918: svmem(total=99932160000, available=82967179264, percent=17.0, used=14097428480, free=64773373952, active=18171289600, inactive=15270694912, buffers=9158656, cached=21052198912, shared=1067294720, slab=713592832) 
2022-04-06 18:31:46.924735: 
epoch:  116 
2022-04-06 18:33:34.037694: train loss : -0.5389 
2022-04-06 18:33:44.690267: validation loss: -0.5169 
2022-04-06 18:33:44.712140: Average global foreground Dice: [0.8201, 0.9343, 0.9629, 0.3734] 
2022-04-06 18:33:44.724321: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:33:45.675747: lr: 0.007867 
2022-04-06 18:33:45.720728: saving scheduled checkpoint file... 
2022-04-06 18:33:45.822974: saving checkpoint... 
2022-04-06 18:33:50.758478: done, saving took 4.99 seconds 
2022-04-06 18:33:50.793918: done 
2022-04-06 18:33:50.881808: saving checkpoint... 
2022-04-06 18:33:55.279460: done, saving took 4.48 seconds 
2022-04-06 18:33:55.301841: This epoch took 128.376348 s
 
2022-04-06 18:33:55.306831: svmem(total=99932160000, available=88311623680, percent=11.6, used=8747675648, free=70172200960, active=17137233920, inactive=10903764992, buffers=9420800, cached=21002862592, shared=1067294720, slab=723333120) 
2022-04-06 18:33:55.311581: 
epoch:  117 
2022-04-06 18:35:43.491179: train loss : -0.5088 
2022-04-06 18:35:54.958140: validation loss: -0.4450 
2022-04-06 18:35:54.961905: Average global foreground Dice: [0.7508, 0.9124, 0.9547, 0.3634] 
2022-04-06 18:35:54.962646: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:35:55.654933: lr: 0.007848 
2022-04-06 18:35:55.664895: saving scheduled checkpoint file... 
2022-04-06 18:35:55.851290: saving checkpoint... 
2022-04-06 18:36:00.550312: done, saving took 4.88 seconds 
2022-04-06 18:36:00.583925: done 
2022-04-06 18:36:00.589813: This epoch took 125.271553 s
 
2022-04-06 18:36:00.596906: svmem(total=99932160000, available=88409567232, percent=11.5, used=8647114752, free=70196350976, active=19528577024, inactive=8485363712, buffers=9830400, cached=21078863872, shared=1067294720, slab=728674304) 
2022-04-06 18:36:00.603302: 
epoch:  118 
2022-04-06 18:37:47.934134: train loss : -0.4910 
2022-04-06 18:37:59.894436: validation loss: -0.5343 
2022-04-06 18:37:59.898952: Average global foreground Dice: [0.7922, 0.9335, 0.9765, 0.3921] 
2022-04-06 18:37:59.903093: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:38:00.819618: lr: 0.00783 
2022-04-06 18:38:00.826031: saving scheduled checkpoint file... 
2022-04-06 18:38:00.933229: saving checkpoint... 
2022-04-06 18:38:05.341777: done, saving took 4.51 seconds 
2022-04-06 18:38:05.361202: done 
2022-04-06 18:38:05.456711: saving checkpoint... 
2022-04-06 18:38:09.851134: done, saving took 4.49 seconds 
2022-04-06 18:38:09.869188: This epoch took 129.256399 s
 
2022-04-06 18:38:09.870381: svmem(total=99932160000, available=88368836608, percent=11.6, used=8686858240, free=70202748928, active=19974512640, inactive=8027049984, buffers=10084352, cached=21032468480, shared=1067294720, slab=731127808) 
2022-04-06 18:38:09.871174: 
epoch:  119 
2022-04-06 18:39:58.335968: train loss : -0.4898 
2022-04-06 18:40:10.596463: validation loss: -0.4473 
2022-04-06 18:40:10.618270: Average global foreground Dice: [0.7862, 0.922, 0.9428, 0.347] 
2022-04-06 18:40:10.629885: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:40:11.867931: lr: 0.007811 
2022-04-06 18:40:11.879196: saving scheduled checkpoint file... 
2022-04-06 18:40:11.979189: saving checkpoint... 
2022-04-06 18:40:16.351292: done, saving took 4.46 seconds 
2022-04-06 18:40:16.388152: done 
2022-04-06 18:40:16.396080: This epoch took 126.523957 s
 
2022-04-06 18:40:16.403089: svmem(total=99932160000, available=88358047744, percent=11.6, used=8696598528, free=70092455936, active=20035846144, inactive=8071757824, buffers=10321920, cached=21132783616, shared=1067294720, slab=732606464) 
2022-04-06 18:40:16.412189: 
epoch:  120 
2022-04-06 18:42:03.468602: train loss : -0.4774 
2022-04-06 18:42:14.669858: validation loss: -0.4822 
2022-04-06 18:42:14.681407: Average global foreground Dice: [0.7399, 0.9288, 0.9587, 0.3559] 
2022-04-06 18:42:14.695597: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:42:15.471592: lr: 0.007793 
2022-04-06 18:42:15.474479: saving scheduled checkpoint file... 
2022-04-06 18:42:15.525718: saving checkpoint... 
2022-04-06 18:42:19.991102: done, saving took 4.51 seconds 
2022-04-06 18:42:20.020947: done 
2022-04-06 18:42:20.026513: This epoch took 123.608790 s
 
2022-04-06 18:42:20.033778: svmem(total=99932160000, available=88232218624, percent=11.7, used=8821817344, free=70090346496, active=19824955392, inactive=8281665536, buffers=10551296, cached=21009444864, shared=1067294720, slab=735358976) 
2022-04-06 18:42:20.046197: 
epoch:  121 
2022-04-06 18:44:05.385479: train loss : -0.5045 
2022-04-06 18:44:16.508157: validation loss: -0.5621 
2022-04-06 18:44:16.523157: Average global foreground Dice: [0.7889, 0.942, 0.972, 0.3764] 
2022-04-06 18:44:16.531545: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:44:17.473602: lr: 0.007774 
2022-04-06 18:44:17.478774: saving scheduled checkpoint file... 
2022-04-06 18:44:17.590342: saving checkpoint... 
2022-04-06 18:44:22.244261: done, saving took 4.76 seconds 
2022-04-06 18:44:22.265692: done 
2022-04-06 18:44:22.318563: saving checkpoint... 
2022-04-06 18:44:26.729725: done, saving took 4.46 seconds 
2022-04-06 18:44:26.744219: This epoch took 126.690057 s
 
2022-04-06 18:44:26.746353: svmem(total=99932160000, available=88176099328, percent=11.8, used=8877256704, free=70092718080, active=19883085824, inactive=8222801920, buffers=10788864, cached=20951396352, shared=1067294720, slab=736022528) 
2022-04-06 18:44:26.748399: 
epoch:  122 
2022-04-06 18:46:15.036246: train loss : -0.5032 
2022-04-06 18:46:26.708798: validation loss: -0.4823 
2022-04-06 18:46:26.722358: Average global foreground Dice: [0.7847, 0.9261, 0.949, 0.3712] 
2022-04-06 18:46:26.731874: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:46:27.590688: lr: 0.007756 
2022-04-06 18:46:27.598670: saving scheduled checkpoint file... 
2022-04-06 18:46:27.691494: saving checkpoint... 
2022-04-06 18:46:32.419908: done, saving took 4.81 seconds 
2022-04-06 18:46:32.432777: done 
2022-04-06 18:46:32.484658: saving checkpoint... 
2022-04-06 18:46:36.866458: done, saving took 4.43 seconds 
2022-04-06 18:46:36.880714: This epoch took 130.130122 s
 
2022-04-06 18:46:36.881872: svmem(total=99932160000, available=88247844864, percent=11.7, used=8805527552, free=70113898496, active=20122886144, inactive=7961407488, buffers=11067392, cached=21001666560, shared=1067294720, slab=736157696) 
2022-04-06 18:46:36.882617: 
epoch:  123 
2022-04-06 18:48:23.705535: train loss : -0.5042 
2022-04-06 18:48:35.926542: validation loss: -0.5181 
2022-04-06 18:48:35.936492: Average global foreground Dice: [0.7727, 0.9379, 0.9643, 0.3667] 
2022-04-06 18:48:35.937404: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:48:36.903661: lr: 0.007737 
2022-04-06 18:48:36.904697: saving scheduled checkpoint file... 
2022-04-06 18:48:37.028593: saving checkpoint... 
2022-04-06 18:48:41.462244: done, saving took 4.56 seconds 
2022-04-06 18:48:41.492502: done 
2022-04-06 18:48:41.582291: saving checkpoint... 
2022-04-06 18:48:46.002084: done, saving took 4.50 seconds 
2022-04-06 18:48:46.016395: This epoch took 129.133067 s
 
2022-04-06 18:48:46.017632: svmem(total=99932160000, available=87366721536, percent=12.6, used=9683894272, free=69120446464, active=26130391040, inactive=2946736128, buffers=11517952, cached=21116301312, shared=1067302912, slab=741330944) 
2022-04-06 18:48:46.018578: 
epoch:  124 
2022-04-06 18:50:33.675012: train loss : -0.5143 
2022-04-06 18:50:44.859463: validation loss: -0.5213 
2022-04-06 18:50:44.869220: Average global foreground Dice: [0.7947, 0.936, 0.9676, 0.3655] 
2022-04-06 18:50:44.871177: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:50:45.975543: lr: 0.007719 
2022-04-06 18:50:45.979077: saving scheduled checkpoint file... 
2022-04-06 18:50:46.096853: saving checkpoint... 
2022-04-06 18:50:50.550903: done, saving took 4.57 seconds 
2022-04-06 18:50:50.574464: done 
2022-04-06 18:50:50.671038: saving checkpoint... 
2022-04-06 18:50:55.082276: done, saving took 4.50 seconds 
2022-04-06 18:50:55.099480: This epoch took 129.080224 s
 
2022-04-06 18:50:55.100450: svmem(total=99932160000, available=88253333504, percent=11.7, used=8797122560, free=70004215808, active=25360740352, inactive=2833940480, buffers=11784192, cached=21119037440, shared=1067298816, slab=743534592) 
2022-04-06 18:50:55.101140: 
epoch:  125 
2022-04-06 18:52:43.456987: train loss : -0.5236 
2022-04-06 18:52:55.851188: validation loss: -0.5289 
2022-04-06 18:52:55.854728: Average global foreground Dice: [0.797, 0.9379, 0.9623, 0.3623] 
2022-04-06 18:52:55.856888: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:52:56.666513: lr: 0.0077 
2022-04-06 18:52:56.670609: saving scheduled checkpoint file... 
2022-04-06 18:52:56.766396: saving checkpoint... 
2022-04-06 18:53:01.120271: done, saving took 4.44 seconds 
2022-04-06 18:53:01.153905: done 
2022-04-06 18:53:01.209142: saving checkpoint... 
2022-04-06 18:53:05.511351: done, saving took 4.35 seconds 
2022-04-06 18:53:05.525106: This epoch took 130.423219 s
 
2022-04-06 18:53:05.526254: svmem(total=99932160000, available=88134217728, percent=11.8, used=8916209664, free=69847334912, active=25401876480, inactive=2949038080, buffers=12029952, cached=21156585472, shared=1067298816, slab=741617664) 
2022-04-06 18:53:05.527054: 
epoch:  126 
2022-04-06 18:54:53.266076: train loss : -0.5234 
2022-04-06 18:55:11.788794: validation loss: -0.5479 
2022-04-06 18:55:11.809892: Average global foreground Dice: [0.7863, 0.9375, 0.9719, 0.3948] 
2022-04-06 18:55:11.823877: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:55:12.348178: lr: 0.007682 
2022-04-06 18:55:12.364006: saving scheduled checkpoint file... 
2022-04-06 18:55:12.557857: saving checkpoint... 
2022-04-06 18:55:16.891493: done, saving took 4.49 seconds 
2022-04-06 18:55:16.940114: done 
2022-04-06 18:55:17.007437: saving checkpoint... 
2022-04-06 18:55:21.244815: done, saving took 4.28 seconds 
2022-04-06 18:55:21.261690: This epoch took 135.733856 s
 
2022-04-06 18:55:21.267217: svmem(total=99932160000, available=85483331584, percent=14.5, used=11574546432, free=77042298880, active=11382169600, inactive=9700147200, buffers=9191424, cached=11306123264, shared=1079881728, slab=711090176) 
2022-04-06 18:55:21.273875: 
epoch:  127 
2022-04-06 18:57:08.551971: train loss : -0.5466 
2022-04-06 18:57:21.673262: validation loss: -0.5558 
2022-04-06 18:57:21.690118: Average global foreground Dice: [0.8002, 0.9446, 0.9732, 0.35] 
2022-04-06 18:57:21.706205: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:57:22.538536: lr: 0.007663 
2022-04-06 18:57:22.567122: saving scheduled checkpoint file... 
2022-04-06 18:57:22.640535: saving checkpoint... 
2022-04-06 18:57:26.996743: done, saving took 4.41 seconds 
2022-04-06 18:57:27.048437: done 
2022-04-06 18:57:27.108327: saving checkpoint... 
2022-04-06 18:57:31.432271: done, saving took 4.37 seconds 
2022-04-06 18:57:31.462543: This epoch took 130.182529 s
 
2022-04-06 18:57:31.473384: svmem(total=99932160000, available=85265838080, percent=14.7, used=11815157760, free=67240615936, active=16408838144, inactive=14551306240, buffers=9568256, cached=20866818048, shared=1042825216, slab=730800128) 
2022-04-06 18:57:31.483583: 
epoch:  128 
2022-04-06 18:59:18.257132: train loss : -0.5521 
2022-04-06 18:59:29.973281: validation loss: -0.5499 
2022-04-06 18:59:29.978336: Average global foreground Dice: [0.8173, 0.9368, 0.9697, 0.3757] 
2022-04-06 18:59:29.979161: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 18:59:30.831039: lr: 0.007645 
2022-04-06 18:59:30.834237: saving scheduled checkpoint file... 
2022-04-06 18:59:30.975169: saving checkpoint... 
2022-04-06 18:59:35.507735: done, saving took 4.67 seconds 
2022-04-06 18:59:35.530260: done 
2022-04-06 18:59:35.582888: saving checkpoint... 
2022-04-06 18:59:39.955505: done, saving took 4.42 seconds 
2022-04-06 18:59:39.970781: This epoch took 128.477040 s
 
2022-04-06 18:59:39.972089: svmem(total=99932160000, available=88270684160, percent=11.7, used=8777494528, free=67500253184, active=17231306752, inactive=13462962176, buffers=9846784, cached=23644565504, shared=1067298816, slab=746123264) 
2022-04-06 18:59:39.972919: 
epoch:  129 
2022-04-06 19:01:29.301099: train loss : -0.4799 
2022-04-06 19:01:40.922547: validation loss: -0.4956 
2022-04-06 19:01:40.924137: Average global foreground Dice: [0.7722, 0.938, 0.9535, 0.3566] 
2022-04-06 19:01:40.925798: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:01:41.740545: lr: 0.007626 
2022-04-06 19:01:41.741596: saving scheduled checkpoint file... 
2022-04-06 19:01:41.808145: saving checkpoint... 
2022-04-06 19:01:46.203526: done, saving took 4.46 seconds 
2022-04-06 19:01:46.226504: done 
2022-04-06 19:01:46.232136: This epoch took 126.258589 s
 
2022-04-06 19:01:46.238545: svmem(total=99932160000, available=87559929856, percent=12.4, used=9486020608, free=66843029504, active=25597308928, inactive=5745664000, buffers=10530816, cached=23592579072, shared=1067307008, slab=752193536) 
2022-04-06 19:01:46.242267: 
epoch:  130 
2022-04-06 19:03:35.173905: train loss : -0.4585 
2022-04-06 19:03:46.207083: validation loss: -0.5106 
2022-04-06 19:03:46.217767: Average global foreground Dice: [0.7981, 0.9297, 0.9654, 0.3863] 
2022-04-06 19:03:46.226506: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:03:46.808295: lr: 0.007608 
2022-04-06 19:03:46.810430: saving scheduled checkpoint file... 
2022-04-06 19:03:46.892888: saving checkpoint... 
2022-04-06 19:03:51.464025: done, saving took 4.65 seconds 
2022-04-06 19:03:51.485192: done 
2022-04-06 19:03:51.567592: saving checkpoint... 
2022-04-06 19:03:55.968443: done, saving took 4.48 seconds 
2022-04-06 19:03:55.989412: This epoch took 129.743077 s
 
2022-04-06 19:03:55.993292: svmem(total=99932160000, available=88173215744, percent=11.8, used=8870825984, free=67556274176, active=25724252160, inactive=4901220352, buffers=10809344, cached=23494250496, shared=1067298816, slab=755200000) 
2022-04-06 19:03:55.997802: 
epoch:  131 
2022-04-06 19:05:42.122149: train loss : -0.4664 
2022-04-06 19:05:59.165652: validation loss: -0.4950 
2022-04-06 19:05:59.269741: Average global foreground Dice: [0.79, 0.9333, 0.9501, 0.378] 
2022-04-06 19:05:59.289093: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:05:59.951940: lr: 0.007589 
2022-04-06 19:05:59.975781: saving scheduled checkpoint file... 
2022-04-06 19:06:00.058768: saving checkpoint... 
2022-04-06 19:06:04.412721: done, saving took 4.42 seconds 
2022-04-06 19:06:04.461326: done 
2022-04-06 19:06:04.528524: saving checkpoint... 
2022-04-06 19:06:09.006661: done, saving took 4.53 seconds 
2022-04-06 19:06:09.035666: This epoch took 133.033924 s
 
2022-04-06 19:06:09.048454: svmem(total=99932160000, available=87960248320, percent=12.0, used=9115111424, free=83155972096, active=8089006080, inactive=7001980928, buffers=8859648, cached=7652216832, shared=1067298816, slab=695566336) 
2022-04-06 19:06:09.059791: 
epoch:  132 
2022-04-06 19:07:58.574525: train loss : -0.4675 
2022-04-06 19:08:16.333574: validation loss: -0.5100 
2022-04-06 19:08:16.350812: Average global foreground Dice: [0.7818, 0.9348, 0.9597, 0.3827] 
2022-04-06 19:08:16.364119: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:08:17.106070: lr: 0.007571 
2022-04-06 19:08:17.121981: saving scheduled checkpoint file... 
2022-04-06 19:08:17.239457: saving checkpoint... 
2022-04-06 19:08:21.839367: done, saving took 4.70 seconds 
2022-04-06 19:08:21.892138: done 
2022-04-06 19:08:21.958473: saving checkpoint... 
2022-04-06 19:08:26.467121: done, saving took 4.55 seconds 
2022-04-06 19:08:26.557084: This epoch took 137.488318 s
 
2022-04-06 19:08:26.580297: svmem(total=99932160000, available=84156735488, percent=15.8, used=12923809792, free=65403187200, active=16144973824, inactive=16653795328, buffers=9408512, cached=21595754496, shared=1042825216, slab=730771456) 
2022-04-06 19:08:26.606588: 
epoch:  133 
2022-04-06 19:10:16.276351: train loss : -0.5028 
2022-04-06 19:10:36.374933: validation loss: -0.5229 
2022-04-06 19:10:36.377779: Average global foreground Dice: [0.7819, 0.9409, 0.9698, 0.358] 
2022-04-06 19:10:36.396978: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:10:37.003413: lr: 0.007552 
2022-04-06 19:10:37.016504: saving scheduled checkpoint file... 
2022-04-06 19:10:37.115400: saving checkpoint... 
2022-04-06 19:10:41.646604: done, saving took 4.62 seconds 
2022-04-06 19:10:41.725729: done 
2022-04-06 19:10:41.831760: saving checkpoint... 
2022-04-06 19:10:46.247248: done, saving took 4.49 seconds 
2022-04-06 19:10:46.298136: This epoch took 139.669935 s
 
2022-04-06 19:10:46.331293: svmem(total=99932160000, available=84029792256, percent=15.9, used=13001166848, free=74357805056, active=12973670400, inactive=10754691072, buffers=7897088, cached=12565291008, shared=1102934016, slab=726872064) 
2022-04-06 19:10:46.368107: 
epoch:  134 
2022-04-06 19:12:44.165828: train loss : -0.5349 
2022-04-06 19:12:56.202316: validation loss: -0.5474 
2022-04-06 19:12:56.205820: Average global foreground Dice: [0.7787, 0.9501, 0.9701, 0.4132] 
2022-04-06 19:12:56.206634: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:12:57.023031: lr: 0.007533 
2022-04-06 19:12:57.030286: saving scheduled checkpoint file... 
2022-04-06 19:12:57.158523: saving checkpoint... 
2022-04-06 19:13:01.528772: done, saving took 4.49 seconds 
2022-04-06 19:13:01.546491: done 
2022-04-06 19:13:01.644810: saving checkpoint... 
2022-04-06 19:13:06.036860: done, saving took 4.49 seconds 
2022-04-06 19:13:06.054639: This epoch took 139.654165 s
 
2022-04-06 19:13:06.055649: svmem(total=99932160000, available=83212734464, percent=16.7, used=13443104768, free=62408810496, active=17877012480, inactive=17655042048, buffers=8560640, cached=24071684096, shared=1463193600, slab=767176704) 
2022-04-06 19:13:06.056690: 
epoch:  135 
2022-04-06 19:14:55.483588: train loss : -0.4989 
2022-04-06 19:15:06.947740: validation loss: -0.5188 
2022-04-06 19:15:06.958072: Average global foreground Dice: [0.7907, 0.9309, 0.9596, 0.3905] 
2022-04-06 19:15:06.962899: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:15:07.763486: lr: 0.007515 
2022-04-06 19:15:07.774808: saving scheduled checkpoint file... 
2022-04-06 19:15:07.897417: saving checkpoint... 
2022-04-06 19:15:12.337866: done, saving took 4.55 seconds 
2022-04-06 19:15:12.379761: done 
2022-04-06 19:15:12.484933: saving checkpoint... 
2022-04-06 19:15:17.029502: done, saving took 4.64 seconds 
2022-04-06 19:15:17.042669: This epoch took 130.985224 s
 
2022-04-06 19:15:17.043645: svmem(total=99932160000, available=83152609280, percent=16.8, used=13498310656, free=62318612480, active=22014464000, inactive=13591678976, buffers=8998912, cached=24106237952, shared=1463431168, slab=777728000) 
2022-04-06 19:15:17.044347: 
epoch:  136 
2022-04-06 19:17:06.860416: train loss : -0.4668 
2022-04-06 19:17:18.908222: validation loss: -0.4644 
2022-04-06 19:17:18.912798: Average global foreground Dice: [0.7749, 0.9205, 0.9546, 0.3704] 
2022-04-06 19:17:18.916877: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:17:19.807114: lr: 0.007496 
2022-04-06 19:17:19.813262: saving scheduled checkpoint file... 
2022-04-06 19:17:19.974917: saving checkpoint... 
2022-04-06 19:17:24.498751: done, saving took 4.68 seconds 
2022-04-06 19:17:24.524670: done 
2022-04-06 19:17:24.529741: This epoch took 127.484681 s
 
2022-04-06 19:17:24.535230: svmem(total=99932160000, available=83122380800, percent=16.8, used=13539733504, free=62314733568, active=24459628544, inactive=11146481664, buffers=9265152, cached=24068427776, shared=1449725952, slab=782540800) 
2022-04-06 19:17:24.538727: 
epoch:  137 
2022-04-06 19:19:14.114173: train loss : -0.4425 
2022-04-06 19:19:25.405439: validation loss: -0.4525 
2022-04-06 19:19:25.415400: Average global foreground Dice: [0.7484, 0.9072, 0.9328, 0.3831] 
2022-04-06 19:19:25.425388: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:19:26.138305: lr: 0.007478 
2022-04-06 19:19:26.139188: saving scheduled checkpoint file... 
2022-04-06 19:19:26.314183: saving checkpoint... 
2022-04-06 19:19:32.994104: done, saving took 6.85 seconds 
2022-04-06 19:19:33.015944: done 
2022-04-06 19:19:33.016727: This epoch took 128.474034 s
 
2022-04-06 19:19:33.017498: svmem(total=99932160000, available=83182977024, percent=16.8, used=13478027264, free=62391902208, active=24460591104, inactive=11059523584, buffers=9506816, cached=24052723712, shared=1449639936, slab=785043456) 
2022-04-06 19:19:33.018176: 
epoch:  138 
2022-04-06 19:21:22.067239: train loss : -0.4236 
2022-04-06 19:21:32.788417: validation loss: -0.4480 
2022-04-06 19:21:32.790138: Average global foreground Dice: [0.7128, 0.9226, 0.9455, 0.3534] 
2022-04-06 19:21:32.790995: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:21:33.527953: lr: 0.007459 
2022-04-06 19:21:33.531815: saving scheduled checkpoint file... 
2022-04-06 19:21:33.643970: saving checkpoint... 
2022-04-06 19:21:38.096644: done, saving took 4.56 seconds 
2022-04-06 19:21:38.137032: done 
2022-04-06 19:21:38.150044: This epoch took 125.131090 s
 
2022-04-06 19:21:38.164775: svmem(total=99932160000, available=83213656064, percent=16.7, used=13445914624, free=62216900608, active=24776294400, inactive=10921824256, buffers=9748480, cached=24259596288, shared=1449758720, slab=787439616) 
2022-04-06 19:21:38.173942: 
epoch:  139 
2022-04-06 19:23:28.759386: train loss : -0.4671 
2022-04-06 19:23:41.190984: validation loss: -0.4659 
2022-04-06 19:23:41.199100: Average global foreground Dice: [0.79, 0.9142, 0.9557, 0.3529] 
2022-04-06 19:23:41.205810: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:23:42.070138: lr: 0.00744 
2022-04-06 19:23:42.076953: saving scheduled checkpoint file... 
2022-04-06 19:23:42.194690: saving checkpoint... 
2022-04-06 19:23:46.686093: done, saving took 4.60 seconds 
2022-04-06 19:23:46.710318: done 
2022-04-06 19:23:46.717085: This epoch took 128.531992 s
 
2022-04-06 19:23:46.723161: svmem(total=99932160000, available=83050172416, percent=16.9, used=13609340928, free=62203584512, active=30400442368, inactive=5309497344, buffers=10055680, cached=24109178880, shared=1449730048, slab=788140032) 
2022-04-06 19:23:46.727732: 
epoch:  140 
2022-04-06 19:25:36.096133: train loss : -0.5089 
2022-04-06 19:25:47.058876: validation loss: -0.5401 
2022-04-06 19:25:47.060430: Average global foreground Dice: [0.7881, 0.9479, 0.9637, 0.3988] 
2022-04-06 19:25:47.061209: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:25:47.929378: lr: 0.007422 
2022-04-06 19:25:47.944286: saving scheduled checkpoint file... 
2022-04-06 19:25:48.151492: saving checkpoint... 
2022-04-06 19:25:52.847724: done, saving took 4.90 seconds 
2022-04-06 19:25:52.876008: done 
2022-04-06 19:25:52.879876: This epoch took 126.147734 s
 
2022-04-06 19:25:52.883119: svmem(total=99932160000, available=83254370304, percent=16.7, used=13404119040, free=62263504896, active=30221885440, inactive=5425565696, buffers=10309632, cached=24254226432, shared=1449816064, slab=789008384) 
2022-04-06 19:25:52.885628: 
epoch:  141 
2022-04-06 19:27:43.102640: train loss : -0.5261 
2022-04-06 19:27:54.857131: validation loss: -0.5299 
2022-04-06 19:27:54.861975: Average global foreground Dice: [0.8058, 0.9416, 0.9575, 0.4046] 
2022-04-06 19:27:54.868143: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:27:55.859227: lr: 0.007403 
2022-04-06 19:27:55.866604: saving scheduled checkpoint file... 
2022-04-06 19:27:56.025488: saving checkpoint... 
2022-04-06 19:28:00.481542: done, saving took 4.61 seconds 
2022-04-06 19:28:00.502177: done 
2022-04-06 19:28:00.505999: This epoch took 127.618203 s
 
2022-04-06 19:28:00.510296: svmem(total=99932160000, available=84787974144, percent=15.2, used=12159483904, free=63900459008, active=28954705920, inactive=5170462720, buffers=10571776, cached=23861645312, shared=1160675328, slab=780562432) 
2022-04-06 19:28:00.513460: 
epoch:  142 
2022-04-06 19:29:50.838503: train loss : -0.4576 
2022-04-06 19:30:03.123764: validation loss: -0.4877 
2022-04-06 19:30:03.133396: Average global foreground Dice: [0.7922, 0.9352, 0.9533, 0.3652] 
2022-04-06 19:30:03.144434: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:30:03.848966: lr: 0.007385 
2022-04-06 19:30:03.853975: saving scheduled checkpoint file... 
2022-04-06 19:30:03.974451: saving checkpoint... 
2022-04-06 19:30:08.377019: done, saving took 4.52 seconds 
2022-04-06 19:30:08.394714: done 
2022-04-06 19:30:08.395670: This epoch took 127.878189 s
 
2022-04-06 19:30:08.396575: svmem(total=99932160000, available=84703449088, percent=15.2, used=12245389312, free=63898947584, active=29232242688, inactive=4896690176, buffers=10829824, cached=23776993280, shared=1159360512, slab=780775424) 
2022-04-06 19:30:08.397348: 
epoch:  143 
2022-04-06 19:31:55.279663: train loss : -0.5030 
2022-04-06 19:32:07.217681: validation loss: -0.4631 
2022-04-06 19:32:07.225686: Average global foreground Dice: [0.7839, 0.9237, 0.9451, 0.3357] 
2022-04-06 19:32:07.236120: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:32:08.280291: lr: 0.007366 
2022-04-06 19:32:08.286165: saving scheduled checkpoint file... 
2022-04-06 19:32:08.422744: saving checkpoint... 
2022-04-06 19:32:13.316730: done, saving took 5.02 seconds 
2022-04-06 19:32:13.341704: done 
2022-04-06 19:32:13.344155: This epoch took 124.946097 s
 
2022-04-06 19:32:13.346971: svmem(total=99932160000, available=84814221312, percent=15.1, used=12141920256, free=63950422016, active=29109252096, inactive=4966014976, buffers=11067392, cached=23828750336, shared=1151926272, slab=781230080) 
2022-04-06 19:32:13.349171: 
epoch:  144 
2022-04-06 19:34:01.609457: train loss : -0.4600 
2022-04-06 19:34:13.675013: validation loss: -0.4720 
2022-04-06 19:34:13.676579: Average global foreground Dice: [0.7794, 0.9201, 0.9375, 0.3827] 
2022-04-06 19:34:13.677321: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:34:14.384742: lr: 0.007347 
2022-04-06 19:34:14.391398: saving scheduled checkpoint file... 
2022-04-06 19:34:14.474928: saving checkpoint... 
2022-04-06 19:34:18.998521: done, saving took 4.60 seconds 
2022-04-06 19:34:19.012124: done 
2022-04-06 19:34:19.013041: This epoch took 125.661642 s
 
2022-04-06 19:34:19.014053: svmem(total=99932160000, available=84842700800, percent=15.1, used=12113059840, free=63808413696, active=29150875648, inactive=5071704064, buffers=11317248, cached=23999369216, shared=1151926272, slab=780255232) 
2022-04-06 19:34:19.014906: 
epoch:  145 
2022-04-06 19:36:06.762541: train loss : -0.4683 
2022-04-06 19:36:18.261305: validation loss: -0.4743 
2022-04-06 19:36:18.272641: Average global foreground Dice: [0.7983, 0.93, 0.936, 0.3413] 
2022-04-06 19:36:18.282643: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:36:19.108452: lr: 0.007329 
2022-04-06 19:36:19.109323: saving scheduled checkpoint file... 
2022-04-06 19:36:19.206721: saving checkpoint... 
2022-04-06 19:36:23.804293: done, saving took 4.69 seconds 
2022-04-06 19:36:23.834207: done 
2022-04-06 19:36:23.838890: This epoch took 124.823008 s
 
2022-04-06 19:36:23.844600: svmem(total=99932160000, available=84779442176, percent=15.2, used=12176437248, free=63810248704, active=29269385216, inactive=4952088576, buffers=11583488, cached=23933890560, shared=1152098304, slab=781422592) 
2022-04-06 19:36:23.850973: 
epoch:  146 
2022-04-06 19:38:12.154728: train loss : -0.5039 
2022-04-06 19:38:22.905383: validation loss: -0.5067 
2022-04-06 19:38:22.913832: Average global foreground Dice: [0.7893, 0.9371, 0.9614, 0.3468] 
2022-04-06 19:38:22.921770: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:38:23.479011: lr: 0.00731 
2022-04-06 19:38:23.485230: saving scheduled checkpoint file... 
2022-04-06 19:38:23.556326: saving checkpoint... 
2022-04-06 19:38:28.289699: done, saving took 4.80 seconds 
2022-04-06 19:38:28.320515: done 
2022-04-06 19:38:28.328984: This epoch took 124.473104 s
 
2022-04-06 19:38:28.337641: svmem(total=99932160000, available=84797771776, percent=15.1, used=12156633088, free=63874514944, active=29133320192, inactive=5017079808, buffers=11825152, cached=23889186816, shared=1152012288, slab=780525568) 
2022-04-06 19:38:28.346328: 
epoch:  147 
2022-04-06 19:40:16.468017: train loss : -0.5484 
2022-04-06 19:40:27.910059: validation loss: -0.5452 
2022-04-06 19:40:27.919669: Average global foreground Dice: [0.8068, 0.9452, 0.9681, 0.3604] 
2022-04-06 19:40:27.921446: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:40:28.808676: lr: 0.007291 
2022-04-06 19:40:28.817299: saving scheduled checkpoint file... 
2022-04-06 19:40:28.961454: saving checkpoint... 
2022-04-06 19:40:33.519387: done, saving took 4.69 seconds 
2022-04-06 19:40:33.564090: done 
2022-04-06 19:40:33.567519: This epoch took 125.213894 s
 
2022-04-06 19:40:33.573839: svmem(total=99932160000, available=84819931136, percent=15.1, used=12136230912, free=63896162304, active=29102284800, inactive=5027098624, buffers=12062720, cached=23887704064, shared=1151840256, slab=780980224) 
2022-04-06 19:40:33.578851: 
epoch:  148 
2022-04-06 19:42:22.412198: train loss : -0.5407 
2022-04-06 19:42:33.768773: validation loss: -0.5672 
2022-04-06 19:42:33.770514: Average global foreground Dice: [0.8207, 0.9469, 0.9711, 0.3833] 
2022-04-06 19:42:33.771320: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:42:34.634476: lr: 0.007273 
2022-04-06 19:42:34.640798: saving scheduled checkpoint file... 
2022-04-06 19:42:34.780871: saving checkpoint... 
2022-04-06 19:42:39.215840: done, saving took 4.57 seconds 
2022-04-06 19:42:39.233887: done 
2022-04-06 19:42:39.234646: This epoch took 125.650153 s
 
2022-04-06 19:42:39.235574: svmem(total=99932160000, available=84728922112, percent=15.2, used=12226580480, free=63905660928, active=29225525248, inactive=4898201600, buffers=12320768, cached=23787597824, shared=1152012288, slab=781025280) 
2022-04-06 19:42:39.236338: 
epoch:  149 
2022-04-06 19:44:28.365675: train loss : -0.5671 
2022-04-06 19:44:40.221059: validation loss: -0.5568 
2022-04-06 19:44:40.222326: Average global foreground Dice: [0.7783, 0.9414, 0.9717, 0.3895] 
2022-04-06 19:44:40.223063: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:44:41.203525: lr: 0.007254 
2022-04-06 19:44:41.206146: saving scheduled checkpoint file... 
2022-04-06 19:44:41.338074: saving checkpoint... 
2022-04-06 19:44:45.932275: done, saving took 4.73 seconds 
2022-04-06 19:44:45.953539: done 
2022-04-06 19:44:45.955918: This epoch took 126.718959 s
 
2022-04-06 19:44:45.958344: svmem(total=99932160000, available=84732411904, percent=15.2, used=12211384320, free=63866167296, active=29164683264, inactive=4992827392, buffers=12570624, cached=23842037760, shared=1163997184, slab=783069184) 
2022-04-06 19:44:45.960167: 
epoch:  150 
2022-04-06 19:46:33.049260: train loss : -0.5071 
2022-04-06 19:46:43.693210: validation loss: -0.4866 
2022-04-06 19:46:43.698071: Average global foreground Dice: [0.7687, 0.9271, 0.9401, 0.3719] 
2022-04-06 19:46:43.704839: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:46:44.489960: lr: 0.007236 
2022-04-06 19:46:44.495626: saving scheduled checkpoint file... 
2022-04-06 19:46:44.640545: saving checkpoint... 
2022-04-06 19:46:49.004098: done, saving took 4.50 seconds 
2022-04-06 19:46:49.022624: done 
2022-04-06 19:46:49.026586: This epoch took 123.064158 s
 
2022-04-06 19:46:49.031954: svmem(total=99932160000, available=84766416896, percent=15.2, used=12213047296, free=63741394944, active=29210398720, inactive=5070036992, buffers=14987264, cached=23962730496, shared=1127370752, slab=784056320) 
2022-04-06 19:46:49.034369: 
epoch:  151 
2022-04-06 19:48:37.583422: train loss : -0.4923 
2022-04-06 19:48:49.346236: validation loss: -0.5251 
2022-04-06 19:48:49.348227: Average global foreground Dice: [0.786, 0.9485, 0.964, 0.3628] 
2022-04-06 19:48:49.349087: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:48:50.316457: lr: 0.007217 
2022-04-06 19:48:50.323835: saving scheduled checkpoint file... 
2022-04-06 19:48:50.382852: saving checkpoint... 
2022-04-06 19:48:54.822141: done, saving took 4.49 seconds 
2022-04-06 19:48:54.838375: done 
2022-04-06 19:48:54.842421: This epoch took 125.803448 s
 
2022-04-06 19:48:54.845622: svmem(total=99932160000, available=84715831296, percent=15.2, used=12238622720, free=63745404928, active=29179113472, inactive=5097476096, buffers=15237120, cached=23932895232, shared=1152102400, slab=784674816) 
2022-04-06 19:48:54.849321: 
epoch:  152 
2022-04-06 19:50:44.095427: train loss : -0.5423 
2022-04-06 19:50:56.966921: validation loss: -0.5519 
2022-04-06 19:50:57.001461: Average global foreground Dice: [0.8173, 0.9428, 0.9695, 0.3491] 
2022-04-06 19:50:57.017833: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:50:58.026762: lr: 0.007198 
2022-04-06 19:50:58.027629: saving scheduled checkpoint file... 
2022-04-06 19:50:58.107148: saving checkpoint... 
2022-04-06 19:51:02.563478: done, saving took 4.54 seconds 
2022-04-06 19:51:02.576679: done 
2022-04-06 19:51:02.578800: This epoch took 127.727147 s
 
2022-04-06 19:51:02.581480: svmem(total=99932160000, available=83319623680, percent=16.6, used=13633961984, free=61550788608, active=31484252160, inactive=4987162624, buffers=15536128, cached=24731873280, shared=1152196608, slab=787779584) 
2022-04-06 19:51:02.583180: 
epoch:  153 
2022-04-06 19:52:52.370512: train loss : -0.5556 
2022-04-06 19:53:04.003565: validation loss: -0.5369 
2022-04-06 19:53:04.007324: Average global foreground Dice: [0.8064, 0.945, 0.9658, 0.3794] 
2022-04-06 19:53:04.010439: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:53:04.859468: lr: 0.00718 
2022-04-06 19:53:04.860390: saving scheduled checkpoint file... 
2022-04-06 19:53:04.926837: saving checkpoint... 
2022-04-06 19:53:09.454515: done, saving took 4.59 seconds 
2022-04-06 19:53:09.500712: done 
2022-04-06 19:53:09.561291: saving checkpoint... 
2022-04-06 19:53:13.956884: done, saving took 4.45 seconds 
2022-04-06 19:53:13.970711: This epoch took 131.385264 s
 
2022-04-06 19:53:13.971857: svmem(total=99932160000, available=83082002432, percent=16.9, used=13871878144, free=61392613376, active=31654535168, inactive=4972068864, buffers=15790080, cached=24651878400, shared=1151852544, slab=787980288) 
2022-04-06 19:53:13.972450: 
epoch:  154 
2022-04-06 19:55:02.512114: train loss : -0.5462 
2022-04-06 19:55:14.266425: validation loss: -0.5745 
2022-04-06 19:55:14.279653: Average global foreground Dice: [0.7922, 0.9526, 0.974, 0.3772] 
2022-04-06 19:55:14.283504: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:55:14.982821: lr: 0.007161 
2022-04-06 19:55:14.991796: saving scheduled checkpoint file... 
2022-04-06 19:55:15.080139: saving checkpoint... 
2022-04-06 19:55:19.582809: done, saving took 4.58 seconds 
2022-04-06 19:55:19.602277: done 
2022-04-06 19:55:19.678432: saving checkpoint... 
2022-04-06 19:55:24.104374: done, saving took 4.50 seconds 
2022-04-06 19:55:24.121368: This epoch took 130.148308 s
 
2022-04-06 19:55:24.122351: svmem(total=99932160000, available=84761911296, percent=15.2, used=12192100352, free=63034580992, active=30010089472, inactive=4975853568, buffers=16097280, cached=24689381376, shared=1151852544, slab=787091456) 
2022-04-06 19:55:24.122884: 
epoch:  155 
2022-04-06 19:57:12.574819: train loss : -0.5375 
2022-04-06 19:57:23.767062: validation loss: -0.4555 
2022-04-06 19:57:23.773496: Average global foreground Dice: [0.7397, 0.9274, 0.9472, 0.3553] 
2022-04-06 19:57:23.777545: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:57:24.963329: lr: 0.007142 
2022-04-06 19:57:24.966202: saving scheduled checkpoint file... 
2022-04-06 19:57:25.045435: saving checkpoint... 
2022-04-06 19:57:29.264325: done, saving took 4.29 seconds 
2022-04-06 19:57:29.296376: done 
2022-04-06 19:57:29.305370: This epoch took 125.181737 s
 
2022-04-06 19:57:29.315294: svmem(total=99932160000, available=84751638528, percent=15.2, used=12201553920, free=62960349184, active=29974376448, inactive=5085310976, buffers=16330752, cached=24753926144, shared=1152303104, slab=785895424) 
2022-04-06 19:57:29.325731: 
epoch:  156 
2022-04-06 19:59:19.846510: train loss : -0.4123 
2022-04-06 19:59:31.600159: validation loss: -0.4678 
2022-04-06 19:59:31.609579: Average global foreground Dice: [0.7569, 0.9358, 0.9526, 0.3216] 
2022-04-06 19:59:31.610668: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 19:59:32.523434: lr: 0.007123 
2022-04-06 19:59:32.524728: saving scheduled checkpoint file... 
2022-04-06 19:59:32.658871: saving checkpoint... 
2022-04-06 19:59:37.252305: done, saving took 4.73 seconds 
2022-04-06 19:59:37.268111: done 
2022-04-06 19:59:37.270000: This epoch took 127.938214 s
 
2022-04-06 19:59:37.271307: svmem(total=99932160000, available=84756471808, percent=15.2, used=12197060608, free=62966628352, active=30145810432, inactive=4910563328, buffers=16572416, cached=24751898624, shared=1151852544, slab=786415616) 
2022-04-06 19:59:37.273367: 
epoch:  157 
2022-04-06 20:01:26.727804: train loss : -0.4821 
2022-04-06 20:01:38.325767: validation loss: -0.5330 
2022-04-06 20:01:38.329172: Average global foreground Dice: [0.808, 0.9294, 0.9584, 0.3869] 
2022-04-06 20:01:38.331179: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:01:39.201865: lr: 0.007105 
2022-04-06 20:01:39.202815: saving scheduled checkpoint file... 
2022-04-06 20:01:39.311199: saving checkpoint... 
2022-04-06 20:01:43.836298: done, saving took 4.63 seconds 
2022-04-06 20:01:43.863925: done 
2022-04-06 20:01:43.866880: This epoch took 126.592432 s
 
2022-04-06 20:01:43.870928: svmem(total=99932160000, available=84839964672, percent=15.1, used=12113182720, free=63031054336, active=30011142144, inactive=4979781632, buffers=16990208, cached=24770932736, shared=1151938560, slab=785395712) 
2022-04-06 20:01:43.874033: 
epoch:  158 
2022-04-06 20:03:34.150559: train loss : -0.3649 
2022-04-06 20:03:45.907233: validation loss: -0.3947 
2022-04-06 20:03:45.910018: Average global foreground Dice: [0.6719, 0.9079, 0.927, 0.3501] 
2022-04-06 20:03:45.913882: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:03:46.970231: lr: 0.007086 
2022-04-06 20:03:46.979657: saving scheduled checkpoint file... 
2022-04-06 20:03:47.119576: saving checkpoint... 
2022-04-06 20:03:51.761993: done, saving took 4.77 seconds 
2022-04-06 20:03:51.779548: done 
2022-04-06 20:03:51.780145: This epoch took 127.902624 s
 
2022-04-06 20:03:51.780833: svmem(total=99932160000, available=84719534080, percent=15.2, used=12221382656, free=63046672384, active=30073315328, inactive=4902981632, buffers=17231872, cached=24646873088, shared=1164824576, slab=786612224) 
2022-04-06 20:03:51.781316: 
epoch:  159 
2022-04-06 20:05:39.637424: train loss : -0.4478 
2022-04-06 20:05:50.774395: validation loss: -0.5016 
2022-04-06 20:05:50.783507: Average global foreground Dice: [0.7406, 0.935, 0.9693, 0.3771] 
2022-04-06 20:05:50.790723: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:05:51.600698: lr: 0.007067 
2022-04-06 20:05:51.616467: saving scheduled checkpoint file... 
2022-04-06 20:05:51.758809: saving checkpoint... 
2022-04-06 20:05:56.135031: done, saving took 4.50 seconds 
2022-04-06 20:05:56.171481: done 
2022-04-06 20:05:56.179958: This epoch took 124.398079 s
 
2022-04-06 20:05:56.189898: svmem(total=99932160000, available=84868390912, percent=15.1, used=12084727808, free=63065927680, active=30048735232, inactive=4909633536, buffers=17506304, cached=24763998208, shared=1152196608, slab=787558400) 
2022-04-06 20:05:56.197347: 
epoch:  160 
2022-04-06 20:07:46.038023: train loss : -0.4815 
2022-04-06 20:07:58.877787: validation loss: -0.4929 
2022-04-06 20:07:58.890481: Average global foreground Dice: [0.7715, 0.9365, 0.9605, 0.3721] 
2022-04-06 20:07:58.901006: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:07:59.611807: lr: 0.007049 
2022-04-06 20:07:59.619788: saving scheduled checkpoint file... 
2022-04-06 20:07:59.760387: saving checkpoint... 
2022-04-06 20:08:04.404515: done, saving took 4.78 seconds 
2022-04-06 20:08:04.457787: done 
2022-04-06 20:08:04.480237: This epoch took 128.273869 s
 
2022-04-06 20:08:04.500380: svmem(total=99932160000, available=84881584128, percent=15.1, used=12125573120, free=82023153664, active=10904256512, inactive=5182861312, buffers=8056832, cached=5775376384, shared=1139429376, slab=702087168) 
2022-04-06 20:08:04.522180: 
epoch:  161 
2022-04-06 20:09:54.656765: train loss : -0.4862 
2022-04-06 20:10:06.517687: validation loss: -0.4914 
2022-04-06 20:10:06.519339: Average global foreground Dice: [0.7747, 0.9305, 0.9632, 0.351] 
2022-04-06 20:10:06.520149: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:10:07.396333: lr: 0.00703 
2022-04-06 20:10:07.401702: saving scheduled checkpoint file... 
2022-04-06 20:10:07.490524: saving checkpoint... 
2022-04-06 20:10:12.024813: done, saving took 4.62 seconds 
2022-04-06 20:10:12.044187: done 
2022-04-06 20:10:12.045383: This epoch took 127.504283 s
 
2022-04-06 20:10:12.050216: svmem(total=99932160000, available=84872650752, percent=15.1, used=12201750528, free=72658763776, active=14412283904, inactive=11020865536, buffers=9498624, cached=15062147072, shared=1061654528, slab=723718144) 
2022-04-06 20:10:12.051997: 
epoch:  162 
2022-04-06 20:11:59.402620: train loss : -0.5347 
2022-04-06 20:12:10.797118: validation loss: -0.5631 
2022-04-06 20:12:10.810145: Average global foreground Dice: [0.7953, 0.9461, 0.9745, 0.3808] 
2022-04-06 20:12:10.822761: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:12:11.669803: lr: 0.007011 
2022-04-06 20:12:11.673779: saving scheduled checkpoint file... 
2022-04-06 20:12:11.870965: saving checkpoint... 
2022-04-06 20:12:16.274641: done, saving took 4.60 seconds 
2022-04-06 20:12:16.294174: done 
2022-04-06 20:12:16.296473: This epoch took 124.241029 s
 
2022-04-06 20:12:16.298952: svmem(total=99932160000, available=84797075456, percent=15.1, used=12174880768, free=70372286464, active=19008548864, inactive=8697163776, buffers=9736192, cached=17375256576, shared=1158418432, slab=734732288) 
2022-04-06 20:12:16.300505: 
epoch:  163 
2022-04-06 20:14:05.634687: train loss : -0.5442 
2022-04-06 20:14:17.337676: validation loss: -0.5795 
2022-04-06 20:14:17.342855: Average global foreground Dice: [0.8229, 0.9422, 0.9747, 0.394] 
2022-04-06 20:14:17.346807: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:14:18.219574: lr: 0.006992 
2022-04-06 20:14:18.233516: saving scheduled checkpoint file... 
2022-04-06 20:14:18.357430: saving checkpoint... 
2022-04-06 20:14:22.740091: done, saving took 4.49 seconds 
2022-04-06 20:14:22.784305: done 
2022-04-06 20:14:22.799941: This epoch took 126.497651 s
 
2022-04-06 20:14:22.818195: svmem(total=99932160000, available=84824879104, percent=15.1, used=12156506112, free=70317117440, active=22309683200, inactive=5442895872, buffers=9994240, cached=17448542208, shared=1145458688, slab=742453248) 
2022-04-06 20:14:22.834260: 
epoch:  164 
2022-04-06 20:16:12.395574: train loss : -0.5525 
2022-04-06 20:16:24.227176: validation loss: -0.5383 
2022-04-06 20:16:24.229313: Average global foreground Dice: [0.8077, 0.9478, 0.9671, 0.3486] 
2022-04-06 20:16:24.230505: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:16:25.031663: lr: 0.006974 
2022-04-06 20:16:25.035543: saving scheduled checkpoint file... 
2022-04-06 20:16:25.234113: saving checkpoint... 
2022-04-06 20:16:29.710056: done, saving took 4.67 seconds 
2022-04-06 20:16:29.718416: done 
2022-04-06 20:16:29.719191: This epoch took 126.870855 s
 
2022-04-06 20:16:29.720153: svmem(total=99932160000, available=84700303360, percent=15.2, used=12278067200, free=70311038976, active=23421849600, inactive=4334456832, buffers=10395648, cached=17332658176, shared=1146413056, slab=746795008) 
2022-04-06 20:16:29.720856: 
epoch:  165 
2022-04-06 20:18:16.957628: train loss : -0.5539 
2022-04-06 20:18:28.954052: validation loss: -0.5701 
2022-04-06 20:18:28.967176: Average global foreground Dice: [0.8103, 0.9456, 0.97, 0.3945] 
2022-04-06 20:18:28.976475: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:18:29.699871: lr: 0.006955 
2022-04-06 20:18:29.702011: saving scheduled checkpoint file... 
2022-04-06 20:18:29.772805: saving checkpoint... 
2022-04-06 20:18:34.427259: done, saving took 4.72 seconds 
2022-04-06 20:18:34.445316: done 
2022-04-06 20:18:34.449884: This epoch took 124.728230 s
 
2022-04-06 20:18:34.457127: svmem(total=99932160000, available=84659494912, percent=15.3, used=12317024256, free=70303272960, active=23395065856, inactive=4361650176, buffers=10674176, cached=17301188608, shared=1147301888, slab=749486080) 
2022-04-06 20:18:34.463922: 
epoch:  166 
2022-04-06 20:20:23.631569: train loss : -0.5300 
2022-04-06 20:20:34.925215: validation loss: -0.5304 
2022-04-06 20:20:34.932092: Average global foreground Dice: [0.8215, 0.9353, 0.959, 0.3574] 
2022-04-06 20:20:34.933841: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:20:35.752342: lr: 0.006936 
2022-04-06 20:20:35.767441: saving scheduled checkpoint file... 
2022-04-06 20:20:35.904189: saving checkpoint... 
2022-04-06 20:20:40.323400: done, saving took 4.54 seconds 
2022-04-06 20:20:40.358437: done 
2022-04-06 20:20:40.373046: This epoch took 125.903549 s
 
2022-04-06 20:20:40.385679: svmem(total=99932160000, available=84851666944, percent=15.1, used=12122509312, free=70378098688, active=23432400896, inactive=4252155904, buffers=10915840, cached=17420636160, shared=1148968960, slab=750657536) 
2022-04-06 20:20:40.395866: 
epoch:  167 
2022-04-06 20:22:28.874066: train loss : -0.5587 
2022-04-06 20:22:41.071111: validation loss: -0.5569 
2022-04-06 20:22:41.082271: Average global foreground Dice: [0.8169, 0.9313, 0.9714, 0.4094] 
2022-04-06 20:22:41.086218: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:22:41.836928: lr: 0.006918 
2022-04-06 20:22:41.837950: saving scheduled checkpoint file... 
2022-04-06 20:22:41.989930: saving checkpoint... 
2022-04-06 20:22:46.616677: done, saving took 4.78 seconds 
2022-04-06 20:22:46.641216: done 
2022-04-06 20:22:46.707444: saving checkpoint... 
2022-04-06 20:22:50.998541: done, saving took 4.35 seconds 
2022-04-06 20:22:51.016226: This epoch took 130.611285 s
 
2022-04-06 20:22:51.017147: svmem(total=99932160000, available=84849651712, percent=15.1, used=12123049984, free=70393004032, active=23425208320, inactive=4245708800, buffers=11161600, cached=17404944384, shared=1149681664, slab=753041408) 
2022-04-06 20:22:51.017700: 
epoch:  168 
2022-04-06 20:24:40.250117: train loss : -0.4506 
2022-04-06 20:24:52.261664: validation loss: -0.4831 
2022-04-06 20:24:52.275844: Average global foreground Dice: [0.746, 0.9308, 0.9622, 0.3764] 
2022-04-06 20:24:52.276664: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:24:53.080695: lr: 0.006899 
2022-04-06 20:24:53.091675: saving scheduled checkpoint file... 
2022-04-06 20:24:53.155137: saving checkpoint... 
2022-04-06 20:24:57.669957: done, saving took 4.58 seconds 
2022-04-06 20:24:57.688614: done 
2022-04-06 20:24:57.693239: This epoch took 126.674948 s
 
2022-04-06 20:24:57.694514: svmem(total=99932160000, available=84771983360, percent=15.2, used=12199567360, free=70296621056, active=23516725248, inactive=4241682432, buffers=11403264, cached=17424568320, shared=1150562304, slab=752406528) 
2022-04-06 20:24:57.697814: 
epoch:  169 
2022-04-06 20:26:46.488932: train loss : -0.4352 
2022-04-06 20:26:59.222013: validation loss: -0.5077 
2022-04-06 20:26:59.225724: Average global foreground Dice: [0.7812, 0.9375, 0.9565, 0.3813] 
2022-04-06 20:26:59.228661: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:27:00.040874: lr: 0.00688 
2022-04-06 20:27:00.054452: saving scheduled checkpoint file... 
2022-04-06 20:27:00.174700: saving checkpoint... 
2022-04-06 20:27:04.760107: done, saving took 4.69 seconds 
2022-04-06 20:27:04.773798: done 
2022-04-06 20:27:04.778961: This epoch took 127.078469 s
 
2022-04-06 20:27:04.780014: svmem(total=99932160000, available=84710387712, percent=15.2, used=12260634624, free=70296010752, active=23172591616, inactive=4587454464, buffers=11669504, cached=17363845120, shared=1150689280, slab=753090560) 
2022-04-06 20:27:04.780606: 
epoch:  170 
2022-04-06 20:28:55.524645: train loss : -0.5023 
2022-04-06 20:29:07.624172: validation loss: -0.5303 
2022-04-06 20:29:07.632316: Average global foreground Dice: [0.7918, 0.9473, 0.9662, 0.3667] 
2022-04-06 20:29:07.633269: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:29:08.427892: lr: 0.006861 
2022-04-06 20:29:08.442699: saving scheduled checkpoint file... 
2022-04-06 20:29:08.551190: saving checkpoint... 
2022-04-06 20:29:13.120103: done, saving took 4.66 seconds 
2022-04-06 20:29:13.154236: done 
2022-04-06 20:29:13.161573: This epoch took 128.380315 s
 
2022-04-06 20:29:13.169222: svmem(total=99932160000, available=84798881792, percent=15.1, used=12161282048, free=70351884288, active=23281364992, inactive=4423643136, buffers=13012992, cached=17405980672, shared=1161252864, slab=753291264) 
2022-04-06 20:29:13.177759: 
epoch:  171 
2022-04-06 20:31:03.567158: train loss : -0.5454 
2022-04-06 20:31:15.938143: validation loss: -0.5713 
2022-04-06 20:31:15.948539: Average global foreground Dice: [0.8071, 0.9466, 0.9736, 0.3694] 
2022-04-06 20:31:15.951145: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:31:16.687983: lr: 0.006842 
2022-04-06 20:31:16.702842: saving scheduled checkpoint file... 
2022-04-06 20:31:16.821054: saving checkpoint... 
2022-04-06 20:31:21.339278: done, saving took 4.62 seconds 
2022-04-06 20:31:21.393575: done 
2022-04-06 20:31:21.492276: saving checkpoint... 
2022-04-06 20:31:26.009256: done, saving took 4.61 seconds 
2022-04-06 20:31:26.025598: This epoch took 132.841947 s
 
2022-04-06 20:31:26.026634: svmem(total=99932160000, available=84771766272, percent=15.2, used=12198223872, free=70362345472, active=23319691264, inactive=4376375296, buffers=15216640, cached=17356374016, shared=1151488000, slab=752721920) 
2022-04-06 20:31:26.027259: 
epoch:  172 
2022-04-06 20:33:15.680096: train loss : -0.5320 
2022-04-06 20:33:27.772898: validation loss: -0.5470 
2022-04-06 20:33:27.776078: Average global foreground Dice: [0.7805, 0.9464, 0.9706, 0.3782] 
2022-04-06 20:33:27.778834: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:33:28.796986: lr: 0.006824 
2022-04-06 20:33:28.798511: saving scheduled checkpoint file... 
2022-04-06 20:33:28.957074: saving checkpoint... 
2022-04-06 20:33:33.375877: done, saving took 4.57 seconds 
2022-04-06 20:33:33.394716: done 
2022-04-06 20:33:33.493655: saving checkpoint... 
2022-04-06 20:33:37.915588: done, saving took 4.52 seconds 
2022-04-06 20:33:37.934203: This epoch took 131.906383 s
 
2022-04-06 20:33:37.935210: svmem(total=99932160000, available=84716937216, percent=15.2, used=12252610560, free=70360498176, active=23438176256, inactive=4257931264, buffers=15466496, cached=17303584768, shared=1151868928, slab=753262592) 
2022-04-06 20:33:37.936032: 
epoch:  173 
2022-04-06 20:35:27.557107: train loss : -0.5261 
2022-04-06 20:35:39.948488: validation loss: -0.5254 
2022-04-06 20:35:39.951904: Average global foreground Dice: [0.7726, 0.9437, 0.9662, 0.3533] 
2022-04-06 20:35:39.952940: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:35:40.692553: lr: 0.006805 
2022-04-06 20:35:40.702124: saving scheduled checkpoint file... 
2022-04-06 20:35:40.802251: saving checkpoint... 
2022-04-06 20:35:45.112772: done, saving took 4.40 seconds 
2022-04-06 20:35:45.125091: done 
2022-04-06 20:35:45.125931: This epoch took 127.189160 s
 
2022-04-06 20:35:45.126711: svmem(total=99932160000, available=84710334464, percent=15.2, used=12259176448, free=70251630592, active=23515869184, inactive=4286099456, buffers=15757312, cached=17405595648, shared=1151651840, slab=753905664) 
2022-04-06 20:35:45.127297: 
epoch:  174 
2022-04-06 20:37:33.858901: train loss : -0.5638 
2022-04-06 20:37:45.510668: validation loss: -0.5598 
2022-04-06 20:37:45.527802: Average global foreground Dice: [0.7978, 0.947, 0.9561, 0.3955] 
2022-04-06 20:37:45.536300: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:37:46.213224: lr: 0.006786 
2022-04-06 20:37:46.214354: saving scheduled checkpoint file... 
2022-04-06 20:37:46.350467: saving checkpoint... 
2022-04-06 20:37:50.995173: done, saving took 4.78 seconds 
2022-04-06 20:37:51.010732: done 
2022-04-06 20:37:51.061901: saving checkpoint... 
2022-04-06 20:37:55.353094: done, saving took 4.34 seconds 
2022-04-06 20:37:55.366629: This epoch took 130.238715 s
 
2022-04-06 20:37:55.367702: svmem(total=99932160000, available=84696944640, percent=15.2, used=12272586752, free=70251225088, active=23372808192, inactive=4430766080, buffers=16003072, cached=17392345088, shared=1151737856, slab=753618944) 
2022-04-06 20:37:55.368390: 
epoch:  175 
2022-04-06 20:39:43.741834: train loss : -0.5478 
2022-04-06 20:39:54.829999: validation loss: -0.5861 
2022-04-06 20:39:54.843529: Average global foreground Dice: [0.8106, 0.9497, 0.976, 0.3906] 
2022-04-06 20:39:54.853559: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:39:55.675473: lr: 0.006767 
2022-04-06 20:39:55.692139: saving scheduled checkpoint file... 
2022-04-06 20:39:55.791356: saving checkpoint... 
2022-04-06 20:40:00.324735: done, saving took 4.62 seconds 
2022-04-06 20:40:00.359328: done 
2022-04-06 20:40:00.420711: saving checkpoint... 
2022-04-06 20:40:04.859377: done, saving took 4.49 seconds 
2022-04-06 20:40:04.868331: This epoch took 129.499247 s
 
2022-04-06 20:40:04.869303: svmem(total=99932160000, available=84717592576, percent=15.2, used=12251955200, free=70250770432, active=23363620864, inactive=4441206784, buffers=16244736, cached=17413189632, shared=1151565824, slab=754110464) 
2022-04-06 20:40:04.869946: 
epoch:  176 
2022-04-06 20:41:53.699023: train loss : -0.5647 
2022-04-06 20:42:04.003308: validation loss: -0.5916 
2022-04-06 20:42:04.009797: Average global foreground Dice: [0.8109, 0.9561, 0.9754, 0.392] 
2022-04-06 20:42:04.013788: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:42:04.973629: lr: 0.006749 
2022-04-06 20:42:04.974831: saving scheduled checkpoint file... 
2022-04-06 20:42:05.030368: saving checkpoint... 
2022-04-06 20:42:09.314356: done, saving took 4.34 seconds 
2022-04-06 20:42:09.335989: done 
2022-04-06 20:42:09.388731: saving checkpoint... 
2022-04-06 20:42:13.811344: done, saving took 4.47 seconds 
2022-04-06 20:42:13.825786: This epoch took 128.955331 s
 
2022-04-06 20:42:13.826787: svmem(total=99932160000, available=84635975680, percent=15.3, used=12324519936, free=70241492992, active=23506821120, inactive=4305420288, buffers=16494592, cached=17349652480, shared=1160523776, slab=753917952) 
2022-04-06 20:42:13.827557: 
epoch:  177 
2022-04-06 20:44:03.397044: train loss : -0.5686 
2022-04-06 20:44:14.600253: validation loss: -0.5922 
2022-04-06 20:44:14.612252: Average global foreground Dice: [0.8283, 0.9487, 0.9732, 0.3861] 
2022-04-06 20:44:14.613053: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:44:15.732502: lr: 0.00673 
2022-04-06 20:44:15.736441: saving scheduled checkpoint file... 
2022-04-06 20:44:15.840842: saving checkpoint... 
2022-04-06 20:44:20.380035: done, saving took 4.64 seconds 
2022-04-06 20:44:20.412873: done 
2022-04-06 20:44:20.503155: saving checkpoint... 
2022-04-06 20:44:24.950548: done, saving took 4.53 seconds 
2022-04-06 20:44:24.969446: This epoch took 131.141258 s
 
2022-04-06 20:44:24.970567: svmem(total=99932160000, available=84809166848, percent=15.1, used=12160081920, free=70332674048, active=23242702848, inactive=4483059712, buffers=16736256, cached=17422667776, shared=1151787008, slab=753328128) 
2022-04-06 20:44:24.971512: 
epoch:  178 
2022-04-06 20:46:13.828718: train loss : -0.5766 
2022-04-06 20:46:25.338658: validation loss: -0.5786 
2022-04-06 20:46:25.354125: Average global foreground Dice: [0.8079, 0.949, 0.9738, 0.4032] 
2022-04-06 20:46:25.355064: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:46:26.209455: lr: 0.006711 
2022-04-06 20:46:26.210964: saving scheduled checkpoint file... 
2022-04-06 20:46:26.373456: saving checkpoint... 
2022-04-06 20:46:30.498005: done, saving took 4.28 seconds 
2022-04-06 20:46:30.530264: done 
2022-04-06 20:46:30.628803: saving checkpoint... 
2022-04-06 20:46:35.054007: done, saving took 4.52 seconds 
2022-04-06 20:46:35.078269: This epoch took 130.105978 s
 
2022-04-06 20:46:35.079355: svmem(total=99932160000, available=84661104640, percent=15.3, used=12308172800, free=70352433152, active=23297339392, inactive=4402315264, buffers=17010688, cached=17254543360, shared=1151811584, slab=754507776) 
2022-04-06 20:46:35.080070: 
epoch:  179 
2022-04-06 20:48:24.397004: train loss : -0.5975 
2022-04-06 20:48:35.393101: validation loss: -0.5867 
2022-04-06 20:48:35.394974: Average global foreground Dice: [0.8336, 0.9556, 0.9747, 0.4004] 
2022-04-06 20:48:35.395729: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:48:37.151182: lr: 0.006692 
2022-04-06 20:48:37.157907: saving scheduled checkpoint file... 
2022-04-06 20:48:37.250668: saving checkpoint... 
2022-04-06 20:48:41.633451: done, saving took 4.47 seconds 
2022-04-06 20:48:41.671066: done 
2022-04-06 20:48:41.729202: saving checkpoint... 
2022-04-06 20:48:46.051798: done, saving took 4.37 seconds 
2022-04-06 20:48:46.081606: This epoch took 131.000835 s
 
2022-04-06 20:48:46.088712: svmem(total=99932160000, available=84748607488, percent=15.2, used=12221046784, free=70243536896, active=23318409216, inactive=4491427840, buffers=17276928, cached=17450299392, shared=1151299584, slab=754671616) 
2022-04-06 20:48:46.096132: 
epoch:  180 
2022-04-06 20:50:33.455290: train loss : -0.5892 
2022-04-06 20:50:44.661533: validation loss: -0.5544 
2022-04-06 20:50:44.675877: Average global foreground Dice: [0.8144, 0.9496, 0.9723, 0.3505] 
2022-04-06 20:50:44.676660: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:50:45.706377: lr: 0.006673 
2022-04-06 20:50:45.715247: saving scheduled checkpoint file... 
2022-04-06 20:50:45.821967: saving checkpoint... 
2022-04-06 20:50:50.235545: done, saving took 4.51 seconds 
2022-04-06 20:50:50.254856: done 
2022-04-06 20:50:50.257566: This epoch took 124.156409 s
 
2022-04-06 20:50:50.261218: svmem(total=99932160000, available=84710420480, percent=15.2, used=12259364864, free=70249050112, active=23557705728, inactive=4250697728, buffers=17518592, cached=17406226432, shared=1151856640, slab=754139136) 
2022-04-06 20:50:50.264287: 
epoch:  181 
2022-04-06 20:52:37.536216: train loss : -0.5765 
2022-04-06 20:52:49.104575: validation loss: -0.6045 
2022-04-06 20:52:49.108300: Average global foreground Dice: [0.8555, 0.9502, 0.9712, 0.3886] 
2022-04-06 20:52:49.109212: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:52:49.817550: lr: 0.006654 
2022-04-06 20:52:49.830549: saving scheduled checkpoint file... 
2022-04-06 20:52:49.887911: saving checkpoint... 
2022-04-06 20:52:54.327360: done, saving took 4.49 seconds 
2022-04-06 20:52:54.344712: done 
2022-04-06 20:52:54.392871: saving checkpoint... 
2022-04-06 20:52:58.671279: done, saving took 4.32 seconds 
2022-04-06 20:52:58.684614: This epoch took 128.416978 s
 
2022-04-06 20:52:58.685682: svmem(total=99932160000, available=84715380736, percent=15.2, used=12253372416, free=70248611840, active=23313022976, inactive=4491223040, buffers=17764352, cached=17412411392, shared=1152053248, slab=755171328) 
2022-04-06 20:52:58.686544: 
epoch:  182 
2022-04-06 20:54:47.227916: train loss : -0.5904 
2022-04-06 20:54:57.792589: validation loss: -0.5862 
2022-04-06 20:54:57.801973: Average global foreground Dice: [0.8176, 0.9547, 0.9748, 0.4117] 
2022-04-06 20:54:57.804886: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:54:58.803092: lr: 0.006636 
2022-04-06 20:54:58.817081: saving scheduled checkpoint file... 
2022-04-06 20:54:58.923957: saving checkpoint... 
2022-04-06 20:55:03.408804: done, saving took 4.58 seconds 
2022-04-06 20:55:03.447240: done 
2022-04-06 20:55:03.506273: saving checkpoint... 
2022-04-06 20:55:07.950752: done, saving took 4.50 seconds 
2022-04-06 20:55:07.964523: This epoch took 129.277267 s
 
2022-04-06 20:55:07.965503: svmem(total=99932160000, available=84684079104, percent=15.3, used=12282277888, free=70240190464, active=23463976960, inactive=4349001728, buffers=18034688, cached=17391656960, shared=1154535424, slab=754286592) 
2022-04-06 20:55:07.966103: 
epoch:  183 
2022-04-06 20:56:56.534668: train loss : -0.5143 
2022-04-06 20:57:07.442664: validation loss: -0.5466 
2022-04-06 20:57:07.452155: Average global foreground Dice: [0.8209, 0.9474, 0.965, 0.3669] 
2022-04-06 20:57:07.460612: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:57:08.192141: lr: 0.006617 
2022-04-06 20:57:08.195324: saving scheduled checkpoint file... 
2022-04-06 20:57:08.257863: saving checkpoint... 
2022-04-06 20:57:12.835646: done, saving took 4.64 seconds 
2022-04-06 20:57:12.860672: done 
2022-04-06 20:57:12.864053: This epoch took 124.897311 s
 
2022-04-06 20:57:12.868268: svmem(total=99932160000, available=84652539904, percent=15.3, used=12318195712, free=70263304192, active=23533559808, inactive=4270452736, buffers=18292736, cached=17332367360, shared=1151963136, slab=745926656) 
2022-04-06 20:57:12.872524: 
epoch:  184 
2022-04-06 20:59:01.250589: train loss : -0.5631 
2022-04-06 20:59:12.228033: validation loss: -0.5583 
2022-04-06 20:59:12.241601: Average global foreground Dice: [0.8332, 0.9429, 0.9555, 0.4016] 
2022-04-06 20:59:12.247177: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 20:59:12.871891: lr: 0.006598 
2022-04-06 20:59:12.881584: saving scheduled checkpoint file... 
2022-04-06 20:59:12.970946: saving checkpoint... 
2022-04-06 20:59:17.433299: done, saving took 4.54 seconds 
2022-04-06 20:59:17.486441: done 
2022-04-06 20:59:17.587594: saving checkpoint... 
2022-04-06 20:59:21.989368: done, saving took 4.48 seconds 
2022-04-06 20:59:22.034537: This epoch took 129.160946 s
 
2022-04-06 20:59:22.053425: svmem(total=99932160000, available=84041314304, percent=15.9, used=12915806208, free=76693987328, active=12190928896, inactive=8849252352, buffers=9224192, cached=10313142272, shared=1185370112, slab=734699520) 
2022-04-06 20:59:22.085431: 
epoch:  185 
2022-04-06 21:01:10.570305: train loss : -0.5744 
2022-04-06 21:01:22.101668: validation loss: -0.5414 
2022-04-06 21:01:22.103197: Average global foreground Dice: [0.8307, 0.9422, 0.9618, 0.3531] 
2022-04-06 21:01:22.105604: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:01:23.200079: lr: 0.006579 
2022-04-06 21:01:23.207486: saving scheduled checkpoint file... 
2022-04-06 21:01:23.327313: saving checkpoint... 
2022-04-06 21:01:28.272443: done, saving took 5.06 seconds 
2022-04-06 21:01:28.290730: done 
2022-04-06 21:01:28.294249: This epoch took 126.176910 s
 
2022-04-06 21:01:28.297944: svmem(total=99932160000, available=84079931392, percent=15.9, used=12840833024, free=69630763008, active=16557707264, inactive=11525042176, buffers=9490432, cached=17451073536, shared=1212190720, slab=753999872) 
2022-04-06 21:01:28.301249: 
epoch:  186 
2022-04-06 21:03:16.869247: train loss : -0.5362 
2022-04-06 21:03:28.310047: validation loss: -0.5326 
2022-04-06 21:03:28.311600: Average global foreground Dice: [0.8053, 0.9381, 0.9649, 0.374] 
2022-04-06 21:03:28.312263: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:03:29.015642: lr: 0.00656 
2022-04-06 21:03:29.021537: saving scheduled checkpoint file... 
2022-04-06 21:03:29.154651: saving checkpoint... 
2022-04-06 21:03:33.709375: done, saving took 4.68 seconds 
2022-04-06 21:03:33.730114: done 
2022-04-06 21:03:33.732421: This epoch took 125.427694 s
 
2022-04-06 21:03:33.734986: svmem(total=99932160000, available=83998093312, percent=15.9, used=12903854080, free=69622448128, active=20960546816, inactive=7129407488, buffers=9723904, cached=17396133888, shared=1227894784, slab=761352192) 
2022-04-06 21:03:33.737144: 
epoch:  187 
2022-04-06 21:05:23.049351: train loss : -0.5224 
2022-04-06 21:05:35.086230: validation loss: -0.5325 
2022-04-06 21:05:35.100343: Average global foreground Dice: [0.7975, 0.9378, 0.949, 0.3887] 
2022-04-06 21:05:35.101293: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:05:35.900818: lr: 0.006541 
2022-04-06 21:05:35.901667: saving scheduled checkpoint file... 
2022-04-06 21:05:36.009547: saving checkpoint... 
2022-04-06 21:05:40.586134: done, saving took 4.68 seconds 
2022-04-06 21:05:40.603334: done 
2022-04-06 21:05:40.604170: This epoch took 126.864825 s
 
2022-04-06 21:05:40.605082: svmem(total=99932160000, available=84053667840, percent=15.9, used=12859097088, free=69707034624, active=23275761664, inactive=4716683264, buffers=10153984, cached=17355874304, shared=1214205952, slab=766910464) 
2022-04-06 21:05:40.605741: 
epoch:  188 
2022-04-06 21:07:29.035158: train loss : -0.5317 
2022-04-06 21:07:40.586656: validation loss: -0.5008 
2022-04-06 21:07:40.602806: Average global foreground Dice: [0.7637, 0.92, 0.9543, 0.3726] 
2022-04-06 21:07:40.616483: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:07:41.291010: lr: 0.006522 
2022-04-06 21:07:41.304597: saving scheduled checkpoint file... 
2022-04-06 21:07:41.419744: saving checkpoint... 
2022-04-06 21:07:46.031953: done, saving took 4.71 seconds 
2022-04-06 21:07:46.061554: done 
2022-04-06 21:07:46.062692: This epoch took 125.456341 s
 
2022-04-06 21:07:46.066442: svmem(total=99932160000, available=84125388800, percent=15.8, used=12785430528, free=69724033024, active=23101235200, inactive=4872548352, buffers=10391552, cached=17412304896, shared=1214316544, slab=770072576) 
2022-04-06 21:07:46.069395: 
epoch:  189 
2022-04-06 21:09:35.434223: train loss : -0.4707 
2022-04-06 21:09:47.163973: validation loss: -0.5231 
2022-04-06 21:09:47.168259: Average global foreground Dice: [0.7803, 0.9469, 0.9605, 0.3737] 
2022-04-06 21:09:47.171945: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:09:48.125252: lr: 0.006504 
2022-04-06 21:09:48.138819: saving scheduled checkpoint file... 
2022-04-06 21:09:48.295199: saving checkpoint... 
2022-04-06 21:09:52.857513: done, saving took 4.71 seconds 
2022-04-06 21:09:52.891894: done 
2022-04-06 21:09:52.901443: This epoch took 126.829214 s
 
2022-04-06 21:09:52.910779: svmem(total=99932160000, available=84042096640, percent=15.9, used=12866322432, free=69711609856, active=23782465536, inactive=4199813120, buffers=10633216, cached=17343594496, shared=1214324736, slab=772308992) 
2022-04-06 21:09:52.918635: 
epoch:  190 
2022-04-06 21:11:41.287728: train loss : -0.5061 
2022-04-06 21:11:52.439837: validation loss: -0.5360 
2022-04-06 21:11:52.450572: Average global foreground Dice: [0.7899, 0.9424, 0.9626, 0.3746] 
2022-04-06 21:11:52.472651: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:11:53.188808: lr: 0.006485 
2022-04-06 21:11:53.189902: saving scheduled checkpoint file... 
2022-04-06 21:11:53.350385: saving checkpoint... 
2022-04-06 21:11:58.043301: done, saving took 4.85 seconds 
2022-04-06 21:11:58.069439: done 
2022-04-06 21:11:58.076381: This epoch took 125.150767 s
 
2022-04-06 21:11:58.080190: svmem(total=99932160000, available=83945648128, percent=16.0, used=12962484224, free=69725548544, active=23752273920, inactive=4214587392, buffers=10866688, cached=17233260544, shared=1215119360, slab=773599232) 
2022-04-06 21:11:58.083947: 
epoch:  191 
2022-04-06 21:13:47.963968: train loss : -0.5654 
2022-04-06 21:13:59.688612: validation loss: -0.5913 
2022-04-06 21:13:59.698312: Average global foreground Dice: [0.8256, 0.9614, 0.9722, 0.3877] 
2022-04-06 21:13:59.703042: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:14:00.867209: lr: 0.006466 
2022-04-06 21:14:00.891800: saving scheduled checkpoint file... 
2022-04-06 21:14:01.000572: saving checkpoint... 
2022-04-06 21:14:05.365237: done, saving took 4.45 seconds 
2022-04-06 21:14:05.383477: done 
2022-04-06 21:14:05.386025: This epoch took 127.299518 s
 
2022-04-06 21:14:05.388418: svmem(total=99932160000, available=83993948160, percent=15.9, used=12913197056, free=69558288384, active=23881420800, inactive=4254154752, buffers=11116544, cached=17449558016, shared=1215504384, slab=774684672) 
2022-04-06 21:14:05.390776: 
epoch:  192 
2022-04-06 21:15:53.316409: train loss : -0.5873 
2022-04-06 21:16:05.406581: validation loss: -0.6157 
2022-04-06 21:16:05.408074: Average global foreground Dice: [0.8219, 0.9577, 0.9785, 0.4158] 
2022-04-06 21:16:05.408878: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:16:06.267449: lr: 0.006447 
2022-04-06 21:16:06.268888: saving scheduled checkpoint file... 
2022-04-06 21:16:06.351453: saving checkpoint... 
2022-04-06 21:16:10.649639: done, saving took 4.38 seconds 
2022-04-06 21:16:10.664479: done 
2022-04-06 21:16:10.665255: This epoch took 125.272178 s
 
2022-04-06 21:16:10.666167: svmem(total=99932160000, available=83884494848, percent=16.1, used=13021716480, free=69545603072, active=23756881920, inactive=4388225024, buffers=11378688, cached=17353461760, shared=1216561152, slab=774189056) 
2022-04-06 21:16:10.666824: 
epoch:  193 
2022-04-06 21:17:58.248602: train loss : -0.5850 
2022-04-06 21:18:09.564464: validation loss: -0.5927 
2022-04-06 21:18:09.576501: Average global foreground Dice: [0.8378, 0.95, 0.9721, 0.4063] 
2022-04-06 21:18:09.586895: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:18:10.131538: lr: 0.006428 
2022-04-06 21:18:10.139238: saving scheduled checkpoint file... 
2022-04-06 21:18:10.237650: saving checkpoint... 
2022-04-06 21:18:14.667106: done, saving took 4.52 seconds 
2022-04-06 21:18:14.698116: done 
2022-04-06 21:18:14.705395: This epoch took 124.037908 s
 
2022-04-06 21:18:14.714652: svmem(total=99932160000, available=83970793472, percent=16.0, used=12934234112, free=69546954752, active=23847407616, inactive=4298813440, buffers=11636736, cached=17439334400, shared=1217818624, slab=775000064) 
2022-04-06 21:18:14.721678: 
epoch:  194 
2022-04-06 21:20:02.331843: train loss : -0.5758 
2022-04-06 21:20:13.437365: validation loss: -0.5432 
2022-04-06 21:20:13.450088: Average global foreground Dice: [0.819, 0.9434, 0.9708, 0.4099] 
2022-04-06 21:20:13.459099: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:20:14.480161: lr: 0.006409 
2022-04-06 21:20:14.492786: saving scheduled checkpoint file... 
2022-04-06 21:20:14.640848: saving checkpoint... 
2022-04-06 21:20:19.100456: done, saving took 4.60 seconds 
2022-04-06 21:20:19.125728: done 
2022-04-06 21:20:19.178501: saving checkpoint... 
2022-04-06 21:20:23.595438: done, saving took 4.47 seconds 
2022-04-06 21:20:23.609392: This epoch took 128.879531 s
 
2022-04-06 21:20:23.610415: svmem(total=99932160000, available=83973922816, percent=16.0, used=12919857152, free=69555523584, active=23722852352, inactive=4415909888, buffers=11886592, cached=17444892672, shared=1228591104, slab=775454720) 
2022-04-06 21:20:23.610991: 
epoch:  195 
2022-04-06 21:22:12.727709: train loss : -0.5687 
2022-04-06 21:22:23.840187: validation loss: -0.5813 
2022-04-06 21:22:23.850364: Average global foreground Dice: [0.8163, 0.9518, 0.9758, 0.3893] 
2022-04-06 21:22:23.851414: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:22:24.723064: lr: 0.00639 
2022-04-06 21:22:24.728441: saving scheduled checkpoint file... 
2022-04-06 21:22:24.838331: saving checkpoint... 
2022-04-06 21:22:29.327754: done, saving took 4.59 seconds 
2022-04-06 21:22:29.354082: done 
2022-04-06 21:22:29.406386: saving checkpoint... 
2022-04-06 21:22:33.832399: done, saving took 4.47 seconds 
2022-04-06 21:22:33.847111: This epoch took 130.235570 s
 
2022-04-06 21:22:33.850815: svmem(total=99932160000, available=83905159168, percent=16.0, used=12985176064, free=69522145280, active=23722831872, inactive=4445032448, buffers=12136448, cached=17412702208, shared=1231900672, slab=776228864) 
2022-04-06 21:22:33.855615: 
epoch:  196 
2022-04-06 21:24:22.604139: train loss : -0.6060 
2022-04-06 21:24:33.997143: validation loss: -0.5773 
2022-04-06 21:24:33.998594: Average global foreground Dice: [0.8092, 0.9473, 0.977, 0.4047] 
2022-04-06 21:24:33.999344: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:24:34.754082: lr: 0.006371 
2022-04-06 21:24:34.757740: saving scheduled checkpoint file... 
2022-04-06 21:24:34.898036: saving checkpoint... 
2022-04-06 21:24:39.388108: done, saving took 4.62 seconds 
2022-04-06 21:24:39.418978: done 
2022-04-06 21:24:39.513024: saving checkpoint... 
2022-04-06 21:24:43.971232: done, saving took 4.54 seconds 
2022-04-06 21:24:43.988491: This epoch took 130.130460 s
 
2022-04-06 21:24:43.989514: svmem(total=99932160000, available=83946221568, percent=16.0, used=12957863936, free=69618368512, active=23769329664, inactive=4301762560, buffers=12394496, cached=17343533056, shared=1218383872, slab=776597504) 
2022-04-06 21:24:43.990123: 
epoch:  197 
2022-04-06 21:26:32.371312: train loss : -0.5973 
2022-04-06 21:26:43.379655: validation loss: -0.6056 
2022-04-06 21:26:43.384454: Average global foreground Dice: [0.8479, 0.9524, 0.9765, 0.4064] 
2022-04-06 21:26:43.386760: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:26:44.534597: lr: 0.006352 
2022-04-06 21:26:44.544753: saving scheduled checkpoint file... 
2022-04-06 21:26:44.658757: saving checkpoint... 
2022-04-06 21:26:49.259383: done, saving took 4.71 seconds 
2022-04-06 21:26:49.271196: done 
2022-04-06 21:26:49.324429: saving checkpoint... 
2022-04-06 21:26:53.718158: done, saving took 4.44 seconds 
2022-04-06 21:26:53.731766: This epoch took 129.741082 s
 
2022-04-06 21:26:53.732561: svmem(total=99932160000, available=84002377728, percent=15.9, used=12901326848, free=69544259584, active=23481995264, inactive=4664717312, buffers=12668928, cached=17473904640, shared=1218215936, slab=775340032) 
2022-04-06 21:26:53.733094: 
epoch:  198 
2022-04-06 21:28:42.351586: train loss : -0.5886 
2022-04-06 21:28:53.512242: validation loss: -0.6098 
2022-04-06 21:28:53.515696: Average global foreground Dice: [0.8629, 0.9472, 0.9763, 0.3852] 
2022-04-06 21:28:53.531460: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:28:54.277744: lr: 0.006333 
2022-04-06 21:28:54.278577: saving scheduled checkpoint file... 
2022-04-06 21:28:54.375336: saving checkpoint... 
2022-04-06 21:28:58.804208: done, saving took 4.52 seconds 
2022-04-06 21:28:58.832530: done 
2022-04-06 21:28:58.890643: saving checkpoint... 
2022-04-06 21:29:03.398811: done, saving took 4.55 seconds 
2022-04-06 21:29:03.412881: This epoch took 129.679225 s
 
2022-04-06 21:29:03.413872: svmem(total=99932160000, available=83956424704, percent=16.0, used=12947726336, free=69545783296, active=23556378624, inactive=4591984640, buffers=12910592, cached=17425739776, shared=1217826816, slab=776237056) 
2022-04-06 21:29:03.414604: 
epoch:  199 
2022-04-06 21:30:53.416596: train loss : -0.5645 
2022-04-06 21:31:05.474640: validation loss: -0.5655 
2022-04-06 21:31:05.479012: Average global foreground Dice: [0.8161, 0.9484, 0.9686, 0.4056] 
2022-04-06 21:31:05.480734: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:31:06.311517: lr: 0.006314 
2022-04-06 21:31:06.317024: saving scheduled checkpoint file... 
2022-04-06 21:31:06.514328: saving checkpoint... 
2022-04-06 21:31:10.974921: done, saving took 4.65 seconds 
2022-04-06 21:31:10.995803: done 
2022-04-06 21:31:11.076585: saving checkpoint... 
2022-04-06 21:31:15.475520: done, saving took 4.48 seconds 
2022-04-06 21:31:15.492652: This epoch took 132.077372 s
 
2022-04-06 21:31:15.493699: svmem(total=99932160000, available=84081131520, percent=15.9, used=12822441984, free=69636509696, active=23761395712, inactive=4291178496, buffers=13176832, cached=17460031488, shared=1218416640, slab=777019392) 
2022-04-06 21:31:15.494472: 
epoch:  200 
2022-04-06 21:33:04.979362: train loss : -0.5718 
2022-04-06 21:33:17.125422: validation loss: -0.5808 
2022-04-06 21:33:17.132278: Average global foreground Dice: [0.8533, 0.9528, 0.968, 0.4031] 
2022-04-06 21:33:17.134099: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:33:17.993242: lr: 0.006296 
2022-04-06 21:33:18.004310: saving scheduled checkpoint file... 
2022-04-06 21:33:18.169178: saving checkpoint... 
2022-04-06 21:33:22.667287: done, saving took 4.65 seconds 
2022-04-06 21:33:22.691595: done 
2022-04-06 21:33:22.803913: saving checkpoint... 
2022-04-06 21:33:27.229899: done, saving took 4.53 seconds 
2022-04-06 21:33:27.248777: This epoch took 131.753663 s
 
2022-04-06 21:33:27.249848: svmem(total=99932160000, available=83965992960, percent=16.0, used=12937801728, free=69640843264, active=23690543104, inactive=4358307840, buffers=13426688, cached=17340088320, shared=1218080768, slab=776642560) 
2022-04-06 21:33:27.250526: 
epoch:  201 
2022-04-06 21:35:16.867955: train loss : -0.5754 
2022-04-06 21:35:28.560632: validation loss: -0.5383 
2022-04-06 21:35:28.575313: Average global foreground Dice: [0.7981, 0.937, 0.9623, 0.3963] 
2022-04-06 21:35:28.587748: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:35:29.532772: lr: 0.006277 
2022-04-06 21:35:29.535791: saving scheduled checkpoint file... 
2022-04-06 21:35:29.730995: saving checkpoint... 
2022-04-06 21:35:34.176563: done, saving took 4.64 seconds 
2022-04-06 21:35:34.204593: done 
2022-04-06 21:35:34.209151: This epoch took 126.957952 s
 
2022-04-06 21:35:34.214387: svmem(total=99932160000, available=83993673728, percent=15.9, used=12909662208, free=69635989504, active=23758118912, inactive=4294385664, buffers=13692928, cached=17372815360, shared=1218260992, slab=776265728) 
2022-04-06 21:35:34.219344: 
epoch:  202 
2022-04-06 21:37:24.194105: train loss : -0.5736 
2022-04-06 21:37:36.100352: validation loss: -0.5830 
2022-04-06 21:37:36.108018: Average global foreground Dice: [0.8183, 0.952, 0.9718, 0.3995] 
2022-04-06 21:37:36.108989: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:37:37.028389: lr: 0.006258 
2022-04-06 21:37:37.029662: saving scheduled checkpoint file... 
2022-04-06 21:37:37.159855: saving checkpoint... 
2022-04-06 21:37:41.655478: done, saving took 4.62 seconds 
2022-04-06 21:37:41.681424: done 
2022-04-06 21:37:41.686984: This epoch took 127.461925 s
 
2022-04-06 21:37:41.691822: svmem(total=99932160000, available=84016234496, percent=15.9, used=12886990848, free=69638336512, active=23769337856, inactive=4284784640, buffers=13938688, cached=17392893952, shared=1218265088, slab=777240576) 
2022-04-06 21:37:41.697653: 
epoch:  203 
2022-04-06 21:39:32.399013: train loss : -0.5901 
2022-04-06 21:39:44.613436: validation loss: -0.6125 
2022-04-06 21:39:44.644541: Average global foreground Dice: [0.8416, 0.9492, 0.9773, 0.3903] 
2022-04-06 21:39:44.655178: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:39:45.527397: lr: 0.006239 
2022-04-06 21:39:45.543944: saving scheduled checkpoint file... 
2022-04-06 21:39:45.644278: saving checkpoint... 
2022-04-06 21:39:50.101812: done, saving took 4.55 seconds 
2022-04-06 21:39:50.118670: done 
2022-04-06 21:39:50.122353: This epoch took 128.421503 s
 
2022-04-06 21:39:50.124543: svmem(total=99932160000, available=83965902848, percent=16.0, used=12936339456, free=69525417984, active=23503175680, inactive=4658462720, buffers=14188544, cached=17456214016, shared=1219428352, slab=777482240) 
2022-04-06 21:39:50.126228: 
epoch:  204 
2022-04-06 21:41:38.745646: train loss : -0.5847 
2022-04-06 21:41:50.450016: validation loss: -0.5638 
2022-04-06 21:41:50.453497: Average global foreground Dice: [0.8319, 0.942, 0.9585, 0.4133] 
2022-04-06 21:41:50.457468: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:41:51.307366: lr: 0.00622 
2022-04-06 21:41:51.316390: saving scheduled checkpoint file... 
2022-04-06 21:41:51.484831: saving checkpoint... 
2022-04-06 21:41:55.842428: done, saving took 4.52 seconds 
2022-04-06 21:41:55.867576: done 
2022-04-06 21:41:55.917115: saving checkpoint... 
2022-04-06 21:42:00.232759: done, saving took 4.36 seconds 
2022-04-06 21:42:00.248347: This epoch took 130.120427 s
 
2022-04-06 21:42:00.250927: svmem(total=99932160000, available=83913084928, percent=16.0, used=12990672896, free=69530460160, active=23544832000, inactive=4616343552, buffers=14438400, cached=17396588544, shared=1218093056, slab=776306688) 
2022-04-06 21:42:00.253409: 
epoch:  205 
2022-04-06 21:43:49.291852: train loss : -0.5695 
2022-04-06 21:44:01.258959: validation loss: -0.5745 
2022-04-06 21:44:01.263671: Average global foreground Dice: [0.8477, 0.9453, 0.976, 0.3773] 
2022-04-06 21:44:01.264457: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:44:02.046826: lr: 0.006201 
2022-04-06 21:44:02.053817: saving scheduled checkpoint file... 
2022-04-06 21:44:02.146318: saving checkpoint... 
2022-04-06 21:44:06.476232: done, saving took 4.41 seconds 
2022-04-06 21:44:06.494865: done 
2022-04-06 21:44:06.549197: saving checkpoint... 
2022-04-06 21:44:10.841681: done, saving took 4.34 seconds 
2022-04-06 21:44:10.849889: This epoch took 130.594831 s
 
2022-04-06 21:44:10.850795: svmem(total=99932160000, available=83924488192, percent=16.0, used=12965765120, free=69523300352, active=23884046336, inactive=4285673472, buffers=14688256, cached=17428406272, shared=1232056320, slab=774803456) 
2022-04-06 21:44:10.851544: 
epoch:  206 
2022-04-06 21:45:59.090248: train loss : -0.5830 
2022-04-06 21:46:09.968443: validation loss: -0.5502 
2022-04-06 21:46:09.970127: Average global foreground Dice: [0.8399, 0.9479, 0.9614, 0.4029] 
2022-04-06 21:46:09.971797: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:46:10.865845: lr: 0.006182 
2022-04-06 21:46:10.869931: saving scheduled checkpoint file... 
2022-04-06 21:46:10.924187: saving checkpoint... 
2022-04-06 21:46:15.510584: done, saving took 4.64 seconds 
2022-04-06 21:46:15.528590: done 
2022-04-06 21:46:15.581408: saving checkpoint... 
2022-04-06 21:46:20.001390: done, saving took 4.47 seconds 
2022-04-06 21:46:20.015754: This epoch took 129.163640 s
 
2022-04-06 21:46:20.016763: svmem(total=99932160000, available=83954888704, percent=16.0, used=12948791296, free=69526626304, active=23850049536, inactive=4313169920, buffers=14950400, cached=17441792000, shared=1218101248, slab=775032832) 
2022-04-06 21:46:20.017500: 
epoch:  207 
2022-04-06 21:48:08.189761: train loss : -0.5664 
2022-04-06 21:48:20.556654: validation loss: -0.5099 
2022-04-06 21:48:20.562903: Average global foreground Dice: [0.8327, 0.9344, 0.9451, 0.39] 
2022-04-06 21:48:20.564095: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:48:21.409418: lr: 0.006163 
2022-04-06 21:48:21.426788: saving scheduled checkpoint file... 
2022-04-06 21:48:21.521831: saving checkpoint... 
2022-04-06 21:48:26.083900: done, saving took 4.65 seconds 
2022-04-06 21:48:26.100546: done 
2022-04-06 21:48:26.102978: This epoch took 126.084818 s
 
2022-04-06 21:48:26.107240: svmem(total=99932160000, available=83889852416, percent=16.1, used=13014384640, free=69522714624, active=23828762624, inactive=4336705536, buffers=15192064, cached=17379868672, shared=1218134016, slab=777134080) 
2022-04-06 21:48:26.111753: 
epoch:  208 
2022-04-06 21:50:13.875377: train loss : -0.5398 
2022-04-06 21:50:25.819350: validation loss: -0.5582 
2022-04-06 21:50:25.837366: Average global foreground Dice: [0.8276, 0.9402, 0.9658, 0.4119] 
2022-04-06 21:50:25.845554: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:50:26.691536: lr: 0.006144 
2022-04-06 21:50:26.692291: saving scheduled checkpoint file... 
2022-04-06 21:50:26.796025: saving checkpoint... 
2022-04-06 21:50:31.218411: done, saving took 4.53 seconds 
2022-04-06 21:50:31.237099: done 
2022-04-06 21:50:31.238925: This epoch took 125.120914 s
 
2022-04-06 21:50:31.241331: svmem(total=99932160000, available=83914158080, percent=16.0, used=12988780544, free=69601038336, active=23737675776, inactive=4347285504, buffers=15433728, cached=17326907392, shared=1218224128, slab=776507392) 
2022-04-06 21:50:31.243715: 
epoch:  209 
2022-04-06 21:52:19.660179: train loss : -0.5791 
2022-04-06 21:52:31.709424: validation loss: -0.5832 
2022-04-06 21:52:31.711371: Average global foreground Dice: [0.8193, 0.9535, 0.9733, 0.39] 
2022-04-06 21:52:31.713263: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:52:32.576840: lr: 0.006125 
2022-04-06 21:52:32.591570: saving scheduled checkpoint file... 
2022-04-06 21:52:32.670171: saving checkpoint... 
2022-04-06 21:52:37.145685: done, saving took 4.54 seconds 
2022-04-06 21:52:37.185880: done 
2022-04-06 21:52:37.202170: This epoch took 125.956018 s
 
2022-04-06 21:52:37.213731: svmem(total=99932160000, available=84094889984, percent=15.8, used=12804378624, free=69529935872, active=23863336960, inactive=4295692288, buffers=15671296, cached=17582174208, shared=1222602752, slab=776368128) 
2022-04-06 21:52:37.226870: 
epoch:  210 
2022-04-06 21:54:27.275299: train loss : -0.5315 
2022-04-06 21:54:38.315938: validation loss: -0.5240 
2022-04-06 21:54:38.321048: Average global foreground Dice: [0.8296, 0.9449, 0.9574, 0.3611] 
2022-04-06 21:54:38.321867: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:54:39.187997: lr: 0.006106 
2022-04-06 21:54:39.191039: saving scheduled checkpoint file... 
2022-04-06 21:54:39.293809: saving checkpoint... 
2022-04-06 21:54:43.768481: done, saving took 4.57 seconds 
2022-04-06 21:54:43.782802: done 
2022-04-06 21:54:43.786571: This epoch took 126.546815 s
 
2022-04-06 21:54:43.789862: svmem(total=99932160000, available=84057591808, percent=15.9, used=12845805568, free=69521522688, active=23770951680, inactive=4396285952, buffers=15929344, cached=17548902400, shared=1218248704, slab=777043968) 
2022-04-06 21:54:43.790714: 
epoch:  211 
2022-04-06 21:56:33.309779: train loss : -0.5363 
2022-04-06 21:56:45.369101: validation loss: -0.5281 
2022-04-06 21:56:45.370779: Average global foreground Dice: [0.7999, 0.9277, 0.9574, 0.3964] 
2022-04-06 21:56:45.371492: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:56:46.408637: lr: 0.006087 
2022-04-06 21:56:46.422887: saving scheduled checkpoint file... 
2022-04-06 21:56:46.530055: saving checkpoint... 
2022-04-06 21:56:50.842222: done, saving took 4.41 seconds 
2022-04-06 21:56:50.874366: done 
2022-04-06 21:56:50.884784: This epoch took 127.093278 s
 
2022-04-06 21:56:50.892830: svmem(total=99932160000, available=83998195712, percent=15.9, used=12904706048, free=69583200256, active=23706222592, inactive=4397674496, buffers=16195584, cached=17428058112, shared=1218252800, slab=776953856) 
2022-04-06 21:56:50.906005: 
epoch:  212 
2022-04-06 21:58:42.096811: train loss : -0.5648 
2022-04-06 21:58:53.512055: validation loss: -0.5268 
2022-04-06 21:58:53.551035: Average global foreground Dice: [0.8228, 0.9379, 0.9549, 0.3764] 
2022-04-06 21:58:53.572769: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 21:58:54.193099: lr: 0.006068 
2022-04-06 21:58:54.199336: saving scheduled checkpoint file... 
2022-04-06 21:58:54.384182: saving checkpoint... 
2022-04-06 21:58:58.840445: done, saving took 4.63 seconds 
2022-04-06 21:58:58.889211: done 
2022-04-06 21:58:58.902708: This epoch took 127.987015 s
 
2022-04-06 21:58:58.918541: svmem(total=99932160000, available=84132958208, percent=15.8, used=12770271232, free=69612494848, active=23696957440, inactive=4380176384, buffers=16437248, cached=17532956672, shared=1218355200, slab=777957376) 
2022-04-06 21:58:58.934025: 
epoch:  213 
2022-04-06 22:00:48.968680: train loss : -0.5198 
2022-04-06 22:01:00.524261: validation loss: -0.3980 
2022-04-06 22:01:00.531533: Average global foreground Dice: [0.784, 0.9176, 0.8866, 0.3891] 
2022-04-06 22:01:00.534659: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:01:01.416025: lr: 0.006049 
2022-04-06 22:01:01.422790: saving scheduled checkpoint file... 
2022-04-06 22:01:01.560402: saving checkpoint... 
2022-04-06 22:01:06.082230: done, saving took 4.65 seconds 
2022-04-06 22:01:06.122169: done 
2022-04-06 22:01:06.130593: This epoch took 127.182719 s
 
2022-04-06 22:01:06.140890: svmem(total=99932160000, available=84064309248, percent=15.9, used=12838920192, free=69617041408, active=23716474880, inactive=4354994176, buffers=16834560, cached=17459363840, shared=1218359296, slab=777838592) 
2022-04-06 22:01:06.149053: 
epoch:  214 
2022-04-06 22:02:55.189363: train loss : -0.4322 
2022-04-06 22:03:07.771114: validation loss: -0.4860 
2022-04-06 22:03:07.785249: Average global foreground Dice: [0.7925, 0.9122, 0.9426, 0.3607] 
2022-04-06 22:03:07.793987: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:03:08.668096: lr: 0.00603 
2022-04-06 22:03:08.670026: saving scheduled checkpoint file... 
2022-04-06 22:03:08.826186: saving checkpoint... 
2022-04-06 22:03:13.414666: done, saving took 4.74 seconds 
2022-04-06 22:03:13.451354: done 
2022-04-06 22:03:13.457623: This epoch took 127.301334 s
 
2022-04-06 22:03:13.462364: svmem(total=99932160000, available=83997286400, percent=15.9, used=12906328064, free=69603938304, active=23543361536, inactive=4539600896, buffers=17076224, cached=17404817408, shared=1218101248, slab=778358784) 
2022-04-06 22:03:13.463097: 
epoch:  215 
2022-04-06 22:05:05.077995: train loss : -0.4800 
2022-04-06 22:05:16.842803: validation loss: -0.5407 
2022-04-06 22:05:16.847324: Average global foreground Dice: [0.7844, 0.9404, 0.969, 0.387] 
2022-04-06 22:05:16.848170: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:05:17.555445: lr: 0.006011 
2022-04-06 22:05:17.561819: saving scheduled checkpoint file... 
2022-04-06 22:05:17.699061: saving checkpoint... 
2022-04-06 22:05:22.577493: done, saving took 5.01 seconds 
2022-04-06 22:05:22.608786: done 
2022-04-06 22:05:22.617858: This epoch took 129.153989 s
 
2022-04-06 22:05:22.626711: svmem(total=99932160000, available=83939999744, percent=16.0, used=12956454912, free=69576499200, active=23711191040, inactive=4404035584, buffers=17342464, cached=17381863424, shared=1225064448, slab=777625600) 
2022-04-06 22:05:22.633760: 
epoch:  216 
2022-04-06 22:07:10.165834: train loss : -0.5218 
2022-04-06 22:07:22.098990: validation loss: -0.5326 
2022-04-06 22:07:22.120029: Average global foreground Dice: [0.8068, 0.9426, 0.9618, 0.4025] 
2022-04-06 22:07:22.125423: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:07:22.861067: lr: 0.005991 
2022-04-06 22:07:22.872995: saving scheduled checkpoint file... 
2022-04-06 22:07:22.999619: saving checkpoint... 
2022-04-06 22:07:27.482344: done, saving took 4.59 seconds 
2022-04-06 22:07:27.500105: done 
2022-04-06 22:07:27.502172: This epoch took 124.859325 s
 
2022-04-06 22:07:27.504839: svmem(total=99932160000, available=83890540544, percent=16.1, used=13013512192, free=69510000640, active=23764107264, inactive=4414361600, buffers=17571840, cached=17391075328, shared=1218289664, slab=777592832) 
2022-04-06 22:07:27.506235: 
epoch:  217 
2022-04-06 22:09:16.035366: train loss : -0.4766 
2022-04-06 22:09:27.347787: validation loss: -0.4835 
2022-04-06 22:09:27.374194: Average global foreground Dice: [0.8029, 0.9329, 0.9458, 0.3558] 
2022-04-06 22:09:27.381669: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:09:28.410449: lr: 0.005972 
2022-04-06 22:09:28.418758: saving scheduled checkpoint file... 
2022-04-06 22:09:28.598972: saving checkpoint... 
2022-04-06 22:09:33.166038: done, saving took 4.74 seconds 
2022-04-06 22:09:33.201331: done 
2022-04-06 22:09:33.213181: This epoch took 125.704151 s
 
2022-04-06 22:09:33.221727: svmem(total=99932160000, available=83976077312, percent=16.0, used=12926693376, free=69506576384, active=23809662976, inactive=4367552512, buffers=17813504, cached=17481076736, shared=1218306048, slab=777920512) 
2022-04-06 22:09:33.230821: 
epoch:  218 
2022-04-06 22:11:22.633407: train loss : -0.5589 
2022-04-06 22:11:34.266269: validation loss: -0.5820 
2022-04-06 22:11:34.269733: Average global foreground Dice: [0.8292, 0.9533, 0.9737, 0.3878] 
2022-04-06 22:11:34.273581: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:11:35.119037: lr: 0.005953 
2022-04-06 22:11:35.125994: saving scheduled checkpoint file... 
2022-04-06 22:11:35.220351: saving checkpoint... 
2022-04-06 22:11:39.579420: done, saving took 4.44 seconds 
2022-04-06 22:11:39.622498: done 
2022-04-06 22:11:39.634750: This epoch took 126.389419 s
 
2022-04-06 22:11:39.650226: svmem(total=99932160000, available=83943940096, percent=16.0, used=12959436800, free=69503438848, active=23672999936, inactive=4505468928, buffers=18079744, cached=17451204608, shared=1218220032, slab=778039296) 
2022-04-06 22:11:39.664542: 
epoch:  219 
2022-04-06 22:13:28.423439: train loss : -0.5852 
2022-04-06 22:13:39.997506: validation loss: -0.5912 
2022-04-06 22:13:40.000798: Average global foreground Dice: [0.8383, 0.9532, 0.9796, 0.3946] 
2022-04-06 22:13:40.004004: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:13:40.826076: lr: 0.005934 
2022-04-06 22:13:40.831828: saving scheduled checkpoint file... 
2022-04-06 22:13:40.922149: saving checkpoint... 
2022-04-06 22:13:45.413681: done, saving took 4.57 seconds 
2022-04-06 22:13:45.430833: done 
2022-04-06 22:13:45.434178: This epoch took 125.754679 s
 
2022-04-06 22:13:45.436956: svmem(total=99932160000, available=83989340160, percent=16.0, used=12913856512, free=69507530752, active=23599915008, inactive=4574785536, buffers=18313216, cached=17492459520, shared=1218392064, slab=777895936) 
2022-04-06 22:13:45.439978: 
epoch:  220 
2022-04-06 22:15:34.520899: train loss : -0.5446 
2022-04-06 22:15:44.787315: validation loss: -0.5554 
2022-04-06 22:15:44.800753: Average global foreground Dice: [0.7965, 0.9458, 0.9733, 0.4025] 
2022-04-06 22:15:44.804824: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:15:45.546325: lr: 0.005915 
2022-04-06 22:15:45.549582: saving scheduled checkpoint file... 
2022-04-06 22:15:45.612139: saving checkpoint... 
2022-04-06 22:15:50.167370: done, saving took 4.61 seconds 
2022-04-06 22:15:50.195151: done 
2022-04-06 22:15:50.210375: This epoch took 124.767836 s
 
2022-04-06 22:15:50.225190: svmem(total=99932160000, available=83954294784, percent=16.0, used=12944543744, free=69499969536, active=23768567808, inactive=4418920448, buffers=18575360, cached=17469071360, shared=1222680576, slab=778768384) 
2022-04-06 22:15:50.233800: 
epoch:  221 
2022-04-06 22:17:38.726965: train loss : -0.5595 
2022-04-06 22:17:49.523425: validation loss: -0.5694 
2022-04-06 22:17:49.526017: Average global foreground Dice: [0.8216, 0.9577, 0.9681, 0.378] 
2022-04-06 22:17:49.529717: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:17:50.386045: lr: 0.005896 
2022-04-06 22:17:50.390682: saving scheduled checkpoint file... 
2022-04-06 22:17:50.527135: saving checkpoint... 
2022-04-06 22:17:55.444410: done, saving took 5.05 seconds 
2022-04-06 22:17:55.471714: done 
2022-04-06 22:17:55.476088: This epoch took 125.229474 s
 
2022-04-06 22:17:55.479037: svmem(total=99932160000, available=83937722368, percent=16.0, used=12965003264, free=69556789248, active=23744266240, inactive=4377616384, buffers=18825216, cached=17391542272, shared=1218392064, slab=778706944) 
2022-04-06 22:17:55.482121: 
epoch:  222 
2022-04-06 22:19:44.011877: train loss : -0.5809 
2022-04-06 22:19:54.883197: validation loss: -0.6020 
2022-04-06 22:19:54.893856: Average global foreground Dice: [0.8431, 0.9574, 0.9704, 0.4] 
2022-04-06 22:19:54.896627: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:19:55.844953: lr: 0.005877 
2022-04-06 22:19:55.875125: saving scheduled checkpoint file... 
2022-04-06 22:19:55.974424: saving checkpoint... 
2022-04-06 22:20:00.548157: done, saving took 4.65 seconds 
2022-04-06 22:20:00.570020: done 
2022-04-06 22:20:00.577310: This epoch took 125.091657 s
 
2022-04-06 22:20:00.583241: svmem(total=99932160000, available=84040396800, percent=15.9, used=12848521216, free=69489139712, active=23634558976, inactive=4561080320, buffers=19066880, cached=17575432192, shared=1232097280, slab=777203712) 
2022-04-06 22:20:00.590156: 
epoch:  223 
2022-04-06 22:21:48.345411: train loss : -0.5926 
2022-04-06 22:22:00.305112: validation loss: -0.6199 
2022-04-06 22:22:00.310581: Average global foreground Dice: [0.844, 0.9528, 0.9804, 0.4074] 
2022-04-06 22:22:00.315611: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:22:01.195460: lr: 0.005858 
2022-04-06 22:22:01.198664: saving scheduled checkpoint file... 
2022-04-06 22:22:01.258634: saving checkpoint... 
2022-04-06 22:22:05.790438: done, saving took 4.59 seconds 
2022-04-06 22:22:05.803259: done 
2022-04-06 22:22:05.804094: This epoch took 125.207747 s
 
2022-04-06 22:22:05.805044: svmem(total=99932160000, available=83978711040, percent=16.0, used=12911038464, free=69489905664, active=23819272192, inactive=4379246592, buffers=19304448, cached=17511911424, shared=1231773696, slab=778223616) 
2022-04-06 22:22:05.805757: 
epoch:  224 
2022-04-06 22:23:55.829874: train loss : -0.5809 
2022-04-06 22:24:17.422831: validation loss: -0.5703 
2022-04-06 22:24:17.439885: Average global foreground Dice: [0.8251, 0.9393, 0.9565, 0.3978] 
2022-04-06 22:24:17.445597: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:24:18.374164: lr: 0.005839 
2022-04-06 22:24:18.382746: saving scheduled checkpoint file... 
2022-04-06 22:24:18.483301: saving checkpoint... 
2022-04-06 22:24:22.897605: done, saving took 4.51 seconds 
2022-04-06 22:24:22.924435: done 
2022-04-06 22:24:22.926730: This epoch took 137.120267 s
 
2022-04-06 22:24:22.929905: svmem(total=99932160000, available=84044918784, percent=15.9, used=12858593280, free=69576445952, active=23472037888, inactive=4640542720, buffers=19566592, cached=17477554176, shared=1218220032, slab=777707520) 
2022-04-06 22:24:22.933479: 
epoch:  225 
2022-04-06 22:26:13.245350: train loss : -0.5065 
2022-04-06 22:26:24.789767: validation loss: -0.5163 
2022-04-06 22:26:24.791298: Average global foreground Dice: [0.8056, 0.943, 0.9369, 0.3901] 
2022-04-06 22:26:24.791989: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:26:25.589975: lr: 0.00582 
2022-04-06 22:26:25.605120: saving scheduled checkpoint file... 
2022-04-06 22:26:25.737687: saving checkpoint... 
2022-04-06 22:26:30.249623: done, saving took 4.64 seconds 
2022-04-06 22:26:30.279699: done 
2022-04-06 22:26:30.281461: This epoch took 127.343531 s
 
2022-04-06 22:26:30.283934: svmem(total=99932160000, available=84105592832, percent=15.8, used=12797353984, free=69597962240, active=23738417152, inactive=4352188416, buffers=19832832, cached=17517010944, shared=1218306048, slab=778575872) 
2022-04-06 22:26:30.285499: 
epoch:  226 
2022-04-06 22:28:19.027587: train loss : -0.4875 
2022-04-06 22:28:31.716730: validation loss: -0.5174 
2022-04-06 22:28:31.720049: Average global foreground Dice: [0.8203, 0.9399, 0.9527, 0.3276] 
2022-04-06 22:28:31.720804: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:28:32.597761: lr: 0.005801 
2022-04-06 22:28:32.601876: saving scheduled checkpoint file... 
2022-04-06 22:28:32.713479: saving checkpoint... 
2022-04-06 22:28:37.166618: done, saving took 4.56 seconds 
2022-04-06 22:28:37.195472: done 
2022-04-06 22:28:37.202293: This epoch took 126.912256 s
 
2022-04-06 22:28:37.211375: svmem(total=99932160000, available=83963502592, percent=16.0, used=12939816960, free=69601382400, active=23758880768, inactive=4328595456, buffers=20070400, cached=17370890240, shared=1218220032, slab=777736192) 
2022-04-06 22:28:37.217870: 
epoch:  227 
2022-04-06 22:30:26.346493: train loss : -0.5545 
2022-04-06 22:30:37.748803: validation loss: -0.5700 
2022-04-06 22:30:37.755751: Average global foreground Dice: [0.8196, 0.9524, 0.9694, 0.3791] 
2022-04-06 22:30:37.765238: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:30:38.650189: lr: 0.005781 
2022-04-06 22:30:38.651197: saving scheduled checkpoint file... 
2022-04-06 22:30:38.790817: saving checkpoint... 
2022-04-06 22:30:43.491226: done, saving took 4.84 seconds 
2022-04-06 22:30:43.524820: done 
2022-04-06 22:30:43.529373: This epoch took 126.303617 s
 
2022-04-06 22:30:43.539330: svmem(total=99932160000, available=83976122368, percent=16.0, used=12927045632, free=69570240512, active=23753314304, inactive=4358754304, buffers=20357120, cached=17414516736, shared=1218220032, slab=779788288) 
2022-04-06 22:30:43.548962: 
epoch:  228 
2022-04-06 22:32:33.597114: train loss : -0.5747 
2022-04-06 22:32:46.234615: validation loss: -0.5888 
2022-04-06 22:32:46.243448: Average global foreground Dice: [0.8289, 0.946, 0.9713, 0.4044] 
2022-04-06 22:32:46.246228: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:32:46.991251: lr: 0.005762 
2022-04-06 22:32:46.992477: saving scheduled checkpoint file... 
2022-04-06 22:32:47.105917: saving checkpoint... 
2022-04-06 22:32:51.538467: done, saving took 4.54 seconds 
2022-04-06 22:32:53.345192: done 
2022-04-06 22:32:53.345969: This epoch took 129.788634 s
 
2022-04-06 22:32:53.346685: svmem(total=99932160000, available=83961020416, percent=16.0, used=12942135296, free=69495611392, active=23739830272, inactive=4450877440, buffers=20606976, cached=17473806336, shared=1218306048, slab=778457088) 
2022-04-06 22:32:53.347185: 
epoch:  229 
2022-04-06 22:34:43.648268: train loss : -0.5610 
2022-04-06 22:34:55.685427: validation loss: -0.5771 
2022-04-06 22:34:55.686827: Average global foreground Dice: [0.8326, 0.9511, 0.9698, 0.4126] 
2022-04-06 22:34:55.687592: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:34:56.468602: lr: 0.005743 
2022-04-06 22:34:56.485656: saving scheduled checkpoint file... 
2022-04-06 22:34:56.554268: saving checkpoint... 
2022-04-06 22:35:00.905613: done, saving took 4.40 seconds 
2022-04-06 22:35:00.962534: done 
2022-04-06 22:35:00.984651: This epoch took 127.636944 s
 
2022-04-06 22:35:01.004768: svmem(total=99932160000, available=84056068096, percent=15.9, used=12872130560, free=69495472128, active=23722684416, inactive=4467146752, buffers=20848640, cached=17543708672, shared=1193730048, slab=777871360) 
2022-04-06 22:35:01.023477: 
epoch:  230 
2022-04-06 22:36:51.062329: train loss : -0.5889 
2022-04-06 22:37:03.554681: validation loss: -0.5943 
2022-04-06 22:37:03.564103: Average global foreground Dice: [0.8091, 0.9522, 0.9772, 0.384] 
2022-04-06 22:37:03.564926: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:37:04.184260: lr: 0.005724 
2022-04-06 22:37:04.187179: saving scheduled checkpoint file... 
2022-04-06 22:37:04.268009: saving checkpoint... 
2022-04-06 22:37:08.707797: done, saving took 4.52 seconds 
2022-04-06 22:37:08.726732: done 
2022-04-06 22:37:08.728945: This epoch took 127.685289 s
 
2022-04-06 22:37:08.731763: svmem(total=99932160000, available=84010160128, percent=15.9, used=12893032448, free=69488054272, active=23681540096, inactive=4512608256, buffers=21127168, cached=17529946112, shared=1218220032, slab=778493952) 
2022-04-06 22:37:08.733924: 
epoch:  231 
2022-04-06 22:38:58.348845: train loss : -0.6049 
2022-04-06 22:39:10.546229: validation loss: -0.5974 
2022-04-06 22:39:10.554525: Average global foreground Dice: [0.8397, 0.9565, 0.9781, 0.3979] 
2022-04-06 22:39:10.566429: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:39:11.251261: lr: 0.005705 
2022-04-06 22:39:11.254571: saving scheduled checkpoint file... 
2022-04-06 22:39:11.315880: saving checkpoint... 
2022-04-06 22:39:15.488415: done, saving took 4.23 seconds 
2022-04-06 22:39:15.518801: done 
2022-04-06 22:39:15.524554: This epoch took 126.788154 s
 
2022-04-06 22:39:15.532310: svmem(total=99932160000, available=83956015104, percent=16.0, used=12947390464, free=69490393088, active=23842951168, inactive=4353937408, buffers=21368832, cached=17473007616, shared=1218306048, slab=777437184) 
2022-04-06 22:39:15.538808: 
epoch:  232 
2022-04-06 22:41:04.454299: train loss : -0.6201 
2022-04-06 22:41:15.319488: validation loss: -0.6138 
2022-04-06 22:41:15.323052: Average global foreground Dice: [0.8389, 0.9552, 0.9802, 0.433] 
2022-04-06 22:41:15.328229: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:41:16.149563: lr: 0.005686 
2022-04-06 22:41:16.155932: saving scheduled checkpoint file... 
2022-04-06 22:41:16.258229: saving checkpoint... 
2022-04-06 22:41:20.930611: done, saving took 4.77 seconds 
2022-04-06 22:41:20.960515: done 
2022-04-06 22:41:20.969220: This epoch took 125.424826 s
 
2022-04-06 22:41:20.977087: svmem(total=99932160000, available=84009873408, percent=15.9, used=12879339520, free=69543755776, active=23733493760, inactive=4411715584, buffers=21610496, cached=17487454208, shared=1232097280, slab=776724480) 
2022-04-06 22:41:20.983845: 
epoch:  233 
2022-04-06 22:43:11.843704: train loss : -0.6049 
2022-04-06 22:43:35.317588: validation loss: -0.6306 
2022-04-06 22:43:35.330709: Average global foreground Dice: [0.8512, 0.9582, 0.976, 0.4033] 
2022-04-06 22:43:35.331499: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:43:36.175017: lr: 0.005667 
2022-04-06 22:43:36.178011: saving scheduled checkpoint file... 
2022-04-06 22:43:36.304928: saving checkpoint... 
2022-04-06 22:43:40.915921: done, saving took 4.73 seconds 
2022-04-06 22:43:40.936785: done 
2022-04-06 22:43:40.997650: saving checkpoint... 
2022-04-06 22:43:47.254966: done, saving took 6.32 seconds 
2022-04-06 22:43:47.265749: This epoch took 146.270074 s
 
2022-04-06 22:43:47.266683: svmem(total=99932160000, available=83966078976, percent=16.0, used=12937048064, free=69501919232, active=23821922304, inactive=4369039360, buffers=21893120, cached=17471299584, shared=1218232320, slab=776126464) 
2022-04-06 22:43:47.267457: 
epoch:  234 
2022-04-06 22:45:36.185468: train loss : -0.6237 
2022-04-06 22:45:48.280931: validation loss: -0.5998 
2022-04-06 22:45:48.282374: Average global foreground Dice: [0.8386, 0.9589, 0.9752, 0.3957] 
2022-04-06 22:45:48.283182: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:45:49.023358: lr: 0.005647 
2022-04-06 22:45:49.039589: saving scheduled checkpoint file... 
2022-04-06 22:45:49.126578: saving checkpoint... 
2022-04-06 22:45:53.718360: done, saving took 4.67 seconds 
2022-04-06 22:45:53.732024: done 
2022-04-06 22:45:53.781574: saving checkpoint... 
2022-04-06 22:45:58.211273: done, saving took 4.48 seconds 
2022-04-06 22:45:58.227412: This epoch took 130.959257 s
 
2022-04-06 22:45:58.231346: svmem(total=99932160000, available=84069232640, percent=15.9, used=12833501184, free=69499715584, active=23582334976, inactive=4608655360, buffers=22163456, cached=17576779776, shared=1218244608, slab=776695808) 
2022-04-06 22:45:58.234369: 
epoch:  235 
2022-04-06 22:47:46.833135: train loss : -0.6064 
2022-04-06 22:47:59.022739: validation loss: -0.6259 
2022-04-06 22:47:59.024409: Average global foreground Dice: [0.8618, 0.9586, 0.9687, 0.4165] 
2022-04-06 22:47:59.025291: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:47:59.803152: lr: 0.005628 
2022-04-06 22:47:59.808950: saving scheduled checkpoint file... 
2022-04-06 22:48:00.052771: saving checkpoint... 
2022-04-06 22:48:04.587922: done, saving took 4.77 seconds 
2022-04-06 22:48:04.597426: done 
2022-04-06 22:48:04.647206: saving checkpoint... 
2022-04-06 22:48:09.062892: done, saving took 4.46 seconds 
2022-04-06 22:48:09.075398: This epoch took 130.837601 s
 
2022-04-06 22:48:09.076458: svmem(total=99932160000, available=84063772672, percent=15.9, used=12839268352, free=69490196480, active=23797272576, inactive=4403666944, buffers=22405120, cached=17580290048, shared=1218416640, slab=775540736) 
2022-04-06 22:48:09.077160: 
epoch:  236 
2022-04-06 22:49:57.231092: train loss : -0.6165 
2022-04-06 22:50:08.805966: validation loss: -0.6165 
2022-04-06 22:50:08.819936: Average global foreground Dice: [0.8492, 0.9571, 0.978, 0.3865] 
2022-04-06 22:50:08.827514: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:50:09.480890: lr: 0.005609 
2022-04-06 22:50:09.482955: saving scheduled checkpoint file... 
2022-04-06 22:50:09.606915: saving checkpoint... 
2022-04-06 22:50:14.258428: done, saving took 4.77 seconds 
2022-04-06 22:50:14.284461: done 
2022-04-06 22:50:14.377900: saving checkpoint... 
2022-04-06 22:50:18.842726: done, saving took 4.55 seconds 
2022-04-06 22:50:18.869784: This epoch took 129.791843 s
 
2022-04-06 22:50:18.874469: svmem(total=99932160000, available=84055183360, percent=15.9, used=12847702016, free=69561921536, active=23745835008, inactive=4381708288, buffers=22663168, cached=17499873280, shared=1218416640, slab=778809344) 
2022-04-06 22:50:18.878628: 
epoch:  237 
2022-04-06 22:52:08.479294: train loss : -0.6175 
2022-04-06 22:52:20.527848: validation loss: -0.6277 
2022-04-06 22:52:20.536064: Average global foreground Dice: [0.8538, 0.956, 0.9757, 0.3919] 
2022-04-06 22:52:20.543294: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:52:21.597877: lr: 0.00559 
2022-04-06 22:52:21.600829: saving scheduled checkpoint file... 
2022-04-06 22:52:21.766975: saving checkpoint... 
2022-04-06 22:52:26.359783: done, saving took 4.76 seconds 
2022-04-06 22:52:26.386577: done 
2022-04-06 22:52:26.488677: saving checkpoint... 
2022-04-06 22:52:30.942414: done, saving took 4.55 seconds 
2022-04-06 22:52:30.958832: This epoch took 132.073980 s
 
2022-04-06 22:52:30.959991: svmem(total=99932160000, available=84059525120, percent=15.9, used=12843556864, free=69624242176, active=23753224192, inactive=4313849856, buffers=22900736, cached=17441460224, shared=1218330624, slab=777854976) 
2022-04-06 22:52:30.960579: 
epoch:  238 
2022-04-06 22:54:19.315198: train loss : -0.6221 
2022-04-06 22:54:30.736391: validation loss: -0.6350 
2022-04-06 22:54:30.746372: Average global foreground Dice: [0.85, 0.9626, 0.9776, 0.421] 
2022-04-06 22:54:30.747173: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:54:31.664554: lr: 0.005571 
2022-04-06 22:54:31.677611: saving scheduled checkpoint file... 
2022-04-06 22:54:31.923902: saving checkpoint... 
2022-04-06 22:54:36.543000: done, saving took 4.85 seconds 
2022-04-06 22:54:36.580388: done 
2022-04-06 22:54:36.687045: saving checkpoint... 
2022-04-06 22:54:41.121141: done, saving took 4.53 seconds 
2022-04-06 22:54:41.135245: This epoch took 130.174110 s
 
2022-04-06 22:54:41.136481: svmem(total=99932160000, available=84056305664, percent=15.9, used=12846927872, free=69585752064, active=23719530496, inactive=4378087424, buffers=23166976, cached=17476313088, shared=1218097152, slab=779988992) 
2022-04-06 22:54:41.137089: 
epoch:  239 
2022-04-06 22:56:30.987402: train loss : -0.6184 
2022-04-06 22:56:41.939470: validation loss: -0.6033 
2022-04-06 22:56:41.950538: Average global foreground Dice: [0.8505, 0.9478, 0.979, 0.3875] 
2022-04-06 22:56:41.960650: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:56:43.091764: lr: 0.005551 
2022-04-06 22:56:43.094764: saving scheduled checkpoint file... 
2022-04-06 22:56:43.197528: saving checkpoint... 
2022-04-06 22:56:47.559819: done, saving took 4.46 seconds 
2022-04-06 22:56:47.589984: done 
2022-04-06 22:56:47.643456: saving checkpoint... 
2022-04-06 22:56:51.943511: done, saving took 4.35 seconds 
2022-04-06 22:56:51.957352: This epoch took 130.819614 s
 
2022-04-06 22:56:51.958301: svmem(total=99932160000, available=83948716032, percent=16.0, used=12954128384, free=69481172992, active=23713964032, inactive=4492992512, buffers=23433216, cached=17473425408, shared=1218527232, slab=777900032) 
2022-04-06 22:56:51.959016: 
epoch:  240 
2022-04-06 22:58:38.879300: train loss : -0.6146 
2022-04-06 22:58:50.079384: validation loss: -0.6108 
2022-04-06 22:58:50.092733: Average global foreground Dice: [0.8383, 0.9601, 0.9808, 0.4003] 
2022-04-06 22:58:50.103909: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 22:58:50.965523: lr: 0.005532 
2022-04-06 22:58:50.967484: saving scheduled checkpoint file... 
2022-04-06 22:58:51.058374: saving checkpoint... 
2022-04-06 22:58:55.402485: done, saving took 4.43 seconds 
2022-04-06 22:58:55.440747: done 
2022-04-06 22:58:55.503453: saving checkpoint... 
2022-04-06 22:58:59.817432: done, saving took 4.36 seconds 
2022-04-06 22:58:59.830965: This epoch took 127.871332 s
 
2022-04-06 22:58:59.831827: svmem(total=99932160000, available=83918786560, percent=16.0, used=12974903296, free=69474070528, active=23589539840, inactive=4622659584, buffers=23674880, cached=17459511296, shared=1227685888, slab=777330688) 
2022-04-06 22:58:59.832354: 
epoch:  241 
2022-04-06 23:00:47.295606: train loss : -0.6285 
2022-04-06 23:00:59.167087: validation loss: -0.6376 
2022-04-06 23:00:59.182782: Average global foreground Dice: [0.8495, 0.958, 0.9804, 0.4456] 
2022-04-06 23:00:59.185937: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:01:00.001162: lr: 0.005513 
2022-04-06 23:01:00.006309: saving scheduled checkpoint file... 
2022-04-06 23:01:00.113336: saving checkpoint... 
2022-04-06 23:01:04.558324: done, saving took 4.55 seconds 
2022-04-06 23:01:04.585551: done 
2022-04-06 23:01:04.643070: saving checkpoint... 
2022-04-06 23:01:09.062532: done, saving took 4.47 seconds 
2022-04-06 23:01:09.073954: This epoch took 129.240798 s
 
2022-04-06 23:01:09.075260: svmem(total=99932160000, available=84020174848, percent=15.9, used=12880871424, free=69490065408, active=23621541888, inactive=4575506432, buffers=23953408, cached=17537269760, shared=1220435968, slab=777199616) 
2022-04-06 23:01:09.076030: 
epoch:  242 
2022-04-06 23:02:57.717439: train loss : -0.6412 
2022-04-06 23:03:10.223900: validation loss: -0.5975 
2022-04-06 23:03:10.225509: Average global foreground Dice: [0.8663, 0.9532, 0.9786, 0.4081] 
2022-04-06 23:03:10.226494: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:03:11.036781: lr: 0.005494 
2022-04-06 23:03:11.056379: saving scheduled checkpoint file... 
2022-04-06 23:03:11.153583: saving checkpoint... 
2022-04-06 23:03:15.598084: done, saving took 4.53 seconds 
2022-04-06 23:03:15.615206: done 
2022-04-06 23:03:15.667249: saving checkpoint... 
2022-04-06 23:03:20.114894: done, saving took 4.50 seconds 
2022-04-06 23:03:20.131783: This epoch took 131.055114 s
 
2022-04-06 23:03:20.132910: svmem(total=99932160000, available=83925254144, percent=16.0, used=12975927296, free=69478854656, active=23736836096, inactive=4475281408, buffers=24195072, cached=17453182976, shared=1219862528, slab=776355840) 
2022-04-06 23:03:20.133713: 
epoch:  243 
2022-04-06 23:05:08.997094: train loss : -0.6415 
2022-04-06 23:05:20.470468: validation loss: -0.6229 
2022-04-06 23:05:20.480422: Average global foreground Dice: [0.8433, 0.9578, 0.9797, 0.3912] 
2022-04-06 23:05:20.485641: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:05:21.328162: lr: 0.005474 
2022-04-06 23:05:21.345371: saving scheduled checkpoint file... 
2022-04-06 23:05:21.461081: saving checkpoint... 
2022-04-06 23:05:26.113844: done, saving took 4.75 seconds 
2022-04-06 23:05:26.132767: done 
2022-04-06 23:05:26.135092: This epoch took 126.000733 s
 
2022-04-06 23:05:26.137236: svmem(total=99932160000, available=83963465728, percent=16.0, used=12940038144, free=69554212864, active=23777910784, inactive=4356325376, buffers=24461312, cached=17413447680, shared=1218060288, slab=777338880) 
2022-04-06 23:05:26.139082: 
epoch:  244 
2022-04-06 23:07:15.456275: train loss : -0.6339 
2022-04-06 23:07:27.479635: validation loss: -0.6485 
2022-04-06 23:07:27.481314: Average global foreground Dice: [0.85, 0.9567, 0.982, 0.4511] 
2022-04-06 23:07:27.482082: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:07:28.220798: lr: 0.005455 
2022-04-06 23:07:28.225898: saving scheduled checkpoint file... 
2022-04-06 23:07:28.353897: saving checkpoint... 
2022-04-06 23:07:32.849379: done, saving took 4.62 seconds 
2022-04-06 23:07:32.875967: done 
2022-04-06 23:07:32.988183: saving checkpoint... 
2022-04-06 23:07:37.505288: done, saving took 4.62 seconds 
2022-04-06 23:07:37.523674: This epoch took 131.382270 s
 
2022-04-06 23:07:37.524652: svmem(total=99932160000, available=84058923008, percent=15.9, used=12843966464, free=69590368256, active=23673389056, inactive=4425764864, buffers=24711168, cached=17473114112, shared=1218469888, slab=777322496) 
2022-04-06 23:07:37.525533: 
epoch:  245 
2022-04-06 23:09:27.579606: train loss : -0.6395 
2022-04-06 23:09:40.188723: validation loss: -0.6327 
2022-04-06 23:09:40.201389: Average global foreground Dice: [0.8496, 0.9591, 0.9791, 0.4108] 
2022-04-06 23:09:40.207568: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:09:41.301512: lr: 0.005436 
2022-04-06 23:09:41.310622: saving scheduled checkpoint file... 
2022-04-06 23:09:41.421005: saving checkpoint... 
2022-04-06 23:09:45.826035: done, saving took 4.50 seconds 
2022-04-06 23:09:45.835818: done 
2022-04-06 23:09:45.882755: saving checkpoint... 
2022-04-06 23:09:50.161880: done, saving took 4.33 seconds 
2022-04-06 23:09:50.175657: This epoch took 132.649380 s
 
2022-04-06 23:09:50.176474: svmem(total=99932160000, available=83978153984, percent=16.0, used=12924899328, free=69472866304, active=23823867904, inactive=4390404096, buffers=24973312, cached=17509421056, shared=1218297856, slab=777670656) 
2022-04-06 23:09:50.177057: 
epoch:  246 
2022-04-06 23:11:38.630697: train loss : -0.6161 
2022-04-06 23:11:50.290556: validation loss: -0.5865 
2022-04-06 23:11:50.306886: Average global foreground Dice: [0.818, 0.9607, 0.9644, 0.418] 
2022-04-06 23:11:50.324575: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:11:50.899523: lr: 0.005417 
2022-04-06 23:11:50.919370: saving scheduled checkpoint file... 
2022-04-06 23:11:51.026564: saving checkpoint... 
2022-04-06 23:11:55.396736: done, saving took 4.46 seconds 
2022-04-06 23:11:55.437401: done 
2022-04-06 23:11:55.448220: This epoch took 125.270559 s
 
2022-04-06 23:11:55.460820: svmem(total=99932160000, available=84076220416, percent=15.9, used=12924575744, free=69473120256, active=23971315712, inactive=4245049344, buffers=25206784, cached=17509257216, shared=1120489472, slab=777322496) 
2022-04-06 23:11:55.475575: 
epoch:  247 
2022-04-06 23:13:43.702713: train loss : -0.6151 
2022-04-06 23:13:55.781235: validation loss: -0.6386 
2022-04-06 23:13:55.791079: Average global foreground Dice: [0.8661, 0.9527, 0.9819, 0.4017] 
2022-04-06 23:13:55.793614: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:13:56.372992: lr: 0.005397 
2022-04-06 23:13:56.380477: saving scheduled checkpoint file... 
2022-04-06 23:13:56.433522: saving checkpoint... 
2022-04-06 23:14:00.907478: done, saving took 4.52 seconds 
2022-04-06 23:14:00.926665: done 
2022-04-06 23:14:00.978260: saving checkpoint... 
2022-04-06 23:14:05.378535: done, saving took 4.45 seconds 
2022-04-06 23:14:05.604683: This epoch took 130.112279 s
 
2022-04-06 23:14:05.605795: svmem(total=99932160000, available=84003557376, percent=15.9, used=12899356672, free=69470568448, active=23663128576, inactive=4554698752, buffers=25456640, cached=17536778240, shared=1218383872, slab=777351168) 
2022-04-06 23:14:05.606557: 
epoch:  248 
2022-04-06 23:15:54.926764: train loss : -0.6157 
2022-04-06 23:16:06.566761: validation loss: -0.6296 
2022-04-06 23:16:06.569188: Average global foreground Dice: [0.8383, 0.9572, 0.9766, 0.4142] 
2022-04-06 23:16:06.571121: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:16:07.596110: lr: 0.005378 
2022-04-06 23:16:07.610523: saving scheduled checkpoint file... 
2022-04-06 23:16:07.730203: saving checkpoint... 
2022-04-06 23:16:12.184311: done, saving took 4.56 seconds 
2022-04-06 23:16:12.204187: done 
2022-04-06 23:16:12.257013: saving checkpoint... 
2022-04-06 23:16:16.674595: done, saving took 4.47 seconds 
2022-04-06 23:16:16.688386: This epoch took 131.081048 s
 
2022-04-06 23:16:16.689335: svmem(total=99932160000, available=83865702400, percent=16.1, used=13037527040, free=69463662592, active=23739498496, inactive=4480311296, buffers=25735168, cached=17405235200, shared=1218060288, slab=778690560) 
2022-04-06 23:16:16.689972: 
epoch:  249 
2022-04-06 23:18:05.126795: train loss : -0.5884 
2022-04-06 23:18:16.110112: validation loss: -0.5237 
2022-04-06 23:18:16.111594: Average global foreground Dice: [0.7798, 0.938, 0.9627, 0.3927] 
2022-04-06 23:18:16.112445: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:18:16.969742: lr: 0.005359 
2022-04-06 23:18:16.982070: saving scheduled checkpoint file... 
2022-04-06 23:18:17.051258: saving checkpoint... 
2022-04-06 23:18:21.229558: done, saving took 4.24 seconds 
2022-04-06 23:18:21.263585: done 
2022-04-06 23:18:21.268316: This epoch took 124.577709 s
 
2022-04-06 23:18:21.280958: svmem(total=99932160000, available=83915997184, percent=16.0, used=12987039744, free=69465997312, active=23707308032, inactive=4511105024, buffers=25993216, cached=17453129728, shared=1218125824, slab=777981952) 
2022-04-06 23:18:21.292374: 
epoch:  250 
2022-04-06 23:20:09.196836: train loss : -0.5571 
2022-04-06 23:20:19.982524: validation loss: -0.6080 
2022-04-06 23:20:19.984105: Average global foreground Dice: [0.8354, 0.9566, 0.9756, 0.4133] 
2022-04-06 23:20:19.984910: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:20:21.240820: lr: 0.00534 
2022-04-06 23:20:21.243896: saving scheduled checkpoint file... 
2022-04-06 23:20:21.321554: saving checkpoint... 
2022-04-06 23:20:25.838915: done, saving took 4.59 seconds 
2022-04-06 23:20:25.857128: done 
2022-04-06 23:20:25.857928: This epoch took 124.553800 s
 
2022-04-06 23:20:25.859029: svmem(total=99932160000, available=84013457408, percent=15.9, used=12889456640, free=69526478848, active=23739420672, inactive=4422840320, buffers=26234880, cached=17489989632, shared=1218383872, slab=777756672) 
2022-04-06 23:20:25.859700: 
epoch:  251 
2022-04-06 23:22:14.632032: train loss : -0.6080 
2022-04-06 23:22:25.843525: validation loss: -0.6319 
2022-04-06 23:22:25.848929: Average global foreground Dice: [0.8674, 0.9552, 0.9784, 0.4038] 
2022-04-06 23:22:25.855687: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:22:26.865016: lr: 0.00532 
2022-04-06 23:22:26.873411: saving scheduled checkpoint file... 
2022-04-06 23:22:26.987392: saving checkpoint... 
2022-04-06 23:22:31.770735: done, saving took 4.89 seconds 
2022-04-06 23:22:31.804450: done 
2022-04-06 23:22:31.809106: This epoch took 125.948805 s
 
2022-04-06 23:22:31.816087: svmem(total=99932160000, available=84067086336, percent=15.9, used=12834701312, free=69646323712, active=23562448896, inactive=4478877696, buffers=26476544, cached=17424658432, shared=1218383872, slab=778674176) 
2022-04-06 23:22:31.820962: 
epoch:  252 
2022-04-06 23:24:21.225492: train loss : -0.6028 
2022-04-06 23:24:32.098057: validation loss: -0.6223 
2022-04-06 23:24:32.111631: Average global foreground Dice: [0.8535, 0.9585, 0.98, 0.3931] 
2022-04-06 23:24:32.120576: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:24:32.711590: lr: 0.005301 
2022-04-06 23:24:32.720027: saving scheduled checkpoint file... 
2022-04-06 23:24:32.783985: saving checkpoint... 
2022-04-06 23:24:37.389966: done, saving took 4.66 seconds 
2022-04-06 23:24:37.410232: done 
2022-04-06 23:24:37.415397: This epoch took 125.588271 s
 
2022-04-06 23:24:37.420085: svmem(total=99932160000, available=84092747776, percent=15.9, used=12810620928, free=69461315584, active=23554510848, inactive=4669640704, buffers=26730496, cached=17633492992, shared=1218469888, slab=778559488) 
2022-04-06 23:24:37.426548: 
epoch:  253 
2022-04-06 23:26:26.835061: train loss : -0.6189 
2022-04-06 23:26:38.150209: validation loss: -0.6307 
2022-04-06 23:26:38.156648: Average global foreground Dice: [0.8503, 0.9577, 0.9798, 0.4258] 
2022-04-06 23:26:38.159050: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:26:38.984880: lr: 0.005282 
2022-04-06 23:26:38.993460: saving scheduled checkpoint file... 
2022-04-06 23:26:39.096749: saving checkpoint... 
2022-04-06 23:26:42.981393: done, saving took 3.98 seconds 
2022-04-06 23:26:43.006376: done 
2022-04-06 23:26:43.012587: This epoch took 125.583245 s
 
2022-04-06 23:26:43.019168: svmem(total=99932160000, available=84016369664, percent=15.9, used=12885807104, free=69462200320, active=23839338496, inactive=4384600064, buffers=26996736, cached=17557155840, shared=1218641920, slab=778584064) 
2022-04-06 23:26:43.024945: 
epoch:  254 
2022-04-06 23:28:32.142288: train loss : -0.6272 
2022-04-06 23:28:43.254295: validation loss: -0.5990 
2022-04-06 23:28:43.256183: Average global foreground Dice: [0.8506, 0.9583, 0.9709, 0.3665] 
2022-04-06 23:28:43.258029: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:28:43.953732: lr: 0.005262 
2022-04-06 23:28:43.957992: saving scheduled checkpoint file... 
2022-04-06 23:28:44.023459: saving checkpoint... 
2022-04-06 23:28:48.826726: done, saving took 4.86 seconds 
2022-04-06 23:28:48.857689: done 
2022-04-06 23:28:48.863908: This epoch took 125.832920 s
 
2022-04-06 23:28:48.874408: svmem(total=99932160000, available=84101459968, percent=15.8, used=12801347584, free=69531934720, active=23796248576, inactive=4360384512, buffers=27238400, cached=17571639296, shared=1218383872, slab=777887744) 
2022-04-06 23:28:48.885167: 
epoch:  255 
2022-04-06 23:30:38.142077: train loss : -0.5820 
2022-04-06 23:30:50.285821: validation loss: -0.5908 
2022-04-06 23:30:50.300765: Average global foreground Dice: [0.7914, 0.9487, 0.9782, 0.4133] 
2022-04-06 23:30:50.301835: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:30:51.318565: lr: 0.005243 
2022-04-06 23:30:51.319506: saving scheduled checkpoint file... 
2022-04-06 23:30:51.441228: saving checkpoint... 
2022-04-06 23:30:55.889476: done, saving took 4.57 seconds 
2022-04-06 23:30:55.916690: done 
2022-04-06 23:30:55.920092: This epoch took 127.027658 s
 
2022-04-06 23:30:55.924935: svmem(total=99932160000, available=84110446592, percent=15.8, used=12792786944, free=69566574592, active=23731564544, inactive=4391342080, buffers=27508736, cached=17545289728, shared=1218297856, slab=776966144) 
2022-04-06 23:30:55.928473: 
epoch:  256 
2022-04-06 23:32:45.412801: train loss : -0.6149 
2022-04-06 23:32:56.956840: validation loss: -0.6295 
2022-04-06 23:32:56.963013: Average global foreground Dice: [0.8596, 0.9627, 0.9764, 0.395] 
2022-04-06 23:32:56.970088: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:32:57.637851: lr: 0.005224 
2022-04-06 23:32:57.639811: saving scheduled checkpoint file... 
2022-04-06 23:32:57.786617: saving checkpoint... 
2022-04-06 23:33:01.864510: done, saving took 4.22 seconds 
2022-04-06 23:33:01.893073: done 
2022-04-06 23:33:01.899269: This epoch took 125.968626 s
 
2022-04-06 23:33:01.908711: svmem(total=99932160000, available=84048789504, percent=15.9, used=12851601408, free=69556887552, active=23763587072, inactive=4365705216, buffers=27742208, cached=17495928832, shared=1220366336, slab=778395648) 
2022-04-06 23:33:01.917342: 
epoch:  257 
2022-04-06 23:34:51.044699: train loss : -0.6262 
2022-04-06 23:35:02.785123: validation loss: -0.6184 
2022-04-06 23:35:02.799495: Average global foreground Dice: [0.8515, 0.9571, 0.9795, 0.4281] 
2022-04-06 23:35:02.811013: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:35:03.664695: lr: 0.005204 
2022-04-06 23:35:03.677647: saving scheduled checkpoint file... 
2022-04-06 23:35:03.792332: saving checkpoint... 
2022-04-06 23:35:08.896825: done, saving took 5.21 seconds 
2022-04-06 23:35:08.922866: done 
2022-04-06 23:35:08.927489: This epoch took 127.002392 s
 
2022-04-06 23:35:08.931378: svmem(total=99932160000, available=84073754624, percent=15.9, used=12853252096, free=69605801984, active=23436476416, inactive=4647702528, buffers=28016640, cached=17445089280, shared=1193910272, slab=776781824) 
2022-04-06 23:35:08.934374: 
epoch:  258 
2022-04-06 23:36:57.739402: train loss : -0.6361 
2022-04-06 23:37:09.945091: validation loss: -0.6348 
2022-04-06 23:37:09.950318: Average global foreground Dice: [0.8707, 0.9626, 0.9794, 0.4193] 
2022-04-06 23:37:09.955867: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:37:10.765788: lr: 0.005185 
2022-04-06 23:37:10.775376: saving scheduled checkpoint file... 
2022-04-06 23:37:10.866067: saving checkpoint... 
2022-04-06 23:37:15.317598: done, saving took 4.53 seconds 
2022-04-06 23:37:15.335278: done 
2022-04-06 23:37:15.388316: saving checkpoint... 
2022-04-06 23:37:19.837632: done, saving took 4.50 seconds 
2022-04-06 23:37:19.850642: This epoch took 130.914262 s
 
2022-04-06 23:37:19.851598: svmem(total=99932160000, available=83972902912, percent=16.0, used=12908683264, free=69454737408, active=23840276480, inactive=4390424576, buffers=28262400, cached=17540476928, shared=1239609344, slab=778260480) 
2022-04-06 23:37:19.852318: 
epoch:  259 
2022-04-06 23:39:08.099391: train loss : -0.6338 
2022-04-06 23:39:19.754772: validation loss: -0.6238 
2022-04-06 23:39:19.771684: Average global foreground Dice: [0.8293, 0.9581, 0.9829, 0.4364] 
2022-04-06 23:39:19.778378: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:39:20.649273: lr: 0.005166 
2022-04-06 23:39:20.663366: saving scheduled checkpoint file... 
2022-04-06 23:39:20.726455: saving checkpoint... 
2022-04-06 23:39:25.241816: done, saving took 4.57 seconds 
2022-04-06 23:39:25.286294: done 
2022-04-06 23:39:25.341185: saving checkpoint... 
2022-04-06 23:39:29.830869: done, saving took 4.54 seconds 
2022-04-06 23:39:29.844580: This epoch took 129.991470 s
 
2022-04-06 23:39:29.845617: svmem(total=99932160000, available=83983777792, percent=16.0, used=12919095296, free=69452386304, active=23826309120, inactive=4403908608, buffers=28504064, cached=17532174336, shared=1218297856, slab=778973184) 
2022-04-06 23:39:29.846210: 
epoch:  260 
2022-04-06 23:41:27.847540: train loss : -0.6313 
2022-04-06 23:41:39.462069: validation loss: -0.6295 
2022-04-06 23:41:39.463795: Average global foreground Dice: [0.8662, 0.9533, 0.9819, 0.3723] 
2022-04-06 23:41:39.465523: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:41:40.253761: lr: 0.005146 
2022-04-06 23:41:40.255361: saving scheduled checkpoint file... 
2022-04-06 23:41:40.382721: saving checkpoint... 
2022-04-06 23:41:44.880273: done, saving took 4.62 seconds 
2022-04-06 23:41:44.914392: done 
2022-04-06 23:41:44.920598: This epoch took 135.073720 s
 
2022-04-06 23:41:44.926241: svmem(total=99932160000, available=84056735744, percent=15.9, used=12845178880, free=69524893696, active=23741362176, inactive=4420759552, buffers=28778496, cached=17533308928, shared=1218555904, slab=780517376) 
2022-04-06 23:41:44.932488: 
epoch:  261 
2022-04-06 23:43:34.432505: train loss : -0.6370 
2022-04-06 23:43:46.479978: validation loss: -0.6314 
2022-04-06 23:43:46.495326: Average global foreground Dice: [0.8525, 0.9568, 0.9805, 0.4101] 
2022-04-06 23:43:46.514622: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:43:47.228041: lr: 0.005127 
2022-04-06 23:43:47.232370: saving scheduled checkpoint file... 
2022-04-06 23:43:47.381405: saving checkpoint... 
2022-04-06 23:43:52.002952: done, saving took 4.77 seconds 
2022-04-06 23:43:52.021827: done 
2022-04-06 23:43:52.117988: saving checkpoint... 
2022-04-06 23:43:56.540806: done, saving took 4.52 seconds 
2022-04-06 23:43:56.559481: This epoch took 131.620327 s
 
2022-04-06 23:43:56.560637: svmem(total=99932160000, available=84003504128, percent=15.9, used=12899119104, free=69558456320, active=23752884224, inactive=4375052288, buffers=29028352, cached=17445556224, shared=1218535424, slab=779587584) 
2022-04-06 23:43:56.561399: 
epoch:  262 
2022-04-06 23:45:45.944498: train loss : -0.6401 
2022-04-06 23:45:57.470279: validation loss: -0.6323 
2022-04-06 23:45:57.480484: Average global foreground Dice: [0.8792, 0.9616, 0.9827, 0.4143] 
2022-04-06 23:45:57.486201: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:45:58.261364: lr: 0.005107 
2022-04-06 23:45:58.262234: saving scheduled checkpoint file... 
2022-04-06 23:45:58.367028: saving checkpoint... 
2022-04-06 23:46:02.802272: done, saving took 4.54 seconds 
2022-04-06 23:46:02.837342: done 
2022-04-06 23:46:02.964692: saving checkpoint... 
2022-04-06 23:46:07.439193: done, saving took 4.59 seconds 
2022-04-06 23:46:07.457127: This epoch took 130.895122 s
 
2022-04-06 23:46:07.457963: svmem(total=99932160000, available=84018495488, percent=15.9, used=12884246528, free=69555027968, active=23716884480, inactive=4415102976, buffers=29298688, cached=17463586816, shared=1218387968, slab=778612736) 
2022-04-06 23:46:07.458480: 
epoch:  263 
2022-04-06 23:47:57.346677: train loss : -0.6216 
2022-04-06 23:48:08.765087: validation loss: -0.6352 
2022-04-06 23:48:08.766644: Average global foreground Dice: [0.8751, 0.961, 0.9776, 0.4092] 
2022-04-06 23:48:08.767449: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:48:09.808288: lr: 0.005088 
2022-04-06 23:48:09.810894: saving scheduled checkpoint file... 
2022-04-06 23:48:09.906364: saving checkpoint... 
2022-04-06 23:48:14.378404: done, saving took 4.57 seconds 
2022-04-06 23:48:14.405980: done 
2022-04-06 23:48:14.467273: saving checkpoint... 
2022-04-06 23:48:18.797518: done, saving took 4.38 seconds 
2022-04-06 23:48:18.811476: This epoch took 131.352490 s
 
2022-04-06 23:48:18.812612: svmem(total=99932160000, available=83948838912, percent=16.0, used=12953796608, free=69446225920, active=23723999232, inactive=4514156544, buffers=29540352, cached=17502597120, shared=1218473984, slab=777834496) 
2022-04-06 23:48:18.813260: 
epoch:  264 
2022-04-06 23:50:06.189755: train loss : -0.6199 
2022-04-06 23:50:17.713907: validation loss: -0.6286 
2022-04-06 23:50:17.718589: Average global foreground Dice: [0.8742, 0.9598, 0.9733, 0.4076] 
2022-04-06 23:50:17.723862: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:50:18.686625: lr: 0.005069 
2022-04-06 23:50:18.693717: saving scheduled checkpoint file... 
2022-04-06 23:50:18.838496: saving checkpoint... 
2022-04-06 23:50:23.307450: done, saving took 4.60 seconds 
2022-04-06 23:50:23.325438: done 
2022-04-06 23:50:23.381579: saving checkpoint... 
2022-04-06 23:50:27.698992: done, saving took 4.37 seconds 
2022-04-06 23:50:27.707999: This epoch took 128.894059 s
 
2022-04-06 23:50:27.709012: svmem(total=99932160000, available=83866697728, percent=16.1, used=13035962368, free=69448511488, active=23910535168, inactive=4330180608, buffers=29798400, cached=17417887744, shared=1218560000, slab=778842112) 
2022-04-06 23:50:27.709721: 
epoch:  265 
2022-04-06 23:52:16.915125: train loss : -0.6318 
2022-04-06 23:52:28.985752: validation loss: -0.6349 
2022-04-06 23:52:29.002643: Average global foreground Dice: [0.8524, 0.9639, 0.9775, 0.416] 
2022-04-06 23:52:29.003611: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:52:29.773733: lr: 0.005049 
2022-04-06 23:52:29.788467: saving scheduled checkpoint file... 
2022-04-06 23:52:29.880465: saving checkpoint... 
2022-04-06 23:52:34.333312: done, saving took 4.53 seconds 
2022-04-06 23:52:34.347752: done 
2022-04-06 23:52:34.399251: saving checkpoint... 
2022-04-06 23:52:38.803488: done, saving took 4.45 seconds 
2022-04-06 23:52:38.816859: This epoch took 131.106513 s
 
2022-04-06 23:52:38.817847: svmem(total=99932160000, available=83887497216, percent=16.1, used=13014622208, free=69449777152, active=23859535872, inactive=4378824704, buffers=30056448, cached=17437704192, shared=1219055616, slab=777719808) 
2022-04-06 23:52:38.818510: 
epoch:  266 
2022-04-06 23:54:28.254141: train loss : -0.6328 
2022-04-06 23:54:40.623110: validation loss: -0.6320 
2022-04-06 23:54:40.627597: Average global foreground Dice: [0.8542, 0.9529, 0.971, 0.3914] 
2022-04-06 23:54:40.629156: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:54:41.381494: lr: 0.00503 
2022-04-06 23:54:41.384762: saving scheduled checkpoint file... 
2022-04-06 23:54:41.473824: saving checkpoint... 
2022-04-06 23:54:45.927234: done, saving took 4.54 seconds 
2022-04-06 23:54:45.955262: done 
2022-04-06 23:54:45.964369: This epoch took 127.145232 s
 
2022-04-06 23:54:45.971950: svmem(total=99932160000, available=83985874944, percent=16.0, used=12917092352, free=69448933376, active=23825211392, inactive=4415893504, buffers=30294016, cached=17535840256, shared=1218306048, slab=776843264) 
2022-04-06 23:54:45.977822: 
epoch:  267 
2022-04-06 23:56:35.542334: train loss : -0.5783 
2022-04-06 23:56:47.602645: validation loss: -0.5524 
2022-04-06 23:56:47.610198: Average global foreground Dice: [0.8348, 0.9412, 0.9633, 0.4115] 
2022-04-06 23:56:47.611145: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:56:48.407370: lr: 0.00501 
2022-04-06 23:56:48.417458: saving scheduled checkpoint file... 
2022-04-06 23:56:48.491277: saving checkpoint... 
2022-04-06 23:56:52.946035: done, saving took 4.51 seconds 
2022-04-06 23:56:52.964279: done 
2022-04-06 23:56:52.966131: This epoch took 126.981011 s
 
2022-04-06 23:56:52.968466: svmem(total=99932160000, available=84015988736, percent=15.9, used=12886142976, free=69516976128, active=23776010240, inactive=4393746432, buffers=30560256, cached=17498480640, shared=1218334720, slab=777687040) 
2022-04-06 23:56:52.970876: 
epoch:  268 
2022-04-06 23:58:41.962935: train loss : -0.6152 
2022-04-06 23:58:53.542070: validation loss: -0.6457 
2022-04-06 23:58:53.548494: Average global foreground Dice: [0.8669, 0.9545, 0.9784, 0.4311] 
2022-04-06 23:58:53.550158: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-06 23:58:54.410383: lr: 0.004991 
2022-04-06 23:58:54.411245: saving scheduled checkpoint file... 
2022-04-06 23:58:54.553108: saving checkpoint... 
2022-04-06 23:58:59.025610: done, saving took 4.61 seconds 
2022-04-06 23:58:59.072157: done 
2022-04-06 23:58:59.082318: This epoch took 126.109588 s
 
2022-04-06 23:58:59.093220: svmem(total=99932160000, available=84111613952, percent=15.8, used=12790267904, free=69536460800, active=23734231040, inactive=4417474560, buffers=30801920, cached=17574629376, shared=1218355200, slab=779517952) 
2022-04-06 23:58:59.102950: 
epoch:  269 
2022-04-07 00:00:48.670954: train loss : -0.6256 
2022-04-07 00:01:00.673045: validation loss: -0.6232 
2022-04-07 00:01:00.685390: Average global foreground Dice: [0.8666, 0.9572, 0.978, 0.4261] 
2022-04-07 00:01:00.686192: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:01:01.516653: lr: 0.004971 
2022-04-07 00:01:01.533443: saving scheduled checkpoint file... 
2022-04-07 00:01:01.763675: saving checkpoint... 
2022-04-07 00:01:06.333977: done, saving took 4.77 seconds 
2022-04-07 00:01:06.380862: done 
2022-04-07 00:01:06.456457: saving checkpoint... 
2022-04-07 00:01:10.764941: done, saving took 4.37 seconds 
2022-04-07 00:01:10.782355: This epoch took 131.669972 s
 
2022-04-07 00:01:10.783197: svmem(total=99932160000, available=84004302848, percent=15.9, used=12887474176, free=69600964608, active=23671095296, inactive=4405067776, buffers=38117376, cached=17405603840, shared=1223397376, slab=790949888) 
2022-04-07 00:01:10.783804: 
epoch:  270 
2022-04-07 00:02:59.016800: train loss : -0.6462 
2022-04-07 00:03:10.759201: validation loss: -0.6599 
2022-04-07 00:03:10.760845: Average global foreground Dice: [0.8664, 0.9622, 0.9799, 0.4391] 
2022-04-07 00:03:10.761648: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:03:11.644367: lr: 0.004952 
2022-04-07 00:03:11.650160: saving scheduled checkpoint file... 
2022-04-07 00:03:11.731366: saving checkpoint... 
2022-04-07 00:03:16.182982: done, saving took 4.53 seconds 
2022-04-07 00:03:16.209966: done 
2022-04-07 00:03:16.262342: saving checkpoint... 
2022-04-07 00:03:20.732966: done, saving took 4.52 seconds 
2022-04-07 00:03:20.743699: This epoch took 129.959239 s
 
2022-04-07 00:03:20.744644: svmem(total=99932160000, available=83944628224, percent=16.0, used=12951310336, free=69419405312, active=23653482496, inactive=4604145664, buffers=38367232, cached=17523077120, shared=1218359296, slab=789524480) 
2022-04-07 00:03:20.745305: 
epoch:  271 
2022-04-07 00:05:10.129612: train loss : -0.6555 
2022-04-07 00:05:21.737801: validation loss: -0.6308 
2022-04-07 00:05:21.751893: Average global foreground Dice: [0.8847, 0.9559, 0.9817, 0.3543] 
2022-04-07 00:05:21.752799: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:05:22.451095: lr: 0.004933 
2022-04-07 00:05:22.462341: saving scheduled checkpoint file... 
2022-04-07 00:05:22.569205: saving checkpoint... 
2022-04-07 00:05:27.168006: done, saving took 4.69 seconds 
2022-04-07 00:05:27.192353: done 
2022-04-07 00:05:27.202494: This epoch took 126.456561 s
 
2022-04-07 00:05:27.207881: svmem(total=99932160000, available=84017143808, percent=15.9, used=12878917632, free=69421019136, active=23905079296, inactive=4353458176, buffers=38641664, cached=17593581568, shared=1218318336, slab=791621632) 
2022-04-07 00:05:27.214922: 
epoch:  272 
2022-04-07 00:07:16.244975: train loss : -0.6435 
2022-04-07 00:07:27.958809: validation loss: -0.6548 
2022-04-07 00:07:27.970439: Average global foreground Dice: [0.8711, 0.9659, 0.9777, 0.4276] 
2022-04-07 00:07:27.971489: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:07:28.792970: lr: 0.004913 
2022-04-07 00:07:28.795095: saving scheduled checkpoint file... 
2022-04-07 00:07:28.859252: saving checkpoint... 
2022-04-07 00:07:33.359239: done, saving took 4.56 seconds 
2022-04-07 00:07:33.382471: done 
2022-04-07 00:07:33.463113: saving checkpoint... 
2022-04-07 00:07:37.883985: done, saving took 4.50 seconds 
2022-04-07 00:07:37.900700: This epoch took 130.679842 s
 
2022-04-07 00:07:37.901682: svmem(total=99932160000, available=83988865024, percent=16.0, used=12895465472, free=69479899136, active=23791386624, inactive=4403957760, buffers=38883328, cached=17517912064, shared=1230839808, slab=791384064) 
2022-04-07 00:07:37.902444: 
epoch:  273 
2022-04-07 00:09:27.889589: train loss : -0.6369 
2022-04-07 00:09:38.953415: validation loss: -0.6517 
2022-04-07 00:09:38.955023: Average global foreground Dice: [0.8575, 0.9613, 0.9832, 0.4176] 
2022-04-07 00:09:38.956020: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:09:39.721531: lr: 0.004894 
2022-04-07 00:09:39.727617: saving scheduled checkpoint file... 
2022-04-07 00:09:39.928859: saving checkpoint... 
2022-04-07 00:09:44.555338: done, saving took 4.82 seconds 
2022-04-07 00:09:44.579818: done 
2022-04-07 00:09:44.681094: saving checkpoint... 
2022-04-07 00:09:49.139078: done, saving took 4.56 seconds 
2022-04-07 00:09:49.160624: This epoch took 131.257386 s
 
2022-04-07 00:09:49.163094: svmem(total=99932160000, available=84042461184, percent=15.9, used=12854018048, free=69509230592, active=23745744896, inactive=4421328896, buffers=39145472, cached=17529765888, shared=1218691072, slab=790028288) 
2022-04-07 00:09:49.165088: 
epoch:  274 
2022-04-07 00:11:38.844166: train loss : -0.6467 
2022-04-07 00:11:51.337500: validation loss: -0.6402 
2022-04-07 00:11:51.345849: Average global foreground Dice: [0.8746, 0.952, 0.9767, 0.4154] 
2022-04-07 00:11:51.347706: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:11:51.912458: lr: 0.004874 
2022-04-07 00:11:51.914405: saving scheduled checkpoint file... 
2022-04-07 00:11:52.138521: saving checkpoint... 
2022-04-07 00:11:56.559842: done, saving took 4.64 seconds 
2022-04-07 00:11:56.585788: done 
2022-04-07 00:11:56.686076: saving checkpoint... 
2022-04-07 00:12:01.100142: done, saving took 4.51 seconds 
2022-04-07 00:12:01.117739: This epoch took 131.950511 s
 
2022-04-07 00:12:01.118675: svmem(total=99932160000, available=83997081600, percent=15.9, used=12899508224, free=69500141568, active=23683190784, inactive=4490878976, buffers=39395328, cached=17493114880, shared=1218457600, slab=792162304) 
2022-04-07 00:12:01.119250: 
epoch:  275 
2022-04-07 00:13:50.406027: train loss : -0.6043 
2022-04-07 00:14:02.186834: validation loss: -0.6178 
2022-04-07 00:14:02.193590: Average global foreground Dice: [0.8535, 0.9501, 0.9703, 0.408] 
2022-04-07 00:14:02.197429: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:14:03.273200: lr: 0.004855 
2022-04-07 00:14:03.285671: saving scheduled checkpoint file... 
2022-04-07 00:14:03.386404: saving checkpoint... 
2022-04-07 00:14:07.746875: done, saving took 4.45 seconds 
2022-04-07 00:14:07.775275: done 
2022-04-07 00:14:07.785965: This epoch took 126.666209 s
 
2022-04-07 00:14:07.795743: svmem(total=99932160000, available=83972825088, percent=16.0, used=12923740160, free=69403222016, active=23656415232, inactive=4608098304, buffers=39636992, cached=17565560832, shared=1218371584, slab=791924736) 
2022-04-07 00:14:07.804970: 
epoch:  276 
2022-04-07 00:15:57.717614: train loss : -0.6264 
2022-04-07 00:16:09.855254: validation loss: -0.6480 
2022-04-07 00:16:09.864188: Average global foreground Dice: [0.8814, 0.9613, 0.9804, 0.4368] 
2022-04-07 00:16:09.864952: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:16:10.785650: lr: 0.004835 
2022-04-07 00:16:10.796388: saving scheduled checkpoint file... 
2022-04-07 00:16:10.893576: saving checkpoint... 
2022-04-07 00:16:15.223380: done, saving took 4.42 seconds 
2022-04-07 00:16:15.263086: done 
2022-04-07 00:16:15.318033: saving checkpoint... 
2022-04-07 00:16:19.623666: done, saving took 4.35 seconds 
2022-04-07 00:16:19.637557: This epoch took 131.823027 s
 
2022-04-07 00:16:19.638471: svmem(total=99932160000, available=83885166592, percent=16.1, used=13003751424, free=69405323264, active=23889326080, inactive=4380012544, buffers=39903232, cached=17483182080, shared=1226063872, slab=791224320) 
2022-04-07 00:16:19.639171: 
epoch:  277 
2022-04-07 00:18:07.391464: train loss : -0.6242 
2022-04-07 00:18:18.085214: validation loss: -0.5925 
2022-04-07 00:18:18.092357: Average global foreground Dice: [0.853, 0.9575, 0.9674, 0.3988] 
2022-04-07 00:18:18.093189: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:18:19.122935: lr: 0.004816 
2022-04-07 00:18:19.125393: saving scheduled checkpoint file... 
2022-04-07 00:18:19.174826: saving checkpoint... 
2022-04-07 00:18:23.720369: done, saving took 4.59 seconds 
2022-04-07 00:18:23.758300: done 
2022-04-07 00:18:23.760929: This epoch took 124.121074 s
 
2022-04-07 00:18:23.762885: svmem(total=99932160000, available=84043575296, percent=15.9, used=12839989248, free=69394591744, active=23787286528, inactive=4494348288, buffers=40157184, cached=17657421824, shared=1232080896, slab=791228416) 
2022-04-07 00:18:23.764666: 
epoch:  278 
2022-04-07 00:20:14.991176: train loss : -0.6140 
2022-04-07 00:20:27.155932: validation loss: -0.6254 
2022-04-07 00:20:27.164739: Average global foreground Dice: [0.8606, 0.9619, 0.9777, 0.4111] 
2022-04-07 00:20:27.169688: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:20:27.974451: lr: 0.004796 
2022-04-07 00:20:27.986118: saving scheduled checkpoint file... 
2022-04-07 00:20:28.098338: saving checkpoint... 
2022-04-07 00:20:32.549498: done, saving took 4.55 seconds 
2022-04-07 00:20:32.595645: done 
2022-04-07 00:20:32.608158: This epoch took 128.842460 s
 
2022-04-07 00:20:32.624455: svmem(total=99932160000, available=83986350080, percent=16.0, used=12909645824, free=69401108480, active=23891132416, inactive=4378099712, buffers=40407040, cached=17580998656, shared=1218461696, slab=791687168) 
2022-04-07 00:20:32.638354: 
epoch:  279 
2022-04-07 00:22:20.971847: train loss : -0.6312 
2022-04-07 00:22:33.425413: validation loss: -0.6068 
2022-04-07 00:22:33.429666: Average global foreground Dice: [0.8486, 0.9612, 0.9796, 0.368] 
2022-04-07 00:22:33.430306: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:22:34.187593: lr: 0.004776 
2022-04-07 00:22:34.192782: saving scheduled checkpoint file... 
2022-04-07 00:22:34.256459: saving checkpoint... 
2022-04-07 00:22:38.680371: done, saving took 4.48 seconds 
2022-04-07 00:22:38.696762: done 
2022-04-07 00:22:38.697582: This epoch took 126.046359 s
 
2022-04-07 00:22:38.698408: svmem(total=99932160000, available=83972861952, percent=16.0, used=12923645952, free=69482246144, active=23626182656, inactive=4566573056, buffers=40669184, cached=17485598720, shared=1218719744, slab=791273472) 
2022-04-07 00:22:38.699137: 
epoch:  280 
2022-04-07 00:24:29.127717: train loss : -0.6350 
2022-04-07 00:24:41.692923: validation loss: -0.6613 
2022-04-07 00:24:41.697940: Average global foreground Dice: [0.8555, 0.9674, 0.9817, 0.4317] 
2022-04-07 00:24:41.698641: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:24:42.659431: lr: 0.004757 
2022-04-07 00:24:42.660303: saving scheduled checkpoint file... 
2022-04-07 00:24:42.814392: saving checkpoint... 
2022-04-07 00:24:47.258229: done, saving took 4.60 seconds 
2022-04-07 00:24:47.292725: done 
2022-04-07 00:24:47.297696: This epoch took 128.597905 s
 
2022-04-07 00:24:47.302572: svmem(total=99932160000, available=83916779520, percent=16.0, used=12979658752, free=69489946624, active=23719124992, inactive=4461723648, buffers=40906752, cached=17421647872, shared=1218461696, slab=790581248) 
2022-04-07 00:24:47.310652: 
epoch:  281 
2022-04-07 00:26:37.755382: train loss : -0.6336 
2022-04-07 00:26:49.097564: validation loss: -0.6343 
2022-04-07 00:26:49.115034: Average global foreground Dice: [0.8765, 0.9621, 0.9778, 0.4028] 
2022-04-07 00:26:49.123342: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:26:50.060876: lr: 0.004737 
2022-04-07 00:26:50.061954: saving scheduled checkpoint file... 
2022-04-07 00:26:50.209031: saving checkpoint... 
2022-04-07 00:26:55.458577: done, saving took 5.40 seconds 
2022-04-07 00:26:55.497116: done 
2022-04-07 00:26:55.503243: This epoch took 128.188211 s
 
2022-04-07 00:26:55.512625: svmem(total=99932160000, available=84103032832, percent=15.8, used=12793020416, free=69577400320, active=23606325248, inactive=4491100160, buffers=41172992, cached=17520566272, shared=1218547712, slab=791031808) 
2022-04-07 00:26:55.519486: 
epoch:  282 
2022-04-07 00:28:43.606204: train loss : -0.6355 
2022-04-07 00:28:54.607368: validation loss: -0.6018 
2022-04-07 00:28:54.619233: Average global foreground Dice: [0.8617, 0.9568, 0.9783, 0.3818] 
2022-04-07 00:28:54.625886: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:28:55.357293: lr: 0.004718 
2022-04-07 00:28:55.361443: saving scheduled checkpoint file... 
2022-04-07 00:28:55.427273: saving checkpoint... 
2022-04-07 00:29:00.311707: done, saving took 4.95 seconds 
2022-04-07 00:29:00.337333: done 
2022-04-07 00:29:00.342009: This epoch took 124.816971 s
 
2022-04-07 00:29:00.350744: svmem(total=99932160000, available=84098867200, percent=15.8, used=12797579264, free=69401346048, active=23827914752, inactive=4445134848, buffers=41406464, cached=17691828224, shared=1218547712, slab=791166976) 
2022-04-07 00:29:00.357147: 
epoch:  283 
2022-04-07 00:30:49.206143: train loss : -0.5311 
2022-04-07 00:31:00.092617: validation loss: -0.5189 
2022-04-07 00:31:00.097457: Average global foreground Dice: [0.7965, 0.9421, 0.9415, 0.4111] 
2022-04-07 00:31:00.099239: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:31:00.979032: lr: 0.004698 
2022-04-07 00:31:00.987216: saving scheduled checkpoint file... 
2022-04-07 00:31:01.110523: saving checkpoint... 
2022-04-07 00:31:05.772321: done, saving took 4.78 seconds 
2022-04-07 00:31:05.801366: done 
2022-04-07 00:31:05.807332: This epoch took 125.445759 s
 
2022-04-07 00:31:05.816149: svmem(total=99932160000, available=84047974400, percent=15.9, used=12834521088, free=69346516992, active=23716188160, inactive=4612386816, buffers=43462656, cached=17707659264, shared=1232338944, slab=791191552) 
2022-04-07 00:31:05.822459: 
epoch:  284 
2022-04-07 00:32:54.764982: train loss : -0.5319 
2022-04-07 00:33:06.622087: validation loss: -0.5370 
2022-04-07 00:33:06.627928: Average global foreground Dice: [0.8135, 0.9472, 0.962, 0.3829] 
2022-04-07 00:33:06.629449: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:33:07.479161: lr: 0.004679 
2022-04-07 00:33:07.488815: saving scheduled checkpoint file... 
2022-04-07 00:33:07.565645: saving checkpoint... 
2022-04-07 00:33:12.046496: done, saving took 4.55 seconds 
2022-04-07 00:33:12.079295: done 
2022-04-07 00:33:12.087748: This epoch took 126.257098 s
 
2022-04-07 00:33:12.093696: svmem(total=99932160000, available=84092600320, percent=15.9, used=12803149824, free=69429706752, active=23623532544, inactive=4619038720, buffers=43704320, cached=17655599104, shared=1218551808, slab=790212608) 
2022-04-07 00:33:12.099544: 
epoch:  285 
2022-04-07 00:35:00.523612: train loss : -0.5803 
2022-04-07 00:35:12.525306: validation loss: -0.5904 
2022-04-07 00:35:12.542687: Average global foreground Dice: [0.8441, 0.9538, 0.9769, 0.3853] 
2022-04-07 00:35:12.561085: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:35:13.165741: lr: 0.004659 
2022-04-07 00:35:13.174802: saving scheduled checkpoint file... 
2022-04-07 00:35:13.480738: saving checkpoint... 
2022-04-07 00:35:17.890570: done, saving took 4.71 seconds 
2022-04-07 00:35:17.908682: done 
2022-04-07 00:35:17.909679: This epoch took 125.805125 s
 
2022-04-07 00:35:17.910566: svmem(total=99932160000, available=84038209536, percent=15.9, used=12844339200, free=69448622080, active=23517958144, inactive=4706553856, buffers=43970560, cached=17595228160, shared=1232343040, slab=791425024) 
2022-04-07 00:35:17.911270: 
epoch:  286 
2022-04-07 00:37:07.724888: train loss : -0.6211 
2022-04-07 00:37:19.466985: validation loss: -0.6334 
2022-04-07 00:37:19.475373: Average global foreground Dice: [0.855, 0.9608, 0.9801, 0.4261] 
2022-04-07 00:37:19.488073: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:37:20.543380: lr: 0.004639 
2022-04-07 00:37:20.555559: saving scheduled checkpoint file... 
2022-04-07 00:37:20.696888: saving checkpoint... 
2022-04-07 00:37:25.248561: done, saving took 4.68 seconds 
2022-04-07 00:37:25.262556: done 
2022-04-07 00:37:25.263332: This epoch took 127.351449 s
 
2022-04-07 00:37:25.264271: svmem(total=99932160000, available=84032983040, percent=15.9, used=12863721472, free=69461344256, active=23798276096, inactive=4416655360, buffers=44212224, cached=17562882048, shared=1218396160, slab=791310336) 
2022-04-07 00:37:25.264953: 
epoch:  287 
2022-04-07 00:39:15.068497: train loss : -0.6298 
2022-04-07 00:39:26.759539: validation loss: -0.5090 
2022-04-07 00:39:26.772055: Average global foreground Dice: [0.7993, 0.9394, 0.9533, 0.3862] 
2022-04-07 00:39:26.774362: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:39:28.334817: lr: 0.00462 
2022-04-07 00:39:28.337704: saving scheduled checkpoint file... 
2022-04-07 00:39:28.408638: saving checkpoint... 
2022-04-07 00:39:32.830712: done, saving took 4.49 seconds 
2022-04-07 00:39:32.844406: done 
2022-04-07 00:39:32.845229: This epoch took 127.579679 s
 
2022-04-07 00:39:32.846043: svmem(total=99932160000, available=83930611712, percent=16.0, used=12965871616, free=69346594816, active=23818448896, inactive=4502749184, buffers=44449792, cached=17575243776, shared=1218482176, slab=791392256) 
2022-04-07 00:39:32.846688: 
epoch:  288 
2022-04-07 00:41:22.559633: train loss : -0.6235 
2022-04-07 00:41:34.503196: validation loss: -0.6260 
2022-04-07 00:41:34.513822: Average global foreground Dice: [0.8676, 0.963, 0.9762, 0.4032] 
2022-04-07 00:41:34.514556: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:41:35.425998: lr: 0.0046 
2022-04-07 00:41:35.428096: saving scheduled checkpoint file... 
2022-04-07 00:41:35.521044: saving checkpoint... 
2022-04-07 00:41:39.929702: done, saving took 4.50 seconds 
2022-04-07 00:41:39.942961: done 
2022-04-07 00:41:39.943669: This epoch took 127.096346 s
 
2022-04-07 00:41:39.945495: svmem(total=99932160000, available=83888861184, percent=16.1, used=12993507328, free=69336928256, active=23628021760, inactive=4706447360, buffers=44707840, cached=17557016576, shared=1232359424, slab=792383488) 
2022-04-07 00:41:39.948573: 
epoch:  289 
2022-04-07 00:43:28.778388: train loss : -0.6466 
2022-04-07 00:43:41.131105: validation loss: -0.6385 
2022-04-07 00:43:41.132830: Average global foreground Dice: [0.8595, 0.9634, 0.9834, 0.4042] 
2022-04-07 00:43:41.133511: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:43:42.163743: lr: 0.004581 
2022-04-07 00:43:42.183350: saving scheduled checkpoint file... 
2022-04-07 00:43:42.278831: saving checkpoint... 
2022-04-07 00:43:46.327360: done, saving took 4.12 seconds 
2022-04-07 00:43:46.350835: done 
2022-04-07 00:43:46.355864: This epoch took 126.404515 s
 
2022-04-07 00:43:46.358565: svmem(total=99932160000, available=83939581952, percent=16.0, used=12956684288, free=69350191104, active=23731585024, inactive=4588396544, buffers=44949504, cached=17580335104, shared=1218310144, slab=792428544) 
2022-04-07 00:43:46.364776: 
epoch:  290 
2022-04-07 00:45:33.973605: train loss : -0.6283 
2022-04-07 00:45:45.118050: validation loss: -0.6233 
2022-04-07 00:45:45.125338: Average global foreground Dice: [0.8455, 0.9579, 0.9706, 0.4043] 
2022-04-07 00:45:45.126065: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:45:46.012635: lr: 0.004561 
2022-04-07 00:45:46.030606: saving scheduled checkpoint file... 
2022-04-07 00:45:46.101608: saving checkpoint... 
2022-04-07 00:45:50.662256: done, saving took 4.62 seconds 
2022-04-07 00:45:50.689256: done 
2022-04-07 00:45:50.697662: This epoch took 124.328924 s
 
2022-04-07 00:45:50.707517: svmem(total=99932160000, available=84011728896, percent=15.9, used=12884373504, free=69353373696, active=23599616000, inactive=4722491392, buffers=45203456, cached=17649209344, shared=1218482176, slab=791990272) 
2022-04-07 00:45:50.719471: 
epoch:  291 
2022-04-07 00:47:40.416204: train loss : -0.6074 
2022-04-07 00:47:52.921995: validation loss: -0.5558 
2022-04-07 00:47:52.923615: Average global foreground Dice: [0.8159, 0.952, 0.9614, 0.3773] 
2022-04-07 00:47:52.924637: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:47:53.829258: lr: 0.004541 
2022-04-07 00:47:53.835745: saving scheduled checkpoint file... 
2022-04-07 00:47:53.917557: saving checkpoint... 
2022-04-07 00:47:58.329781: done, saving took 4.49 seconds 
2022-04-07 00:47:58.357649: done 
2022-04-07 00:47:58.362240: This epoch took 127.634979 s
 
2022-04-07 00:47:58.370523: svmem(total=99932160000, available=84020826112, percent=15.9, used=12924125184, free=69418475520, active=23813595136, inactive=4443463680, buffers=45445120, cached=17544114176, shared=1169707008, slab=792076288) 
2022-04-07 00:47:58.377869: 
epoch:  292 
2022-04-07 00:49:46.727694: train loss : -0.5897 
2022-04-07 00:49:57.653318: validation loss: -0.5809 
2022-04-07 00:49:57.658627: Average global foreground Dice: [0.8467, 0.9576, 0.9761, 0.3814] 
2022-04-07 00:49:57.662409: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:49:58.417116: lr: 0.004522 
2022-04-07 00:49:58.422223: saving scheduled checkpoint file... 
2022-04-07 00:49:58.564668: saving checkpoint... 
2022-04-07 00:50:03.143978: done, saving took 4.72 seconds 
2022-04-07 00:50:03.173042: done 
2022-04-07 00:50:03.180484: This epoch took 124.799329 s
 
2022-04-07 00:50:03.189893: svmem(total=99932160000, available=83891089408, percent=16.1, used=13005213696, free=69444665344, active=23690354688, inactive=4540108800, buffers=45686784, cached=17436594176, shared=1218396160, slab=791883776) 
2022-04-07 00:50:03.198393: 
epoch:  293 
2022-04-07 00:51:53.740662: train loss : -0.5995 
2022-04-07 00:52:05.561422: validation loss: -0.6050 
2022-04-07 00:52:05.563169: Average global foreground Dice: [0.8355, 0.9606, 0.9743, 0.3794] 
2022-04-07 00:52:05.563843: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:52:06.393930: lr: 0.004502 
2022-04-07 00:52:06.410124: saving scheduled checkpoint file... 
2022-04-07 00:52:06.578164: saving checkpoint... 
2022-04-07 00:52:11.220248: done, saving took 4.79 seconds 
2022-04-07 00:52:11.259622: done 
2022-04-07 00:52:11.265811: This epoch took 128.060086 s
 
2022-04-07 00:52:11.271395: svmem(total=99932160000, available=84047036416, percent=15.9, used=12850171904, free=69424476160, active=23663669248, inactive=4583874560, buffers=45932544, cached=17611579392, shared=1218404352, slab=790769664) 
2022-04-07 00:52:11.277387: 
epoch:  294 
2022-04-07 00:53:59.033020: train loss : -0.5949 
2022-04-07 00:54:10.021322: validation loss: -0.6288 
2022-04-07 00:54:10.039901: Average global foreground Dice: [0.8438, 0.9587, 0.9759, 0.4038] 
2022-04-07 00:54:10.055073: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:54:10.921324: lr: 0.004482 
2022-04-07 00:54:10.922677: saving scheduled checkpoint file... 
2022-04-07 00:54:11.015477: saving checkpoint... 
2022-04-07 00:54:15.505941: done, saving took 4.58 seconds 
2022-04-07 00:54:15.524178: done 
2022-04-07 00:54:15.530376: This epoch took 124.248068 s
 
2022-04-07 00:54:15.537874: svmem(total=99932160000, available=84016771072, percent=15.9, used=12928253952, free=69341691904, active=23913283584, inactive=4414189568, buffers=46166016, cached=17616048128, shared=1169641472, slab=791052288) 
2022-04-07 00:54:15.544891: 
epoch:  295 
2022-04-07 00:56:03.971366: train loss : -0.6157 
2022-04-07 00:56:14.629788: validation loss: -0.6603 
2022-04-07 00:56:14.635008: Average global foreground Dice: [0.8745, 0.9647, 0.9811, 0.4004] 
2022-04-07 00:56:14.638035: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:56:15.711842: lr: 0.004463 
2022-04-07 00:56:15.718301: saving scheduled checkpoint file... 
2022-04-07 00:56:15.824995: saving checkpoint... 
2022-04-07 00:56:20.158362: done, saving took 4.43 seconds 
2022-04-07 00:56:20.181539: done 
2022-04-07 00:56:20.186606: This epoch took 124.634644 s
 
2022-04-07 00:56:20.193053: svmem(total=99932160000, available=83936722944, percent=16.0, used=12959432704, free=69343178752, active=23889190912, inactive=4438540288, buffers=46424064, cached=17583124480, shared=1218424832, slab=792788992) 
2022-04-07 00:56:20.198429: 
epoch:  296 
2022-04-07 00:58:09.819825: train loss : -0.6414 
2022-04-07 00:58:21.041446: validation loss: -0.6451 
2022-04-07 00:58:21.043132: Average global foreground Dice: [0.8851, 0.9601, 0.9763, 0.4082] 
2022-04-07 00:58:21.043827: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 00:58:22.055742: lr: 0.004443 
2022-04-07 00:58:22.066546: saving scheduled checkpoint file... 
2022-04-07 00:58:22.161526: saving checkpoint... 
2022-04-07 00:58:26.725449: done, saving took 4.65 seconds 
2022-04-07 00:58:26.750043: done 
2022-04-07 00:58:26.755705: This epoch took 126.553405 s
 
2022-04-07 00:58:26.763694: svmem(total=99932160000, available=83866116096, percent=16.1, used=13030060032, free=69344227328, active=23752892416, inactive=4572311552, buffers=46665728, cached=17511206912, shared=1218342912, slab=792592384) 
2022-04-07 00:58:26.769151: 
epoch:  297 
2022-04-07 01:00:16.728736: train loss : -0.6424 
2022-04-07 01:00:28.313213: validation loss: -0.6783 
2022-04-07 01:00:28.334607: Average global foreground Dice: [0.8722, 0.9658, 0.9834, 0.4431] 
2022-04-07 01:00:28.353701: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:00:28.938925: lr: 0.004423 
2022-04-07 01:00:28.952941: saving scheduled checkpoint file... 
2022-04-07 01:00:29.011044: saving checkpoint... 
2022-04-07 01:00:33.619081: done, saving took 4.66 seconds 
2022-04-07 01:00:33.641240: done 
2022-04-07 01:00:33.648004: This epoch took 126.873261 s
 
2022-04-07 01:00:33.653818: svmem(total=99932160000, available=83905114112, percent=16.0, used=12980641792, free=69329436672, active=23693406208, inactive=4648075264, buffers=46927872, cached=17575153664, shared=1228836864, slab=792489984) 
2022-04-07 01:00:33.658202: 
epoch:  298 
2022-04-07 01:02:22.876474: train loss : -0.6513 
2022-04-07 01:02:34.862505: validation loss: -0.6617 
2022-04-07 01:02:34.867535: Average global foreground Dice: [0.8813, 0.9649, 0.9826, 0.4088] 
2022-04-07 01:02:34.875938: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:02:36.300384: lr: 0.004404 
2022-04-07 01:02:36.322803: saving scheduled checkpoint file... 
2022-04-07 01:02:36.401254: saving checkpoint... 
2022-04-07 01:02:40.900156: done, saving took 4.55 seconds 
2022-04-07 01:02:40.931725: done 
2022-04-07 01:02:40.945850: This epoch took 127.283608 s
 
2022-04-07 01:02:40.958211: svmem(total=99932160000, available=83851157504, percent=16.1, used=13045121024, free=69341057024, active=23707451392, inactive=4624998400, buffers=47194112, cached=17498787840, shared=1218600960, slab=792952832) 
2022-04-07 01:02:40.970184: 
epoch:  299 
2022-04-07 01:04:30.526928: train loss : -0.6613 
2022-04-07 01:04:42.146573: validation loss: -0.6597 
2022-04-07 01:04:42.148114: Average global foreground Dice: [0.8795, 0.9652, 0.9807, 0.4137] 
2022-04-07 01:04:42.148957: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:04:42.960298: lr: 0.004384 
2022-04-07 01:04:42.967302: saving scheduled checkpoint file... 
2022-04-07 01:04:43.092630: saving checkpoint... 
2022-04-07 01:04:47.842228: done, saving took 4.86 seconds 
2022-04-07 01:04:47.881072: done 
2022-04-07 01:04:47.891118: This epoch took 126.910290 s
 
2022-04-07 01:04:47.901542: svmem(total=99932160000, available=83965751296, percent=16.0, used=12930715648, free=69404037120, active=23703195648, inactive=4559392768, buffers=47435776, cached=17549971456, shared=1218691072, slab=793788416) 
2022-04-07 01:04:47.908978: 
epoch:  300 
2022-04-07 01:06:34.786829: train loss : -0.6475 
2022-04-07 01:06:46.793356: validation loss: -0.6489 
2022-04-07 01:06:46.805263: Average global foreground Dice: [0.8762, 0.9625, 0.9704, 0.4237] 
2022-04-07 01:06:46.812507: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:06:47.572586: lr: 0.004364 
2022-04-07 01:06:47.587730: saving scheduled checkpoint file... 
2022-04-07 01:06:47.692654: saving checkpoint... 
2022-04-07 01:06:52.340477: done, saving took 4.74 seconds 
2022-04-07 01:06:52.371918: done 
2022-04-07 01:06:52.380673: This epoch took 124.466055 s
 
2022-04-07 01:06:52.391017: svmem(total=99932160000, available=83989708800, percent=16.0, used=12906168320, free=69330669568, active=23772274688, inactive=4563824640, buffers=47689728, cached=17647632384, shared=1218519040, slab=794161152) 
2022-04-07 01:06:52.401891: 
epoch:  301 
2022-04-07 01:08:40.743554: train loss : -0.6646 
2022-04-07 01:08:53.155800: validation loss: -0.6710 
2022-04-07 01:08:53.165606: Average global foreground Dice: [0.8786, 0.9659, 0.9817, 0.4195] 
2022-04-07 01:08:53.166526: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:08:54.124718: lr: 0.004344 
2022-04-07 01:08:54.125522: saving scheduled checkpoint file... 
2022-04-07 01:08:54.192243: saving checkpoint... 
2022-04-07 01:08:58.670687: done, saving took 4.54 seconds 
2022-04-07 01:08:58.691903: done 
2022-04-07 01:08:58.698123: This epoch took 126.286092 s
 
2022-04-07 01:08:58.703258: svmem(total=99932160000, available=84023701504, percent=15.9, used=12872183808, free=69330358272, active=23831683072, inactive=4504989696, buffers=47931392, cached=17681686528, shared=1218605056, slab=794378240) 
2022-04-07 01:08:58.710322: 
epoch:  302 
2022-04-07 01:10:48.474622: train loss : -0.6588 
2022-04-07 01:11:00.428638: validation loss: -0.6403 
2022-04-07 01:11:00.438270: Average global foreground Dice: [0.8667, 0.9589, 0.9791, 0.4111] 
2022-04-07 01:11:00.441991: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:11:01.376913: lr: 0.004325 
2022-04-07 01:11:01.378804: saving scheduled checkpoint file... 
2022-04-07 01:11:01.492146: saving checkpoint... 
2022-04-07 01:11:05.956600: done, saving took 4.58 seconds 
2022-04-07 01:11:05.974829: done 
2022-04-07 01:11:05.977124: This epoch took 127.262422 s
 
2022-04-07 01:11:05.978606: svmem(total=99932160000, available=83986857984, percent=16.0, used=12908654592, free=69404069888, active=23755747328, inactive=4509741056, buffers=48173056, cached=17571262464, shared=1218605056, slab=794169344) 
2022-04-07 01:11:05.980939: 
epoch:  303 
2022-04-07 01:12:56.569440: train loss : -0.6111 
2022-04-07 01:13:07.927110: validation loss: -0.6345 
2022-04-07 01:13:07.931429: Average global foreground Dice: [0.8842, 0.9574, 0.9724, 0.4091] 
2022-04-07 01:13:07.938530: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:13:08.821607: lr: 0.004305 
2022-04-07 01:13:08.822628: saving scheduled checkpoint file... 
2022-04-07 01:13:08.976279: saving checkpoint... 
2022-04-07 01:13:13.448692: done, saving took 4.63 seconds 
2022-04-07 01:13:13.472991: done 
2022-04-07 01:13:13.476056: This epoch took 127.492984 s
 
2022-04-07 01:13:13.477000: svmem(total=99932160000, available=84075319296, percent=15.9, used=12813340672, free=69428305920, active=23759781888, inactive=4482777088, buffers=48410624, cached=17642102784, shared=1225543680, slab=794927104) 
2022-04-07 01:13:13.477810: 
epoch:  304 
2022-04-07 01:15:03.751885: train loss : -0.6212 
2022-04-07 01:15:15.595630: validation loss: -0.6321 
2022-04-07 01:15:15.611726: Average global foreground Dice: [0.8583, 0.9575, 0.9703, 0.419] 
2022-04-07 01:15:15.613572: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:15:16.228512: lr: 0.004285 
2022-04-07 01:15:16.242797: saving scheduled checkpoint file... 
2022-04-07 01:15:16.359991: saving checkpoint... 
2022-04-07 01:15:20.729293: done, saving took 4.48 seconds 
2022-04-07 01:15:20.742312: done 
2022-04-07 01:15:20.743016: This epoch took 127.264563 s
 
2022-04-07 01:15:20.743749: svmem(total=99932160000, available=83880718336, percent=16.1, used=13014786048, free=69333569536, active=23910518784, inactive=4429869056, buffers=48685056, cached=17535119360, shared=1218433024, slab=793718784) 
2022-04-07 01:15:20.744293: 
epoch:  305 
2022-04-07 01:17:10.539705: train loss : -0.5973 
2022-04-07 01:17:22.210661: validation loss: -0.5396 
2022-04-07 01:17:22.213725: Average global foreground Dice: [0.7961, 0.9468, 0.9531, 0.371] 
2022-04-07 01:17:22.216578: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:17:23.563671: lr: 0.004265 
2022-04-07 01:17:23.569584: saving scheduled checkpoint file... 
2022-04-07 01:17:23.672225: saving checkpoint... 
2022-04-07 01:17:28.007380: done, saving took 4.43 seconds 
2022-04-07 01:17:28.021986: done 
2022-04-07 01:17:28.022761: This epoch took 127.277950 s
 
2022-04-07 01:17:28.023562: svmem(total=99932160000, available=83973935104, percent=16.0, used=12910071808, free=69322469376, active=23926632448, inactive=4425125888, buffers=48951296, cached=17650667520, shared=1231081472, slab=793460736) 
2022-04-07 01:17:28.024236: 
epoch:  306 
2022-04-07 01:19:15.864169: train loss : -0.5519 
2022-04-07 01:19:27.712059: validation loss: -0.5446 
2022-04-07 01:19:27.720286: Average global foreground Dice: [0.8194, 0.9502, 0.9624, 0.3802] 
2022-04-07 01:19:27.728054: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:19:28.703538: lr: 0.004245 
2022-04-07 01:19:28.704323: saving scheduled checkpoint file... 
2022-04-07 01:19:28.789901: saving checkpoint... 
2022-04-07 01:19:33.173363: done, saving took 4.47 seconds 
2022-04-07 01:19:33.194810: done 
2022-04-07 01:19:33.203859: This epoch took 125.178774 s
 
2022-04-07 01:19:33.208615: svmem(total=99932160000, available=83924508672, percent=16.0, used=12970135552, free=69334200320, active=23376306176, inactive=4956712960, buffers=49180672, cached=17578643456, shared=1218433024, slab=793300992) 
2022-04-07 01:19:33.214642: 
epoch:  307 
2022-04-07 01:21:22.847640: train loss : -0.5783 
2022-04-07 01:21:34.151608: validation loss: -0.6261 
2022-04-07 01:21:34.153263: Average global foreground Dice: [0.8594, 0.9567, 0.9758, 0.3967] 
2022-04-07 01:21:34.159565: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:21:34.840286: lr: 0.004226 
2022-04-07 01:21:34.842067: saving scheduled checkpoint file... 
2022-04-07 01:21:34.937664: saving checkpoint... 
2022-04-07 01:21:39.599354: done, saving took 4.76 seconds 
2022-04-07 01:21:39.614142: done 
2022-04-07 01:21:39.616520: This epoch took 126.398560 s
 
2022-04-07 01:21:39.619211: svmem(total=99932160000, available=83905572864, percent=16.0, used=12990418944, free=69329469440, active=23866195968, inactive=4477591552, buffers=49438720, cached=17562832896, shared=1218523136, slab=792924160) 
2022-04-07 01:21:39.621421: 
epoch:  308 
2022-04-07 01:23:29.581972: train loss : -0.6177 
2022-04-07 01:23:40.672660: validation loss: -0.6526 
2022-04-07 01:23:40.675189: Average global foreground Dice: [0.8661, 0.962, 0.9795, 0.408] 
2022-04-07 01:23:40.676206: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:23:41.556563: lr: 0.004206 
2022-04-07 01:23:41.567705: saving scheduled checkpoint file... 
2022-04-07 01:23:41.628600: saving checkpoint... 
2022-04-07 01:23:46.144933: done, saving took 4.57 seconds 
2022-04-07 01:23:46.179392: done 
2022-04-07 01:23:46.191207: This epoch took 126.567371 s
 
2022-04-07 01:23:46.202173: svmem(total=99932160000, available=84009992192, percent=15.9, used=12896985088, free=69315178496, active=23709515776, inactive=4649295872, buffers=49688576, cached=17670307840, shared=1208012800, slab=792117248) 
2022-04-07 01:23:46.211221: 
epoch:  309 
2022-04-07 01:25:35.434328: train loss : -0.6434 
2022-04-07 01:25:47.532487: validation loss: -0.6385 
2022-04-07 01:25:47.539635: Average global foreground Dice: [0.879, 0.9598, 0.9743, 0.4016] 
2022-04-07 01:25:47.547235: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:25:48.138890: lr: 0.004186 
2022-04-07 01:25:48.156580: saving scheduled checkpoint file... 
2022-04-07 01:25:48.380543: saving checkpoint... 
2022-04-07 01:25:52.858323: done, saving took 4.68 seconds 
2022-04-07 01:25:52.891110: done 
2022-04-07 01:25:52.901738: This epoch took 126.677605 s
 
2022-04-07 01:25:52.911742: svmem(total=99932160000, available=84132945920, percent=15.8, used=12787560448, free=69400395776, active=23690563584, inactive=4583202816, buffers=49963008, cached=17694240768, shared=1193811968, slab=792383488) 
2022-04-07 01:25:52.919083: 
epoch:  310 
2022-04-07 01:27:42.157983: train loss : -0.6211 
2022-04-07 01:27:53.739603: validation loss: -0.6335 
2022-04-07 01:27:53.741447: Average global foreground Dice: [0.8501, 0.9627, 0.9792, 0.3844] 
2022-04-07 01:27:53.742258: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:27:54.594116: lr: 0.004166 
2022-04-07 01:27:54.599944: saving scheduled checkpoint file... 
2022-04-07 01:27:54.707698: saving checkpoint... 
2022-04-07 01:27:59.185055: done, saving took 4.58 seconds 
2022-04-07 01:27:59.207891: done 
2022-04-07 01:27:59.210179: This epoch took 126.281302 s
 
2022-04-07 01:27:59.212509: svmem(total=99932160000, available=83949215744, percent=16.0, used=12946411520, free=69414326272, active=23718612992, inactive=4538478592, buffers=50192384, cached=17521229824, shared=1218609152, slab=792936448) 
2022-04-07 01:27:59.214508: 
epoch:  311 
2022-04-07 01:29:47.877685: train loss : -0.6369 
2022-04-07 01:29:59.635313: validation loss: -0.6420 
2022-04-07 01:29:59.636998: Average global foreground Dice: [0.8762, 0.9608, 0.9762, 0.4293] 
2022-04-07 01:29:59.637917: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:30:00.455149: lr: 0.004146 
2022-04-07 01:30:00.464037: saving scheduled checkpoint file... 
2022-04-07 01:30:00.605967: saving checkpoint... 
2022-04-07 01:30:05.339510: done, saving took 4.86 seconds 
2022-04-07 01:30:05.378105: done 
2022-04-07 01:30:05.386303: This epoch took 126.169107 s
 
2022-04-07 01:30:05.394829: svmem(total=99932160000, available=84043378688, percent=15.9, used=12852166656, free=69493800960, active=23605497856, inactive=4572835840, buffers=50442240, cached=17535750144, shared=1218437120, slab=793538560) 
2022-04-07 01:30:05.401501: 
epoch:  312 
2022-04-07 01:31:54.426526: train loss : -0.6167 
2022-04-07 01:32:06.474127: validation loss: -0.6346 
2022-04-07 01:32:06.486296: Average global foreground Dice: [0.8675, 0.9632, 0.9783, 0.3837] 
2022-04-07 01:32:06.488775: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:32:07.259857: lr: 0.004127 
2022-04-07 01:32:07.269414: saving scheduled checkpoint file... 
2022-04-07 01:32:07.332539: saving checkpoint... 
2022-04-07 01:32:11.854861: done, saving took 4.58 seconds 
2022-04-07 01:32:11.879969: done 
2022-04-07 01:32:11.887214: This epoch took 126.480585 s
 
2022-04-07 01:32:11.893198: svmem(total=99932160000, available=84010192896, percent=15.9, used=12885458944, free=69316534272, active=23673831424, inactive=4676587520, buffers=50704384, cached=17679462400, shared=1218695168, slab=794112000) 
2022-04-07 01:32:11.899670: 
epoch:  313 
2022-04-07 01:34:00.786010: train loss : -0.6451 
2022-04-07 01:34:13.493240: validation loss: -0.6279 
2022-04-07 01:34:13.505906: Average global foreground Dice: [0.8631, 0.9607, 0.9742, 0.4036] 
2022-04-07 01:34:13.515129: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:34:14.146448: lr: 0.004107 
2022-04-07 01:34:14.148515: saving scheduled checkpoint file... 
2022-04-07 01:34:14.244660: saving checkpoint... 
2022-04-07 01:34:18.974349: done, saving took 4.82 seconds 
2022-04-07 01:34:18.989050: done 
2022-04-07 01:34:18.989799: This epoch took 127.084001 s
 
2022-04-07 01:34:18.990746: svmem(total=99932160000, available=84001460224, percent=15.9, used=12884975616, free=69303267328, active=23880503296, inactive=4484034560, buffers=50933760, cached=17692983296, shared=1227939840, slab=793714688) 
2022-04-07 01:34:18.991596: 
epoch:  314 
2022-04-07 01:36:08.159578: train loss : -0.6354 
2022-04-07 01:36:19.743355: validation loss: -0.6486 
2022-04-07 01:36:19.753565: Average global foreground Dice: [0.8732, 0.9603, 0.9793, 0.409] 
2022-04-07 01:36:19.754438: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:36:20.472384: lr: 0.004087 
2022-04-07 01:36:20.473699: saving scheduled checkpoint file... 
2022-04-07 01:36:20.543293: saving checkpoint... 
2022-04-07 01:36:25.182210: done, saving took 4.71 seconds 
2022-04-07 01:36:25.211819: done 
2022-04-07 01:36:25.219105: This epoch took 126.226774 s
 
2022-04-07 01:36:25.225208: svmem(total=99932160000, available=84036276224, percent=15.9, used=12859568128, free=69385158656, active=23774474240, inactive=4512976896, buffers=51212288, cached=17636220928, shared=1218437120, slab=794677248) 
2022-04-07 01:36:25.230891: 
epoch:  315 
2022-04-07 01:38:13.870343: train loss : -0.6495 
2022-04-07 01:38:25.004375: validation loss: -0.6652 
2022-04-07 01:38:25.019798: Average global foreground Dice: [0.8775, 0.9602, 0.9825, 0.4157] 
2022-04-07 01:38:25.020684: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:38:25.875295: lr: 0.004067 
2022-04-07 01:38:25.887152: saving scheduled checkpoint file... 
2022-04-07 01:38:26.046764: saving checkpoint... 
2022-04-07 01:38:30.504359: done, saving took 4.61 seconds 
2022-04-07 01:38:30.544308: done 
2022-04-07 01:38:30.551073: This epoch took 125.314057 s
 
2022-04-07 01:38:30.565808: svmem(total=99932160000, available=84098756608, percent=15.8, used=12796080128, free=69419597824, active=23796498432, inactive=4452782080, buffers=51453952, cached=17665028096, shared=1218437120, slab=794058752) 
2022-04-07 01:38:30.572590: 
epoch:  316 
2022-04-07 01:40:20.366467: train loss : -0.6583 
2022-04-07 01:40:32.487095: validation loss: -0.6432 
2022-04-07 01:40:32.490281: Average global foreground Dice: [0.8792, 0.9629, 0.9818, 0.4176] 
2022-04-07 01:40:32.495322: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:40:33.432086: lr: 0.004047 
2022-04-07 01:40:33.440727: saving scheduled checkpoint file... 
2022-04-07 01:40:33.557683: saving checkpoint... 
2022-04-07 01:40:38.163975: done, saving took 4.71 seconds 
2022-04-07 01:40:38.183767: done 
2022-04-07 01:40:38.186081: This epoch took 127.599652 s
 
2022-04-07 01:40:38.187508: svmem(total=99932160000, available=83997507584, percent=15.9, used=12886147072, free=69411749888, active=23641903104, inactive=4616548352, buffers=51695616, cached=17582567424, shared=1230741504, slab=795361280) 
2022-04-07 01:40:38.188308: 
epoch:  317 
2022-04-07 01:42:28.343360: train loss : -0.6477 
2022-04-07 01:42:40.465590: validation loss: -0.6351 
2022-04-07 01:42:40.476435: Average global foreground Dice: [0.8688, 0.9655, 0.9769, 0.3656] 
2022-04-07 01:42:40.484480: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:42:41.505298: lr: 0.004027 
2022-04-07 01:42:41.508592: saving scheduled checkpoint file... 
2022-04-07 01:42:41.600325: saving checkpoint... 
2022-04-07 01:42:46.032297: done, saving took 4.52 seconds 
2022-04-07 01:42:46.047570: done 
2022-04-07 01:42:46.049928: This epoch took 127.860993 s
 
2022-04-07 01:42:46.051860: svmem(total=99932160000, available=84035952640, percent=15.9, used=12859846656, free=69311488000, active=23925374976, inactive=4431671296, buffers=51949568, cached=17708875776, shared=1218523136, slab=795709440) 
2022-04-07 01:42:46.054094: 
epoch:  318 
2022-04-07 01:44:32.478042: train loss : -0.6665 
2022-04-07 01:44:44.260162: validation loss: -0.6666 
2022-04-07 01:44:44.270339: Average global foreground Dice: [0.888, 0.9667, 0.9846, 0.4166] 
2022-04-07 01:44:44.272404: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:44:45.199813: lr: 0.004007 
2022-04-07 01:44:45.207943: saving scheduled checkpoint file... 
2022-04-07 01:44:45.355877: saving checkpoint... 
2022-04-07 01:44:49.725735: done, saving took 4.51 seconds 
2022-04-07 01:44:49.740058: done 
2022-04-07 01:44:49.742544: This epoch took 123.687226 s
 
2022-04-07 01:44:49.745130: svmem(total=99932160000, available=83872755712, percent=16.1, used=13008969728, free=69301960704, active=23563841536, inactive=4809703424, buffers=52183040, cached=17569046528, shared=1232400384, slab=793120768) 
2022-04-07 01:44:49.745908: 
epoch:  319 
2022-04-07 01:46:38.179675: train loss : -0.6660 
2022-04-07 01:46:50.619280: validation loss: -0.6808 
2022-04-07 01:46:50.622169: Average global foreground Dice: [0.8736, 0.9676, 0.984, 0.4188] 
2022-04-07 01:46:50.624919: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:46:51.389615: lr: 0.003987 
2022-04-07 01:46:51.391245: saving scheduled checkpoint file... 
2022-04-07 01:46:51.442075: saving checkpoint... 
2022-04-07 01:46:55.879920: done, saving took 4.49 seconds 
2022-04-07 01:46:55.902919: done 
2022-04-07 01:46:55.954366: saving checkpoint... 
2022-04-07 01:47:00.505517: done, saving took 4.60 seconds 
2022-04-07 01:47:00.537145: This epoch took 130.790583 s
 
2022-04-07 01:47:00.541125: svmem(total=99932160000, available=83892756480, percent=16.1, used=13001830400, free=69318750208, active=23762567168, inactive=4590710784, buffers=52461568, cached=17559117824, shared=1219690496, slab=793731072) 
2022-04-07 01:47:00.544106: 
epoch:  320 
2022-04-07 01:48:50.301944: train loss : -0.6643 
2022-04-07 01:49:02.270216: validation loss: -0.6634 
2022-04-07 01:49:02.285771: Average global foreground Dice: [0.8923, 0.9685, 0.9772, 0.3969] 
2022-04-07 01:49:02.303005: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:49:03.105404: lr: 0.003967 
2022-04-07 01:49:03.108398: saving scheduled checkpoint file... 
2022-04-07 01:49:03.223688: saving checkpoint... 
2022-04-07 01:49:07.831450: done, saving took 4.72 seconds 
2022-04-07 01:49:07.861639: done 
2022-04-07 01:49:07.917131: saving checkpoint... 
2022-04-07 01:49:12.347595: done, saving took 4.48 seconds 
2022-04-07 01:49:12.361509: This epoch took 131.812708 s
 
2022-04-07 01:49:12.362503: svmem(total=99932160000, available=83942125568, percent=16.0, used=12953604096, free=69308751872, active=23765028864, inactive=4596125696, buffers=52699136, cached=17617104896, shared=1218527232, slab=792047616) 
2022-04-07 01:49:12.363174: 
epoch:  321 
2022-04-07 01:51:01.667938: train loss : -0.6698 
2022-04-07 01:51:13.355688: validation loss: -0.6625 
2022-04-07 01:51:13.372863: Average global foreground Dice: [0.8892, 0.9699, 0.983, 0.3921] 
2022-04-07 01:51:13.374951: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:51:14.348891: lr: 0.003947 
2022-04-07 01:51:14.361702: saving scheduled checkpoint file... 
2022-04-07 01:51:14.429327: saving checkpoint... 
2022-04-07 01:51:18.860019: done, saving took 4.49 seconds 
2022-04-07 01:51:18.878979: done 
2022-04-07 01:51:18.960034: saving checkpoint... 
2022-04-07 01:51:23.386240: done, saving took 4.50 seconds 
2022-04-07 01:51:23.403873: This epoch took 131.039969 s
 
2022-04-07 01:51:23.404952: svmem(total=99932160000, available=83929956352, percent=16.0, used=12951945216, free=69382701056, active=23790788608, inactive=4498542592, buffers=52957184, cached=17544556544, shared=1232404480, slab=793313280) 
2022-04-07 01:51:23.405577: 
epoch:  322 
2022-04-07 01:53:13.837898: train loss : -0.6716 
2022-04-07 01:53:25.922249: validation loss: -0.6860 
2022-04-07 01:53:25.927287: Average global foreground Dice: [0.8999, 0.969, 0.9838, 0.4082] 
2022-04-07 01:53:25.929931: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:53:26.761511: lr: 0.003927 
2022-04-07 01:53:26.774442: saving scheduled checkpoint file... 
2022-04-07 01:53:26.914545: saving checkpoint... 
2022-04-07 01:53:31.488015: done, saving took 4.70 seconds 
2022-04-07 01:53:31.511841: done 
2022-04-07 01:53:31.609403: saving checkpoint... 
2022-04-07 01:53:36.053599: done, saving took 4.54 seconds 
2022-04-07 01:53:36.071496: This epoch took 132.665181 s
 
2022-04-07 01:53:36.072441: svmem(total=99932160000, available=83925721088, percent=16.0, used=12970258432, free=69410250752, active=23694184448, inactive=4565803008, buffers=53207040, cached=17498443776, shared=1218396160, slab=793112576) 
2022-04-07 01:53:36.073079: 
epoch:  323 
2022-04-07 01:55:25.387445: train loss : -0.6565 
2022-04-07 01:55:37.105787: validation loss: -0.6671 
2022-04-07 01:55:37.113794: Average global foreground Dice: [0.8696, 0.9631, 0.9813, 0.4088] 
2022-04-07 01:55:37.120001: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:55:38.358252: lr: 0.003907 
2022-04-07 01:55:38.362087: saving scheduled checkpoint file... 
2022-04-07 01:55:38.437081: saving checkpoint... 
2022-04-07 01:55:42.502610: done, saving took 4.14 seconds 
2022-04-07 01:55:42.526098: done 
2022-04-07 01:55:42.577944: saving checkpoint... 
2022-04-07 01:55:46.918278: done, saving took 4.39 seconds 
2022-04-07 01:55:46.930632: This epoch took 130.856824 s
 
2022-04-07 01:55:46.931434: svmem(total=99932160000, available=83909726208, percent=16.0, used=12972220416, free=69296959488, active=23933202432, inactive=4442218496, buffers=53506048, cached=17609474048, shared=1232318464, slab=791891968) 
2022-04-07 01:55:46.931957: 
epoch:  324 
2022-04-07 01:57:34.676721: train loss : -0.6627 
2022-04-07 01:57:46.558572: validation loss: -0.6400 
2022-04-07 01:57:46.565437: Average global foreground Dice: [0.8853, 0.9679, 0.9758, 0.3814] 
2022-04-07 01:57:46.566439: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:57:47.401109: lr: 0.003887 
2022-04-07 01:57:47.413625: saving scheduled checkpoint file... 
2022-04-07 01:57:47.552798: saving checkpoint... 
2022-04-07 01:57:51.951008: done, saving took 4.54 seconds 
2022-04-07 01:57:51.965497: done 
2022-04-07 01:57:51.966461: This epoch took 125.033969 s
 
2022-04-07 01:57:51.967448: svmem(total=99932160000, available=83941265408, percent=16.0, used=12954288128, free=69302267904, active=23849631744, inactive=4517761024, buffers=53735424, cached=17621868544, shared=1218613248, slab=792571904) 
2022-04-07 01:57:51.968154: 
epoch:  325 
2022-04-07 01:59:39.664427: train loss : -0.6691 
2022-04-07 01:59:51.339742: validation loss: -0.6557 
2022-04-07 01:59:51.347793: Average global foreground Dice: [0.8625, 0.9673, 0.9835, 0.418] 
2022-04-07 01:59:51.348875: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 01:59:52.208169: lr: 0.003867 
2022-04-07 01:59:52.216565: saving scheduled checkpoint file... 
2022-04-07 01:59:52.287539: saving checkpoint... 
2022-04-07 01:59:56.676445: done, saving took 4.45 seconds 
2022-04-07 01:59:56.690112: done 
2022-04-07 01:59:56.736792: saving checkpoint... 
2022-04-07 02:00:01.025716: done, saving took 4.33 seconds 
2022-04-07 02:00:01.034429: This epoch took 129.065531 s
 
2022-04-07 02:00:01.035537: svmem(total=99932160000, available=83892744192, percent=16.1, used=13002956800, free=69307125760, active=23672926208, inactive=4694224896, buffers=53977088, cached=17568100352, shared=1218613248, slab=791683072) 
2022-04-07 02:00:01.036303: 
epoch:  326 
2022-04-07 02:01:51.356637: train loss : -0.6741 
2022-04-07 02:02:04.788416: validation loss: -0.6616 
2022-04-07 02:02:04.790665: Average global foreground Dice: [0.8836, 0.9685, 0.977, 0.3913] 
2022-04-07 02:02:04.794357: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:02:05.767730: lr: 0.003847 
2022-04-07 02:02:05.775176: saving scheduled checkpoint file... 
2022-04-07 02:02:05.838133: saving checkpoint... 
2022-04-07 02:02:10.312686: done, saving took 4.53 seconds 
2022-04-07 02:02:10.357326: done 
2022-04-07 02:02:10.370354: This epoch took 129.333323 s
 
2022-04-07 02:02:10.378891: svmem(total=99932160000, available=83994415104, percent=15.9, used=12900990976, free=69307277312, active=23724789760, inactive=4642263040, buffers=54247424, cached=17669644288, shared=1218453504, slab=791457792) 
2022-04-07 02:02:10.388041: 
epoch:  327 
2022-04-07 02:04:01.053884: train loss : -0.6508 
2022-04-07 02:04:12.802928: validation loss: -0.6707 
2022-04-07 02:04:12.805748: Average global foreground Dice: [0.901, 0.965, 0.9842, 0.4258] 
2022-04-07 02:04:12.807670: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:04:13.820522: lr: 0.003827 
2022-04-07 02:04:13.823353: saving scheduled checkpoint file... 
2022-04-07 02:04:14.036048: saving checkpoint... 
2022-04-07 02:04:18.510423: done, saving took 4.68 seconds 
2022-04-07 02:04:18.530790: done 
2022-04-07 02:04:18.581426: saving checkpoint... 
2022-04-07 02:04:23.004967: done, saving took 4.47 seconds 
2022-04-07 02:04:23.018292: This epoch took 132.625382 s
 
2022-04-07 02:04:23.019141: svmem(total=99932160000, available=83857440768, percent=16.1, used=13024288768, free=69287686144, active=23741382656, inactive=4643069952, buffers=54493184, cached=17565691904, shared=1232502784, slab=793812992) 
2022-04-07 02:04:23.019643: 
epoch:  328 
2022-04-07 02:06:11.043840: train loss : -0.6449 
2022-04-07 02:06:23.301992: validation loss: -0.6705 
2022-04-07 02:06:23.303964: Average global foreground Dice: [0.8964, 0.9666, 0.9847, 0.4065] 
2022-04-07 02:06:23.304751: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:06:24.130808: lr: 0.003807 
2022-04-07 02:06:24.134135: saving scheduled checkpoint file... 
2022-04-07 02:06:24.192839: saving checkpoint... 
2022-04-07 02:06:28.703018: done, saving took 4.57 seconds 
2022-04-07 02:06:28.723971: done 
2022-04-07 02:06:28.805574: saving checkpoint... 
2022-04-07 02:06:33.265023: done, saving took 4.54 seconds 
2022-04-07 02:06:33.280638: This epoch took 130.260502 s
 
2022-04-07 02:06:33.281490: svmem(total=99932160000, available=83885555712, percent=16.1, used=13010165760, free=69372755968, active=23600959488, inactive=4697747456, buffers=54775808, cached=17494462464, shared=1218453504, slab=792473600) 
2022-04-07 02:06:33.282069: 
epoch:  329 
2022-04-07 02:08:22.937730: train loss : -0.6710 
2022-04-07 02:08:34.122658: validation loss: -0.6751 
2022-04-07 02:08:34.124348: Average global foreground Dice: [0.8861, 0.9623, 0.9822, 0.4174] 
2022-04-07 02:08:34.125127: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:08:35.269141: lr: 0.003787 
2022-04-07 02:08:35.270870: saving scheduled checkpoint file... 
2022-04-07 02:08:35.371050: saving checkpoint... 
2022-04-07 02:08:40.028950: done, saving took 4.75 seconds 
2022-04-07 02:08:40.043713: done 
2022-04-07 02:08:40.092830: saving checkpoint... 
2022-04-07 02:08:44.525628: done, saving took 4.48 seconds 
2022-04-07 02:08:44.545819: This epoch took 131.263181 s
 
2022-04-07 02:08:44.549690: svmem(total=99932160000, available=83919527936, percent=16.0, used=12976140288, free=69299449856, active=23896133632, inactive=4476211200, buffers=55042048, cached=17601527808, shared=1218453504, slab=793378816) 
2022-04-07 02:08:44.552915: 
epoch:  330 
2022-04-07 02:10:33.524437: train loss : -0.6446 
2022-04-07 02:10:44.557478: validation loss: -0.6433 
2022-04-07 02:10:44.561729: Average global foreground Dice: [0.8645, 0.9675, 0.979, 0.4168] 
2022-04-07 02:10:44.565734: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:10:45.471378: lr: 0.003767 
2022-04-07 02:10:45.472270: saving scheduled checkpoint file... 
2022-04-07 02:10:45.527884: saving checkpoint... 
2022-04-07 02:10:50.000448: done, saving took 4.53 seconds 
2022-04-07 02:10:50.021752: done 
2022-04-07 02:10:50.025723: This epoch took 125.470128 s
 
2022-04-07 02:10:50.029607: svmem(total=99932160000, available=84023746560, percent=15.9, used=12871798784, free=69301706752, active=23874564096, inactive=4495187968, buffers=55275520, cached=17703378944, shared=1218539520, slab=793911296) 
2022-04-07 02:10:50.030530: 
epoch:  331 
2022-04-07 02:12:39.363584: train loss : -0.6620 
2022-04-07 02:12:51.190731: validation loss: -0.6650 
2022-04-07 02:12:51.200887: Average global foreground Dice: [0.8921, 0.9667, 0.9825, 0.4023] 
2022-04-07 02:12:51.209395: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:12:52.001383: lr: 0.003747 
2022-04-07 02:12:52.003172: saving scheduled checkpoint file... 
2022-04-07 02:12:52.067531: saving checkpoint... 
2022-04-07 02:12:56.714462: done, saving took 4.71 seconds 
2022-04-07 02:12:56.745866: done 
2022-04-07 02:12:56.833482: saving checkpoint... 
2022-04-07 02:13:01.302817: done, saving took 4.55 seconds 
2022-04-07 02:13:01.318884: This epoch took 131.287606 s
 
2022-04-07 02:13:01.319934: svmem(total=99932160000, available=83958136832, percent=16.0, used=12925095936, free=69359734784, active=23833251840, inactive=4478095360, buffers=55521280, cached=17591808000, shared=1230843904, slab=793362432) 
2022-04-07 02:13:01.320552: 
epoch:  332 
2022-04-07 02:14:50.824886: train loss : -0.6701 
2022-04-07 02:15:04.357735: validation loss: -0.6548 
2022-04-07 02:15:04.376209: Average global foreground Dice: [0.8874, 0.9623, 0.975, 0.3758] 
2022-04-07 02:15:04.386396: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:15:05.504858: lr: 0.003727 
2022-04-07 02:15:05.507970: saving scheduled checkpoint file... 
2022-04-07 02:15:05.645087: saving checkpoint... 
2022-04-07 02:15:09.851201: done, saving took 4.34 seconds 
2022-04-07 02:15:09.869597: done 
2022-04-07 02:15:09.871291: This epoch took 128.550082 s
 
2022-04-07 02:15:09.873237: svmem(total=99932160000, available=84003725312, percent=15.9, used=12890972160, free=69347307520, active=23301816320, inactive=5018456064, buffers=56348672, cached=17637531648, shared=1218867200, slab=795697152) 
2022-04-07 02:15:09.875526: 
epoch:  333 
2022-04-07 02:16:59.966888: train loss : -0.6650 
2022-04-07 02:17:12.292183: validation loss: -0.6852 
2022-04-07 02:17:12.294127: Average global foreground Dice: [0.9016, 0.9671, 0.9833, 0.3852] 
2022-04-07 02:17:12.295928: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:17:13.011483: lr: 0.003707 
2022-04-07 02:17:13.023196: saving scheduled checkpoint file... 
2022-04-07 02:17:13.196650: saving checkpoint... 
2022-04-07 02:17:17.791525: done, saving took 4.76 seconds 
2022-04-07 02:17:17.832943: done 
2022-04-07 02:17:17.841330: This epoch took 127.963493 s
 
2022-04-07 02:17:17.851240: svmem(total=99932160000, available=84021878784, percent=15.9, used=12873011200, free=69349019648, active=23753244672, inactive=4570017792, buffers=56614912, cached=17653514240, shared=1218547712, slab=794710016) 
2022-04-07 02:17:17.859236: 
epoch:  334 
2022-04-07 02:19:07.091265: train loss : -0.6709 
2022-04-07 02:19:18.050081: validation loss: -0.6769 
2022-04-07 02:19:18.062865: Average global foreground Dice: [0.9, 0.9699, 0.9832, 0.4515] 
2022-04-07 02:19:18.067986: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:19:18.792406: lr: 0.003687 
2022-04-07 02:19:18.802997: saving scheduled checkpoint file... 
2022-04-07 02:19:18.941896: saving checkpoint... 
2022-04-07 02:19:23.663355: done, saving took 4.85 seconds 
2022-04-07 02:19:23.697820: done 
2022-04-07 02:19:23.803224: saving checkpoint... 
2022-04-07 02:19:28.318495: done, saving took 4.61 seconds 
2022-04-07 02:19:28.340336: This epoch took 130.471560 s
 
2022-04-07 02:19:28.343866: svmem(total=99932160000, available=83956883456, percent=16.0, used=12930961408, free=69332561920, active=23839031296, inactive=4501286912, buffers=56852480, cached=17611784192, shared=1225572352, slab=794050560) 
2022-04-07 02:19:28.347099: 
epoch:  335 
2022-04-07 02:21:17.295714: train loss : -0.6728 
2022-04-07 02:21:29.870480: validation loss: -0.6657 
2022-04-07 02:21:29.879663: Average global foreground Dice: [0.8863, 0.9747, 0.9779, 0.3919] 
2022-04-07 02:21:29.883946: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:21:31.222183: lr: 0.003667 
2022-04-07 02:21:31.223004: saving scheduled checkpoint file... 
2022-04-07 02:21:31.298240: saving checkpoint... 
2022-04-07 02:21:35.670991: done, saving took 4.45 seconds 
2022-04-07 02:21:35.689580: done 
2022-04-07 02:21:35.692101: This epoch took 127.343816 s
 
2022-04-07 02:21:35.695895: svmem(total=99932160000, available=83947356160, percent=16.0, used=12947361792, free=69239296000, active=23800639488, inactive=4630282240, buffers=57102336, cached=17688399872, shared=1218719744, slab=796147712) 
2022-04-07 02:21:35.697733: 
epoch:  336 
2022-04-07 02:23:22.192377: train loss : -0.6791 
2022-04-07 02:23:33.643325: validation loss: -0.6745 
2022-04-07 02:23:33.651829: Average global foreground Dice: [0.9036, 0.9629, 0.9789, 0.4169] 
2022-04-07 02:23:33.652588: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:23:34.636995: lr: 0.003647 
2022-04-07 02:23:34.643758: saving scheduled checkpoint file... 
2022-04-07 02:23:34.711298: saving checkpoint... 
2022-04-07 02:23:39.236417: done, saving took 4.59 seconds 
2022-04-07 02:23:39.271089: done 
2022-04-07 02:23:39.330144: saving checkpoint... 
2022-04-07 02:23:43.693441: done, saving took 4.41 seconds 
2022-04-07 02:23:43.706253: This epoch took 128.005706 s
 
2022-04-07 02:23:43.707193: svmem(total=99932160000, available=83791773696, percent=16.2, used=13094199296, free=69227724800, active=23636377600, inactive=4802158592, buffers=57364480, cached=17552871424, shared=1227468800, slab=794460160) 
2022-04-07 02:23:43.707829: 
epoch:  337 
2022-04-07 02:25:31.707387: train loss : -0.6628 
2022-04-07 02:25:43.049284: validation loss: -0.6745 
2022-04-07 02:25:43.067623: Average global foreground Dice: [0.8835, 0.9651, 0.98, 0.4152] 
2022-04-07 02:25:43.082625: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:25:43.793156: lr: 0.003627 
2022-04-07 02:25:43.803070: saving scheduled checkpoint file... 
2022-04-07 02:25:43.982237: saving checkpoint... 
2022-04-07 02:25:48.409969: done, saving took 4.60 seconds 
2022-04-07 02:25:48.430336: done 
2022-04-07 02:25:48.483014: saving checkpoint... 
2022-04-07 02:25:52.812731: done, saving took 4.38 seconds 
2022-04-07 02:25:52.828677: This epoch took 129.120278 s
 
2022-04-07 02:25:52.830483: svmem(total=99932160000, available=83849953280, percent=16.1, used=13044944896, free=69240049664, active=23802884096, inactive=4626874368, buffers=57643008, cached=17589522432, shared=1218461696, slab=794460160) 
2022-04-07 02:25:52.833860: 
epoch:  338 
2022-04-07 02:27:41.280069: train loss : -0.6781 
2022-04-07 02:27:52.000792: validation loss: -0.6787 
2022-04-07 02:27:52.008079: Average global foreground Dice: [0.9033, 0.9732, 0.9802, 0.4163] 
2022-04-07 02:27:52.014843: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:27:52.834226: lr: 0.003606 
2022-04-07 02:27:52.846404: saving scheduled checkpoint file... 
2022-04-07 02:27:52.906727: saving checkpoint... 
2022-04-07 02:27:57.510360: done, saving took 4.65 seconds 
2022-04-07 02:27:57.524069: done 
2022-04-07 02:27:57.575919: saving checkpoint... 
2022-04-07 02:28:01.994715: done, saving took 4.47 seconds 
2022-04-07 02:28:02.011942: This epoch took 129.176182 s
 
2022-04-07 02:28:02.014850: svmem(total=99932160000, available=83979251712, percent=16.0, used=12915593216, free=69240176640, active=23689830400, inactive=4742782976, buffers=57880576, cached=17718509568, shared=1218547712, slab=795004928) 
2022-04-07 02:28:02.017905: 
epoch:  339 
2022-04-07 02:29:50.666257: train loss : -0.6765 
2022-04-07 02:30:01.976955: validation loss: -0.6471 
2022-04-07 02:30:01.989428: Average global foreground Dice: [0.8977, 0.9699, 0.9823, 0.3936] 
2022-04-07 02:30:01.990201: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:30:02.871444: lr: 0.003586 
2022-04-07 02:30:02.885749: saving scheduled checkpoint file... 
2022-04-07 02:30:02.955098: saving checkpoint... 
2022-04-07 02:30:07.465349: done, saving took 4.57 seconds 
2022-04-07 02:30:07.478881: done 
2022-04-07 02:30:07.527686: saving checkpoint... 
2022-04-07 02:30:11.963542: done, saving took 4.48 seconds 
2022-04-07 02:30:12.247654: This epoch took 130.228567 s
 
2022-04-07 02:30:12.248854: svmem(total=99932160000, available=83820208128, percent=16.1, used=13074731008, free=69237084160, active=23799951360, inactive=4634460160, buffers=58146816, cached=17562198016, shared=1218461696, slab=793886720) 
2022-04-07 02:30:12.249752: 
epoch:  340 
2022-04-07 02:32:01.684755: train loss : -0.6871 
2022-04-07 02:32:12.805582: validation loss: -0.6816 
2022-04-07 02:32:12.810497: Average global foreground Dice: [0.8858, 0.9666, 0.9786, 0.4334] 
2022-04-07 02:32:12.816838: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:32:13.685072: lr: 0.003566 
2022-04-07 02:32:13.696896: saving scheduled checkpoint file... 
2022-04-07 02:32:13.814017: saving checkpoint... 
2022-04-07 02:32:18.557665: done, saving took 4.85 seconds 
2022-04-07 02:32:18.600136: done 
2022-04-07 02:32:18.690089: saving checkpoint... 
2022-04-07 02:32:23.178191: done, saving took 4.57 seconds 
2022-04-07 02:32:23.199186: This epoch took 130.948756 s
 
2022-04-07 02:32:23.200051: svmem(total=99932160000, available=83875340288, percent=16.1, used=13019435008, free=69307449344, active=23515750400, inactive=4846972928, buffers=58404864, cached=17546870784, shared=1218547712, slab=795750400) 
2022-04-07 02:32:23.200568: 
epoch:  341 
2022-04-07 02:34:13.401706: train loss : -0.6616 
2022-04-07 02:34:25.007837: validation loss: -0.6429 
2022-04-07 02:34:25.009551: Average global foreground Dice: [0.8818, 0.9628, 0.9759, 0.3931] 
2022-04-07 02:34:25.010238: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:34:26.301698: lr: 0.003546 
2022-04-07 02:34:26.304749: saving scheduled checkpoint file... 
2022-04-07 02:34:26.377349: saving checkpoint... 
2022-04-07 02:34:30.738902: done, saving took 4.43 seconds 
2022-04-07 02:34:30.753355: done 
2022-04-07 02:34:30.756703: This epoch took 127.555594 s
 
2022-04-07 02:34:30.759635: svmem(total=99932160000, available=83914432512, percent=16.0, used=12979994624, free=69239013376, active=23854133248, inactive=4575895552, buffers=58646528, cached=17654505472, shared=1218891776, slab=795492352) 
2022-04-07 02:34:30.762865: 
epoch:  342 
2022-04-07 02:36:18.679765: train loss : -0.6761 
2022-04-07 02:36:30.181012: validation loss: -0.6788 
2022-04-07 02:36:30.183669: Average global foreground Dice: [0.8855, 0.9662, 0.978, 0.4199] 
2022-04-07 02:36:30.187488: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:36:30.950702: lr: 0.003526 
2022-04-07 02:36:30.971173: saving scheduled checkpoint file... 
2022-04-07 02:36:31.082257: saving checkpoint... 
2022-04-07 02:36:34.985496: done, saving took 3.99 seconds 
2022-04-07 02:36:35.010182: done 
2022-04-07 02:36:35.017969: This epoch took 124.252672 s
 
2022-04-07 02:36:35.026669: svmem(total=99932160000, available=83983335424, percent=16.0, used=12935888896, free=69225390080, active=23874572288, inactive=4564123648, buffers=58900480, cached=17711980544, shared=1194074112, slab=794169344) 
2022-04-07 02:36:35.034771: 
epoch:  343 
2022-04-07 02:38:23.434594: train loss : -0.6805 
2022-04-07 02:38:34.485004: validation loss: -0.6896 
2022-04-07 02:38:34.493687: Average global foreground Dice: [0.8894, 0.9657, 0.9814, 0.4259] 
2022-04-07 02:38:34.499779: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:38:35.457868: lr: 0.003505 
2022-04-07 02:38:35.470590: saving scheduled checkpoint file... 
2022-04-07 02:38:35.579031: saving checkpoint... 
2022-04-07 02:38:40.091434: done, saving took 4.61 seconds 
2022-04-07 02:38:40.121753: done 
2022-04-07 02:38:40.130718: This epoch took 125.091521 s
 
2022-04-07 02:38:40.139532: svmem(total=99932160000, available=84040478720, percent=15.9, used=12878245888, free=69231710208, active=23796072448, inactive=4642160640, buffers=59158528, cached=17763045376, shared=1193988096, slab=794521600) 
2022-04-07 02:38:40.146550: 
epoch:  344 
2022-04-07 02:40:31.409248: train loss : -0.6762 
2022-04-07 02:40:43.238487: validation loss: -0.6623 
2022-04-07 02:40:43.247138: Average global foreground Dice: [0.8583, 0.9679, 0.9833, 0.427] 
2022-04-07 02:40:43.252832: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:40:44.060667: lr: 0.003485 
2022-04-07 02:40:44.069199: saving scheduled checkpoint file... 
2022-04-07 02:40:44.145108: saving checkpoint... 
2022-04-07 02:40:48.607352: done, saving took 4.53 seconds 
2022-04-07 02:40:48.630863: done 
2022-04-07 02:40:48.636525: This epoch took 128.483202 s
 
2022-04-07 02:40:48.641839: svmem(total=99932160000, available=84026544128, percent=15.9, used=12878143488, free=69218426880, active=23879032832, inactive=4572020736, buffers=59400192, cached=17776189440, shared=1207865344, slab=795119616) 
2022-04-07 02:40:48.649348: 
epoch:  345 
2022-04-07 02:42:37.906674: train loss : -0.6797 
2022-04-07 02:42:49.469792: validation loss: -0.6789 
2022-04-07 02:42:49.473323: Average global foreground Dice: [0.9043, 0.9733, 0.9823, 0.4045] 
2022-04-07 02:42:49.475865: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:42:50.106641: lr: 0.003465 
2022-04-07 02:42:50.112393: saving scheduled checkpoint file... 
2022-04-07 02:42:50.251047: saving checkpoint... 
2022-04-07 02:42:54.809391: done, saving took 4.69 seconds 
2022-04-07 02:42:54.828305: done 
2022-04-07 02:42:54.901571: saving checkpoint... 
2022-04-07 02:42:59.302763: done, saving took 4.47 seconds 
2022-04-07 02:42:59.322946: This epoch took 130.669689 s
 
2022-04-07 02:42:59.325620: svmem(total=99932160000, available=83986075648, percent=16.0, used=12908257280, free=69296971776, active=23686451200, inactive=4681158656, buffers=59645952, cached=17667284992, shared=1218461696, slab=794300416) 
2022-04-07 02:42:59.326507: 
epoch:  346 
2022-04-07 02:44:48.405437: train loss : -0.6732 
2022-04-07 02:45:00.585632: validation loss: -0.6658 
2022-04-07 02:45:00.602032: Average global foreground Dice: [0.8615, 0.9688, 0.9854, 0.4339] 
2022-04-07 02:45:00.620927: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:45:01.208561: lr: 0.003445 
2022-04-07 02:45:01.217953: saving scheduled checkpoint file... 
2022-04-07 02:45:01.331766: saving checkpoint... 
2022-04-07 02:45:05.886603: done, saving took 4.66 seconds 
2022-04-07 02:45:05.932238: done 
2022-04-07 02:45:06.143982: saving checkpoint... 
2022-04-07 02:45:10.641858: done, saving took 4.70 seconds 
2022-04-07 02:45:10.667761: This epoch took 131.340395 s
 
2022-04-07 02:45:10.671516: svmem(total=99932160000, available=83998105600, percent=15.9, used=12914364416, free=69317705728, active=23655419904, inactive=4694949888, buffers=59920384, cached=17640169472, shared=1200775168, slab=794103808) 
2022-04-07 02:45:10.674856: 
epoch:  347 
2022-04-07 02:47:00.791475: train loss : -0.6842 
2022-04-07 02:47:12.352580: validation loss: -0.6704 
2022-04-07 02:47:12.354142: Average global foreground Dice: [0.9118, 0.9728, 0.983, 0.3782] 
2022-04-07 02:47:12.354896: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:47:13.533725: lr: 0.003424 
2022-04-07 02:47:13.539963: saving scheduled checkpoint file... 
2022-04-07 02:47:13.624067: saving checkpoint... 
2022-04-07 02:47:17.945263: done, saving took 4.40 seconds 
2022-04-07 02:47:17.968732: done 
2022-04-07 02:47:17.989506: This epoch took 127.310052 s
 
2022-04-07 02:47:17.995087: svmem(total=99932160000, available=83892609024, percent=16.1, used=12987514880, free=69209317376, active=23682752512, inactive=4774379520, buffers=60170240, cached=17675157504, shared=1232769024, slab=795394048) 
2022-04-07 02:47:18.001525: 
epoch:  348 
2022-04-07 02:49:05.928344: train loss : -0.6831 
2022-04-07 02:49:18.274244: validation loss: -0.6869 
2022-04-07 02:49:18.278681: Average global foreground Dice: [0.8988, 0.9701, 0.9813, 0.4006] 
2022-04-07 02:49:18.282788: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:49:19.148977: lr: 0.003404 
2022-04-07 02:49:19.165538: saving scheduled checkpoint file... 
2022-04-07 02:49:19.259552: saving checkpoint... 
2022-04-07 02:49:23.720196: done, saving took 4.55 seconds 
2022-04-07 02:49:23.733865: done 
2022-04-07 02:49:23.785424: saving checkpoint... 
2022-04-07 02:49:28.074609: done, saving took 4.34 seconds 
2022-04-07 02:49:28.088692: This epoch took 130.080974 s
 
2022-04-07 02:49:28.089525: svmem(total=99932160000, available=83846373376, percent=16.1, used=13048258560, free=69219061760, active=23882862592, inactive=4564127744, buffers=60407808, cached=17604431872, shared=1218473984, slab=794791936) 
2022-04-07 02:49:28.090201: 
epoch:  349 
2022-04-07 02:51:17.672940: train loss : -0.6748 
2022-04-07 02:51:29.080026: validation loss: -0.6593 
2022-04-07 02:51:29.091212: Average global foreground Dice: [0.8632, 0.9652, 0.9804, 0.4326] 
2022-04-07 02:51:29.095720: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:51:29.886345: lr: 0.003384 
2022-04-07 02:51:29.887090: saving scheduled checkpoint file... 
2022-04-07 02:51:29.938273: saving checkpoint... 
2022-04-07 02:51:34.358986: done, saving took 4.47 seconds 
2022-04-07 02:51:34.382829: done 
2022-04-07 02:51:34.387254: This epoch took 126.296525 s
 
2022-04-07 02:51:34.408540: svmem(total=99932160000, available=83846447104, percent=16.1, used=13048066048, free=69223673856, active=23879360512, inactive=4569063424, buffers=60657664, cached=17599762432, shared=1218547712, slab=795152384) 
2022-04-07 02:51:34.414972: 
epoch:  350 
2022-04-07 02:53:24.769400: train loss : -0.6785 
2022-04-07 02:53:35.729968: validation loss: -0.6851 
2022-04-07 02:53:35.732799: Average global foreground Dice: [0.8954, 0.9678, 0.981, 0.4195] 
2022-04-07 02:53:35.734303: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:53:36.674644: lr: 0.003364 
2022-04-07 02:53:36.679079: saving scheduled checkpoint file... 
2022-04-07 02:53:36.733990: saving checkpoint... 
2022-04-07 02:53:41.227368: done, saving took 4.54 seconds 
2022-04-07 02:53:41.264364: done 
2022-04-07 02:53:41.329392: saving checkpoint... 
2022-04-07 02:53:45.801723: done, saving took 4.52 seconds 
2022-04-07 02:53:45.828733: This epoch took 131.409756 s
 
2022-04-07 02:53:45.832894: svmem(total=99932160000, available=83956912128, percent=16.0, used=12937691136, free=69220155392, active=23845351424, inactive=4604387328, buffers=60923904, cached=17713389568, shared=1218461696, slab=795435008) 
2022-04-07 02:53:45.837193: 
epoch:  351 
2022-04-07 02:55:34.670431: train loss : -0.6842 
2022-04-07 02:55:46.827610: validation loss: -0.7016 
2022-04-07 02:55:46.841723: Average global foreground Dice: [0.8979, 0.9707, 0.9797, 0.4299] 
2022-04-07 02:55:46.845616: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:55:47.759066: lr: 0.003343 
2022-04-07 02:55:47.788168: saving scheduled checkpoint file... 
2022-04-07 02:55:47.881581: saving checkpoint... 
2022-04-07 02:55:52.342986: done, saving took 4.54 seconds 
2022-04-07 02:55:52.357505: done 
2022-04-07 02:55:52.405686: saving checkpoint... 
2022-04-07 02:55:56.819626: done, saving took 4.46 seconds 
2022-04-07 02:55:56.832590: This epoch took 130.991987 s
 
2022-04-07 02:55:56.833587: svmem(total=99932160000, available=83878293504, percent=16.1, used=13016059904, free=69224660992, active=23795585024, inactive=4651794432, buffers=61194240, cached=17630244864, shared=1218633728, slab=796188672) 
2022-04-07 02:55:56.834216: 
epoch:  352 
2022-04-07 02:57:44.723885: train loss : -0.6777 
2022-04-07 02:57:56.496247: validation loss: -0.6833 
2022-04-07 02:57:56.497791: Average global foreground Dice: [0.8964, 0.9706, 0.9841, 0.4293] 
2022-04-07 02:57:56.498522: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 02:57:57.541469: lr: 0.003323 
2022-04-07 02:57:57.550576: saving scheduled checkpoint file... 
2022-04-07 02:57:57.701698: saving checkpoint... 
2022-04-07 02:58:01.883516: done, saving took 4.32 seconds 
2022-04-07 02:58:01.914305: done 
2022-04-07 02:58:01.994352: saving checkpoint... 
2022-04-07 02:58:06.452721: done, saving took 4.53 seconds 
2022-04-07 02:58:06.472134: This epoch took 129.637293 s
 
2022-04-07 02:58:06.475184: svmem(total=99932160000, available=83928920064, percent=16.0, used=12965212160, free=69277929472, active=23836213248, inactive=4554919936, buffers=61440000, cached=17627578368, shared=1218646016, slab=795852800) 
2022-04-07 02:58:06.478303: 
epoch:  353 
2022-04-07 02:59:54.102364: train loss : -0.6821 
2022-04-07 03:00:04.788432: validation loss: -0.6740 
2022-04-07 03:00:04.790134: Average global foreground Dice: [0.8989, 0.9647, 0.9823, 0.4257] 
2022-04-07 03:00:04.790923: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:00:05.743495: lr: 0.003303 
2022-04-07 03:00:05.750238: saving scheduled checkpoint file... 
2022-04-07 03:00:05.893058: saving checkpoint... 
2022-04-07 03:00:09.902360: done, saving took 4.15 seconds 
2022-04-07 03:00:09.953696: done 
2022-04-07 03:00:10.024281: saving checkpoint... 
2022-04-07 03:00:14.606463: done, saving took 4.63 seconds 
2022-04-07 03:00:14.630168: This epoch took 128.148432 s
 
2022-04-07 03:00:14.632685: svmem(total=99932160000, available=84022661120, percent=15.9, used=12857942016, free=69203898368, active=23797923840, inactive=4668510208, buffers=61706240, cached=17808613376, shared=1232437248, slab=795488256) 
2022-04-07 03:00:14.633666: 
epoch:  354 
2022-04-07 03:02:05.060598: train loss : -0.6848 
2022-04-07 03:02:17.136477: validation loss: -0.6945 
2022-04-07 03:02:17.141309: Average global foreground Dice: [0.8966, 0.9728, 0.9844, 0.4333] 
2022-04-07 03:02:17.142036: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:02:18.143921: lr: 0.003282 
2022-04-07 03:02:18.147343: saving scheduled checkpoint file... 
2022-04-07 03:02:18.201289: saving checkpoint... 
2022-04-07 03:02:22.957601: done, saving took 4.81 seconds 
2022-04-07 03:02:22.986403: done 
2022-04-07 03:02:23.039871: saving checkpoint... 
2022-04-07 03:02:27.477763: done, saving took 4.49 seconds 
2022-04-07 03:02:27.490864: This epoch took 132.856539 s
 
2022-04-07 03:02:27.491764: svmem(total=99932160000, available=83927052288, percent=16.0, used=12967411712, free=69242707968, active=23726292992, inactive=4702138368, buffers=61956096, cached=17660084224, shared=1218473984, slab=796016640) 
2022-04-07 03:02:27.492405: 
epoch:  355 
2022-04-07 03:04:17.756202: train loss : -0.6879 
2022-04-07 03:04:29.317695: validation loss: -0.7009 
2022-04-07 03:04:29.320712: Average global foreground Dice: [0.9138, 0.9747, 0.987, 0.4373] 
2022-04-07 03:04:29.326708: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:04:30.265097: lr: 0.003262 
2022-04-07 03:04:30.267888: saving scheduled checkpoint file... 
2022-04-07 03:04:30.369623: saving checkpoint... 
2022-04-07 03:04:34.836057: done, saving took 4.57 seconds 
2022-04-07 03:04:34.871786: done 
2022-04-07 03:04:34.963053: saving checkpoint... 
2022-04-07 03:04:39.399332: done, saving took 4.52 seconds 
2022-04-07 03:04:39.416567: This epoch took 131.923578 s
 
2022-04-07 03:04:39.417562: svmem(total=99932160000, available=83970146304, percent=16.0, used=12924366848, free=69309947904, active=23812714496, inactive=4545249280, buffers=62205952, cached=17635639296, shared=1218498560, slab=796135424) 
2022-04-07 03:04:39.418223: 
epoch:  356 
2022-04-07 03:06:28.776355: train loss : -0.6660 
2022-04-07 03:06:40.960585: validation loss: -0.6566 
2022-04-07 03:06:41.004156: Average global foreground Dice: [0.8879, 0.9652, 0.9794, 0.3977] 
2022-04-07 03:06:41.013818: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:06:41.843394: lr: 0.003241 
2022-04-07 03:06:41.847625: saving scheduled checkpoint file... 
2022-04-07 03:06:41.983381: saving checkpoint... 
2022-04-07 03:06:46.464422: done, saving took 4.61 seconds 
2022-04-07 03:06:46.492233: done 
2022-04-07 03:06:46.497925: This epoch took 127.079103 s
 
2022-04-07 03:06:46.503847: svmem(total=99932160000, available=84039786496, percent=15.9, used=12854321152, free=69315977216, active=23677997056, inactive=4675620864, buffers=62484480, cached=17699377152, shared=1218646016, slab=794525696) 
2022-04-07 03:06:46.511333: 
epoch:  357 
2022-04-07 03:08:35.519023: train loss : -0.6707 
2022-04-07 03:08:47.114039: validation loss: -0.6736 
2022-04-07 03:08:47.117698: Average global foreground Dice: [0.8963, 0.9645, 0.9808, 0.4017] 
2022-04-07 03:08:47.120673: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:08:47.942223: lr: 0.003221 
2022-04-07 03:08:47.948374: saving scheduled checkpoint file... 
2022-04-07 03:08:48.082079: saving checkpoint... 
2022-04-07 03:08:52.137162: done, saving took 4.18 seconds 
2022-04-07 03:08:52.183105: done 
2022-04-07 03:08:52.194463: This epoch took 125.675897 s
 
2022-04-07 03:08:52.206405: svmem(total=99932160000, available=84054736896, percent=15.9, used=12863635456, free=69316124672, active=23804096512, inactive=4550012928, buffers=62717952, cached=17689681920, shared=1194000384, slab=794411008) 
2022-04-07 03:08:52.212979: 
epoch:  358 
2022-04-07 03:10:41.723804: train loss : -0.6933 
2022-04-07 03:10:54.554973: validation loss: -0.6827 
2022-04-07 03:10:54.575664: Average global foreground Dice: [0.8987, 0.9707, 0.9836, 0.4045] 
2022-04-07 03:10:54.579566: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:10:55.554363: lr: 0.003201 
2022-04-07 03:10:55.555140: saving scheduled checkpoint file... 
2022-04-07 03:10:55.741145: saving checkpoint... 
2022-04-07 03:11:00.111135: done, saving took 4.56 seconds 
2022-04-07 03:11:00.132453: done 
2022-04-07 03:11:00.134849: This epoch took 127.918378 s
 
2022-04-07 03:11:00.137394: svmem(total=99932160000, available=84011204608, percent=15.9, used=12883607552, free=69306417152, active=23787614208, inactive=4571414528, buffers=62992384, cached=17679142912, shared=1218473984, slab=795201536) 
2022-04-07 03:11:00.139567: 
epoch:  359 
2022-04-07 03:12:49.515163: train loss : -0.6745 
2022-04-07 03:13:01.665833: validation loss: -0.6464 
2022-04-07 03:13:01.689389: Average global foreground Dice: [0.9022, 0.9664, 0.9755, 0.4027] 
2022-04-07 03:13:01.708182: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:13:02.370355: lr: 0.00318 
2022-04-07 03:13:02.375049: saving scheduled checkpoint file... 
2022-04-07 03:13:02.512174: saving checkpoint... 
2022-04-07 03:13:07.506673: done, saving took 5.13 seconds 
2022-04-07 03:13:07.531648: done 
2022-04-07 03:13:07.538478: This epoch took 127.397228 s
 
2022-04-07 03:13:07.542670: svmem(total=99932160000, available=84042256384, percent=15.9, used=12850438144, free=69378011136, active=23717171200, inactive=4571623424, buffers=63221760, cached=17640488960, shared=1218560000, slab=796667904) 
2022-04-07 03:13:07.547476: 
epoch:  360 
2022-04-07 03:14:56.064327: train loss : -0.6696 
2022-04-07 03:15:08.372934: validation loss: -0.6666 
2022-04-07 03:15:08.377433: Average global foreground Dice: [0.9022, 0.9694, 0.9769, 0.4062] 
2022-04-07 03:15:08.383274: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:15:09.119144: lr: 0.00316 
2022-04-07 03:15:09.125095: saving scheduled checkpoint file... 
2022-04-07 03:15:09.212885: saving checkpoint... 
2022-04-07 03:15:13.659578: done, saving took 4.53 seconds 
2022-04-07 03:15:13.685184: done 
2022-04-07 03:15:13.690519: This epoch took 126.136156 s
 
2022-04-07 03:15:13.695518: svmem(total=99932160000, available=83958611968, percent=16.0, used=12935622656, free=69203423232, active=23787155456, inactive=4674494464, buffers=63488000, cached=17729626112, shared=1218473984, slab=796225536) 
2022-04-07 03:15:13.702767: 
epoch:  361 
2022-04-07 03:17:02.923375: train loss : -0.6413 
2022-04-07 03:17:14.572233: validation loss: -0.6196 
2022-04-07 03:17:14.582134: Average global foreground Dice: [0.8792, 0.9573, 0.9709, 0.4139] 
2022-04-07 03:17:14.588721: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:17:15.671165: lr: 0.003139 
2022-04-07 03:17:15.678727: saving scheduled checkpoint file... 
2022-04-07 03:17:15.789792: saving checkpoint... 
2022-04-07 03:17:20.253884: done, saving took 4.57 seconds 
2022-04-07 03:17:20.264634: done 
2022-04-07 03:17:20.266364: This epoch took 126.559133 s
 
2022-04-07 03:17:20.268739: svmem(total=99932160000, available=84081414144, percent=15.9, used=12812980224, free=69209038848, active=23910789120, inactive=4553228288, buffers=63746048, cached=17846394880, shared=1218473984, slab=796176384) 
2022-04-07 03:17:20.270154: 
epoch:  362 
2022-04-07 03:19:09.247852: train loss : -0.6565 
2022-04-07 03:19:20.210042: validation loss: -0.6738 
2022-04-07 03:19:20.225050: Average global foreground Dice: [0.8905, 0.9667, 0.9804, 0.4128] 
2022-04-07 03:19:20.226674: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:19:21.126585: lr: 0.003119 
2022-04-07 03:19:21.140324: saving scheduled checkpoint file... 
2022-04-07 03:19:21.255767: saving checkpoint... 
2022-04-07 03:19:25.819938: done, saving took 4.67 seconds 
2022-04-07 03:19:25.867015: done 
2022-04-07 03:19:25.875589: This epoch took 125.603803 s
 
2022-04-07 03:19:25.888068: svmem(total=99932160000, available=84039041024, percent=15.9, used=12841709568, free=69262364672, active=23829037056, inactive=4579610624, buffers=63975424, cached=17764110336, shared=1231704064, slab=796053504) 
2022-04-07 03:19:25.900655: 
epoch:  363 
2022-04-07 03:21:15.349355: train loss : -0.6722 
2022-04-07 03:21:27.042258: validation loss: -0.6772 
2022-04-07 03:21:27.061848: Average global foreground Dice: [0.9054, 0.9623, 0.9802, 0.4479] 
2022-04-07 03:21:27.080636: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:21:27.830037: lr: 0.003098 
2022-04-07 03:21:27.851478: saving scheduled checkpoint file... 
2022-04-07 03:21:27.992081: saving checkpoint... 
2022-04-07 03:21:32.751003: done, saving took 4.88 seconds 
2022-04-07 03:21:32.769619: done 
2022-04-07 03:21:32.770356: This epoch took 126.861823 s
 
2022-04-07 03:21:32.771349: svmem(total=99932160000, available=84021153792, percent=15.9, used=12873633792, free=69303599104, active=23794442240, inactive=4566347776, buffers=64225280, cached=17690701824, shared=1218236416, slab=795488256) 
2022-04-07 03:21:32.771969: 
epoch:  364 
2022-04-07 03:23:22.674842: train loss : -0.6652 
2022-04-07 03:23:34.328964: validation loss: -0.6675 
2022-04-07 03:23:34.338393: Average global foreground Dice: [0.8742, 0.966, 0.9815, 0.438] 
2022-04-07 03:23:34.339526: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:23:35.397447: lr: 0.003078 
2022-04-07 03:23:35.416408: saving scheduled checkpoint file... 
2022-04-07 03:23:35.564759: saving checkpoint... 
2022-04-07 03:23:40.109615: done, saving took 4.68 seconds 
2022-04-07 03:23:40.139561: done 
2022-04-07 03:23:40.147411: This epoch took 127.374812 s
 
2022-04-07 03:23:40.154326: svmem(total=99932160000, available=84082446336, percent=15.9, used=12811853824, free=69306773504, active=23666360320, inactive=4697645056, buffers=64487424, cached=17749045248, shared=1218646016, slab=795922432) 
2022-04-07 03:23:40.162765: 
epoch:  365 
2022-04-07 03:25:29.732898: train loss : -0.6741 
2022-04-07 03:25:41.191433: validation loss: -0.6674 
2022-04-07 03:25:41.202115: Average global foreground Dice: [0.8887, 0.9674, 0.9839, 0.4131] 
2022-04-07 03:25:41.211561: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:25:42.049951: lr: 0.003057 
2022-04-07 03:25:42.051381: saving scheduled checkpoint file... 
2022-04-07 03:25:42.214914: saving checkpoint... 
2022-04-07 03:25:46.941475: done, saving took 4.89 seconds 
2022-04-07 03:25:46.969260: done 
2022-04-07 03:25:46.971468: This epoch took 126.805242 s
 
2022-04-07 03:25:46.973907: svmem(total=99932160000, available=84073562112, percent=15.9, used=12820619264, free=69346254848, active=23760076800, inactive=4564135936, buffers=64761856, cached=17700524032, shared=1218904064, slab=795049984) 
2022-04-07 03:25:46.975741: 
epoch:  366 
2022-04-07 03:27:35.941176: train loss : -0.6797 
2022-04-07 03:27:46.689741: validation loss: -0.6671 
2022-04-07 03:27:46.693398: Average global foreground Dice: [0.9032, 0.9676, 0.9782, 0.3823] 
2022-04-07 03:27:46.695348: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:27:47.564992: lr: 0.003037 
2022-04-07 03:27:47.580602: saving scheduled checkpoint file... 
2022-04-07 03:27:47.702509: saving checkpoint... 
2022-04-07 03:27:52.346445: done, saving took 4.75 seconds 
2022-04-07 03:27:52.376200: done 
2022-04-07 03:27:52.386061: This epoch took 125.407982 s
 
2022-04-07 03:27:52.397342: svmem(total=99932160000, available=84093640704, percent=15.8, used=12800270336, free=69193265152, active=23902380032, inactive=4571848704, buffers=65855488, cached=17872769024, shared=1218473984, slab=795754496) 
2022-04-07 03:27:52.404918: 
epoch:  367 
2022-04-07 03:29:41.324554: train loss : -0.6531 
2022-04-07 03:29:53.192700: validation loss: -0.6641 
2022-04-07 03:29:53.208163: Average global foreground Dice: [0.891, 0.9617, 0.9746, 0.3963] 
2022-04-07 03:29:53.224373: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:29:53.803328: lr: 0.003016 
2022-04-07 03:29:53.810219: saving scheduled checkpoint file... 
2022-04-07 03:29:53.911346: saving checkpoint... 
2022-04-07 03:29:58.626381: done, saving took 4.81 seconds 
2022-04-07 03:29:58.657150: done 
2022-04-07 03:29:58.667717: This epoch took 126.253057 s
 
2022-04-07 03:29:58.676278: svmem(total=99932160000, available=83985731584, percent=16.0, used=12909289472, free=69193465856, active=23881408512, inactive=4594376704, buffers=66088960, cached=17763315712, shared=1218478080, slab=797077504) 
2022-04-07 03:29:58.684391: 
epoch:  368 
2022-04-07 03:31:48.921778: train loss : -0.6693 
2022-04-07 03:32:00.199010: validation loss: -0.6647 
2022-04-07 03:32:00.216494: Average global foreground Dice: [0.8815, 0.9663, 0.9791, 0.3971] 
2022-04-07 03:32:00.239256: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:32:00.850342: lr: 0.002996 
2022-04-07 03:32:00.860608: saving scheduled checkpoint file... 
2022-04-07 03:32:00.951694: saving checkpoint... 
2022-04-07 03:32:05.414581: done, saving took 4.54 seconds 
2022-04-07 03:32:05.479868: done 
2022-04-07 03:32:05.502209: This epoch took 126.809916 s
 
2022-04-07 03:32:05.519802: svmem(total=99932160000, available=84039524352, percent=15.9, used=12854185984, free=69258895360, active=23705960448, inactive=4705304576, buffers=66351104, cached=17752727552, shared=1218588672, slab=796626944) 
2022-04-07 03:32:05.537294: 
epoch:  369 
2022-04-07 03:33:56.153925: train loss : -0.6532 
2022-04-07 03:34:08.270221: validation loss: -0.6504 
2022-04-07 03:34:08.280229: Average global foreground Dice: [0.8907, 0.9678, 0.9768, 0.3998] 
2022-04-07 03:34:08.280993: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:34:09.083541: lr: 0.002975 
2022-04-07 03:34:09.096932: saving scheduled checkpoint file... 
2022-04-07 03:34:09.321903: saving checkpoint... 
2022-04-07 03:34:13.791563: done, saving took 4.69 seconds 
2022-04-07 03:34:13.816046: done 
2022-04-07 03:34:13.819925: This epoch took 128.268762 s
 
2022-04-07 03:34:13.822426: svmem(total=99932160000, available=84105723904, percent=15.8, used=12811538432, free=69283590144, active=23751032832, inactive=4634394624, buffers=66588672, cached=17770442752, shared=1194287104, slab=797982720) 
2022-04-07 03:34:13.825537: 
epoch:  370 
2022-04-07 03:36:04.624646: train loss : -0.6383 
2022-04-07 03:36:16.473262: validation loss: -0.6547 
2022-04-07 03:36:16.475567: Average global foreground Dice: [0.8702, 0.9631, 0.9798, 0.4017] 
2022-04-07 03:36:16.477520: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:36:17.407048: lr: 0.002954 
2022-04-07 03:36:17.407785: saving scheduled checkpoint file... 
2022-04-07 03:36:17.542168: saving checkpoint... 
2022-04-07 03:36:22.050685: done, saving took 4.64 seconds 
2022-04-07 03:36:22.069504: done 
2022-04-07 03:36:22.070247: This epoch took 128.241872 s
 
2022-04-07 03:36:22.071043: svmem(total=99932160000, available=84046548992, percent=15.9, used=12841725952, free=69290008576, active=23787577344, inactive=4591783936, buffers=66863104, cached=17733562368, shared=1224040448, slab=797933568) 
2022-04-07 03:36:22.071543: 
epoch:  371 
2022-04-07 03:38:11.346626: train loss : -0.6536 
2022-04-07 03:38:22.577253: validation loss: -0.6991 
2022-04-07 03:38:22.581322: Average global foreground Dice: [0.9094, 0.9696, 0.9826, 0.423] 
2022-04-07 03:38:22.585157: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:38:23.367625: lr: 0.002934 
2022-04-07 03:38:23.368332: saving scheduled checkpoint file... 
2022-04-07 03:38:23.502883: saving checkpoint... 
2022-04-07 03:38:28.435666: done, saving took 5.07 seconds 
2022-04-07 03:38:28.486234: done 
2022-04-07 03:38:28.499738: This epoch took 126.427722 s
 
2022-04-07 03:38:28.514515: svmem(total=99932160000, available=84058198016, percent=15.9, used=12835655680, free=69332365312, active=23737733120, inactive=4602454016, buffers=67104768, cached=17697034240, shared=1218330624, slab=796540928) 
2022-04-07 03:38:28.529755: 
epoch:  372 
2022-04-07 03:40:17.768580: train loss : -0.6704 
2022-04-07 03:40:29.316617: validation loss: -0.6731 
2022-04-07 03:40:29.325086: Average global foreground Dice: [0.8939, 0.9613, 0.9822, 0.4102] 
2022-04-07 03:40:29.334013: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:40:30.190375: lr: 0.002913 
2022-04-07 03:40:30.196630: saving scheduled checkpoint file... 
2022-04-07 03:40:30.336762: saving checkpoint... 
2022-04-07 03:40:34.867451: done, saving took 4.66 seconds 
2022-04-07 03:40:34.885749: done 
2022-04-07 03:40:34.890646: This epoch took 126.348681 s
 
2022-04-07 03:40:34.894373: svmem(total=99932160000, available=83933814784, percent=16.0, used=12960301056, free=69190127616, active=23892021248, inactive=4584984576, buffers=67354624, cached=17714376704, shared=1218523136, slab=798261248) 
2022-04-07 03:40:34.897711: 
epoch:  373 
2022-04-07 03:42:23.926587: train loss : -0.6747 
2022-04-07 03:42:35.213634: validation loss: -0.6899 
2022-04-07 03:42:35.218373: Average global foreground Dice: [0.9023, 0.9694, 0.9836, 0.4292] 
2022-04-07 03:42:35.223909: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:42:36.171431: lr: 0.002892 
2022-04-07 03:42:36.174414: saving scheduled checkpoint file... 
2022-04-07 03:42:36.355879: saving checkpoint... 
2022-04-07 03:42:40.882513: done, saving took 4.71 seconds 
2022-04-07 03:42:40.911557: done 
2022-04-07 03:42:40.916693: This epoch took 126.015389 s
 
2022-04-07 03:42:40.922487: svmem(total=99932160000, available=83942559744, percent=16.0, used=12950872064, free=69184278528, active=23928356864, inactive=4556812288, buffers=67645440, cached=17729363968, shared=1218912256, slab=797614080) 
2022-04-07 03:42:40.930606: 
epoch:  374 
2022-04-07 03:44:29.363590: train loss : -0.6906 
2022-04-07 03:44:40.140357: validation loss: -0.6975 
2022-04-07 03:44:40.151098: Average global foreground Dice: [0.9099, 0.9705, 0.9847, 0.414] 
2022-04-07 03:44:40.154962: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:44:41.008327: lr: 0.002872 
2022-04-07 03:44:41.020405: saving scheduled checkpoint file... 
2022-04-07 03:44:41.135377: saving checkpoint... 
2022-04-07 03:44:45.759776: done, saving took 4.73 seconds 
2022-04-07 03:44:45.791581: done 
2022-04-07 03:44:45.799514: This epoch took 124.860553 s
 
2022-04-07 03:44:45.808340: svmem(total=99932160000, available=84143759360, percent=15.8, used=12774387712, free=69257330688, active=23860215808, inactive=4552339456, buffers=67887104, cached=17832554496, shared=1194115072, slab=796704768) 
2022-04-07 03:44:45.816988: 
epoch:  375 
2022-04-07 03:46:35.189608: train loss : -0.6822 
2022-04-07 03:46:46.873369: validation loss: -0.6805 
2022-04-07 03:46:46.889990: Average global foreground Dice: [0.8974, 0.9701, 0.9821, 0.4163] 
2022-04-07 03:46:46.891789: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:46:47.736786: lr: 0.002851 
2022-04-07 03:46:47.749861: saving scheduled checkpoint file... 
2022-04-07 03:46:47.870981: saving checkpoint... 
2022-04-07 03:46:52.528603: done, saving took 4.76 seconds 
2022-04-07 03:46:52.552647: done 
2022-04-07 03:46:52.556157: This epoch took 126.730152 s
 
2022-04-07 03:46:52.559891: svmem(total=99932160000, available=84055474176, percent=15.9, used=12862214144, free=69288390656, active=23879487488, inactive=4500660224, buffers=68153344, cached=17713401856, shared=1194201088, slab=797675520) 
2022-04-07 03:46:52.562018: 
epoch:  376 
2022-04-07 03:48:42.087362: train loss : -0.6904 
2022-04-07 03:48:53.940030: validation loss: -0.7061 
2022-04-07 03:48:53.943032: Average global foreground Dice: [0.9181, 0.9701, 0.9839, 0.4137] 
2022-04-07 03:48:53.943841: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:48:54.964528: lr: 0.00283 
2022-04-07 03:48:54.974652: saving scheduled checkpoint file... 
2022-04-07 03:48:55.113166: saving checkpoint... 
2022-04-07 03:48:59.563640: done, saving took 4.58 seconds 
2022-04-07 03:49:00.747061: done 
2022-04-07 03:49:00.747836: This epoch took 128.185133 s
 
2022-04-07 03:49:00.748728: svmem(total=99932160000, available=84009185280, percent=15.9, used=12872675328, free=69165752320, active=23918497792, inactive=4581814272, buffers=68390912, cached=17825341440, shared=1230397440, slab=797990912) 
2022-04-07 03:49:00.749355: 
epoch:  377 
2022-04-07 03:50:50.539203: train loss : -0.6862 
2022-04-07 03:51:01.062177: validation loss: -0.6771 
2022-04-07 03:51:01.076785: Average global foreground Dice: [0.8888, 0.9705, 0.981, 0.4333] 
2022-04-07 03:51:01.079784: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:51:02.381331: lr: 0.00281 
2022-04-07 03:51:02.387742: saving scheduled checkpoint file... 
2022-04-07 03:51:02.493196: saving checkpoint... 
2022-04-07 03:51:06.549199: done, saving took 4.15 seconds 
2022-04-07 03:51:06.589442: done 
2022-04-07 03:51:06.601792: This epoch took 125.851815 s
 
2022-04-07 03:51:06.614167: svmem(total=99932160000, available=83985498112, percent=16.0, used=12908150784, free=69174755328, active=23874191360, inactive=4613939200, buffers=68632576, cached=17780621312, shared=1218502656, slab=798167040) 
2022-04-07 03:51:06.627690: 
epoch:  378 
2022-04-07 03:52:55.442961: train loss : -0.6917 
2022-04-07 03:53:06.415061: validation loss: -0.6954 
2022-04-07 03:53:06.419699: Average global foreground Dice: [0.9038, 0.9712, 0.9842, 0.4387] 
2022-04-07 03:53:06.424121: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:53:07.332011: lr: 0.002789 
2022-04-07 03:53:07.336063: saving scheduled checkpoint file... 
2022-04-07 03:53:07.390639: saving checkpoint... 
2022-04-07 03:53:11.761907: done, saving took 4.42 seconds 
2022-04-07 03:53:11.782521: done 
2022-04-07 03:53:11.786651: This epoch took 125.149936 s
 
2022-04-07 03:53:11.788738: svmem(total=99932160000, available=83997011968, percent=15.9, used=12907778048, free=69160058880, active=23799824384, inactive=4702965760, buffers=68866048, cached=17795457024, shared=1207410688, slab=797044736) 
2022-04-07 03:53:11.803786: 
epoch:  379 
2022-04-07 03:55:00.915736: train loss : -0.6962 
2022-04-07 03:55:12.034545: validation loss: -0.6793 
2022-04-07 03:55:12.044289: Average global foreground Dice: [0.9025, 0.9718, 0.9817, 0.4232] 
2022-04-07 03:55:12.044926: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:55:12.853363: lr: 0.002768 
2022-04-07 03:55:12.867998: saving scheduled checkpoint file... 
2022-04-07 03:55:12.989566: saving checkpoint... 
2022-04-07 03:55:17.400017: done, saving took 4.52 seconds 
2022-04-07 03:55:17.444172: done 
2022-04-07 03:55:17.509322: saving checkpoint... 
2022-04-07 03:55:21.963473: done, saving took 4.50 seconds 
2022-04-07 03:55:21.976670: This epoch took 130.171145 s
 
2022-04-07 03:55:21.977722: svmem(total=99932160000, available=83936161792, percent=16.0, used=12957331456, free=69179785216, active=23867764736, inactive=4617195520, buffers=69140480, cached=17725902848, shared=1218846720, slab=796299264) 
2022-04-07 03:55:21.978357: 
epoch:  380 
2022-04-07 03:57:12.530172: train loss : -0.6994 
2022-04-07 03:57:25.038184: validation loss: -0.6867 
2022-04-07 03:57:25.041809: Average global foreground Dice: [0.9093, 0.9697, 0.9824, 0.4004] 
2022-04-07 03:57:25.045762: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:57:25.950562: lr: 0.002747 
2022-04-07 03:57:25.955053: saving scheduled checkpoint file... 
2022-04-07 03:57:26.050986: saving checkpoint... 
2022-04-07 03:57:30.515589: done, saving took 4.55 seconds 
2022-04-07 03:57:30.535309: done 
2022-04-07 03:57:30.540434: This epoch took 128.561440 s
 
2022-04-07 03:57:30.544599: svmem(total=99932160000, available=84003823616, percent=15.9, used=12889444352, free=69174231040, active=23715553280, inactive=4778876928, buffers=69378048, cached=17799106560, shared=1218846720, slab=797085696) 
2022-04-07 03:57:30.548910: 
epoch:  381 
2022-04-07 03:59:20.245099: train loss : -0.6980 
2022-04-07 03:59:31.366974: validation loss: -0.6698 
2022-04-07 03:59:31.384325: Average global foreground Dice: [0.8995, 0.9707, 0.9792, 0.3918] 
2022-04-07 03:59:31.386589: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 03:59:32.134831: lr: 0.002727 
2022-04-07 03:59:32.140591: saving scheduled checkpoint file... 
2022-04-07 03:59:32.228308: saving checkpoint... 
2022-04-07 03:59:36.826462: done, saving took 4.68 seconds 
2022-04-07 03:59:36.859940: done 
2022-04-07 03:59:36.868525: This epoch took 126.315800 s
 
2022-04-07 03:59:36.877755: svmem(total=99932160000, available=84030349312, percent=15.9, used=12887072768, free=69231534080, active=23922982912, inactive=4510158848, buffers=69619712, cached=17743933440, shared=1194029056, slab=796016640) 
2022-04-07 03:59:36.886500: 
epoch:  382 
2022-04-07 04:01:27.923971: train loss : -0.6911 
2022-04-07 04:01:39.481626: validation loss: -0.6810 
2022-04-07 04:01:39.485334: Average global foreground Dice: [0.9086, 0.9713, 0.9801, 0.4059] 
2022-04-07 04:01:39.486344: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:01:40.417747: lr: 0.002706 
2022-04-07 04:01:40.419787: saving scheduled checkpoint file... 
2022-04-07 04:01:40.555398: saving checkpoint... 
2022-04-07 04:01:45.184514: done, saving took 4.76 seconds 
2022-04-07 04:01:45.212113: done 
2022-04-07 04:01:45.217113: This epoch took 128.321142 s
 
2022-04-07 04:01:45.224787: svmem(total=99932160000, available=83981754368, percent=16.0, used=12912377856, free=69269000192, active=23859646464, inactive=4537294848, buffers=69910528, cached=17680871424, shared=1218138112, slab=798224384) 
2022-04-07 04:01:45.229460: 
epoch:  383 
2022-04-07 04:03:35.387092: train loss : -0.7062 
2022-04-07 04:03:47.234691: validation loss: -0.7157 
2022-04-07 04:03:47.240088: Average global foreground Dice: [0.925, 0.9732, 0.9878, 0.4271] 
2022-04-07 04:03:47.245738: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:03:48.301252: lr: 0.002685 
2022-04-07 04:03:48.308214: saving scheduled checkpoint file... 
2022-04-07 04:03:48.387735: saving checkpoint... 
2022-04-07 04:03:52.786229: done, saving took 4.47 seconds 
2022-04-07 04:03:52.794358: done 
2022-04-07 04:03:52.843803: saving checkpoint... 
2022-04-07 04:03:57.147557: done, saving took 4.35 seconds 
2022-04-07 04:03:57.161828: This epoch took 131.927881 s
 
2022-04-07 04:03:57.162921: svmem(total=99932160000, available=83911917568, percent=16.0, used=12976017408, free=69166059520, active=23608487936, inactive=4893986816, buffers=70156288, cached=17719926784, shared=1224364032, slab=796774400) 
2022-04-07 04:03:57.163751: 
epoch:  384 
2022-04-07 04:05:43.573811: train loss : -0.7098 
2022-04-07 04:05:53.907871: validation loss: -0.7050 
2022-04-07 04:05:53.931904: Average global foreground Dice: [0.9134, 0.9734, 0.9861, 0.4385] 
2022-04-07 04:05:53.953069: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:05:54.572092: lr: 0.002664 
2022-04-07 04:05:54.572914: saving scheduled checkpoint file... 
2022-04-07 04:05:54.679244: saving checkpoint... 
2022-04-07 04:05:59.251772: done, saving took 4.68 seconds 
2022-04-07 04:05:59.289915: done 
2022-04-07 04:05:59.347293: saving checkpoint... 
2022-04-07 04:06:03.694520: done, saving took 4.39 seconds 
2022-04-07 04:06:03.711841: This epoch took 126.547518 s
 
2022-04-07 04:06:03.714918: svmem(total=99932160000, available=83916353536, percent=16.0, used=12974706688, free=69164556288, active=23641530368, inactive=4860792832, buffers=70414336, cached=17722482688, shared=1221238784, slab=797315072) 
2022-04-07 04:06:03.716822: 
epoch:  385 
2022-04-07 04:07:54.434563: train loss : -0.6904 
2022-04-07 04:08:06.485045: validation loss: -0.6807 
2022-04-07 04:08:06.486662: Average global foreground Dice: [0.9067, 0.9707, 0.9799, 0.4083] 
2022-04-07 04:08:06.487985: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:08:07.312089: lr: 0.002643 
2022-04-07 04:08:07.316372: saving scheduled checkpoint file... 
2022-04-07 04:08:07.377173: saving checkpoint... 
2022-04-07 04:08:12.892817: done, saving took 5.57 seconds 
2022-04-07 04:08:12.918516: done 
2022-04-07 04:08:12.922064: This epoch took 129.203500 s
 
2022-04-07 04:08:12.925742: svmem(total=99932160000, available=83866148864, percent=16.1, used=13027524608, free=69162225664, active=23867420672, inactive=4632211456, buffers=70664192, cached=17671745536, shared=1218502656, slab=800145408) 
2022-04-07 04:08:12.930262: 
epoch:  386 
2022-04-07 04:10:01.810961: train loss : -0.7057 
2022-04-07 04:10:13.680559: validation loss: -0.6810 
2022-04-07 04:10:13.683865: Average global foreground Dice: [0.9043, 0.9727, 0.9815, 0.4215] 
2022-04-07 04:10:13.684565: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:10:14.385093: lr: 0.002622 
2022-04-07 04:10:14.398264: saving scheduled checkpoint file... 
2022-04-07 04:10:14.475931: saving checkpoint... 
2022-04-07 04:10:18.532799: done, saving took 4.12 seconds 
2022-04-07 04:10:18.568030: done 
2022-04-07 04:10:18.629490: saving checkpoint... 
2022-04-07 04:10:23.148557: done, saving took 4.57 seconds 
2022-04-07 04:10:23.165101: This epoch took 130.230874 s
 
2022-04-07 04:10:23.167242: svmem(total=99932160000, available=83867643904, percent=16.1, used=13026279424, free=69168533504, active=23841062912, inactive=4655472640, buffers=70905856, cached=17666441216, shared=1218502656, slab=797683712) 
2022-04-07 04:10:23.168745: 
epoch:  387 
2022-04-07 04:12:12.578523: train loss : -0.7047 
2022-04-07 04:12:24.080105: validation loss: -0.7045 
2022-04-07 04:12:24.084345: Average global foreground Dice: [0.9218, 0.9736, 0.9852, 0.4075] 
2022-04-07 04:12:24.085281: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:12:24.881743: lr: 0.002601 
2022-04-07 04:12:24.890734: saving scheduled checkpoint file... 
2022-04-07 04:12:24.977329: saving checkpoint... 
2022-04-07 04:12:29.567773: done, saving took 4.67 seconds 
2022-04-07 04:12:29.583833: done 
2022-04-07 04:12:29.632066: saving checkpoint... 
2022-04-07 04:12:34.061628: done, saving took 4.48 seconds 
2022-04-07 04:12:34.073127: This epoch took 130.902229 s
 
2022-04-07 04:12:34.074201: svmem(total=99932160000, available=83859263488, percent=16.1, used=13034246144, free=69167968256, active=23858118656, inactive=4640915456, buffers=71151616, cached=17658793984, shared=1218674688, slab=798007296) 
2022-04-07 04:12:34.074911: 
epoch:  388 
2022-04-07 04:14:23.003404: train loss : -0.6977 
2022-04-07 04:14:35.152388: validation loss: -0.6826 
2022-04-07 04:14:35.153921: Average global foreground Dice: [0.8888, 0.9716, 0.985, 0.4199] 
2022-04-07 04:14:35.154706: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:14:36.036883: lr: 0.002581 
2022-04-07 04:14:36.042968: saving scheduled checkpoint file... 
2022-04-07 04:14:36.106526: saving checkpoint... 
2022-04-07 04:14:40.590971: done, saving took 4.54 seconds 
2022-04-07 04:14:40.615218: done 
2022-04-07 04:14:40.620256: This epoch took 126.544684 s
 
2022-04-07 04:14:40.627509: svmem(total=99932160000, available=84008206336, percent=15.9, used=12885041152, free=69230764032, active=23666941952, inactive=4765585408, buffers=71401472, cached=17744953344, shared=1218592768, slab=797990912) 
2022-04-07 04:14:40.632097: 
epoch:  389 
2022-04-07 04:16:30.032604: train loss : -0.6975 
2022-04-07 04:16:41.318556: validation loss: -0.7104 
2022-04-07 04:16:41.327020: Average global foreground Dice: [0.9149, 0.9725, 0.9852, 0.4269] 
2022-04-07 04:16:41.334644: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:16:42.844044: lr: 0.00256 
2022-04-07 04:16:42.849559: saving scheduled checkpoint file... 
2022-04-07 04:16:42.919560: saving checkpoint... 
2022-04-07 04:16:49.654523: done, saving took 6.80 seconds 
2022-04-07 04:16:49.666485: done 
2022-04-07 04:16:49.715973: saving checkpoint... 
2022-04-07 04:16:54.161091: done, saving took 4.49 seconds 
2022-04-07 04:16:54.174409: This epoch took 133.537947 s
 
2022-04-07 04:16:54.175242: svmem(total=99932160000, available=83956760576, percent=16.0, used=12936552448, free=69176631296, active=23868137472, inactive=4623118336, buffers=71675904, cached=17747300352, shared=1218936832, slab=797048832) 
2022-04-07 04:16:54.175905: 
epoch:  390 
2022-04-07 04:18:44.947975: train loss : -0.6994 
2022-04-07 04:18:56.558690: validation loss: -0.6904 
2022-04-07 04:18:56.572268: Average global foreground Dice: [0.8937, 0.9728, 0.9816, 0.4094] 
2022-04-07 04:18:56.573081: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:18:57.524082: lr: 0.002539 
2022-04-07 04:18:57.536569: saving scheduled checkpoint file... 
2022-04-07 04:18:57.663175: saving checkpoint... 
2022-04-07 04:19:02.151700: done, saving took 4.60 seconds 
2022-04-07 04:19:02.180240: done 
2022-04-07 04:19:02.188634: This epoch took 128.012175 s
 
2022-04-07 04:19:02.193951: svmem(total=99932160000, available=84011909120, percent=15.9, used=12905979904, free=69162688512, active=23891738624, inactive=4613853184, buffers=71938048, cached=17791553536, shared=1194033152, slab=797020160) 
2022-04-07 04:19:02.199362: 
epoch:  391 
2022-04-07 04:20:51.934849: train loss : -0.7030 
2022-04-07 04:21:03.647771: validation loss: -0.7043 
2022-04-07 04:21:03.649896: Average global foreground Dice: [0.9179, 0.9677, 0.9841, 0.3882] 
2022-04-07 04:21:03.652206: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:21:04.468459: lr: 0.002518 
2022-04-07 04:21:04.471792: saving scheduled checkpoint file... 
2022-04-07 04:21:04.595635: saving checkpoint... 
2022-04-07 04:21:09.097055: done, saving took 4.62 seconds 
2022-04-07 04:21:09.119495: done 
2022-04-07 04:21:09.123080: This epoch took 126.918532 s
 
2022-04-07 04:21:09.126754: svmem(total=99932160000, available=83983290368, percent=16.0, used=12910972928, free=69228699648, active=23780098048, inactive=4655116288, buffers=72187904, cached=17720299520, shared=1218592768, slab=795811840) 
2022-04-07 04:21:09.129960: 
epoch:  392 
2022-04-07 04:22:58.497875: train loss : -0.7027 
2022-04-07 04:23:10.638186: validation loss: -0.7102 
2022-04-07 04:23:10.652472: Average global foreground Dice: [0.9167, 0.9687, 0.9851, 0.4283] 
2022-04-07 04:23:10.654373: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:23:11.527114: lr: 0.002497 
2022-04-07 04:23:11.535044: saving scheduled checkpoint file... 
2022-04-07 04:23:11.673613: saving checkpoint... 
2022-04-07 04:23:16.260606: done, saving took 4.72 seconds 
2022-04-07 04:23:16.279329: done 
2022-04-07 04:23:16.280042: This epoch took 127.145515 s
 
2022-04-07 04:23:16.280789: svmem(total=99932160000, available=83976781824, percent=16.0, used=12917207040, free=69264072704, active=23807561728, inactive=4596793344, buffers=72433664, cached=17678446592, shared=1218420736, slab=796717056) 
2022-04-07 04:23:16.281391: 
epoch:  393 
2022-04-07 04:25:04.318365: train loss : -0.6999 
2022-04-07 04:25:15.900256: validation loss: -0.6904 
2022-04-07 04:25:15.902837: Average global foreground Dice: [0.8919, 0.9684, 0.9808, 0.4131] 
2022-04-07 04:25:15.912074: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:25:16.869234: lr: 0.002476 
2022-04-07 04:25:16.881139: saving scheduled checkpoint file... 
2022-04-07 04:25:17.025510: saving checkpoint... 
2022-04-07 04:25:21.553295: done, saving took 4.66 seconds 
2022-04-07 04:25:21.576540: done 
2022-04-07 04:25:21.580895: This epoch took 125.298891 s
 
2022-04-07 04:25:21.583486: svmem(total=99932160000, available=83955453952, percent=16.0, used=12938002432, free=69264232448, active=23821926400, inactive=4582453248, buffers=72691712, cached=17657233408, shared=1218678784, slab=796758016) 
2022-04-07 04:25:21.584934: 
epoch:  394 
2022-04-07 04:27:12.398920: train loss : -0.6873 
2022-04-07 04:27:23.764950: validation loss: -0.7183 
2022-04-07 04:27:23.774199: Average global foreground Dice: [0.9143, 0.973, 0.9866, 0.452] 
2022-04-07 04:27:23.775864: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:27:24.579942: lr: 0.002455 
2022-04-07 04:27:24.583472: saving scheduled checkpoint file... 
2022-04-07 04:27:24.718820: saving checkpoint... 
2022-04-07 04:27:30.120123: done, saving took 5.53 seconds 
2022-04-07 04:27:30.161340: done 
2022-04-07 04:27:30.273199: saving checkpoint... 
2022-04-07 04:27:34.726552: done, saving took 4.56 seconds 
2022-04-07 04:27:34.749477: This epoch took 133.162108 s
 
2022-04-07 04:27:34.750612: svmem(total=99932160000, available=83870482432, percent=16.1, used=13022728192, free=69264965632, active=23739752448, inactive=4663009280, buffers=72937472, cached=17571528704, shared=1218850816, slab=797949952) 
2022-04-07 04:27:34.751334: 
epoch:  395 
2022-04-07 04:29:23.230419: train loss : -0.6939 
2022-04-07 04:29:35.032856: validation loss: -0.7078 
2022-04-07 04:29:35.044875: Average global foreground Dice: [0.9205, 0.9715, 0.9845, 0.4312] 
2022-04-07 04:29:35.048824: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:29:36.071832: lr: 0.002434 
2022-04-07 04:29:36.073522: saving scheduled checkpoint file... 
2022-04-07 04:29:36.167926: saving checkpoint... 
2022-04-07 04:29:40.544625: done, saving took 4.47 seconds 
2022-04-07 04:29:40.561990: done 
2022-04-07 04:29:40.615426: saving checkpoint... 
2022-04-07 04:29:44.955877: done, saving took 4.39 seconds 
2022-04-07 04:29:44.969693: This epoch took 130.217692 s
 
2022-04-07 04:29:44.970657: svmem(total=99932160000, available=83903115264, percent=16.0, used=12990418944, free=69160099840, active=24033701888, inactive=4479221760, buffers=73224192, cached=17708417024, shared=1218506752, slab=797356032) 
2022-04-07 04:29:44.971277: 
epoch:  396 
2022-04-07 04:31:32.612411: train loss : -0.6981 
2022-04-07 04:31:44.326518: validation loss: -0.7098 
2022-04-07 04:31:44.332834: Average global foreground Dice: [0.9168, 0.9764, 0.9849, 0.4302] 
2022-04-07 04:31:44.337907: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:31:45.360491: lr: 0.002413 
2022-04-07 04:31:45.371133: saving scheduled checkpoint file... 
2022-04-07 04:31:45.427657: saving checkpoint... 
2022-04-07 04:31:49.882231: done, saving took 4.51 seconds 
2022-04-07 04:31:49.905172: done 
2022-04-07 04:31:49.955667: saving checkpoint... 
2022-04-07 04:31:54.275119: done, saving took 4.37 seconds 
2022-04-07 04:31:54.289031: This epoch took 129.317135 s
 
2022-04-07 04:31:54.289839: svmem(total=99932160000, available=83887620096, percent=16.1, used=12993490944, free=69141942272, active=23823093760, inactive=4702797824, buffers=73498624, cached=17723228160, shared=1230983168, slab=797659136) 
2022-04-07 04:31:54.290463: 
epoch:  397 
2022-04-07 04:33:42.236026: train loss : -0.7037 
2022-04-07 04:33:53.622339: validation loss: -0.6723 
2022-04-07 04:33:53.631346: Average global foreground Dice: [0.8957, 0.9727, 0.9859, 0.4382] 
2022-04-07 04:33:53.634042: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:33:54.534985: lr: 0.002391 
2022-04-07 04:33:54.539080: saving scheduled checkpoint file... 
2022-04-07 04:33:54.647743: saving checkpoint... 
2022-04-07 04:33:58.808082: done, saving took 4.27 seconds 
2022-04-07 04:33:58.821295: done 
2022-04-07 04:33:58.870071: saving checkpoint... 
2022-04-07 04:34:03.305269: done, saving took 4.48 seconds 
2022-04-07 04:34:03.318680: This epoch took 129.027690 s
 
2022-04-07 04:34:03.319654: svmem(total=99932160000, available=83880677376, percent=16.1, used=13012828160, free=69151674368, active=23997419520, inactive=4519002112, buffers=73744384, cached=17693913088, shared=1218506752, slab=797675520) 
2022-04-07 04:34:03.320364: 
epoch:  398 
2022-04-07 04:35:52.043531: train loss : -0.6952 
2022-04-07 04:36:03.939464: validation loss: -0.6555 
2022-04-07 04:36:03.946558: Average global foreground Dice: [0.8919, 0.9668, 0.9773, 0.4169] 
2022-04-07 04:36:03.948575: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:36:04.634933: lr: 0.00237 
2022-04-07 04:36:04.635902: saving scheduled checkpoint file... 
2022-04-07 04:36:04.687289: saving checkpoint... 
2022-04-07 04:36:09.175605: done, saving took 4.54 seconds 
2022-04-07 04:36:09.200676: done 
2022-04-07 04:36:09.204720: This epoch took 125.883743 s
 
2022-04-07 04:36:09.210375: svmem(total=99932160000, available=84009213952, percent=15.9, used=12883918848, free=69143875584, active=23912181760, inactive=4603494400, buffers=74010624, cached=17830354944, shared=1218678784, slab=797814784) 
2022-04-07 04:36:09.215101: 
epoch:  399 
2022-04-07 04:37:58.587200: train loss : -0.6950 
2022-04-07 04:38:10.977111: validation loss: -0.6998 
2022-04-07 04:38:10.986768: Average global foreground Dice: [0.9051, 0.9736, 0.9826, 0.423] 
2022-04-07 04:38:10.989625: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:38:11.737227: lr: 0.002349 
2022-04-07 04:38:11.742129: saving scheduled checkpoint file... 
2022-04-07 04:38:11.843185: saving checkpoint... 
2022-04-07 04:38:16.343801: done, saving took 4.60 seconds 
2022-04-07 04:38:16.361149: done 
2022-04-07 04:38:16.362002: This epoch took 127.140650 s
 
2022-04-07 04:38:16.363047: svmem(total=99932160000, available=83961438208, percent=16.0, used=12931112960, free=69153611776, active=23955759104, inactive=4559941632, buffers=74252288, cached=17773182976, shared=1218592768, slab=795725824) 
2022-04-07 04:38:16.363791: 
epoch:  400 
2022-04-07 04:40:04.784379: train loss : -0.6978 
2022-04-07 04:40:15.533462: validation loss: -0.6561 
2022-04-07 04:40:15.535020: Average global foreground Dice: [0.9008, 0.9624, 0.9716, 0.4447] 
2022-04-07 04:40:15.535812: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:40:16.532562: lr: 0.002328 
2022-04-07 04:40:16.534511: saving scheduled checkpoint file... 
2022-04-07 04:40:16.607137: saving checkpoint... 
2022-04-07 04:40:21.231729: done, saving took 4.69 seconds 
2022-04-07 04:40:21.268925: done 
2022-04-07 04:40:21.278623: This epoch took 124.914027 s
 
2022-04-07 04:40:21.284649: svmem(total=99932160000, available=83965378560, percent=16.0, used=12927692800, free=69226450944, active=23592128512, inactive=4851720192, buffers=74493952, cached=17703522304, shared=1218592768, slab=796598272) 
2022-04-07 04:40:21.290164: 
epoch:  401 
2022-04-07 04:42:09.103144: train loss : -0.6965 
2022-04-07 04:42:20.067908: validation loss: -0.6834 
2022-04-07 04:42:20.076465: Average global foreground Dice: [0.9047, 0.9667, 0.984, 0.4132] 
2022-04-07 04:42:20.086139: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:42:21.558627: lr: 0.002307 
2022-04-07 04:42:21.571666: saving scheduled checkpoint file... 
2022-04-07 04:42:21.700570: saving checkpoint... 
2022-04-07 04:42:25.691958: done, saving took 4.11 seconds 
2022-04-07 04:42:25.711868: done 
2022-04-07 04:42:25.713395: This epoch took 124.415792 s
 
2022-04-07 04:42:25.715799: svmem(total=99932160000, available=84015755264, percent=15.9, used=12864798720, free=69134221312, active=23937814528, inactive=4595834880, buffers=74723328, cached=17858416640, shared=1231327232, slab=795820032) 
2022-04-07 04:42:25.717405: 
epoch:  402 
2022-04-07 04:44:13.356803: train loss : -0.7085 
2022-04-07 04:44:24.598343: validation loss: -0.6982 
2022-04-07 04:44:24.601245: Average global foreground Dice: [0.9076, 0.9738, 0.9818, 0.4263] 
2022-04-07 04:44:24.602022: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:44:25.433634: lr: 0.002286 
2022-04-07 04:44:25.448992: saving scheduled checkpoint file... 
2022-04-07 04:44:25.522342: saving checkpoint... 
2022-04-07 04:44:30.130806: done, saving took 4.67 seconds 
2022-04-07 04:44:30.145038: done 
2022-04-07 04:44:30.149485: This epoch took 124.429561 s
 
2022-04-07 04:44:30.151832: svmem(total=99932160000, available=83997741056, percent=15.9, used=12895477760, free=69146255360, active=23973875712, inactive=4548321280, buffers=74956800, cached=17815470080, shared=1218592768, slab=796475392) 
2022-04-07 04:44:30.154050: 
epoch:  403 
2022-04-07 04:46:19.151856: train loss : -0.6994 
2022-04-07 04:46:31.913796: validation loss: -0.6812 
2022-04-07 04:46:31.915409: Average global foreground Dice: [0.9089, 0.9728, 0.9789, 0.3974] 
2022-04-07 04:46:31.916162: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:46:32.651188: lr: 0.002264 
2022-04-07 04:46:32.672389: saving scheduled checkpoint file... 
2022-04-07 04:46:32.766103: saving checkpoint... 
2022-04-07 04:46:36.950828: done, saving took 4.26 seconds 
2022-04-07 04:46:36.968644: done 
2022-04-07 04:46:36.969506: This epoch took 126.813807 s
 
2022-04-07 04:46:36.970371: svmem(total=99932160000, available=84032045056, percent=15.9, used=12860944384, free=69226139648, active=23855710208, inactive=4584849408, buffers=75223040, cached=17769852928, shared=1218936832, slab=796905472) 
2022-04-07 04:46:36.971000: 
epoch:  404 
2022-04-07 04:48:26.958963: train loss : -0.6973 
2022-04-07 04:48:39.115420: validation loss: -0.7151 
2022-04-07 04:48:39.117847: Average global foreground Dice: [0.9084, 0.9729, 0.9856, 0.4338] 
2022-04-07 04:48:39.118955: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:48:39.803498: lr: 0.002243 
2022-04-07 04:48:39.810877: saving scheduled checkpoint file... 
2022-04-07 04:48:39.944277: saving checkpoint... 
2022-04-07 04:48:44.434627: done, saving took 4.62 seconds 
2022-04-07 04:48:44.461976: done 
2022-04-07 04:48:44.463658: This epoch took 127.492043 s
 
2022-04-07 04:48:44.466190: svmem(total=99932160000, available=84159811584, percent=15.8, used=12732952576, free=69246959616, active=23857954816, inactive=4563726336, buffers=75476992, cached=17876770816, shared=1218764800, slab=798445568) 
2022-04-07 04:48:44.469097: 
epoch:  405 
2022-04-07 04:50:34.219264: train loss : -0.7016 
2022-04-07 04:50:45.765319: validation loss: -0.7279 
2022-04-07 04:50:45.768385: Average global foreground Dice: [0.9182, 0.9723, 0.9874, 0.4447] 
2022-04-07 04:50:45.769462: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:50:46.565327: lr: 0.002222 
2022-04-07 04:50:46.568343: saving scheduled checkpoint file... 
2022-04-07 04:50:46.728555: saving checkpoint... 
2022-04-07 04:50:51.214200: done, saving took 4.64 seconds 
2022-04-07 04:50:51.250315: done 
2022-04-07 04:50:51.362049: saving checkpoint... 
2022-04-07 04:50:55.766068: done, saving took 4.51 seconds 
2022-04-07 04:50:55.783457: This epoch took 131.310207 s
 
2022-04-07 04:50:55.784371: svmem(total=99932160000, available=83969454080, percent=16.0, used=12923940864, free=69261795328, active=23833731072, inactive=4567994368, buffers=75735040, cached=17670688768, shared=1218420736, slab=798810112) 
2022-04-07 04:50:55.785001: 
epoch:  406 
2022-04-07 04:52:45.836455: train loss : -0.7049 
2022-04-07 04:52:57.433300: validation loss: -0.6949 
2022-04-07 04:52:57.439130: Average global foreground Dice: [0.9256, 0.9712, 0.9801, 0.4021] 
2022-04-07 04:52:57.443620: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:52:58.370028: lr: 0.002201 
2022-04-07 04:52:58.372768: saving scheduled checkpoint file... 
2022-04-07 04:52:58.508534: saving checkpoint... 
2022-04-07 04:53:02.977118: done, saving took 4.60 seconds 
2022-04-07 04:53:03.013261: done 
2022-04-07 04:53:03.017361: This epoch took 127.231812 s
 
2022-04-07 04:53:03.022669: svmem(total=99932160000, available=84032774144, percent=15.9, used=12860395520, free=69238390784, active=23895916544, inactive=4537839616, buffers=75976704, cached=17757396992, shared=1218764800, slab=798117888) 
2022-04-07 04:53:03.031917: 
epoch:  407 
2022-04-07 04:54:53.655834: train loss : -0.7155 
2022-04-07 04:55:06.052026: validation loss: -0.7032 
2022-04-07 04:55:06.056261: Average global foreground Dice: [0.8987, 0.9733, 0.9841, 0.4365] 
2022-04-07 04:55:06.059173: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:55:07.495136: lr: 0.002179 
2022-04-07 04:55:07.496560: saving scheduled checkpoint file... 
2022-04-07 04:55:07.584678: saving checkpoint... 
2022-04-07 04:55:11.906948: done, saving took 4.41 seconds 
2022-04-07 04:55:11.919307: done 
2022-04-07 04:55:11.970097: saving checkpoint... 
2022-04-07 04:55:16.284483: done, saving took 4.36 seconds 
2022-04-07 04:55:16.297956: This epoch took 133.257110 s
 
2022-04-07 04:55:16.299013: svmem(total=99932160000, available=83896619008, percent=16.0, used=12996616192, free=69141278720, active=24003100672, inactive=4523597824, buffers=76251136, cached=17718013952, shared=1218592768, slab=798846976) 
2022-04-07 04:55:16.299663: 
epoch:  408 
2022-04-07 04:57:04.166795: train loss : -0.7024 
2022-04-07 04:57:15.277184: validation loss: -0.6981 
2022-04-07 04:57:15.281771: Average global foreground Dice: [0.9041, 0.9751, 0.9856, 0.4352] 
2022-04-07 04:57:15.286252: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:57:16.108230: lr: 0.002158 
2022-04-07 04:57:16.109394: saving scheduled checkpoint file... 
2022-04-07 04:57:16.214862: saving checkpoint... 
2022-04-07 04:57:20.693831: done, saving took 4.58 seconds 
2022-04-07 04:57:20.728855: done 
2022-04-07 04:57:20.786413: saving checkpoint... 
2022-04-07 04:57:25.117054: done, saving took 4.38 seconds 
2022-04-07 04:57:25.126843: This epoch took 128.826534 s
 
2022-04-07 04:57:25.129155: svmem(total=99932160000, available=83914350592, percent=16.0, used=12978524160, free=69134053376, active=23931453440, inactive=4600725504, buffers=76488704, cached=17743093760, shared=1218850816, slab=797392896) 
2022-04-07 04:57:25.130083: 
epoch:  409 
2022-04-07 04:59:13.542321: train loss : -0.7088 
2022-04-07 04:59:24.991236: validation loss: -0.6978 
2022-04-07 04:59:24.997900: Average global foreground Dice: [0.9025, 0.9744, 0.9833, 0.4302] 
2022-04-07 04:59:25.001860: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 04:59:26.048385: lr: 0.002137 
2022-04-07 04:59:26.057247: saving scheduled checkpoint file... 
2022-04-07 04:59:26.249277: saving checkpoint... 
2022-04-07 04:59:30.207712: done, saving took 4.14 seconds 
2022-04-07 04:59:30.255448: done 
2022-04-07 04:59:30.316575: saving checkpoint... 
2022-04-07 04:59:34.684886: done, saving took 4.41 seconds 
2022-04-07 04:59:34.705051: This epoch took 129.574299 s
 
2022-04-07 04:59:34.709919: svmem(total=99932160000, available=83910488064, percent=16.0, used=12982530048, free=69132619776, active=23631925248, inactive=4900933632, buffers=76730368, cached=17740279808, shared=1218678784, slab=798257152) 
2022-04-07 04:59:34.712002: 
epoch:  410 
2022-04-07 05:01:24.380025: train loss : -0.7167 
2022-04-07 05:01:36.149075: validation loss: -0.6940 
2022-04-07 05:01:36.171547: Average global foreground Dice: [0.8876, 0.9757, 0.9846, 0.4374] 
2022-04-07 05:01:36.172434: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:01:36.808550: lr: 0.002115 
2022-04-07 05:01:36.812436: saving scheduled checkpoint file... 
2022-04-07 05:01:36.884300: saving checkpoint... 
2022-04-07 05:01:41.671197: done, saving took 4.86 seconds 
2022-04-07 05:01:41.696721: done 
2022-04-07 05:01:41.704240: This epoch took 126.990069 s
 
2022-04-07 05:01:41.709996: svmem(total=99932160000, available=83942096896, percent=16.0, used=12975493120, free=69137321984, active=23977066496, inactive=4552626176, buffers=77021184, cached=17742323712, shared=1194119168, slab=796880896) 
2022-04-07 05:01:41.712922: 
epoch:  411 
2022-04-07 05:03:30.369851: train loss : -0.7172 
2022-04-07 05:03:42.429981: validation loss: -0.7080 
2022-04-07 05:03:42.434408: Average global foreground Dice: [0.9162, 0.9729, 0.9854, 0.423] 
2022-04-07 05:03:42.438486: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:03:43.231841: lr: 0.002094 
2022-04-07 05:03:43.239887: saving scheduled checkpoint file... 
2022-04-07 05:03:43.334399: saving checkpoint... 
2022-04-07 05:03:47.887790: done, saving took 4.64 seconds 
2022-04-07 05:03:47.922966: done 
2022-04-07 05:03:47.978437: saving checkpoint... 
2022-04-07 05:03:52.428847: done, saving took 4.50 seconds 
2022-04-07 05:03:52.442364: This epoch took 130.724461 s
 
2022-04-07 05:03:52.443437: svmem(total=99932160000, available=83896496128, percent=16.0, used=12996460544, free=69136674816, active=23937544192, inactive=4595847168, buffers=77266944, cached=17721757696, shared=1218764800, slab=798707712) 
2022-04-07 05:03:52.444121: 
epoch:  412 
2022-04-07 05:05:41.159343: train loss : -0.7194 
2022-04-07 05:05:52.131517: validation loss: -0.6969 
2022-04-07 05:05:52.133015: Average global foreground Dice: [0.9209, 0.9747, 0.9861, 0.3911] 
2022-04-07 05:05:52.133700: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:05:53.016754: lr: 0.002072 
2022-04-07 05:05:53.025741: saving scheduled checkpoint file... 
2022-04-07 05:05:53.111912: saving checkpoint... 
2022-04-07 05:05:59.021935: done, saving took 5.98 seconds 
2022-04-07 05:05:59.057501: done 
2022-04-07 05:05:59.070447: This epoch took 126.625728 s
 
2022-04-07 05:05:59.078868: svmem(total=99932160000, available=83967307776, percent=16.0, used=12925874176, free=69125165056, active=23880556544, inactive=4655407104, buffers=77533184, cached=17803587584, shared=1218592768, slab=798679040) 
2022-04-07 05:05:59.087161: 
epoch:  413 
2022-04-07 05:07:46.725178: train loss : -0.6991 
2022-04-07 05:07:57.710226: validation loss: -0.6554 
2022-04-07 05:07:57.713722: Average global foreground Dice: [0.8988, 0.966, 0.9659, 0.3982] 
2022-04-07 05:07:57.714477: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:07:59.128855: lr: 0.002051 
2022-04-07 05:07:59.141883: saving scheduled checkpoint file... 
2022-04-07 05:07:59.249060: saving checkpoint... 
2022-04-07 05:08:03.730035: done, saving took 4.58 seconds 
2022-04-07 05:08:03.770327: done 
2022-04-07 05:08:03.785517: This epoch took 124.697464 s
 
2022-04-07 05:08:03.796817: svmem(total=99932160000, available=84117942272, percent=15.8, used=12799344640, free=69127962624, active=23833329664, inactive=4701548544, buffers=77774848, cached=17927077888, shared=1194123264, slab=798949376) 
2022-04-07 05:08:03.809069: 
epoch:  414 
2022-04-07 05:09:54.237232: train loss : -0.6850 
2022-04-07 05:10:06.381921: validation loss: -0.6818 
2022-04-07 05:10:06.397808: Average global foreground Dice: [0.898, 0.9705, 0.9807, 0.4072] 
2022-04-07 05:10:06.415743: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:10:07.054742: lr: 0.00203 
2022-04-07 05:10:07.068259: saving scheduled checkpoint file... 
2022-04-07 05:10:07.136120: saving checkpoint... 
2022-04-07 05:10:11.727819: done, saving took 4.65 seconds 
2022-04-07 05:10:13.762672: done 
2022-04-07 05:10:13.763578: This epoch took 129.944319 s
 
2022-04-07 05:10:13.764349: svmem(total=99932160000, available=84082278400, percent=15.9, used=12832186368, free=74335182848, active=13289070592, inactive=10101137408, buffers=9547776, cached=12755243008, shared=1225179136, slab=741109760) 
2022-04-07 05:10:13.764912: 
epoch:  415 
2022-04-07 05:12:03.059762: train loss : -0.6987 
2022-04-07 05:12:13.825891: validation loss: -0.7009 
2022-04-07 05:12:13.828008: Average global foreground Dice: [0.9074, 0.9732, 0.9827, 0.4341] 
2022-04-07 05:12:13.828990: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:12:14.691093: lr: 0.002008 
2022-04-07 05:12:14.691986: saving scheduled checkpoint file... 
2022-04-07 05:12:14.772243: saving checkpoint... 
2022-04-07 05:12:19.133754: done, saving took 4.44 seconds 
2022-04-07 05:12:19.175659: done 
2022-04-07 05:12:19.189678: This epoch took 125.424051 s
 
2022-04-07 05:12:19.202803: svmem(total=99932160000, available=84209586176, percent=15.7, used=12797485056, free=69696577536, active=17607229440, inactive=10410082304, buffers=10182656, cached=17427914752, shared=1125675008, slab=753680384) 
2022-04-07 05:12:19.214742: 
epoch:  416 
2022-04-07 05:14:08.803017: train loss : -0.6975 
2022-04-07 05:14:20.311739: validation loss: -0.6732 
2022-04-07 05:14:20.316313: Average global foreground Dice: [0.9009, 0.9721, 0.9835, 0.4137] 
2022-04-07 05:14:20.321510: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:14:21.229478: lr: 0.001987 
2022-04-07 05:14:21.231876: saving scheduled checkpoint file... 
2022-04-07 05:14:21.338992: saving checkpoint... 
2022-04-07 05:14:25.758468: done, saving took 4.52 seconds 
2022-04-07 05:14:25.773311: done 
2022-04-07 05:14:25.775459: This epoch took 126.550683 s
 
2022-04-07 05:14:25.777884: svmem(total=99932160000, available=84035383296, percent=15.9, used=12878639104, free=69661495296, active=21257064448, inactive=6786797568, buffers=10452992, cached=17381572608, shared=1214615552, slab=761917440) 
2022-04-07 05:14:25.779574: 
epoch:  417 
2022-04-07 05:16:14.391240: train loss : -0.6970 
2022-04-07 05:16:26.232682: validation loss: -0.6928 
2022-04-07 05:16:26.242270: Average global foreground Dice: [0.9107, 0.9714, 0.9826, 0.4081] 
2022-04-07 05:16:26.246642: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:16:27.314966: lr: 0.001965 
2022-04-07 05:16:27.315965: saving scheduled checkpoint file... 
2022-04-07 05:16:27.477161: saving checkpoint... 
2022-04-07 05:16:31.992077: done, saving took 4.68 seconds 
2022-04-07 05:16:32.013259: done 
2022-04-07 05:16:32.016742: This epoch took 126.235011 s
 
2022-04-07 05:16:32.018139: svmem(total=99932160000, available=83951546368, percent=16.0, used=12958511104, free=69659570176, active=23492956160, inactive=4548386816, buffers=10850304, cached=17303228416, shared=1215995904, slab=767430656) 
2022-04-07 05:16:32.022765: 
epoch:  418 
2022-04-07 05:18:21.276485: train loss : -0.7155 
2022-04-07 05:18:32.891036: validation loss: -0.7143 
2022-04-07 05:18:32.899702: Average global foreground Dice: [0.9173, 0.9714, 0.9855, 0.4275] 
2022-04-07 05:18:32.900675: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:18:33.697585: lr: 0.001943 
2022-04-07 05:18:33.703934: saving scheduled checkpoint file... 
2022-04-07 05:18:33.880420: saving checkpoint... 
2022-04-07 05:18:38.319068: done, saving took 4.61 seconds 
2022-04-07 05:18:38.343363: done 
2022-04-07 05:18:38.344661: This epoch took 126.318629 s
 
2022-04-07 05:18:38.345495: svmem(total=99932160000, available=84054020096, percent=15.9, used=12854755328, free=69654454272, active=23719665664, inactive=4324429824, buffers=11128832, cached=17411821568, shared=1216729088, slab=768860160) 
2022-04-07 05:18:38.346082: 
epoch:  419 
2022-04-07 05:20:28.419658: train loss : -0.7023 
2022-04-07 05:20:42.196597: validation loss: -0.6706 
2022-04-07 05:20:42.308465: Average global foreground Dice: [0.9057, 0.9718, 0.9854, 0.416] 
2022-04-07 05:20:42.334822: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:20:43.590189: lr: 0.001922 
2022-04-07 05:20:43.599897: saving scheduled checkpoint file... 
2022-04-07 05:20:43.706288: saving checkpoint... 
2022-04-07 05:20:48.147455: done, saving took 4.53 seconds 
2022-04-07 05:20:48.192724: done 
2022-04-07 05:20:48.210094: This epoch took 129.863443 s
 
2022-04-07 05:20:48.225014: svmem(total=99932160000, available=84207362048, percent=15.7, used=12786249728, free=82557677568, active=10939568128, inactive=4243210240, buffers=8175616, cached=4580057088, shared=1155080192, slab=720723968) 
2022-04-07 05:20:48.242572: 
epoch:  420 
2022-04-07 05:22:37.216901: train loss : -0.7159 
2022-04-07 05:22:48.443937: validation loss: -0.6909 
2022-04-07 05:22:48.447651: Average global foreground Dice: [0.9139, 0.9744, 0.9828, 0.4114] 
2022-04-07 05:22:48.454364: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:22:49.354563: lr: 0.0019 
2022-04-07 05:22:49.373865: saving scheduled checkpoint file... 
2022-04-07 05:22:49.451544: saving checkpoint... 
2022-04-07 05:22:53.762111: done, saving took 4.37 seconds 
2022-04-07 05:22:53.782920: done 
2022-04-07 05:22:53.789145: This epoch took 125.528639 s
 
2022-04-07 05:22:53.793344: svmem(total=99932160000, available=84040716288, percent=15.9, used=12914999296, free=72968036352, active=14558990336, inactive=10200109056, buffers=9842688, cached=14039281664, shared=1182081024, slab=742592512) 
2022-04-07 05:22:53.798065: 
epoch:  421 
2022-04-07 05:24:42.883311: train loss : -0.7172 
2022-04-07 05:24:54.343560: validation loss: -0.6875 
2022-04-07 05:24:54.351187: Average global foreground Dice: [0.9133, 0.9715, 0.9793, 0.4202] 
2022-04-07 05:24:54.354411: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:24:55.364390: lr: 0.001879 
2022-04-07 05:24:55.367079: saving scheduled checkpoint file... 
2022-04-07 05:24:55.466592: saving checkpoint... 
2022-04-07 05:24:59.913612: done, saving took 4.54 seconds 
2022-04-07 05:24:59.931006: done 
2022-04-07 05:24:59.935525: This epoch took 126.131322 s
 
2022-04-07 05:24:59.941356: svmem(total=99932160000, available=84029911040, percent=15.9, used=12889587712, free=69629825024, active=18785067008, inactive=9296871424, buffers=10080256, cached=17402667008, shared=1211998208, slab=756449280) 
2022-04-07 05:24:59.945543: 
epoch:  422 
2022-04-07 05:26:48.104451: train loss : -0.7035 
2022-04-07 05:26:59.268278: validation loss: -0.6894 
2022-04-07 05:26:59.272127: Average global foreground Dice: [0.9057, 0.9745, 0.9838, 0.4043] 
2022-04-07 05:26:59.273074: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:27:00.148566: lr: 0.001857 
2022-04-07 05:27:00.149321: saving scheduled checkpoint file... 
2022-04-07 05:27:00.201169: saving checkpoint... 
2022-04-07 05:27:04.741987: done, saving took 4.59 seconds 
2022-04-07 05:27:04.765508: done 
2022-04-07 05:27:04.769020: This epoch took 124.819553 s
 
2022-04-07 05:27:04.773570: svmem(total=99932160000, available=84050235392, percent=15.9, used=12862730240, free=69619765248, active=22083686400, inactive=5995401216, buffers=10366976, cached=17439297536, shared=1214672896, slab=764063744) 
2022-04-07 05:27:04.777460: 
epoch:  423 
2022-04-07 05:28:54.854262: train loss : -0.7186 
2022-04-07 05:29:06.277921: validation loss: -0.7141 
2022-04-07 05:29:06.282279: Average global foreground Dice: [0.9121, 0.9776, 0.9863, 0.4425] 
2022-04-07 05:29:06.286134: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:29:07.228961: lr: 0.001835 
2022-04-07 05:29:07.229913: saving scheduled checkpoint file... 
2022-04-07 05:29:07.320283: saving checkpoint... 
2022-04-07 05:29:11.942247: done, saving took 4.71 seconds 
2022-04-07 05:29:11.965826: done 
2022-04-07 05:29:11.968700: This epoch took 127.185164 s
 
2022-04-07 05:29:11.972900: svmem(total=99932160000, available=83908755456, percent=16.0, used=12993286144, free=69613723648, active=23762620416, inactive=4322336768, buffers=10608640, cached=17314541568, shared=1224204288, slab=767688704) 
2022-04-07 05:29:11.977695: 
epoch:  424 
2022-04-07 05:31:01.078238: train loss : -0.7113 
2022-04-07 05:31:12.526902: validation loss: -0.7151 
2022-04-07 05:31:12.538697: Average global foreground Dice: [0.9157, 0.9733, 0.9808, 0.4237] 
2022-04-07 05:31:12.541176: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:31:13.531833: lr: 0.001813 
2022-04-07 05:31:13.532681: saving scheduled checkpoint file... 
2022-04-07 05:31:13.599148: saving checkpoint... 
2022-04-07 05:31:18.226664: done, saving took 4.69 seconds 
2022-04-07 05:31:18.259562: done 
2022-04-07 05:31:18.269712: This epoch took 126.287481 s
 
2022-04-07 05:31:18.280803: svmem(total=99932160000, available=84041523200, percent=15.9, used=12870975488, free=69706788864, active=23750344704, inactive=4240490496, buffers=10883072, cached=17343512576, shared=1212641280, slab=769318912) 
2022-04-07 05:31:18.289486: 
epoch:  425 
2022-04-07 05:33:06.652454: train loss : -0.7118 
2022-04-07 05:33:18.106081: validation loss: -0.6951 
2022-04-07 05:33:18.107628: Average global foreground Dice: [0.9063, 0.9728, 0.9854, 0.4255] 
2022-04-07 05:33:18.108384: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:33:19.137905: lr: 0.001792 
2022-04-07 05:33:19.140615: saving scheduled checkpoint file... 
2022-04-07 05:33:19.234427: saving checkpoint... 
2022-04-07 05:33:23.602025: done, saving took 4.46 seconds 
2022-04-07 05:33:23.616255: done 
2022-04-07 05:33:23.617050: This epoch took 125.316803 s
 
2022-04-07 05:33:23.618420: svmem(total=99932160000, available=84011016192, percent=15.9, used=12898545664, free=69621153792, active=23864082432, inactive=4209885184, buffers=11112448, cached=17401348096, shared=1213632512, slab=771489792) 
2022-04-07 05:33:23.620424: 
epoch:  426 
2022-04-07 05:35:11.658633: train loss : -0.7180 
2022-04-07 05:35:22.913371: validation loss: -0.6789 
2022-04-07 05:35:22.921018: Average global foreground Dice: [0.9043, 0.9734, 0.9834, 0.3914] 
2022-04-07 05:35:22.925446: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:35:23.657919: lr: 0.00177 
2022-04-07 05:35:23.658840: saving scheduled checkpoint file... 
2022-04-07 05:35:23.740016: saving checkpoint... 
2022-04-07 05:35:28.229384: done, saving took 4.57 seconds 
2022-04-07 05:35:28.250495: done 
2022-04-07 05:35:28.254485: This epoch took 124.632473 s
 
2022-04-07 05:35:28.258050: svmem(total=99932160000, available=83919114240, percent=16.0, used=12989321216, free=69619838976, active=23658991616, inactive=4417175552, buffers=11513856, cached=17311485952, shared=1214758912, slab=772526080) 
2022-04-07 05:35:28.262294: 
epoch:  427 
2022-04-07 05:37:18.664279: train loss : -0.7250 
2022-04-07 05:37:30.202446: validation loss: -0.6934 
2022-04-07 05:37:30.206771: Average global foreground Dice: [0.9059, 0.9712, 0.9789, 0.4264] 
2022-04-07 05:37:30.210329: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:37:30.968077: lr: 0.001748 
2022-04-07 05:37:30.975039: saving scheduled checkpoint file... 
2022-04-07 05:37:31.036041: saving checkpoint... 
2022-04-07 05:37:35.483332: done, saving took 4.50 seconds 
2022-04-07 05:37:35.501475: done 
2022-04-07 05:37:35.503135: This epoch took 127.235298 s
 
2022-04-07 05:37:35.503804: svmem(total=99932160000, available=84093440000, percent=15.8, used=12815036416, free=69698359296, active=23738298368, inactive=4256731136, buffers=11755520, cached=17407008768, shared=1214685184, slab=773300224) 
2022-04-07 05:37:35.504361: 
epoch:  428 
2022-04-07 05:39:25.548101: train loss : -0.7241 
2022-04-07 05:39:37.980993: validation loss: -0.7075 
2022-04-07 05:39:37.987638: Average global foreground Dice: [0.907, 0.9749, 0.9852, 0.4268] 
2022-04-07 05:39:37.989074: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:39:38.903476: lr: 0.001726 
2022-04-07 05:39:38.904314: saving scheduled checkpoint file... 
2022-04-07 05:39:39.047622: saving checkpoint... 
2022-04-07 05:39:43.547838: done, saving took 4.64 seconds 
2022-04-07 05:39:43.566580: done 
2022-04-07 05:39:43.567423: This epoch took 128.062534 s
 
2022-04-07 05:39:43.568297: svmem(total=99932160000, available=84070424576, percent=15.9, used=12823674880, free=69710422016, active=23802028032, inactive=4179333120, buffers=12009472, cached=17386053632, shared=1228890112, slab=772292608) 
2022-04-07 05:39:43.569048: 
epoch:  429 
2022-04-07 05:41:33.650087: train loss : -0.7123 
2022-04-07 05:41:44.700114: validation loss: -0.7135 
2022-04-07 05:41:44.715649: Average global foreground Dice: [0.9189, 0.9735, 0.9866, 0.4143] 
2022-04-07 05:41:44.716661: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:41:45.630963: lr: 0.001704 
2022-04-07 05:41:45.640908: saving scheduled checkpoint file... 
2022-04-07 05:41:45.775045: saving checkpoint... 
2022-04-07 05:41:50.239820: done, saving took 4.59 seconds 
2022-04-07 05:41:50.291806: done 
2022-04-07 05:41:50.307998: This epoch took 126.738299 s
 
2022-04-07 05:41:50.325567: svmem(total=99932160000, available=84111970304, percent=15.8, used=12795002880, free=69713035264, active=23592448000, inactive=4384456704, buffers=12259328, cached=17411862528, shared=1215340544, slab=773718016) 
2022-04-07 05:41:50.342135: 
epoch:  430 
2022-04-07 05:43:39.225098: train loss : -0.7183 
2022-04-07 05:43:51.348701: validation loss: -0.7061 
2022-04-07 05:43:51.353708: Average global foreground Dice: [0.8995, 0.9757, 0.9811, 0.4357] 
2022-04-07 05:43:51.355583: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:43:52.220802: lr: 0.001682 
2022-04-07 05:43:52.243511: saving scheduled checkpoint file... 
2022-04-07 05:43:52.386804: saving checkpoint... 
2022-04-07 05:43:56.824353: done, saving took 4.56 seconds 
2022-04-07 05:43:56.858885: done 
2022-04-07 05:43:56.865106: This epoch took 126.510602 s
 
2022-04-07 05:43:56.870338: svmem(total=99932160000, available=84050468864, percent=15.9, used=12879011840, free=69639786496, active=23767957504, inactive=4282757120, buffers=12492800, cached=17400868864, shared=1192632320, slab=775278592) 
2022-04-07 05:43:56.873965: 
epoch:  431 
2022-04-07 05:45:46.563622: train loss : -0.7094 
2022-04-07 05:45:57.700693: validation loss: -0.7134 
2022-04-07 05:45:57.707322: Average global foreground Dice: [0.9215, 0.9744, 0.9852, 0.4152] 
2022-04-07 05:45:57.708643: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:45:58.692167: lr: 0.00166 
2022-04-07 05:45:58.704206: saving scheduled checkpoint file... 
2022-04-07 05:45:58.809029: saving checkpoint... 
2022-04-07 05:46:03.116294: done, saving took 4.40 seconds 
2022-04-07 05:46:03.131901: done 
2022-04-07 05:46:03.135472: This epoch took 126.258188 s
 
2022-04-07 05:46:03.140516: svmem(total=99932160000, available=83954601984, percent=16.0, used=12975968256, free=69535465472, active=23704825856, inactive=4447477760, buffers=12759040, cached=17407967232, shared=1191821312, slab=774615040) 
2022-04-07 05:46:03.143800: 
epoch:  432 
2022-04-07 05:47:50.275343: train loss : -0.7121 
2022-04-07 05:48:01.335480: validation loss: -0.7112 
2022-04-07 05:48:01.338599: Average global foreground Dice: [0.9083, 0.9739, 0.9852, 0.422] 
2022-04-07 05:48:01.341797: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:48:02.496401: lr: 0.001638 
2022-04-07 05:48:02.507303: saving scheduled checkpoint file... 
2022-04-07 05:48:02.567667: saving checkpoint... 
2022-04-07 05:48:07.012335: done, saving took 4.50 seconds 
2022-04-07 05:48:07.027944: done 
2022-04-07 05:48:07.030081: This epoch took 123.882814 s
 
2022-04-07 05:48:07.031435: svmem(total=99932160000, available=83944357888, percent=16.0, used=12961136640, free=69534224384, active=23611801600, inactive=4541845504, buffers=13000704, cached=17423798272, shared=1216745472, slab=775843840) 
2022-04-07 05:48:07.033382: 
epoch:  433 
2022-04-07 05:49:56.209708: train loss : -0.7195 
2022-04-07 05:50:07.933340: validation loss: -0.7094 
2022-04-07 05:50:07.941697: Average global foreground Dice: [0.9236, 0.9772, 0.982, 0.4119] 
2022-04-07 05:50:07.942817: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:50:08.987455: lr: 0.001616 
2022-04-07 05:50:08.994692: saving scheduled checkpoint file... 
2022-04-07 05:50:09.115943: saving checkpoint... 
2022-04-07 05:50:13.478724: done, saving took 4.48 seconds 
2022-04-07 05:50:13.505900: done 
2022-04-07 05:50:13.512715: This epoch took 126.477697 s
 
2022-04-07 05:50:13.518100: svmem(total=99932160000, available=84040671232, percent=15.9, used=12864303104, free=69531152384, active=23679102976, inactive=4477612032, buffers=13242368, cached=17523462144, shared=1217155072, slab=775049216) 
2022-04-07 05:50:13.524579: 
epoch:  434 
2022-04-07 05:52:02.121156: train loss : -0.7227 
2022-04-07 05:52:14.335765: validation loss: -0.7256 
2022-04-07 05:52:14.349828: Average global foreground Dice: [0.9216, 0.9772, 0.9878, 0.438] 
2022-04-07 05:52:14.364910: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:52:15.063947: lr: 0.001594 
2022-04-07 05:52:15.070838: saving scheduled checkpoint file... 
2022-04-07 05:52:15.126114: saving checkpoint... 
2022-04-07 05:52:19.683536: done, saving took 4.61 seconds 
2022-04-07 05:52:19.704703: done 
2022-04-07 05:52:19.756645: saving checkpoint... 
2022-04-07 05:52:24.123207: done, saving took 4.41 seconds 
2022-04-07 05:52:24.136986: This epoch took 130.607349 s
 
2022-04-07 05:52:24.137927: svmem(total=99932160000, available=83992707072, percent=16.0, used=12911656960, free=69536632832, active=23797702656, inactive=4356055040, buffers=13492224, cached=17470377984, shared=1217302528, slab=776192000) 
2022-04-07 05:52:24.138572: 
epoch:  435 
2022-04-07 05:54:14.463457: train loss : -0.7141 
2022-04-07 05:54:26.242580: validation loss: -0.7080 
2022-04-07 05:54:26.259499: Average global foreground Dice: [0.9188, 0.9766, 0.9879, 0.4477] 
2022-04-07 05:54:26.264615: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:54:26.985180: lr: 0.001572 
2022-04-07 05:54:26.988190: saving scheduled checkpoint file... 
2022-04-07 05:54:27.083874: saving checkpoint... 
2022-04-07 05:54:31.753240: done, saving took 4.76 seconds 
2022-04-07 05:54:31.769081: done 
2022-04-07 05:54:31.816084: saving checkpoint... 
2022-04-07 05:54:36.243061: done, saving took 4.47 seconds 
2022-04-07 05:54:36.251279: This epoch took 132.112066 s
 
2022-04-07 05:54:36.252150: svmem(total=99932160000, available=83924508672, percent=16.0, used=12979826688, free=69532725248, active=23880609792, inactive=4276805632, buffers=13737984, cached=17405870080, shared=1217396736, slab=776323072) 
2022-04-07 05:54:36.252924: 
epoch:  436 
2022-04-07 05:56:25.477460: train loss : -0.7158 
2022-04-07 05:56:37.498216: validation loss: -0.6989 
2022-04-07 05:56:37.505102: Average global foreground Dice: [0.9177, 0.9747, 0.9841, 0.4278] 
2022-04-07 05:56:37.512815: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:56:38.329480: lr: 0.00155 
2022-04-07 05:56:38.335719: saving scheduled checkpoint file... 
2022-04-07 05:56:38.404603: saving checkpoint... 
2022-04-07 05:56:43.009435: done, saving took 4.67 seconds 
2022-04-07 05:56:43.027871: done 
2022-04-07 05:56:43.107245: saving checkpoint... 
2022-04-07 05:56:47.539910: done, saving took 4.51 seconds 
2022-04-07 05:56:47.559780: This epoch took 131.306175 s
 
2022-04-07 05:56:47.560945: svmem(total=99932160000, available=84002377728, percent=15.9, used=12901900288, free=69605392384, active=23717699584, inactive=4367863808, buffers=14036992, cached=17410830336, shared=1217478656, slab=776507392) 
2022-04-07 05:56:47.561861: 
epoch:  437 
2022-04-07 05:58:36.270499: train loss : -0.7344 
2022-04-07 05:58:47.829837: validation loss: -0.7040 
2022-04-07 05:58:47.843908: Average global foreground Dice: [0.9121, 0.9728, 0.9844, 0.426] 
2022-04-07 05:58:47.844802: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 05:58:49.006338: lr: 0.001528 
2022-04-07 05:58:49.009372: saving scheduled checkpoint file... 
2022-04-07 05:58:49.116530: saving checkpoint... 
2022-04-07 05:58:53.588621: done, saving took 4.58 seconds 
2022-04-07 05:58:53.605983: done 
2022-04-07 05:58:53.609576: This epoch took 126.047048 s
 
2022-04-07 05:58:53.613396: svmem(total=99932160000, available=84009512960, percent=15.9, used=12894556160, free=69530648576, active=23827779584, inactive=4330770432, buffers=14270464, cached=17492684800, shared=1217519616, slab=776957952) 
2022-04-07 05:58:53.616983: 
epoch:  438 
2022-04-07 06:00:43.667180: train loss : -0.7261 
2022-04-07 06:00:55.033313: validation loss: -0.7372 
2022-04-07 06:00:55.035316: Average global foreground Dice: [0.9244, 0.9774, 0.9841, 0.4616] 
2022-04-07 06:00:55.036205: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:00:55.812581: lr: 0.001506 
2022-04-07 06:00:55.831295: saving scheduled checkpoint file... 
2022-04-07 06:00:55.898561: saving checkpoint... 
2022-04-07 06:01:00.425691: done, saving took 4.58 seconds 
2022-04-07 06:01:00.457032: done 
2022-04-07 06:01:00.512897: saving checkpoint... 
2022-04-07 06:01:04.977007: done, saving took 4.51 seconds 
2022-04-07 06:01:04.990136: This epoch took 131.369975 s
 
2022-04-07 06:01:04.991526: svmem(total=99932160000, available=84033978368, percent=15.9, used=12869812224, free=69565341696, active=23695073280, inactive=4434812928, buffers=14540800, cached=17482465280, shared=1217773568, slab=777027584) 
2022-04-07 06:01:04.992476: 
epoch:  439 
2022-04-07 06:02:54.652025: train loss : -0.7306 
2022-04-07 06:03:06.104725: validation loss: -0.6911 
2022-04-07 06:03:06.107518: Average global foreground Dice: [0.892, 0.9717, 0.9842, 0.4314] 
2022-04-07 06:03:06.108479: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:03:07.027200: lr: 0.001483 
2022-04-07 06:03:07.043874: saving scheduled checkpoint file... 
2022-04-07 06:03:07.220539: saving checkpoint... 
2022-04-07 06:03:11.718185: done, saving took 4.66 seconds 
2022-04-07 06:03:11.738585: done 
2022-04-07 06:03:11.741820: This epoch took 126.748662 s
 
2022-04-07 06:03:11.745251: svmem(total=99932160000, available=84045750272, percent=15.9, used=12858077184, free=69609840640, active=23809773568, inactive=4274245632, buffers=14778368, cached=17449463808, shared=1217466368, slab=777711616) 
2022-04-07 06:03:11.748317: 
epoch:  440 
2022-04-07 06:05:00.341318: train loss : -0.7206 
2022-04-07 06:05:12.680690: validation loss: -0.7259 
2022-04-07 06:05:12.683461: Average global foreground Dice: [0.9057, 0.9779, 0.9829, 0.4506] 
2022-04-07 06:05:12.684178: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:05:13.511601: lr: 0.001461 
2022-04-07 06:05:13.513488: saving scheduled checkpoint file... 
2022-04-07 06:05:13.624462: saving checkpoint... 
2022-04-07 06:05:18.123471: done, saving took 4.61 seconds 
2022-04-07 06:05:18.141521: done 
2022-04-07 06:05:18.142424: This epoch took 126.390678 s
 
2022-04-07 06:05:18.143219: svmem(total=99932160000, available=84110794752, percent=15.8, used=12792578048, free=69637197824, active=23764717568, inactive=4288450560, buffers=15052800, cached=17487331328, shared=1217933312, slab=776577024) 
2022-04-07 06:05:18.143743: 
epoch:  441 
2022-04-07 06:07:09.402906: train loss : -0.7177 
2022-04-07 06:07:20.382967: validation loss: -0.7058 
2022-04-07 06:07:20.392601: Average global foreground Dice: [0.9283, 0.9688, 0.9816, 0.4147] 
2022-04-07 06:07:20.396691: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:07:21.180620: lr: 0.001439 
2022-04-07 06:07:21.188466: saving scheduled checkpoint file... 
2022-04-07 06:07:21.332040: saving checkpoint... 
2022-04-07 06:07:25.845884: done, saving took 4.65 seconds 
2022-04-07 06:07:25.875390: done 
2022-04-07 06:07:25.882275: This epoch took 127.737990 s
 
2022-04-07 06:07:25.887028: svmem(total=99932160000, available=84039077888, percent=15.9, used=12864520192, free=69626019840, active=23788937216, inactive=4278947840, buffers=15294464, cached=17426325504, shared=1218019328, slab=775852032) 
2022-04-07 06:07:25.892376: 
epoch:  442 
2022-04-07 06:09:14.920311: train loss : -0.7087 
2022-04-07 06:09:26.392495: validation loss: -0.7335 
2022-04-07 06:09:26.400226: Average global foreground Dice: [0.9249, 0.9797, 0.9864, 0.4497] 
2022-04-07 06:09:26.400957: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:09:27.176127: lr: 0.001416 
2022-04-07 06:09:27.192925: saving scheduled checkpoint file... 
2022-04-07 06:09:27.384662: saving checkpoint... 
2022-04-07 06:09:31.946587: done, saving took 4.74 seconds 
2022-04-07 06:09:31.992152: done 
2022-04-07 06:09:32.106231: saving checkpoint... 
2022-04-07 06:09:36.554355: done, saving took 4.55 seconds 
2022-04-07 06:09:36.572285: This epoch took 130.672069 s
 
2022-04-07 06:09:36.573195: svmem(total=99932160000, available=83921526784, percent=16.0, used=12981899264, free=69640192000, active=23676145664, inactive=4369772544, buffers=15540224, cached=17294528512, shared=1218129920, slab=776044544) 
2022-04-07 06:09:36.573786: 
epoch:  443 
2022-04-07 06:11:26.014661: train loss : -0.7112 
2022-04-07 06:11:37.417505: validation loss: -0.7271 
2022-04-07 06:11:37.444428: Average global foreground Dice: [0.925, 0.9734, 0.9882, 0.4458] 
2022-04-07 06:11:37.445589: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:11:38.526488: lr: 0.001394 
2022-04-07 06:11:38.528601: saving scheduled checkpoint file... 
2022-04-07 06:11:38.612187: saving checkpoint... 
2022-04-07 06:11:42.498506: done, saving took 3.97 seconds 
2022-04-07 06:11:42.534311: done 
2022-04-07 06:11:42.593336: saving checkpoint... 
2022-04-07 06:11:46.902124: done, saving took 4.36 seconds 
2022-04-07 06:11:46.919744: This epoch took 130.345373 s
 
2022-04-07 06:11:46.921690: svmem(total=99932160000, available=84002156544, percent=15.9, used=12887437312, free=69504204800, active=23723388928, inactive=4458692608, buffers=15806464, cached=17524711424, shared=1231835136, slab=776052736) 
2022-04-07 06:11:46.923739: 
epoch:  444 
2022-04-07 06:13:35.093197: train loss : -0.7221 
2022-04-07 06:13:46.957086: validation loss: -0.7076 
2022-04-07 06:13:46.958678: Average global foreground Dice: [0.9228, 0.9725, 0.988, 0.4266] 
2022-04-07 06:13:46.959403: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:13:47.740785: lr: 0.001372 
2022-04-07 06:13:47.752465: saving scheduled checkpoint file... 
2022-04-07 06:13:47.968305: saving checkpoint... 
2022-04-07 06:13:52.438875: done, saving took 4.68 seconds 
2022-04-07 06:13:52.479736: done 
2022-04-07 06:13:52.539504: saving checkpoint... 
2022-04-07 06:13:56.861757: done, saving took 4.37 seconds 
2022-04-07 06:13:56.875991: This epoch took 129.951088 s
 
2022-04-07 06:13:56.878367: svmem(total=99932160000, available=83924541440, percent=16.0, used=12978782208, free=69520863232, active=23723831296, inactive=4444327936, buffers=16056320, cached=17416458240, shared=1218068480, slab=776515584) 
2022-04-07 06:13:56.880563: 
epoch:  445 
2022-04-07 06:15:45.751102: train loss : -0.7227 
2022-04-07 06:15:58.474689: validation loss: -0.7178 
2022-04-07 06:15:58.483155: Average global foreground Dice: [0.922, 0.9762, 0.985, 0.4105] 
2022-04-07 06:15:58.491579: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:15:59.367562: lr: 0.001349 
2022-04-07 06:15:59.370272: saving scheduled checkpoint file... 
2022-04-07 06:15:59.424237: saving checkpoint... 
2022-04-07 06:16:03.605613: done, saving took 4.23 seconds 
2022-04-07 06:16:03.615244: done 
2022-04-07 06:16:03.616028: This epoch took 126.733099 s
 
2022-04-07 06:16:03.616920: svmem(total=99932160000, available=83915296768, percent=16.0, used=12983312384, free=69524410368, active=23834079232, inactive=4334153728, buffers=16326656, cached=17408110592, shared=1222541312, slab=775438336) 
2022-04-07 06:16:03.617622: 
epoch:  446 
2022-04-07 06:17:52.934916: train loss : -0.7266 
2022-04-07 06:18:03.872118: validation loss: -0.7164 
2022-04-07 06:18:03.874020: Average global foreground Dice: [0.9207, 0.9754, 0.9838, 0.4184] 
2022-04-07 06:18:03.875535: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:18:05.044932: lr: 0.001327 
2022-04-07 06:18:05.052340: saving scheduled checkpoint file... 
2022-04-07 06:18:05.145994: saving checkpoint... 
2022-04-07 06:18:09.745786: done, saving took 4.69 seconds 
2022-04-07 06:18:09.759295: done 
2022-04-07 06:18:09.761498: This epoch took 126.143195 s
 
2022-04-07 06:18:09.762796: svmem(total=99932160000, available=83931770880, percent=16.0, used=12971560960, free=69521829888, active=23866060800, inactive=4307656704, buffers=16592896, cached=17422176256, shared=1218457600, slab=776531968) 
2022-04-07 06:18:09.764669: 
epoch:  447 
2022-04-07 06:19:58.571121: train loss : -0.7326 
2022-04-07 06:20:09.484289: validation loss: -0.7202 
2022-04-07 06:20:09.493520: Average global foreground Dice: [0.9171, 0.9764, 0.9859, 0.4143] 
2022-04-07 06:20:09.495311: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:20:10.334031: lr: 0.001304 
2022-04-07 06:20:10.335080: saving scheduled checkpoint file... 
2022-04-07 06:20:10.396404: saving checkpoint... 
2022-04-07 06:20:14.495046: done, saving took 4.16 seconds 
2022-04-07 06:20:14.530632: done 
2022-04-07 06:20:14.539064: This epoch took 124.770701 s
 
2022-04-07 06:20:14.545633: svmem(total=99932160000, available=84090224640, percent=15.9, used=12813840384, free=69514489856, active=23735488512, inactive=4439175168, buffers=16842752, cached=17586987008, shared=1217945600, slab=776527872) 
2022-04-07 06:20:14.553331: 
epoch:  448 
2022-04-07 06:22:03.234261: train loss : -0.7295 
2022-04-07 06:22:14.410805: validation loss: -0.7260 
2022-04-07 06:22:14.424218: Average global foreground Dice: [0.9307, 0.9757, 0.985, 0.4236] 
2022-04-07 06:22:14.425017: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:22:15.305143: lr: 0.001282 
2022-04-07 06:22:15.315819: saving scheduled checkpoint file... 
2022-04-07 06:22:15.394620: saving checkpoint... 
2022-04-07 06:22:19.969071: done, saving took 4.64 seconds 
2022-04-07 06:22:20.009026: done 
2022-04-07 06:22:20.018262: This epoch took 125.459183 s
 
2022-04-07 06:22:20.029463: svmem(total=99932160000, available=84026855424, percent=15.9, used=12876447744, free=69593305088, active=23744663552, inactive=4351258624, buffers=17084416, cached=17445322752, shared=1217966080, slab=776847360) 
2022-04-07 06:22:20.040812: 
epoch:  449 
2022-04-07 06:24:08.780981: train loss : -0.7201 
2022-04-07 06:24:20.434741: validation loss: -0.7092 
2022-04-07 06:24:20.482405: Average global foreground Dice: [0.9207, 0.9727, 0.9869, 0.4289] 
2022-04-07 06:24:20.490505: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:24:21.645466: lr: 0.001259 
2022-04-07 06:24:21.651900: saving scheduled checkpoint file... 
2022-04-07 06:24:21.725549: saving checkpoint... 
2022-04-07 06:24:26.148967: done, saving took 4.49 seconds 
2022-04-07 06:24:26.163466: done 
2022-04-07 06:24:26.164155: This epoch took 126.113226 s
 
2022-04-07 06:24:26.164969: svmem(total=99932160000, available=84006268928, percent=15.9, used=12896833536, free=69512572928, active=23702650880, inactive=4472401920, buffers=17313792, cached=17505439744, shared=1218150400, slab=777134080) 
2022-04-07 06:24:26.165589: 
epoch:  450 
2022-04-07 06:26:15.492085: train loss : -0.7211 
2022-04-07 06:26:26.811273: validation loss: -0.6979 
2022-04-07 06:26:26.825253: Average global foreground Dice: [0.9124, 0.9725, 0.9837, 0.4147] 
2022-04-07 06:26:26.838495: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:26:27.424890: lr: 0.001236 
2022-04-07 06:26:27.427755: saving scheduled checkpoint file... 
2022-04-07 06:26:27.536380: saving checkpoint... 
2022-04-07 06:26:32.245791: done, saving took 4.81 seconds 
2022-04-07 06:26:32.280267: done 
2022-04-07 06:26:32.285804: This epoch took 126.119686 s
 
2022-04-07 06:26:32.289549: svmem(total=99932160000, available=84028772352, percent=15.9, used=12873867264, free=69418848256, active=23857651712, inactive=4403589120, buffers=21037056, cached=17618407424, shared=1217978368, slab=779292672) 
2022-04-07 06:26:32.293593: 
epoch:  451 
2022-04-07 06:28:21.600792: train loss : -0.7208 
2022-04-07 06:28:33.164392: validation loss: -0.7041 
2022-04-07 06:28:33.176237: Average global foreground Dice: [0.9233, 0.9772, 0.9858, 0.4103] 
2022-04-07 06:28:33.177153: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:28:34.087811: lr: 0.001214 
2022-04-07 06:28:34.089904: saving scheduled checkpoint file... 
2022-04-07 06:28:34.203393: saving checkpoint... 
2022-04-07 06:28:38.683857: done, saving took 4.59 seconds 
2022-04-07 06:28:38.703999: done 
2022-04-07 06:28:38.710210: This epoch took 126.412079 s
 
2022-04-07 06:28:38.715043: svmem(total=99932160000, available=84082130944, percent=15.9, used=12819849216, free=69507411968, active=23788494848, inactive=4391440384, buffers=21295104, cached=17583603712, shared=1218064384, slab=778166272) 
2022-04-07 06:28:38.719630: 
epoch:  452 
2022-04-07 06:30:27.456665: train loss : -0.7302 
2022-04-07 06:30:38.896577: validation loss: -0.7106 
2022-04-07 06:30:38.898338: Average global foreground Dice: [0.9228, 0.9783, 0.9863, 0.4093] 
2022-04-07 06:30:38.899923: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:30:39.721722: lr: 0.001191 
2022-04-07 06:30:39.734684: saving scheduled checkpoint file... 
2022-04-07 06:30:39.922876: saving checkpoint... 
2022-04-07 06:30:44.257447: done, saving took 4.52 seconds 
2022-04-07 06:30:44.282098: done 
2022-04-07 06:30:44.285547: This epoch took 125.561971 s
 
2022-04-07 06:30:44.289227: svmem(total=99932160000, available=84083019776, percent=15.9, used=12819582976, free=69526024192, active=23764770816, inactive=4392177664, buffers=21557248, cached=17564995584, shared=1217998848, slab=778190848) 
2022-04-07 06:30:44.294020: 
epoch:  453 
2022-04-07 06:32:34.906826: train loss : -0.7186 
2022-04-07 06:32:46.469192: validation loss: -0.6994 
2022-04-07 06:32:46.476230: Average global foreground Dice: [0.9162, 0.975, 0.9772, 0.4236] 
2022-04-07 06:32:46.484554: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:32:47.252744: lr: 0.001168 
2022-04-07 06:32:47.267883: saving scheduled checkpoint file... 
2022-04-07 06:32:47.411632: saving checkpoint... 
2022-04-07 06:32:52.069892: done, saving took 4.79 seconds 
2022-04-07 06:32:52.096339: done 
2022-04-07 06:32:52.099209: This epoch took 127.801325 s
 
2022-04-07 06:32:52.102272: svmem(total=99932160000, available=83984846848, percent=16.0, used=12917862400, free=69526302720, active=23755411456, inactive=4402073600, buffers=21798912, cached=17466195968, shared=1217912832, slab=778555392) 
2022-04-07 06:32:52.104724: 
epoch:  454 
2022-04-07 06:34:41.108932: train loss : -0.7324 
2022-04-07 06:34:53.721451: validation loss: -0.7297 
2022-04-07 06:34:53.734993: Average global foreground Dice: [0.9241, 0.9758, 0.9853, 0.422] 
2022-04-07 06:34:53.739342: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:34:54.625855: lr: 0.001145 
2022-04-07 06:34:54.629362: saving scheduled checkpoint file... 
2022-04-07 06:34:54.797840: saving checkpoint... 
2022-04-07 06:34:59.232186: done, saving took 4.60 seconds 
2022-04-07 06:34:59.252421: done 
2022-04-07 06:34:59.254578: This epoch took 127.146445 s
 
2022-04-07 06:34:59.256424: svmem(total=99932160000, available=84077506560, percent=15.9, used=12824719360, free=69525094400, active=23795564544, inactive=4365438976, buffers=22040576, cached=17560305664, shared=1218084864, slab=779182080) 
2022-04-07 06:34:59.258227: 
epoch:  455 
2022-04-07 06:36:48.552594: train loss : -0.7270 
2022-04-07 06:37:00.229945: validation loss: -0.7122 
2022-04-07 06:37:00.240733: Average global foreground Dice: [0.9152, 0.9762, 0.9851, 0.4235] 
2022-04-07 06:37:00.253474: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:37:01.338927: lr: 0.001122 
2022-04-07 06:37:01.344183: saving scheduled checkpoint file... 
2022-04-07 06:37:01.442808: saving checkpoint... 
2022-04-07 06:37:05.891019: done, saving took 4.54 seconds 
2022-04-07 06:37:05.906155: done 
2022-04-07 06:37:05.909635: This epoch took 126.650212 s
 
2022-04-07 06:37:05.912603: svmem(total=99932160000, available=83852713984, percent=16.1, used=13049794560, free=69422088192, active=23757553664, inactive=4505133056, buffers=22315008, cached=17437962240, shared=1218187264, slab=779059200) 
2022-04-07 06:37:05.915804: 
epoch:  456 
2022-04-07 06:38:53.815308: train loss : -0.7273 
2022-04-07 06:39:04.832077: validation loss: -0.7381 
2022-04-07 06:39:04.841591: Average global foreground Dice: [0.9247, 0.9742, 0.9879, 0.4636] 
2022-04-07 06:39:04.843974: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:39:05.531180: lr: 0.001099 
2022-04-07 06:39:05.538304: saving scheduled checkpoint file... 
2022-04-07 06:39:05.657841: saving checkpoint... 
2022-04-07 06:39:10.139996: done, saving took 4.59 seconds 
2022-04-07 06:39:10.193652: done 
2022-04-07 06:39:10.213213: This epoch took 124.294980 s
 
2022-04-07 06:39:10.234917: svmem(total=99932160000, available=83932291072, percent=16.0, used=12969918464, free=69413355520, active=23588937728, inactive=4678844416, buffers=22544384, cached=17526341632, shared=1218015232, slab=779698176) 
2022-04-07 06:39:10.251951: 
epoch:  457 
2022-04-07 06:40:59.640545: train loss : -0.7357 
2022-04-07 06:41:11.090882: validation loss: -0.7069 
2022-04-07 06:41:11.092431: Average global foreground Dice: [0.9333, 0.9762, 0.9857, 0.3977] 
2022-04-07 06:41:11.093184: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:41:11.914568: lr: 0.001076 
2022-04-07 06:41:11.920901: saving scheduled checkpoint file... 
2022-04-07 06:41:12.029097: saving checkpoint... 
2022-04-07 06:41:16.633209: done, saving took 4.71 seconds 
2022-04-07 06:41:16.650993: done 
2022-04-07 06:41:16.653357: This epoch took 126.385216 s
 
2022-04-07 06:41:16.656114: svmem(total=99932160000, available=83955892224, percent=16.0, used=12946108416, free=69413720064, active=23861788672, inactive=4407177216, buffers=22786048, cached=17549545472, shared=1218367488, slab=780320768) 
2022-04-07 06:41:16.658148: 
epoch:  458 
2022-04-07 06:43:06.732414: train loss : -0.7339 
2022-04-07 06:43:18.715750: validation loss: -0.7333 
2022-04-07 06:43:18.719881: Average global foreground Dice: [0.9279, 0.9759, 0.9854, 0.4455] 
2022-04-07 06:43:18.721490: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:43:19.604500: lr: 0.001053 
2022-04-07 06:43:19.619662: saving scheduled checkpoint file... 
2022-04-07 06:43:19.791443: saving checkpoint... 
2022-04-07 06:43:24.391270: done, saving took 4.76 seconds 
2022-04-07 06:43:24.409854: done 
2022-04-07 06:43:24.466003: saving checkpoint... 
2022-04-07 06:43:28.909156: done, saving took 4.50 seconds 
2022-04-07 06:43:28.921825: This epoch took 132.262423 s
 
2022-04-07 06:43:28.923053: svmem(total=99932160000, available=83934445568, percent=16.0, used=12967936000, free=69424291840, active=23891496960, inactive=4369842176, buffers=23035904, cached=17516896256, shared=1218281472, slab=780320768) 
2022-04-07 06:43:28.923799: 
epoch:  459 
2022-04-07 06:45:20.362155: train loss : -0.7278 
2022-04-07 06:45:31.613200: validation loss: -0.7305 
2022-04-07 06:45:31.627952: Average global foreground Dice: [0.9238, 0.9749, 0.9883, 0.4436] 
2022-04-07 06:45:31.628858: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:45:32.364248: lr: 0.00103 
2022-04-07 06:45:32.377710: saving scheduled checkpoint file... 
2022-04-07 06:45:32.448566: saving checkpoint... 
2022-04-07 06:45:37.067895: done, saving took 4.68 seconds 
2022-04-07 06:45:37.097970: done 
2022-04-07 06:45:37.151766: saving checkpoint... 
2022-04-07 06:45:41.606788: done, saving took 4.50 seconds 
2022-04-07 06:45:41.623942: This epoch took 132.699443 s
 
2022-04-07 06:45:41.626102: svmem(total=99932160000, available=83981668352, percent=16.0, used=12920766464, free=69408755712, active=23777914880, inactive=4493979648, buffers=23322624, cached=17579315200, shared=1218142208, slab=779833344) 
2022-04-07 06:45:41.628342: 
epoch:  460 
2022-04-07 06:47:29.211507: train loss : -0.7385 
2022-04-07 06:47:41.448230: validation loss: -0.7239 
2022-04-07 06:47:41.454064: Average global foreground Dice: [0.929, 0.9772, 0.9864, 0.4202] 
2022-04-07 06:47:41.459063: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:47:42.283206: lr: 0.001007 
2022-04-07 06:47:42.294114: saving scheduled checkpoint file... 
2022-04-07 06:47:42.387642: saving checkpoint... 
2022-04-07 06:47:46.852105: done, saving took 4.55 seconds 
2022-04-07 06:47:46.880046: done 
2022-04-07 06:47:46.936600: saving checkpoint... 
2022-04-07 06:47:51.345583: done, saving took 4.46 seconds 
2022-04-07 06:47:51.363003: This epoch took 129.732635 s
 
2022-04-07 06:47:51.364017: svmem(total=99932160000, available=83964813312, percent=16.0, used=12937646080, free=69500862464, active=23751147520, inactive=4431507456, buffers=23564288, cached=17470087168, shared=1218203648, slab=780460032) 
2022-04-07 06:47:51.364806: 
epoch:  461 
2022-04-07 06:49:40.413003: train loss : -0.7326 
2022-04-07 06:49:52.246261: validation loss: -0.7186 
2022-04-07 06:49:52.257226: Average global foreground Dice: [0.9191, 0.9777, 0.9869, 0.4336] 
2022-04-07 06:49:52.259013: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:49:53.205311: lr: 0.000983 
2022-04-07 06:49:53.218225: saving scheduled checkpoint file... 
2022-04-07 06:49:53.324995: saving checkpoint... 
2022-04-07 06:49:57.930012: done, saving took 4.70 seconds 
2022-04-07 06:49:57.944836: done 
2022-04-07 06:49:57.994002: saving checkpoint... 
2022-04-07 06:50:02.431755: done, saving took 4.49 seconds 
2022-04-07 06:50:02.445242: This epoch took 131.079821 s
 
2022-04-07 06:50:02.447973: svmem(total=99932160000, available=84030455808, percent=15.9, used=12871962624, free=69416603648, active=23755694080, inactive=4518105088, buffers=23822336, cached=17619771392, shared=1218244608, slab=780947456) 
2022-04-07 06:50:02.450459: 
epoch:  462 
2022-04-07 06:51:50.882744: train loss : -0.7281 
2022-04-07 06:52:02.510765: validation loss: -0.7430 
2022-04-07 06:52:02.518622: Average global foreground Dice: [0.9336, 0.9741, 0.9847, 0.4533] 
2022-04-07 06:52:02.522710: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:52:03.512256: lr: 0.00096 
2022-04-07 06:52:03.513794: saving scheduled checkpoint file... 
2022-04-07 06:52:03.669201: saving checkpoint... 
2022-04-07 06:52:08.206274: done, saving took 4.69 seconds 
2022-04-07 06:52:08.224148: done 
2022-04-07 06:52:08.276703: saving checkpoint... 
2022-04-07 06:52:12.694460: done, saving took 4.47 seconds 
2022-04-07 06:52:12.708740: This epoch took 130.255710 s
 
2022-04-07 06:52:12.709662: svmem(total=99932160000, available=83950940160, percent=16.0, used=12951175168, free=69419278336, active=23840948224, inactive=4426416128, buffers=24072192, cached=17537634304, shared=1218510848, slab=780468224) 
2022-04-07 06:52:12.710444: 
epoch:  463 
2022-04-07 06:54:00.488274: train loss : -0.7266 
2022-04-07 06:54:11.489182: validation loss: -0.7105 
2022-04-07 06:54:11.500895: Average global foreground Dice: [0.9122, 0.9758, 0.9844, 0.4012] 
2022-04-07 06:54:11.507965: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:54:12.270167: lr: 0.000937 
2022-04-07 06:54:12.276229: saving scheduled checkpoint file... 
2022-04-07 06:54:12.388233: saving checkpoint... 
2022-04-07 06:54:16.906534: done, saving took 4.63 seconds 
2022-04-07 06:54:16.924255: done 
2022-04-07 06:54:16.925068: This epoch took 124.213978 s
 
2022-04-07 06:54:16.926045: svmem(total=99932160000, available=84057362432, percent=15.9, used=12844744704, free=69484085248, active=23821635584, inactive=4385017856, buffers=24301568, cached=17579028480, shared=1218453504, slab=779382784) 
2022-04-07 06:54:16.926837: 
epoch:  464 
2022-04-07 06:56:05.437679: train loss : -0.7286 
2022-04-07 06:56:18.177222: validation loss: -0.7321 
2022-04-07 06:56:18.193539: Average global foreground Dice: [0.9285, 0.9762, 0.9822, 0.4386] 
2022-04-07 06:56:18.196690: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:56:18.837025: lr: 0.000913 
2022-04-07 06:56:18.851338: saving scheduled checkpoint file... 
2022-04-07 06:56:18.988062: saving checkpoint... 
2022-04-07 06:56:23.442982: done, saving took 4.58 seconds 
2022-04-07 06:56:23.464613: done 
2022-04-07 06:56:23.465401: This epoch took 126.537927 s
 
2022-04-07 06:56:23.466286: svmem(total=99932160000, available=83996971008, percent=15.9, used=12904968192, free=69518659584, active=23805427712, inactive=4361572352, buffers=24567808, cached=17483964416, shared=1218543616, slab=780509184) 
2022-04-07 06:56:23.466977: 
epoch:  465 
2022-04-07 06:58:13.285655: train loss : -0.7224 
2022-04-07 06:58:23.990191: validation loss: -0.7191 
2022-04-07 06:58:24.004284: Average global foreground Dice: [0.9265, 0.9769, 0.9861, 0.4414] 
2022-04-07 06:58:24.004965: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 06:58:25.060236: lr: 0.00089 
2022-04-07 06:58:25.070095: saving scheduled checkpoint file... 
2022-04-07 06:58:25.204988: saving checkpoint... 
2022-04-07 06:58:29.833331: done, saving took 4.76 seconds 
2022-04-07 06:58:29.851753: done 
2022-04-07 06:58:29.852545: This epoch took 126.384976 s
 
2022-04-07 06:58:29.853419: svmem(total=99932160000, available=84007313408, percent=15.9, used=12880969728, free=69493166080, active=23797727232, inactive=4392710144, buffers=24809472, cached=17533214720, shared=1232175104, slab=780992512) 
2022-04-07 06:58:29.854142: 
epoch:  466 
2022-04-07 07:00:20.095745: train loss : -0.7316 
2022-04-07 07:00:32.957902: validation loss: -0.7174 
2022-04-07 07:00:32.968836: Average global foreground Dice: [0.9221, 0.9738, 0.985, 0.4139] 
2022-04-07 07:00:32.976348: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:00:33.582706: lr: 0.000866 
2022-04-07 07:00:33.595070: saving scheduled checkpoint file... 
2022-04-07 07:00:33.737806: saving checkpoint... 
2022-04-07 07:00:38.173807: done, saving took 4.57 seconds 
2022-04-07 07:00:38.192621: done 
2022-04-07 07:00:38.193467: This epoch took 128.338834 s
 
2022-04-07 07:00:38.194482: svmem(total=99932160000, available=83934060544, percent=16.0, used=12967673856, free=69509992448, active=23762472960, inactive=4414750720, buffers=25096192, cached=17429397504, shared=1218387968, slab=780320768) 
2022-04-07 07:00:38.195320: 
epoch:  467 
2022-04-07 07:02:28.358886: train loss : -0.7335 
2022-04-07 07:02:39.874541: validation loss: -0.7286 
2022-04-07 07:02:39.879435: Average global foreground Dice: [0.9214, 0.9781, 0.9851, 0.4472] 
2022-04-07 07:02:39.882268: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:02:40.799865: lr: 0.000842 
2022-04-07 07:02:40.812032: saving scheduled checkpoint file... 
2022-04-07 07:02:40.917214: saving checkpoint... 
2022-04-07 07:02:45.316558: done, saving took 4.49 seconds 
2022-04-07 07:02:45.356680: done 
2022-04-07 07:02:45.369676: This epoch took 127.173723 s
 
2022-04-07 07:02:45.384467: svmem(total=99932160000, available=84015923200, percent=15.9, used=12886208512, free=69401477120, active=23451779072, inactive=4831207424, buffers=25337856, cached=17619136512, shared=1218301952, slab=778694656) 
2022-04-07 07:02:45.398757: 
epoch:  468 
2022-04-07 07:04:32.609921: train loss : -0.7408 
2022-04-07 07:04:44.592467: validation loss: -0.7188 
2022-04-07 07:04:44.597708: Average global foreground Dice: [0.9284, 0.9739, 0.9862, 0.4409] 
2022-04-07 07:04:44.601557: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:04:45.540532: lr: 0.000819 
2022-04-07 07:04:45.548709: saving scheduled checkpoint file... 
2022-04-07 07:04:45.604254: saving checkpoint... 
2022-04-07 07:04:49.964016: done, saving took 4.41 seconds 
2022-04-07 07:04:49.987771: done 
2022-04-07 07:04:50.041354: saving checkpoint... 
2022-04-07 07:04:54.427409: done, saving took 4.43 seconds 
2022-04-07 07:04:54.443226: This epoch took 129.027787 s
 
2022-04-07 07:04:54.446641: svmem(total=99932160000, available=83902259200, percent=16.0, used=12999618560, free=69398155264, active=23804465152, inactive=4477353984, buffers=25587712, cached=17508798464, shared=1218568192, slab=778473472) 
2022-04-07 07:04:54.447705: 
epoch:  469 
2022-04-07 07:06:42.779548: train loss : -0.7370 
2022-04-07 07:06:55.322692: validation loss: -0.7302 
2022-04-07 07:06:55.324379: Average global foreground Dice: [0.9171, 0.9759, 0.9881, 0.4643] 
2022-04-07 07:06:55.325366: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:06:55.905916: lr: 0.000795 
2022-04-07 07:06:55.908669: saving scheduled checkpoint file... 
2022-04-07 07:06:55.996185: saving checkpoint... 
2022-04-07 07:07:00.259872: done, saving took 4.35 seconds 
2022-04-07 07:07:00.299579: done 
2022-04-07 07:07:00.359444: saving checkpoint... 
2022-04-07 07:07:04.707234: done, saving took 4.40 seconds 
2022-04-07 07:07:04.742044: This epoch took 130.291734 s
 
2022-04-07 07:07:04.748466: svmem(total=99932160000, available=83929198592, percent=16.0, used=12969648128, free=69397319680, active=23856766976, inactive=4428234752, buffers=25862144, cached=17539330048, shared=1221369856, slab=780517376) 
2022-04-07 07:07:04.754415: 
epoch:  470 
2022-04-07 07:08:53.033411: train loss : -0.7370 
2022-04-07 07:09:05.949476: validation loss: -0.6902 
2022-04-07 07:09:05.960546: Average global foreground Dice: [0.9189, 0.9736, 0.9792, 0.4075] 
2022-04-07 07:09:05.961457: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:09:06.693840: lr: 0.000771 
2022-04-07 07:09:06.704948: saving scheduled checkpoint file... 
2022-04-07 07:09:06.764592: saving checkpoint... 
2022-04-07 07:09:11.321850: done, saving took 4.61 seconds 
2022-04-07 07:09:11.343689: done 
2022-04-07 07:09:11.347042: This epoch took 126.587662 s
 
2022-04-07 07:09:11.351954: svmem(total=99932160000, available=84013674496, percent=15.9, used=12888195072, free=69394874368, active=23749169152, inactive=4537294848, buffers=26099712, cached=17622990848, shared=1218568192, slab=780582912) 
2022-04-07 07:09:11.355954: 
epoch:  471 
2022-04-07 07:11:00.421126: train loss : -0.7326 
2022-04-07 07:11:12.201029: validation loss: -0.7325 
2022-04-07 07:11:12.243274: Average global foreground Dice: [0.929, 0.9781, 0.9855, 0.4424] 
2022-04-07 07:11:12.254264: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:11:13.294013: lr: 0.000747 
2022-04-07 07:11:13.295917: saving scheduled checkpoint file... 
2022-04-07 07:11:13.366084: saving checkpoint... 
2022-04-07 07:11:17.982633: done, saving took 4.68 seconds 
2022-04-07 07:11:18.002556: done 
2022-04-07 07:11:18.008227: This epoch took 126.650028 s
 
2022-04-07 07:11:18.010740: svmem(total=99932160000, available=83873677312, percent=16.1, used=13029376000, free=69391892480, active=23895166976, inactive=4395159552, buffers=26341376, cached=17484550144, shared=1218482176, slab=780849152) 
2022-04-07 07:11:18.014641: 
epoch:  472 
2022-04-07 07:13:06.457970: train loss : -0.7364 
2022-04-07 07:13:17.632378: validation loss: -0.7253 
2022-04-07 07:13:17.638180: Average global foreground Dice: [0.9153, 0.9754, 0.9852, 0.4394] 
2022-04-07 07:13:17.643178: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:13:18.654557: lr: 0.000723 
2022-04-07 07:13:18.666538: saving scheduled checkpoint file... 
2022-04-07 07:13:18.764086: saving checkpoint... 
2022-04-07 07:13:23.249583: done, saving took 4.57 seconds 
2022-04-07 07:13:23.287106: done 
2022-04-07 07:13:23.297302: This epoch took 125.275303 s
 
2022-04-07 07:13:23.309570: svmem(total=99932160000, available=84027985920, percent=15.9, used=12874248192, free=69465747456, active=23663656960, inactive=4558032896, buffers=26574848, cached=17565589504, shared=1218310144, slab=779300864) 
2022-04-07 07:13:23.319636: 
epoch:  473 
2022-04-07 07:15:13.379670: train loss : -0.7334 
2022-04-07 07:15:26.830674: validation loss: -0.7130 
2022-04-07 07:15:26.836833: Average global foreground Dice: [0.9189, 0.9792, 0.987, 0.4531] 
2022-04-07 07:15:26.837676: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:15:27.729034: lr: 0.000699 
2022-04-07 07:15:27.736546: saving scheduled checkpoint file... 
2022-04-07 07:15:27.809445: saving checkpoint... 
2022-04-07 07:15:32.235292: done, saving took 4.49 seconds 
2022-04-07 07:15:32.248596: done 
2022-04-07 07:15:32.295965: saving checkpoint... 
2022-04-07 07:15:36.709769: done, saving took 4.46 seconds 
2022-04-07 07:15:36.723613: This epoch took 133.392228 s
 
2022-04-07 07:15:36.724652: svmem(total=99932160000, available=83928453120, percent=16.0, used=12968656896, free=69384552448, active=23900381184, inactive=4396429312, buffers=26853376, cached=17552097280, shared=1223524352, slab=780595200) 
2022-04-07 07:15:36.725234: 
epoch:  474 
2022-04-07 07:17:25.253726: train loss : -0.7325 
2022-04-07 07:17:36.928267: validation loss: -0.7403 
2022-04-07 07:17:36.933796: Average global foreground Dice: [0.9403, 0.9765, 0.9875, 0.4289] 
2022-04-07 07:17:36.936732: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:17:37.764239: lr: 0.000675 
2022-04-07 07:17:37.775918: saving scheduled checkpoint file... 
2022-04-07 07:17:37.873371: saving checkpoint... 
2022-04-07 07:17:42.454390: done, saving took 4.67 seconds 
2022-04-07 07:17:42.475034: done 
2022-04-07 07:17:42.528092: saving checkpoint... 
2022-04-07 07:17:46.960634: done, saving took 4.48 seconds 
2022-04-07 07:17:46.974207: This epoch took 130.248300 s
 
2022-04-07 07:17:46.975088: svmem(total=99932160000, available=84041310208, percent=15.9, used=12860825600, free=69391826944, active=23648546816, inactive=4643622912, buffers=27148288, cached=17652359168, shared=1218396160, slab=780730368) 
2022-04-07 07:17:46.975705: 
epoch:  475 
2022-04-07 07:19:34.520949: train loss : -0.7454 
2022-04-07 07:19:46.364691: validation loss: -0.7052 
2022-04-07 07:19:46.372448: Average global foreground Dice: [0.9166, 0.9758, 0.9855, 0.4036] 
2022-04-07 07:19:46.380544: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:19:47.192178: lr: 0.00065 
2022-04-07 07:19:47.203496: saving scheduled checkpoint file... 
2022-04-07 07:19:47.265687: saving checkpoint... 
2022-04-07 07:19:51.991482: done, saving took 4.78 seconds 
2022-04-07 07:19:52.008091: done 
2022-04-07 07:19:52.008814: This epoch took 125.032482 s
 
2022-04-07 07:19:52.009623: svmem(total=99932160000, available=84090990592, percent=15.9, used=12810952704, free=69462605824, active=23810883584, inactive=4410224640, buffers=27389952, cached=17631211520, shared=1218482176, slab=781598720) 
2022-04-07 07:19:52.010231: 
epoch:  476 
2022-04-07 07:21:42.313125: train loss : -0.7380 
2022-04-07 07:21:54.017823: validation loss: -0.7260 
2022-04-07 07:21:54.025389: Average global foreground Dice: [0.9255, 0.9809, 0.9866, 0.4581] 
2022-04-07 07:21:54.033324: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:21:54.908310: lr: 0.000626 
2022-04-07 07:21:54.911582: saving scheduled checkpoint file... 
2022-04-07 07:21:55.048438: saving checkpoint... 
2022-04-07 07:21:59.491442: done, saving took 4.58 seconds 
2022-04-07 07:21:59.527459: done 
2022-04-07 07:21:59.533654: This epoch took 127.522843 s
 
2022-04-07 07:21:59.544258: svmem(total=99932160000, available=84118458368, percent=15.8, used=12783575040, free=69495197696, active=23773884416, inactive=4413005824, buffers=27639808, cached=17625747456, shared=1218138112, slab=782237696) 
2022-04-07 07:21:59.547682: 
epoch:  477 
2022-04-07 07:23:48.687981: train loss : -0.7391 
2022-04-07 07:24:01.160118: validation loss: -0.7147 
2022-04-07 07:24:01.163529: Average global foreground Dice: [0.9209, 0.9749, 0.9861, 0.4257] 
2022-04-07 07:24:01.167784: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:24:01.978185: lr: 0.000601 
2022-04-07 07:24:01.982229: saving scheduled checkpoint file... 
2022-04-07 07:24:02.094248: saving checkpoint... 
2022-04-07 07:24:06.585978: done, saving took 4.60 seconds 
2022-04-07 07:24:06.600585: done 
2022-04-07 07:24:06.602915: This epoch took 127.051754 s
 
2022-04-07 07:24:06.604666: svmem(total=99932160000, available=83981361152, percent=16.0, used=12917669888, free=69496344576, active=23820427264, inactive=4369137664, buffers=27877376, cached=17490268160, shared=1221455872, slab=782151680) 
2022-04-07 07:24:06.605353: 
epoch:  478 
2022-04-07 07:25:56.941660: train loss : -0.7355 
2022-04-07 07:26:09.144156: validation loss: -0.7259 
2022-04-07 07:26:09.145754: Average global foreground Dice: [0.9201, 0.9751, 0.9853, 0.4432] 
2022-04-07 07:26:09.146480: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:26:09.686994: lr: 0.000577 
2022-04-07 07:26:09.687838: saving scheduled checkpoint file... 
2022-04-07 07:26:09.795049: saving checkpoint... 
2022-04-07 07:26:14.428232: done, saving took 4.74 seconds 
2022-04-07 07:26:14.455836: done 
2022-04-07 07:26:14.457546: This epoch took 127.851611 s
 
2022-04-07 07:26:14.459974: svmem(total=99932160000, available=83980169216, percent=16.0, used=12921421824, free=69496545280, active=23748636672, inactive=4441210880, buffers=28151808, cached=17486041088, shared=1218482176, slab=781500416) 
2022-04-07 07:26:14.462095: 
epoch:  479 
2022-04-07 07:28:03.996677: train loss : -0.7373 
2022-04-07 07:28:16.722370: validation loss: -0.7284 
2022-04-07 07:28:16.728132: Average global foreground Dice: [0.9136, 0.9767, 0.9836, 0.4311] 
2022-04-07 07:28:16.728869: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:28:17.763774: lr: 0.000552 
2022-04-07 07:28:17.768067: saving scheduled checkpoint file... 
2022-04-07 07:28:17.859608: saving checkpoint... 
2022-04-07 07:28:22.188543: done, saving took 4.42 seconds 
2022-04-07 07:28:22.202113: done 
2022-04-07 07:28:22.205660: This epoch took 127.742470 s
 
2022-04-07 07:28:22.206660: svmem(total=99932160000, available=83914166272, percent=16.0, used=12987613184, free=69383946240, active=23867858944, inactive=4430884864, buffers=28393472, cached=17532207104, shared=1218494464, slab=780161024) 
2022-04-07 07:28:22.208407: 
epoch:  480 
2022-04-07 07:30:10.411127: train loss : -0.7387 
2022-04-07 07:30:21.295035: validation loss: -0.7280 
2022-04-07 07:30:21.306624: Average global foreground Dice: [0.9241, 0.9788, 0.984, 0.4254] 
2022-04-07 07:30:21.307335: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:30:22.132059: lr: 0.000527 
2022-04-07 07:30:22.133861: saving scheduled checkpoint file... 
2022-04-07 07:30:22.191631: saving checkpoint... 
2022-04-07 07:30:26.663083: done, saving took 4.53 seconds 
2022-04-07 07:30:26.700576: done 
2022-04-07 07:30:26.709060: This epoch took 124.497926 s
 
2022-04-07 07:30:26.715957: svmem(total=99932160000, available=83968921600, percent=16.0, used=12933898240, free=69385003008, active=23905062912, inactive=4396027904, buffers=28655616, cached=17584603136, shared=1218465792, slab=781328384) 
2022-04-07 07:30:26.718054: 
epoch:  481 
2022-04-07 07:32:16.371021: train loss : -0.7441 
2022-04-07 07:32:27.655831: validation loss: -0.7125 
2022-04-07 07:32:27.657497: Average global foreground Dice: [0.919, 0.9766, 0.9862, 0.4416] 
2022-04-07 07:32:27.658376: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:32:28.533526: lr: 0.000502 
2022-04-07 07:32:28.554210: saving scheduled checkpoint file... 
2022-04-07 07:32:28.673158: saving checkpoint... 
2022-04-07 07:32:33.036288: done, saving took 4.46 seconds 
2022-04-07 07:32:33.070610: done 
2022-04-07 07:32:33.075725: This epoch took 126.355426 s
 
2022-04-07 07:32:33.085798: svmem(total=99932160000, available=83907641344, percent=16.0, used=12994191360, free=69376950272, active=23861989376, inactive=4440121344, buffers=28897280, cached=17532121088, shared=1218506752, slab=781680640) 
2022-04-07 07:32:33.098253: 
epoch:  482 
2022-04-07 07:34:22.473836: train loss : -0.7359 
2022-04-07 07:34:33.364457: validation loss: -0.7307 
2022-04-07 07:34:33.372891: Average global foreground Dice: [0.9258, 0.9786, 0.9879, 0.4435] 
2022-04-07 07:34:33.382247: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:34:34.200261: lr: 0.000477 
2022-04-07 07:34:34.208188: saving scheduled checkpoint file... 
2022-04-07 07:34:34.273196: saving checkpoint... 
2022-04-07 07:34:38.734029: done, saving took 4.52 seconds 
2022-04-07 07:34:38.758412: done 
2022-04-07 07:34:38.768555: This epoch took 125.659007 s
 
2022-04-07 07:34:38.776222: svmem(total=99932160000, available=84156182528, percent=15.8, used=12770054144, free=69377392640, active=23709081600, inactive=4594724864, buffers=29147136, cached=17755566080, shared=1193840640, slab=781344768) 
2022-04-07 07:34:38.782909: 
epoch:  483 
2022-04-07 07:36:27.957924: train loss : -0.7467 
2022-04-07 07:36:39.038619: validation loss: -0.7271 
2022-04-07 07:36:39.064611: Average global foreground Dice: [0.9316, 0.9779, 0.987, 0.4564] 
2022-04-07 07:36:39.089396: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:36:39.626383: lr: 0.000451 
2022-04-07 07:36:39.635548: saving scheduled checkpoint file... 
2022-04-07 07:36:39.726836: saving checkpoint... 
2022-04-07 07:36:44.411488: done, saving took 4.77 seconds 
2022-04-07 07:36:44.437138: done 
2022-04-07 07:36:44.488896: saving checkpoint... 
2022-04-07 07:36:48.951179: done, saving took 4.51 seconds 
2022-04-07 07:36:48.968819: This epoch took 130.177932 s
 
2022-04-07 07:36:48.973603: svmem(total=99932160000, available=83974950912, percent=16.0, used=12925992960, free=69386653696, active=23625375744, inactive=4673159168, buffers=29421568, cached=17590091776, shared=1218572288, slab=782213120) 
2022-04-07 07:36:48.977836: 
epoch:  484 
2022-04-07 07:38:36.324320: train loss : -0.7488 
2022-04-07 07:38:47.604185: validation loss: -0.7341 
2022-04-07 07:38:47.605869: Average global foreground Dice: [0.9294, 0.9805, 0.9848, 0.4158] 
2022-04-07 07:38:47.606608: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:38:48.190878: lr: 0.000426 
2022-04-07 07:38:48.193746: saving scheduled checkpoint file... 
2022-04-07 07:38:48.257118: saving checkpoint... 
2022-04-07 07:38:53.173555: done, saving took 4.98 seconds 
2022-04-07 07:38:53.195074: done 
2022-04-07 07:38:53.197503: This epoch took 124.217438 s
 
2022-04-07 07:38:53.199728: svmem(total=99932160000, available=83954946048, percent=16.0, used=12941910016, free=69452058624, active=23632543744, inactive=4599263232, buffers=29650944, cached=17508540416, shared=1223364608, slab=781946880) 
2022-04-07 07:38:53.203068: 
epoch:  485 
2022-04-07 07:40:40.826501: train loss : -0.7418 
2022-04-07 07:40:51.585248: validation loss: -0.7357 
2022-04-07 07:40:51.592778: Average global foreground Dice: [0.9184, 0.9771, 0.9862, 0.4466] 
2022-04-07 07:40:51.595662: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:40:52.723344: lr: 0.0004 
2022-04-07 07:40:52.724333: saving scheduled checkpoint file... 
2022-04-07 07:40:52.803342: saving checkpoint... 
2022-04-07 07:40:57.321623: done, saving took 4.60 seconds 
2022-04-07 07:40:57.346380: done 
2022-04-07 07:40:57.350003: This epoch took 124.143934 s
 
2022-04-07 07:40:57.354933: svmem(total=99932160000, available=84051943424, percent=15.9, used=12850286592, free=69378269184, active=23771787264, inactive=4534460416, buffers=29884416, cached=17673719808, shared=1218494464, slab=781627392) 
2022-04-07 07:40:57.357497: 
epoch:  486 
2022-04-07 07:42:46.990076: train loss : -0.7423 
2022-04-07 07:42:57.969370: validation loss: -0.7131 
2022-04-07 07:42:57.975946: Average global foreground Dice: [0.9071, 0.9748, 0.9862, 0.4307] 
2022-04-07 07:42:57.982846: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:42:58.860424: lr: 0.000375 
2022-04-07 07:42:58.866636: saving scheduled checkpoint file... 
2022-04-07 07:42:59.068841: saving checkpoint... 
2022-04-07 07:43:03.679995: done, saving took 4.80 seconds 
2022-04-07 07:43:03.713243: done 
2022-04-07 07:43:03.723274: This epoch took 126.362184 s
 
2022-04-07 07:43:03.733132: svmem(total=99932160000, available=83994079232, percent=15.9, used=12907810816, free=69382455296, active=23884427264, inactive=4420313088, buffers=30126080, cached=17611767808, shared=1218580480, slab=781271040) 
2022-04-07 07:43:03.741878: 
epoch:  487 
2022-04-07 07:44:52.740103: train loss : -0.7460 
2022-04-07 07:45:04.910681: validation loss: -0.7096 
2022-04-07 07:45:04.914983: Average global foreground Dice: [0.9203, 0.9762, 0.986, 0.4398] 
2022-04-07 07:45:04.915850: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:45:05.709842: lr: 0.000348 
2022-04-07 07:45:05.714724: saving scheduled checkpoint file... 
2022-04-07 07:45:05.811861: saving checkpoint... 
2022-04-07 07:45:10.329014: done, saving took 4.61 seconds 
2022-04-07 07:45:10.361336: done 
2022-04-07 07:45:10.370904: This epoch took 126.620119 s
 
2022-04-07 07:45:10.378041: svmem(total=99932160000, available=83943235584, percent=16.0, used=12958662656, free=69373558784, active=23833542656, inactive=4474892288, buffers=30388224, cached=17569550336, shared=1218494464, slab=781266944) 
2022-04-07 07:45:10.384064: 
epoch:  488 
2022-04-07 07:46:58.485855: train loss : -0.7367 
2022-04-07 07:47:09.124447: validation loss: -0.7245 
2022-04-07 07:47:09.136942: Average global foreground Dice: [0.9415, 0.9805, 0.9854, 0.4195] 
2022-04-07 07:47:09.153698: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:47:09.916575: lr: 0.000322 
2022-04-07 07:47:09.928350: saving scheduled checkpoint file... 
2022-04-07 07:47:10.000190: saving checkpoint... 
2022-04-07 07:47:14.551153: done, saving took 4.61 seconds 
2022-04-07 07:47:14.579764: done 
2022-04-07 07:47:14.585655: This epoch took 124.196925 s
 
2022-04-07 07:47:14.592298: svmem(total=99932160000, available=84092674048, percent=15.9, used=12794155008, free=69429428224, active=23672373248, inactive=4580319232, buffers=30621696, cached=17677955072, shared=1232371712, slab=782024704) 
2022-04-07 07:47:14.595245: 
epoch:  489 
2022-04-07 07:49:04.284621: train loss : -0.7370 
2022-04-07 07:49:15.440737: validation loss: -0.7011 
2022-04-07 07:49:15.447123: Average global foreground Dice: [0.9213, 0.9791, 0.9827, 0.4117] 
2022-04-07 07:49:15.454158: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:49:16.414048: lr: 0.000296 
2022-04-07 07:49:16.423955: saving scheduled checkpoint file... 
2022-04-07 07:49:16.610076: saving checkpoint... 
2022-04-07 07:49:21.357627: done, saving took 4.93 seconds 
2022-04-07 07:49:21.385231: done 
2022-04-07 07:49:21.390074: This epoch took 126.792024 s
 
2022-04-07 07:49:21.393735: svmem(total=99932160000, available=83974524928, percent=16.0, used=12927332352, free=69477265408, active=23852761088, inactive=4356702208, buffers=30863360, cached=17496698880, shared=1218437120, slab=780480512) 
2022-04-07 07:49:21.396741: 
epoch:  490 
2022-04-07 07:51:10.446715: train loss : -0.7395 
2022-04-07 07:51:22.659184: validation loss: -0.7445 
2022-04-07 07:51:22.667119: Average global foreground Dice: [0.9336, 0.9778, 0.9868, 0.4347] 
2022-04-07 07:51:22.676597: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:51:23.472933: lr: 0.000269 
2022-04-07 07:51:23.485555: saving scheduled checkpoint file... 
2022-04-07 07:51:23.628019: saving checkpoint... 
2022-04-07 07:51:27.782391: done, saving took 4.28 seconds 
2022-04-07 07:51:27.831804: done 
2022-04-07 07:51:27.849050: This epoch took 126.448840 s
 
2022-04-07 07:51:27.862349: svmem(total=99932160000, available=84055121920, percent=15.9, used=12846366720, free=69469937664, active=23788199936, inactive=4426113024, buffers=31109120, cached=17584746496, shared=1218502656, slab=781877248) 
2022-04-07 07:51:27.873823: 
epoch:  491 
2022-04-07 07:53:17.545831: train loss : -0.7464 
2022-04-07 07:53:30.158794: validation loss: -0.7241 
2022-04-07 07:53:30.160359: Average global foreground Dice: [0.9263, 0.9769, 0.9874, 0.4229] 
2022-04-07 07:53:30.161052: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:53:31.026019: lr: 0.000242 
2022-04-07 07:53:31.028063: saving scheduled checkpoint file... 
2022-04-07 07:53:31.103979: saving checkpoint... 
2022-04-07 07:53:35.418029: done, saving took 4.39 seconds 
2022-04-07 07:53:35.429888: done 
2022-04-07 07:53:35.431891: This epoch took 127.542753 s
 
2022-04-07 07:53:35.433443: svmem(total=99932160000, available=83967737856, percent=16.0, used=12934135808, free=69369348096, active=23945277440, inactive=4366503936, buffers=31350784, cached=17597325312, shared=1218510848, slab=783331328) 
2022-04-07 07:53:35.434967: 
epoch:  492 
2022-04-07 07:55:24.401379: train loss : -0.7385 
2022-04-07 07:55:36.070111: validation loss: -0.7095 
2022-04-07 07:55:36.080460: Average global foreground Dice: [0.9131, 0.9757, 0.9792, 0.414] 
2022-04-07 07:55:36.082100: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:55:36.791222: lr: 0.000215 
2022-04-07 07:55:36.793169: saving scheduled checkpoint file... 
2022-04-07 07:55:36.951442: saving checkpoint... 
2022-04-07 07:55:41.429509: done, saving took 4.62 seconds 
2022-04-07 07:55:41.447699: done 
2022-04-07 07:55:41.449826: This epoch took 126.014088 s
 
2022-04-07 07:55:41.452245: svmem(total=99932160000, available=83968872448, percent=16.0, used=12932804608, free=69368016896, active=23805751296, inactive=4508094464, buffers=31633408, cached=17599705088, shared=1218519040, slab=783003648) 
2022-04-07 07:55:41.453738: 
epoch:  493 
2022-04-07 07:57:29.164326: train loss : -0.7458 
2022-04-07 07:57:41.576488: validation loss: -0.7089 
2022-04-07 07:57:41.585516: Average global foreground Dice: [0.9117, 0.9756, 0.9873, 0.4128] 
2022-04-07 07:57:41.590518: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:57:42.442904: lr: 0.000187 
2022-04-07 07:57:42.443760: saving scheduled checkpoint file... 
2022-04-07 07:57:42.537954: saving checkpoint... 
2022-04-07 07:57:46.656867: done, saving took 4.21 seconds 
2022-04-07 07:57:46.682930: done 
2022-04-07 07:57:46.685621: This epoch took 125.229522 s
 
2022-04-07 07:57:46.688192: svmem(total=99932160000, available=83896197120, percent=16.0, used=13005643776, free=69369741312, active=23809069056, inactive=4503248896, buffers=31875072, cached=17524899840, shared=1218519040, slab=782344192) 
2022-04-07 07:57:46.692533: 
epoch:  494 
2022-04-07 07:59:35.192908: train loss : -0.7423 
2022-04-07 07:59:46.709036: validation loss: -0.7389 
2022-04-07 07:59:46.720963: Average global foreground Dice: [0.9269, 0.9781, 0.9872, 0.4561] 
2022-04-07 07:59:46.727861: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 07:59:47.624291: lr: 0.000158 
2022-04-07 07:59:47.625274: saving scheduled checkpoint file... 
2022-04-07 07:59:47.737281: saving checkpoint... 
2022-04-07 07:59:52.222216: done, saving took 4.60 seconds 
2022-04-07 07:59:52.237328: done 
2022-04-07 07:59:52.239513: This epoch took 125.544804 s
 
2022-04-07 07:59:52.240780: svmem(total=99932160000, available=83979493376, percent=16.0, used=12917264384, free=69363601408, active=23910748160, inactive=4412043264, buffers=32104448, cached=17619189760, shared=1223475200, slab=783425536) 
2022-04-07 07:59:52.243079: 
epoch:  495 
2022-04-07 08:01:41.509322: train loss : -0.7345 
2022-04-07 08:01:52.970505: validation loss: -0.7140 
2022-04-07 08:01:52.975996: Average global foreground Dice: [0.9297, 0.9791, 0.9857, 0.4295] 
2022-04-07 08:01:52.981775: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 08:01:53.874081: lr: 0.00013 
2022-04-07 08:01:53.877672: saving scheduled checkpoint file... 
2022-04-07 08:01:53.971697: saving checkpoint... 
2022-04-07 08:01:58.587481: done, saving took 4.71 seconds 
2022-04-07 08:01:58.603307: done 
2022-04-07 08:01:58.605641: This epoch took 126.360415 s
 
2022-04-07 08:01:58.606928: svmem(total=99932160000, available=83839430656, percent=16.1, used=13057110016, free=69364760576, active=23885217792, inactive=4437794816, buffers=32366592, cached=17477922816, shared=1223733248, slab=782979072) 
2022-04-07 08:01:58.608963: 
epoch:  496 
2022-04-07 08:03:47.375947: train loss : -0.7484 
2022-04-07 08:04:03.274945: validation loss: -0.7329 
2022-04-07 08:04:03.276530: Average global foreground Dice: [0.9354, 0.9805, 0.9873, 0.4226] 
2022-04-07 08:04:03.277236: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 08:04:04.077100: lr: 0.0001 
2022-04-07 08:04:04.090393: saving scheduled checkpoint file... 
2022-04-07 08:04:04.179097: saving checkpoint... 
2022-04-07 08:04:08.686797: done, saving took 4.59 seconds 
2022-04-07 08:04:08.723930: done 
2022-04-07 08:04:08.734771: This epoch took 130.124215 s
 
2022-04-07 08:04:08.745250: svmem(total=99932160000, available=84036177920, percent=15.9, used=12864643072, free=69438750720, active=23612493824, inactive=4630814720, buffers=32608256, cached=17596157952, shared=1218949120, slab=782651392) 
2022-04-07 08:04:08.759070: 
epoch:  497 
2022-04-07 08:05:58.163171: train loss : -0.7503 
2022-04-07 08:06:09.721695: validation loss: -0.7221 
2022-04-07 08:06:09.728468: Average global foreground Dice: [0.9265, 0.979, 0.9871, 0.4201] 
2022-04-07 08:06:09.732931: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 08:06:10.850956: lr: 6.9e-05 
2022-04-07 08:06:10.854235: saving scheduled checkpoint file... 
2022-04-07 08:06:10.967556: saving checkpoint... 
2022-04-07 08:06:15.472679: done, saving took 4.61 seconds 
2022-04-07 08:06:15.499477: done 
2022-04-07 08:06:15.505080: This epoch took 126.737855 s
 
2022-04-07 08:06:15.511076: svmem(total=99932160000, available=84032212992, percent=15.9, used=12869193728, free=69358301184, active=23710433280, inactive=4611149824, buffers=32870400, cached=17671794688, shared=1218691072, slab=782217216) 
2022-04-07 08:06:15.512881: 
epoch:  498 
2022-04-07 08:08:04.899803: train loss : -0.7377 
2022-04-07 08:08:16.155411: validation loss: -0.7295 
2022-04-07 08:08:16.161540: Average global foreground Dice: [0.9263, 0.9782, 0.9889, 0.4385] 
2022-04-07 08:08:16.163309: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 08:08:16.829793: lr: 3.7e-05 
2022-04-07 08:08:16.841186: saving scheduled checkpoint file... 
2022-04-07 08:08:17.036692: saving checkpoint... 
2022-04-07 08:08:21.532523: done, saving took 4.68 seconds 
2022-04-07 08:08:21.562206: done 
2022-04-07 08:08:21.564994: This epoch took 126.049239 s
 
2022-04-07 08:08:21.570035: svmem(total=99932160000, available=83994206208, percent=15.9, used=12906717184, free=69359841280, active=23869677568, inactive=4453449728, buffers=33112064, cached=17632489472, shared=1218605056, slab=781656064) 
2022-04-07 08:08:21.574134: 
epoch:  499 
2022-04-07 08:10:12.974231: train loss : -0.7451 
2022-04-07 08:10:24.667933: validation loss: -0.7239 
2022-04-07 08:10:24.681424: Average global foreground Dice: [0.9062, 0.976, 0.9879, 0.4327] 
2022-04-07 08:10:24.696075: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 
2022-04-07 08:10:25.683539: lr: 0.0 
2022-04-07 08:10:25.686127: saving scheduled checkpoint file... 
2022-04-07 08:10:25.801157: saving checkpoint... 
2022-04-07 08:10:30.435123: done, saving took 4.75 seconds 
2022-04-07 08:10:30.479410: done 
2022-04-07 08:10:30.492340: This epoch took 128.914845 s
 
2022-04-07 08:10:30.591941: saving checkpoint... 
2022-04-07 08:10:35.242154: done, saving took 4.73 seconds 
2022-04-07 08:21:15.412867: finished prediction 
2022-04-07 08:21:15.414446: evaluation of raw predictions