Starting... 2022-04-11 16:28:44.759222: Using splits from existing split file: /home/woody/iwi5/iwi5039h/nnUNet_data/nnUNet_preprocessed/Task500_Glacier_zonefronts/splits_final.pkl 2022-04-11 16:28:44.796048: The split file contains 5 splits. 2022-04-11 16:28:44.796841: Desired fold for training: 3 2022-04-11 16:28:44.797617: This split has 447 training and 112 validation cases. 2022-04-11 16:28:45.544957: TRAINING KEYS: odict_keys(['Crane_2002-11-09_ERS_20_2_061', 'Crane_2006-11-27_ERS_20_1_195', 'Crane_2007-02-05_ERS_20_1_195', 'Crane_2007-03-31_ENVISAT_20_3_467', 'Crane_2007-05-05_ENVISAT_20_1_467', 'Crane_2007-06-09_ENVISAT_20_1_467', 'Crane_2007-07-14_ENVISAT_20_1_467', 'Crane_2007-09-22_ENVISAT_20_1_467', 'Crane_2007-09-24_PALSAR_17_1_137', 'Crane_2007-10-27_ENVISAT_20_2_467', 'Crane_2007-11-21_PALSAR_17_2_135', 'Crane_2008-01-05_ENVISAT_20_2_467', 'Crane_2008-10-13_TSX_7_3_034', 'Crane_2008-11-04_TSX_7_1_034', 'Crane_2008-11-23_PALSAR_17_2_135', 'Crane_2009-09-29_PALSAR_17_1_137', 'Crane_2009-11-02_TSX_7_3_034', 'Crane_2009-11-13_TSX_7_1_034', 'Crane_2010-05-29_ENVISAT_20_3_467', 'Crane_2010-07-03_ENVISAT_20_2_467', 'Crane_2010-09-15_PALSAR_17_4_138', 'Crane_2010-11-27_TSX_7_1_110', 'Crane_2010-11-29_PALSAR_17_1_135', 'Crane_2010-12-08_TSX_7_1_110', 'Crane_2010-12-14_TSX_7_1_034', 'Crane_2010-12-19_TSX_7_1_110', 'Crane_2010-12-30_TSX_7_1_110', 'Crane_2011-06-02_TSX_7_2_110', 'Crane_2011-06-24_TSX_7_1_110', 'Crane_2011-07-16_TSX_7_1_110', 'Crane_2011-09-04_TSX_7_1_034', 'Crane_2011-10-18_TSX_7_1_034', 'Crane_2011-12-01_TSX_7_1_034', 'Crane_2011-12-12_TSX_7_2_034', 'Crane_2012-05-14_TSX_7_2_034', 'Crane_2012-06-16_TSX_7_2_034', 'Crane_2012-06-27_TSX_7_2_034', 'Crane_2012-07-03_TSX_7_1_125', 'Crane_2012-07-25_TSX_7_2_125', 'Crane_2012-07-30_TSX_7_2_034', 'Crane_2012-08-10_TSX_7_3_034', 'Crane_2012-09-01_TSX_7_1_034', 'Crane_2012-10-04_TSX_7_3_034', 'Crane_2012-11-06_TSX_7_2_034', 'Crane_2013-02-24_TSX_7_2_034', 'Crane_2013-03-07_TSX_7_2_034', 'Crane_2013-03-29_TSX_7_2_034', 'Crane_2013-06-30_TSX_7_2_110', 'Crane_2013-07-17_TSX_7_3_034', 'Crane_2013-07-22_TSX_7_3_110', 'Crane_2013-08-02_TSX_7_3_110', 'Crane_2013-08-19_TSX_7_4_034', 'Crane_2013-11-15_TSX_7_3_034', 'Crane_2013-11-26_TSX_7_4_034', 'Crane_2013-12-07_TSX_7_5_034', 'Crane_2013-12-18_TSX_7_4_034', 'Crane_2013-12-29_TSX_7_4_034', 'Crane_2014-01-20_TSX_7_2_034', 'Crane_2014-02-11_TSX_7_3_034', 'DBE_1996-01-25_ERS_20_1_152', 'DBE_1997-01-29_ERS_20_2_424', 'DBE_1997-03-05_ERS_20_1_424', 'DBE_1998-02-18_ERS_20_1_424', 'DBE_1999-11-10_ERS_20_4_424', 'DBE_2000-09-10_RSAT_20_3_282', 'DBE_2000-10-04_RSAT_20_4_282', 'DBE_2000-10-06_RSAT_12_3_306', 'DBE_2002-11-16_ERS_20_3_161', 'DBE_2002-12-21_ERS_20_4_161', 'DBE_2003-01-08_ERS_20_4_424', 'DBE_2003-02-12_ERS_20_3_424', 'DBE_2003-11-19_ERS_20_3_424', 'DBE_2004-03-03_ENVISAT_20_3_424', 'DBE_2004-04-07_ENVISAT_20_3_424', 'DBE_2004-05-12_ENVISAT_20_3_424', 'DBE_2004-06-16_ENVISAT_20_3_424', 'DBE_2004-07-21_ENVISAT_20_3_424', 'DBE_2004-08-25_ENVISAT_20_3_424', 'DBE_2004-11-09_RSAT_12_3_063', 'DBE_2004-12-08_ENVISAT_20_5_424', 'DBE_2004-12-08_ERS_20_5_424', 'DBE_2004-12-11_RSAT_12_4_182', 'DBE_2005-01-12_ERS_20_6_424', 'DBE_2005-02-16_ENVISAT_20_1_424', 'DBE_2005-03-23_ENVISAT_20_1_424', 'DBE_2005-04-27_ENVISAT_20_2_424', 'DBE_2005-05-11_RSAT_12_3_282', 'DBE_2005-06-01_ENVISAT_20_2_424', 'DBE_2005-08-10_ENVISAT_20_2_424', 'DBE_2006-02-23_RSAT_12_3_282', 'DBE_2006-03-08_ENVISAT_20_1_424', 'DBE_2006-03-19_RSAT_12_4_282', 'DBE_2006-04-12_ENVISAT_20_1_424', 'DBE_2006-05-18_PALSAR_17_2_135', 'DBE_2006-06-21_ENVISAT_20_2_424', 'DBE_2006-06-21_PALSAR_17_2_137', 'DBE_2006-07-03_PALSAR_17_3_135', 'DBE_2006-08-06_PALSAR_17_2_137', 'DBE_2006-08-10_RSAT_20_4_282', 'DBE_2006-10-04_ENVISAT_20_3_424', 'DBE_2006-11-06_PALSAR_17_3_137', 'DBE_2006-11-08_ENVISAT_20_3_424', 'DBE_2007-01-01_RSAT_12_5_282', 'DBE_2007-02-18_RSAT_12_4_282', 'DBE_2007-03-14_RSAT_12_3_282', 'DBE_2007-05-01_RSAT_12_4_282', 'DBE_2007-08-15_ENVISAT_20_5_424', 'DBE_2007-09-19_ENVISAT_20_6_424', 'DBE_2007-11-11_PALSAR_17_4_129', 'DBE_2007-12-03_RSAT_12_4_282', 'DBE_2007-12-27_RSAT_12_5_282', 'DBE_2008-01-20_RSAT_12_4_282', 'DBE_2008-03-30_TSX_7_3_049', 'DBE_2008-09-28_PALSAR_17_5_129', 'DBE_2008-10-14_TSX_7_3_049', 'DBE_2008-10-25_TSX_7_2_049', 'DBE_2008-11-05_TSX_7_2_049', 'DBE_2008-12-29_PALSAR_17_5_129', 'DBE_2009-02-13_PALSAR_17_5_129', 'DBE_2009-10-23_TSX_7_2_049', 'DBE_2009-12-03_PALSAR_17_5_132', 'DBE_2010-01-22_ERS_20_6_152', 'DBE_2010-03-17_ENVISAT_20_6_424', 'DBE_2010-04-21_ENVISAT_20_6_424', 'DBE_2010-10-10_TSX_7_2_049', 'DBE_2010-10-16_PALSAR_17_4_127', 'DBE_2010-10-21_PALSAR_17_4_132', 'DBE_2010-11-12_TSX_7_4_049', 'DBE_2010-11-19_PALSAR_17_4_129', 'DBE_2010-11-23_TSX_7_5_049', 'DBE_2010-12-01_PALSAR_17_4_127', 'DBE_2010-12-06_PALSAR_17_3_132', 'DBE_2011-01-04_PALSAR_17_4_129', 'DBE_2011-01-16_PALSAR_17_4_127', 'DBE_2011-01-28_TSX_7_2_049', 'DBE_2011-07-05_TSX_7_1_110', 'DBE_2011-09-09_TSX_7_1_110', 'DBE_2011-09-20_TSX_7_1_110', 'DBE_2011-10-23_TSX_7_1_110', 'DBE_2011-11-14_TSX_7_1_110', 'DBE_2011-12-06_TSX_7_5_110', 'DBE_2012-03-14_TSX_7_1_110', 'DBE_2012-04-05_TSX_7_1_110', 'DBE_2012-06-10_TSX_7_1_110', 'DBE_2012-08-04_TSX_7_1_110', 'DBE_2012-08-15_TSX_7_1_110', 'DBE_2012-10-31_TSX_7_1_110', 'DBE_2012-11-11_TSX_7_1_110', 'DBE_2013-01-16_TSX_7_5_110', 'DBE_2013-03-23_TSX_7_1_110', 'DBE_2013-04-03_TSX_7_1_110', 'DBE_2013-05-28_TSX_7_1_110', 'DBE_2013-06-19_TSX_7_1_110', 'DBE_2013-10-07_TSX_7_1_110', 'DBE_2013-11-09_TSX_7_1_110', 'DBE_2013-12-01_TSX_7_2_110', 'DBE_2013-12-12_TSX_7_3_110', 'DBE_2014-01-03_TSX_7_5_110', 'DBE_2014-03-21_TSX_7_2_110', 'JAC_2009-04-16_TSX_6_1_005', 'JAC_2009-04-27_TSX_6_1_005', 'JAC_2009-05-08_TSX_6_1_005', 'JAC_2009-05-30_TSX_6_1_005', 'JAC_2009-06-10_TSX_6_1_005', 'JAC_2009-06-21_TSX_6_1_005', 'JAC_2009-07-02_TSX_6_1_005', 'JAC_2009-08-04_TSX_6_1_005', 'JAC_2009-08-15_TSX_6_1_005', 'JAC_2009-08-26_TSX_6_1_005', 'JAC_2009-09-06_TSX_6_1_005', 'JAC_2009-09-17_TSX_6_1_005', 'JAC_2009-09-28_TSX_6_1_005', 'JAC_2009-10-09_TSX_6_1_005', 'JAC_2009-10-20_TSX_6_1_005', 'JAC_2009-10-31_TSX_6_1_005', 'JAC_2009-11-11_TSX_6_1_005', 'JAC_2009-11-22_TSX_6_1_005', 'JAC_2009-12-03_TSX_6_1_005', 'JAC_2009-12-14_TSX_6_1_005', 'JAC_2010-01-05_TSX_6_1_005', 'JAC_2010-01-16_TSX_6_1_005', 'JAC_2010-02-18_TSX_6_1_005', 'JAC_2010-03-12_TSX_6_1_005', 'JAC_2010-03-23_TSX_6_1_005', 'JAC_2010-04-14_TSX_6_1_005', 'JAC_2010-05-06_TSX_6_1_005', 'JAC_2010-05-17_TSX_6_1_005', 'JAC_2010-06-19_TSX_6_1_005', 'JAC_2010-06-30_TSX_6_1_005', 'JAC_2010-07-22_TSX_6_1_005', 'JAC_2010-09-04_TSX_6_1_005', 'JAC_2010-09-15_TSX_6_1_005', 'JAC_2010-10-18_TSX_6_1_005', 'JAC_2010-11-09_TSX_6_1_005', 'JAC_2010-12-01_TSX_6_1_005', 'JAC_2010-12-23_TSX_6_1_005', 'JAC_2011-01-03_TSX_6_1_005', 'JAC_2011-02-16_TSX_6_1_005', 'JAC_2011-03-10_TSX_6_1_005', 'JAC_2011-03-21_TSX_6_1_005', 'JAC_2011-04-12_TSX_6_1_005', 'JAC_2011-04-23_TSX_6_1_005', 'JAC_2011-05-15_TSX_6_1_005', 'JAC_2011-05-26_TSX_6_1_005', 'JAC_2011-06-17_TSX_6_1_005', 'JAC_2011-07-20_TSX_6_1_005', 'JAC_2011-08-23_TSX_6_1_027', 'JAC_2011-09-14_TSX_6_1_027', 'JAC_2011-10-05_TSX_6_1_005', 'JAC_2011-10-16_TSX_6_1_005', 'JAC_2011-11-18_TSX_6_1_005', 'JAC_2011-12-10_TSX_6_1_005', 'JAC_2011-12-21_TSX_6_1_005', 'JAC_2012-02-14_TSX_6_1_005', 'JAC_2012-02-25_TSX_6_1_005', 'JAC_2012-03-07_TSX_6_1_005', 'JAC_2012-04-20_TSX_6_1_005', 'JAC_2012-05-24_TSX_6_1_027', 'JAC_2012-06-03_TSX_6_1_005', 'JAC_2012-06-14_TSX_6_1_005', 'JAC_2012-07-07_TSX_6_1_027', 'JAC_2012-07-18_TSX_6_1_027', 'JAC_2012-08-20_TSX_6_1_027', 'JAC_2012-09-11_TSX_6_1_027', 'JAC_2012-09-21_TSX_6_1_005', 'JAC_2012-10-14_TSX_6_1_027', 'JAC_2012-11-04_TSX_6_1_005', 'JAC_2012-11-15_TSX_6_1_005', 'JAC_2012-12-07_TSX_6_1_005', 'JAC_2012-12-19_TSX_6_1_027', 'JAC_2013-01-09_TSX_6_1_005', 'JAC_2013-01-20_TSX_6_1_005', 'JAC_2013-02-11_TSX_6_1_005', 'JAC_2013-02-22_TSX_6_1_005', 'JAC_2013-03-06_TSX_6_1_027', 'JAC_2013-03-17_TSX_6_1_027', 'JAC_2013-04-07_TSX_6_1_005', 'JAC_2013-04-18_TSX_6_1_005', 'JAC_2013-05-10_TSX_6_1_005', 'JAC_2013-05-22_TSX_6_1_027', 'JAC_2013-06-13_TSX_6_1_027', 'JAC_2013-07-05_TSX_6_1_027', 'JAC_2013-07-27_TSX_6_1_027', 'JAC_2013-08-17_TSX_6_1_005', 'JAC_2013-08-28_TSX_6_1_005', 'JAC_2013-09-08_TSX_6_1_005', 'JAC_2013-10-12_TSX_6_1_027', 'JAC_2013-10-23_TSX_6_1_027', 'JAC_2013-11-14_TSX_6_1_027', 'JAC_2013-11-25_TSX_6_1_027', 'JAC_2013-12-17_TSX_6_1_027', 'JAC_2014-01-29_TSX_6_1_005', 'JAC_2014-03-03_TSX_6_1_005', 'JAC_2014-03-25_TSX_6_1_005', 'JAC_2014-04-05_TSX_6_1_005', 'JAC_2014-04-16_TSX_6_1_005', 'JAC_2014-04-27_TSX_6_1_005', 'JAC_2014-07-02_TSX_6_1_005', 'JAC_2014-07-03_TSX_6_1_027', 'JAC_2014-07-25_TSX_6_1_027', 'JAC_2014-08-04_TSX_6_1_005', 'JAC_2014-09-18_TSX_6_1_027', 'JAC_2014-09-28_TSX_6_1_005', 'JAC_2014-10-09_TSX_6_1_005', 'JAC_2014-10-20_TSX_6_1_005', 'JAC_2014-11-11_TSX_6_1_005', 'JAC_2014-12-14_TSX_6_1_005', 'JAC_2014-12-25_TSX_6_1_005', 'JAC_2015-01-05_TSX_6_1_005', 'JAC_2015-01-16_TSX_6_1_005', 'JAC_2015-01-27_TSX_6_1_005', 'JAC_2015-03-12_TSX_6_1_005', 'JAC_2015-03-23_TSX_6_1_005', 'JAC_2015-04-14_TSX_6_1_005', 'JAC_2015-04-25_TSX_6_1_005', 'JAC_2015-05-06_TSX_6_1_005', 'JAC_2015-06-08_TSX_6_1_005', 'JAC_2015-06-19_TSX_6_1_005', 'JAC_2015-07-11_TSX_6_1_005', 'JAC_2015-07-22_TSX_6_1_005', 'JAC_2015-08-02_TSX_6_1_005', 'JAC_2015-08-13_TSX_6_1_005', 'JAC_2015-09-04_TSX_6_1_005', 'JAC_2015-09-26_TSX_6_1_005', 'JAC_2015-10-18_TSX_6_1_005', 'JAC_2015-10-29_TSX_6_1_005', 'JAC_2015-11-09_TSX_6_1_005', 'JAC_2015-11-20_TSX_6_1_005', 'JAC_2015-12-12_TSX_6_1_005', 'Jorum_2003-01-27_ERS_20_1_195', 'Jorum_2006-11-27_ERS_20_3_195', 'Jorum_2007-02-05_ERS_20_1_195', 'Jorum_2007-05-05_ENVISAT_20_2_467', 'Jorum_2007-06-09_ENVISAT_20_2_467', 'Jorum_2007-07-14_ENVISAT_20_2_467', 'Jorum_2007-09-22_ENVISAT_20_3_467', 'Jorum_2007-10-27_ENVISAT_20_3_467', 'Jorum_2007-11-21_PALSAR_17_2_135', 'Jorum_2008-01-05_ENVISAT_20_2_467', 'Jorum_2008-11-23_PALSAR_17_2_135', 'Jorum_2010-03-20_ENVISAT_20_2_467', 'Jorum_2010-05-29_ENVISAT_20_2_467', 'Jorum_2010-07-03_ENVISAT_20_2_467', 'Jorum_2010-10-14_PALSAR_17_2_135', 'Jorum_2010-11-05_TSX_7_1_110', 'Jorum_2010-11-16_TSX_7_2_110', 'Jorum_2010-11-27_TSX_7_2_110', 'Jorum_2010-11-29_PALSAR_17_1_135', 'Jorum_2010-12-08_TSX_7_2_110', 'Jorum_2010-12-14_TSX_7_1_034', 'Jorum_2010-12-19_TSX_7_1_110', 'Jorum_2010-12-25_TSX_7_1_034', 'Jorum_2010-12-30_TSX_7_1_110', 'Jorum_2011-06-02_TSX_7_3_110', 'Jorum_2011-06-24_TSX_7_2_110', 'Jorum_2011-07-16_TSX_7_3_110', 'Jorum_2011-12-01_TSX_7_1_034', 'Jorum_2011-12-12_TSX_7_3_034', 'Jorum_2012-05-14_TSX_7_2_034', 'Jorum_2012-06-16_TSX_7_2_034', 'Jorum_2012-06-27_TSX_7_2_034', 'Jorum_2012-07-03_TSX_7_2_125', 'Jorum_2012-07-25_TSX_7_2_125', 'Jorum_2012-07-30_TSX_7_2_034', 'Jorum_2012-08-21_TSX_7_2_034', 'Jorum_2012-09-01_TSX_7_2_034', 'Jorum_2012-10-04_TSX_7_2_034', 'Jorum_2012-11-06_TSX_7_2_034', 'Jorum_2013-02-24_TSX_7_3_034', 'Jorum_2013-03-07_TSX_7_2_034', 'Jorum_2013-06-30_TSX_7_2_110', 'Jorum_2013-07-17_TSX_7_1_034', 'Jorum_2013-07-22_TSX_7_1_110', 'Jorum_2013-08-02_TSX_7_2_110', 'Jorum_2013-08-19_TSX_7_2_034', 'Jorum_2013-11-04_TSX_7_2_034', 'Jorum_2013-11-15_TSX_7_2_034', 'Jorum_2013-11-26_TSX_7_5_034', 'Jorum_2013-12-29_TSX_7_4_034', 'Jorum_2014-01-20_TSX_7_2_034', 'Jorum_2014-02-11_TSX_7_2_034', 'Jorum_2016-07-03_S1_20_2_009', 'Jorum_2016-09-01_S1_20_2_009', 'Jorum_2016-12-30_S1_20_2_009', 'Jorum_2017-06-16_S1_20_2_009', 'Jorum_2018-06-11_S1_20_2_009', 'Jorum_2019-01-01_S1_20_3_009', 'Jorum_2019-03-02_S1_20_3_009', 'Jorum_2019-09-22_S1_20_3_009', 'Jorum_2020-01-08_S1_20_3_009', 'Jorum_2020-03-08_S1_20_3_009', 'Jorum_2020-06-12_S1_20_3_009', 'SI_1995-11-13_ERS_20_3_109', 'SI_1995-11-14_ERS_20_4_118', 'SI_1995-11-16_ERS_20_4_152', 'SI_1996-01-22_ERS_20_3_109', 'SI_1996-01-23_ERS_20_5_109', 'SI_1996-01-24_ERS_20_4_118', 'SI_1996-02-29_ERS_20_2_152', 'SI_1997-01-27_ERS_20_4_390', 'SI_1997-03-03_ERS_20_4_390', 'SI_1999-11-07_ERS_20_2_381', 'SI_2000-02-20_ERS_20_3_381', 'SI_2000-09-10_RSAT_20_5_282', 'SI_2000-10-13_RSAT_20_5_063', 'SI_2002-01-21_ERS_20_2_390', 'SI_2002-02-25_ERS_20_3_390', 'SI_2002-11-12_ERS_20_4_109', 'SI_2002-12-21_ERS_20_3_161', 'SI_2003-01-25_ERS_20_3_161', 'SI_2003-12-05_ENVISAT_20_4_152', 'SI_2004-02-13_ENVISAT_20_4_152', 'SI_2004-09-13_RSAT_12_5_282', 'SI_2004-10-15_ERS_20_5_152', 'SI_2004-11-09_RSAT_12_5_063', 'SI_2004-11-19_ERS_20_4_152', 'SI_2004-11-24_RSAT_12_4_282', 'SI_2004-12-03_RSAT_12_5_063', 'SI_2004-12-24_ENVISAT_20_5_152', 'SI_2005-01-28_ERS_20_5_152', 'SI_2005-03-04_ENVISAT_20_3_152', 'SI_2005-04-08_ENVISAT_20_2_152', 'SI_2005-04-17_RSAT_12_3_282', 'SI_2005-05-11_RSAT_12_2_282', 'SI_2005-07-22_ENVISAT_20_2_152', 'SI_2006-01-06_RSAT_12_2_282', 'SI_2006-01-29_ENVISAT_20_3_381', 'SI_2006-02-23_RSAT_12_1_282', 'SI_2006-03-05_ENVISAT_20_2_381', 'SI_2006-03-19_RSAT_12_2_282', 'SI_2006-04-12_RSAT_12_1_282', 'SI_2006-05-18_PALSAR_17_2_135', 'SI_2006-07-23_ENVISAT_20_4_381', 'SI_2006-08-10_RSAT_20_3_282', 'SI_2006-08-27_ENVISAT_20_3_381', 'SI_2006-09-03_RSAT_20_2_282', 'SI_2006-11-21_ERS_20_3_109', 'SI_2007-01-01_RSAT_12_3_282', 'SI_2007-01-14_ENVISAT_20_3_381', 'SI_2007-01-25_RSAT_12_1_282', 'SI_2007-07-08_ENVISAT_20_3_381', 'SI_2007-09-16_ENVISAT_20_3_381', 'SI_2007-10-21_ENVISAT_20_3_381', 'SI_2007-12-03_RSAT_12_4_282', 'SI_2007-12-27_RSAT_12_2_282', 'SI_2008-01-20_RSAT_12_2_282', 'SI_2008-04-13_ENVISAT_20_2_381', 'SI_2008-08-31_ENVISAT_20_4_381', 'SI_2009-03-29_ENVISAT_20_4_381', 'SI_2009-07-12_ENVISAT_20_3_381', 'SI_2009-08-16_ENVISAT_20_3_381', 'SI_2010-01-13_PALSAR_17_2_127', 'SI_2010-01-22_ERS_20_3_152', 'SI_2010-02-26_ERS_20_2_152', 'SI_2010-10-16_PALSAR_17_3_127', 'SI_2010-12-01_PALSAR_17_3_127', 'SI_2011-01-16_PALSAR_17_2_127', 'SI_2011-06-14_TSX_7_1_125', 'SI_2011-07-06_TSX_7_1_125', 'SI_2011-07-28_TSX_7_1_125', 'SI_2011-08-19_TSX_7_1_125', 'SI_2011-09-10_TSX_7_1_125', 'SI_2011-11-26_TSX_7_2_125', 'SI_2011-12-29_TSX_7_1_125', 'SI_2012-01-09_TSX_7_1_125', 'SI_2012-03-04_TSX_7_1_125', 'SI_2012-03-15_TSX_7_1_125', 'SI_2012-03-26_TSX_7_1_125', 'SI_2012-04-28_TSX_7_1_125', 'SI_2012-05-20_TSX_7_1_125', 'SI_2012-06-22_TSX_7_1_125', 'SI_2012-07-03_TSX_7_1_125', 'SI_2012-08-05_TSX_7_1_125', 'SI_2012-08-16_TSX_7_1_125', 'SI_2013-07-12_TSX_7_1_125', 'SI_2013-08-14_TSX_7_1_125', 'SI_2013-09-16_TSX_7_1_125', 'SI_2013-10-28_TSX_7_1_104', 'SI_2013-11-08_TSX_7_2_104', 'SI_2013-11-21_TSX_7_1_125', 'SI_2013-12-02_TSX_7_1_125', 'SI_2013-12-13_TSX_7_2_125', 'SI_2014-01-04_TSX_7_2_125', 'SI_2014-03-22_TSX_7_1_125', 'SI_2014-04-02_TSX_7_1_125', 'SI_2014-12-11_TSX_7_4_125', 'SI_2014-12-22_TSX_7_2_125']) 2022-04-11 16:28:45.546127: VALIDATION KEYS: odict_keys(['Crane_2002-11-18_ERS_20_3_195', 'Crane_2003-01-18_ERS_20_1_061', 'Crane_2008-10-24_TSX_7_2_034', 'Crane_2010-10-14_PALSAR_17_2_135', 'Crane_2010-11-05_TSX_7_2_110', 'Crane_2010-11-16_TSX_7_2_110', 'Crane_2010-12-25_TSX_7_1_034', 'Crane_2011-08-13_TSX_7_1_034', 'Crane_2012-08-21_TSX_7_3_034', 'Crane_2013-11-04_TSX_7_4_034', 'DBE_1995-11-16_ERS_20_1_152', 'DBE_2000-10-13_RSAT_20_4_063', 'DBE_2002-02-27_ERS_20_1_424', 'DBE_2004-03-03_ERS_20_3_424', 'DBE_2004-09-13_RSAT_12_3_282', 'DBE_2004-09-29_ENVISAT_20_3_424', 'DBE_2004-10-07_RSAT_12_4_282', 'DBE_2004-10-16_RSAT_12_3_063', 'DBE_2004-11-03_ENVISAT_20_4_424', 'DBE_2004-11-24_RSAT_12_3_282', 'DBE_2004-12-18_RSAT_12_3_282', 'DBE_2005-07-06_ENVISAT_20_3_424', 'DBE_2006-01-06_RSAT_12_4_282', 'DBE_2006-04-12_RSAT_12_1_282', 'DBE_2006-05-06_RSAT_12_3_282', 'DBE_2006-07-26_ENVISAT_20_3_424', 'DBE_2006-08-30_ENVISAT_20_2_424', 'DBE_2006-09-03_RSAT_20_5_282', 'DBE_2007-01-25_RSAT_12_5_282', 'DBE_2007-12-27_PALSAR_17_5_129', 'DBE_2008-11-13_PALSAR_17_5_129', 'DBE_2009-11-03_TSX_7_2_049', 'DBE_2010-01-18_PALSAR_17_5_132', 'DBE_2011-01-17_TSX_7_4_049', 'DBE_2011-07-27_TSX_7_1_110', 'DBE_2011-12-28_TSX_7_2_110', 'DBE_2012-05-08_TSX_7_1_110', 'DBE_2012-06-21_TSX_7_1_110', 'DBE_2013-03-12_TSX_7_1_110', 'DBE_2013-07-11_TSX_7_1_110', 'DBE_2013-09-04_TSX_7_1_110', 'DBE_2013-12-23_TSX_7_2_110', 'DBE_2014-02-05_TSX_7_1_110', 'JAC_2009-05-19_TSX_6_1_005', 'JAC_2009-07-13_TSX_6_1_005', 'JAC_2009-07-24_TSX_6_1_005', 'JAC_2009-12-25_TSX_6_1_005', 'JAC_2010-02-07_TSX_6_1_005', 'JAC_2010-04-25_TSX_6_1_005', 'JAC_2010-07-11_TSX_6_1_005', 'JAC_2010-10-07_TSX_6_1_005', 'JAC_2010-11-20_TSX_6_1_005', 'JAC_2011-01-25_TSX_6_1_005', 'JAC_2011-09-03_TSX_6_1_027', 'JAC_2011-11-08_TSX_6_1_027', 'JAC_2012-01-13_TSX_6_1_027', 'JAC_2012-01-23_TSX_6_1_005', 'JAC_2012-03-29_TSX_6_1_005', 'JAC_2012-08-09_TSX_6_1_027', 'JAC_2012-10-03_TSX_6_1_027', 'JAC_2013-06-24_TSX_6_1_027', 'JAC_2013-09-20_TSX_6_1_027', 'JAC_2013-12-06_TSX_6_1_027', 'JAC_2014-03-14_TSX_6_1_005', 'JAC_2014-06-10_TSX_6_1_005', 'JAC_2014-06-21_TSX_6_1_005', 'JAC_2014-07-14_TSX_6_1_027', 'JAC_2014-12-03_TSX_6_1_005', 'JAC_2015-02-07_TSX_6_1_005', 'JAC_2015-02-19_TSX_6_1_027', 'JAC_2015-05-17_TSX_6_1_005', 'JAC_2015-12-23_TSX_6_1_005', 'Jorum_2007-03-31_ENVISAT_20_2_467', 'Jorum_2009-12-25_PALSAR_17_1_136', 'Jorum_2010-09-15_PALSAR_17_2_138', 'Jorum_2011-08-13_TSX_7_2_034', 'Jorum_2011-09-04_TSX_7_4_034', 'Jorum_2011-10-18_TSX_7_3_034', 'Jorum_2012-08-10_TSX_7_2_034', 'Jorum_2013-03-29_TSX_7_2_034', 'Jorum_2013-12-07_TSX_7_5_034', 'Jorum_2013-12-18_TSX_7_5_034', 'Jorum_2017-09-08_S1_20_2_009', 'Jorum_2018-01-06_S1_20_3_009', 'Jorum_2018-03-07_S1_20_3_009', 'Jorum_2018-09-03_S1_20_3_009', 'SI_1995-11-15_ERS_20_3_118', 'SI_1996-02-12_ERS_20_2_390', 'SI_1996-03-18_ERS_20_2_390', 'SI_2000-09-19_RSAT_20_5_063', 'SI_2000-10-04_RSAT_20_5_282', 'SI_2004-07-02_ENVISAT_20_4_152', 'SI_2004-08-06_ENVISAT_20_4_152', 'SI_2004-12-18_RSAT_12_4_282', 'SI_2004-12-24_ERS_20_5_152', 'SI_2005-02-28_RSAT_12_5_282', 'SI_2005-08-26_ENVISAT_20_2_152', 'SI_2006-05-06_RSAT_12_3_282', 'SI_2006-07-03_PALSAR_17_3_135', 'SI_2006-09-03_RSAT_12_2_282', 'SI_2006-10-18_ERS_20_4_118', 'SI_2007-03-14_RSAT_12_2_282', 'SI_2007-03-25_ENVISAT_20_3_381', 'SI_2007-05-01_RSAT_12_2_282', 'SI_2007-06-03_ENVISAT_20_3_381', 'SI_2008-05-18_ENVISAT_20_3_381', 'SI_2008-07-27_ENVISAT_20_4_381', 'SI_2009-02-22_ENVISAT_20_2_381', 'SI_2010-04-02_ERS_20_2_152', 'SI_2011-03-03_PALSAR_17_2_127', 'SI_2011-09-21_TSX_7_1_125', 'SI_2013-10-19_TSX_7_1_125']) 2022-04-11 16:29:02.292370: lr: 0.01 2022-04-11 16:29:22.545760: Unable to plot network architecture: 2022-04-11 16:29:22.692704: No module named 'hiddenlayer' 2022-04-11 16:29:22.813418: printing the network instead: 2022-04-11 16:29:22.933979: Generic_UNet( (conv_blocks_localization): ModuleList( (0): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) (1): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) (2): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) (3): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) (4): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(512, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) (5): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(256, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) (6): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(128, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) (7): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(64, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) ) (conv_blocks_context): ModuleList( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(1, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) (1): ConvDropoutNormNonlin( (conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(32, 64, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) (1): ConvDropoutNormNonlin( (conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (2): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(64, 128, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) (1): ConvDropoutNormNonlin( (conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (3): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(128, 256, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) (1): ConvDropoutNormNonlin( (conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (4): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(256, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) (1): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (5): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) (1): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (6): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) (1): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (7): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) (1): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (8): Sequential( (0): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) (1): StackedConvLayers( (blocks): Sequential( (0): ConvDropoutNormNonlin( (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) ) ) ) ) ) (td): ModuleList() (tu): ModuleList( (0): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False) (1): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False) (2): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False) (3): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False) (4): ConvTranspose2d(480, 256, kernel_size=[2, 2], stride=[2, 2], bias=False) (5): ConvTranspose2d(256, 128, kernel_size=[2, 2], stride=[2, 2], bias=False) (6): ConvTranspose2d(128, 64, kernel_size=[2, 2], stride=[2, 2], bias=False) (7): ConvTranspose2d(64, 32, kernel_size=[2, 2], stride=[2, 2], bias=False) ) (seg_outputs): ModuleList( (0): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) (2): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) (3): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): Conv2d(256, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) (5): Conv2d(128, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) (6): Conv2d(64, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) (7): Conv2d(32, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) ) ) 2022-04-11 16:29:23.076799: 2022-04-11 16:29:23.214236: svmem(total=99932160000, available=83735687168, percent=16.2, used=13592068096, free=78474977280, active=13686714368, inactive=6144872448, buffers=10305536, cached=7854809088, shared=819576832, slab=667951104) 2022-04-11 16:29:23.350417: epoch: 0 2022-04-11 16:32:22.377419: train loss : 0.8238 2022-04-11 16:32:34.364377: validation loss: 0.6482 2022-04-11 16:32:34.366233: Average global foreground Dice: [0.1365, 0.5274, 0.4768, 0.0] 2022-04-11 16:32:34.367073: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:32:35.116948: lr: 0.009982 2022-04-11 16:32:35.117700: saving scheduled checkpoint file... 2022-04-11 16:32:35.302009: saving checkpoint... 2022-04-11 16:32:39.765702: done, saving took 4.65 seconds 2022-04-11 16:32:39.812358: done 2022-04-11 16:32:39.829616: This epoch took 196.333712 s 2022-04-11 16:32:39.844140: svmem(total=99932160000, available=79044161536, percent=20.9, used=17110863872, free=58140999680, active=26688561152, inactive=13375238144, buffers=11227136, cached=24669069312, shared=1971765248, slab=725995520) 2022-04-11 16:32:39.861673: epoch: 1 2022-04-11 16:34:30.169090: train loss : 0.5814 2022-04-11 16:34:41.335785: validation loss: 0.5843 2022-04-11 16:34:41.341337: Average global foreground Dice: [0.1099, 0.6281, 0.449, 0.0] 2022-04-11 16:34:41.346972: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:34:42.271893: lr: 0.009964 2022-04-11 16:34:42.272796: saving scheduled checkpoint file... 2022-04-11 16:34:42.431505: saving checkpoint... 2022-04-11 16:34:46.810588: done, saving took 4.54 seconds 2022-04-11 16:34:46.859161: done 2022-04-11 16:34:46.921122: saving checkpoint... 2022-04-11 16:34:51.201470: done, saving took 4.33 seconds 2022-04-11 16:34:51.231859: This epoch took 131.358964 s 2022-04-11 16:34:51.237672: svmem(total=99932160000, available=78568419328, percent=21.4, used=17386455040, free=52682633216, active=33331380224, inactive=12166164480, buffers=11280384, cached=29851791360, shared=2167574528, slab=747536384) 2022-04-11 16:34:51.242666: epoch: 2 2022-04-11 16:36:40.251366: train loss : 0.4983 2022-04-11 16:36:51.912824: validation loss: 0.3913 2022-04-11 16:36:51.924124: Average global foreground Dice: [0.2355, 0.7223, 0.5552, 0.0] 2022-04-11 16:36:51.925142: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:36:52.869706: lr: 0.009946 2022-04-11 16:36:52.870851: saving scheduled checkpoint file... 2022-04-11 16:36:53.028961: saving checkpoint... 2022-04-11 16:36:57.420493: done, saving took 4.55 seconds 2022-04-11 16:36:57.428782: done 2022-04-11 16:36:57.477852: saving checkpoint... 2022-04-11 16:37:04.475106: done, saving took 7.05 seconds 2022-04-11 16:37:04.797405: This epoch took 133.549628 s 2022-04-11 16:37:04.932465: svmem(total=99932160000, available=78369193984, percent=21.6, used=17583161344, free=50787315712, active=36907515904, inactive=10475245568, buffers=11366400, cached=31550316544, shared=2167488512, slab=757088256) 2022-04-11 16:37:05.056723: epoch: 3 2022-04-11 16:38:53.843630: train loss : 0.4510 2022-04-11 16:39:06.532329: validation loss: 0.3249 2022-04-11 16:39:06.538953: Average global foreground Dice: [0.368, 0.7077, 0.612, 0.0] 2022-04-11 16:39:06.539779: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:39:07.422154: lr: 0.009928 2022-04-11 16:39:07.423250: saving scheduled checkpoint file... 2022-04-11 16:39:07.581586: saving checkpoint... 2022-04-11 16:39:11.939366: done, saving took 4.52 seconds 2022-04-11 16:39:11.961097: done 2022-04-11 16:39:12.104056: saving checkpoint... 2022-04-11 16:39:17.486875: done, saving took 5.52 seconds 2022-04-11 16:39:17.827994: This epoch took 132.655202 s 2022-04-11 16:39:17.952953: svmem(total=99932160000, available=78456627200, percent=21.5, used=17492574208, free=49666019328, active=38652391424, inactive=9849757696, buffers=11419648, cached=32762146816, shared=2169384960, slab=759009280) 2022-04-11 16:39:18.094858: epoch: 4 2022-04-11 16:41:08.074464: train loss : 0.4095 2022-04-11 16:41:19.928014: validation loss: 0.4233 2022-04-11 16:41:19.929910: Average global foreground Dice: [0.401, 0.6334, 0.6498, 0.0] 2022-04-11 16:41:19.933225: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:41:20.782740: lr: 0.00991 2022-04-11 16:41:20.783747: saving scheduled checkpoint file... 2022-04-11 16:41:20.943076: saving checkpoint... 2022-04-11 16:41:25.307407: done, saving took 4.52 seconds 2022-04-11 16:41:25.329277: done 2022-04-11 16:41:25.475191: saving checkpoint... 2022-04-11 16:41:33.372874: done, saving took 8.04 seconds 2022-04-11 16:41:33.393566: This epoch took 135.158985 s 2022-04-11 16:41:33.394790: svmem(total=99932160000, available=78241517568, percent=21.7, used=17708875776, free=49057492992, active=40142872576, inactive=8980979712, buffers=11481088, cached=33154310144, shared=2167488512, slab=747163648) 2022-04-11 16:41:33.395427: epoch: 5 2022-04-11 16:43:23.123893: train loss : 0.4038 2022-04-11 16:43:34.841165: validation loss: 0.2959 2022-04-11 16:43:34.848693: Average global foreground Dice: [0.4321, 0.7177, 0.6236, 0.0] 2022-04-11 16:43:34.849535: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:43:35.880467: lr: 0.009892 2022-04-11 16:43:35.881387: saving scheduled checkpoint file... 2022-04-11 16:43:36.012734: saving checkpoint... 2022-04-11 16:43:40.388546: done, saving took 4.51 seconds 2022-04-11 16:43:40.414715: done 2022-04-11 16:43:40.565311: saving checkpoint... 2022-04-11 16:43:48.459023: done, saving took 8.04 seconds 2022-04-11 16:43:48.483974: This epoch took 135.087964 s 2022-04-11 16:43:48.486491: svmem(total=99932160000, available=78172225536, percent=21.8, used=17768009728, free=48825991168, active=40806719488, inactive=8548659200, buffers=11538432, cached=33326620672, shared=2177400832, slab=747896832) 2022-04-11 16:43:48.488877: epoch: 6 2022-04-11 16:45:38.107710: train loss : 0.3433 2022-04-11 16:45:50.700424: validation loss: 0.2512 2022-04-11 16:45:50.710503: Average global foreground Dice: [0.4365, 0.7714, 0.6671, 0.0] 2022-04-11 16:45:50.711750: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:45:51.600446: lr: 0.009874 2022-04-11 16:45:51.601555: saving scheduled checkpoint file... 2022-04-11 16:45:51.759733: saving checkpoint... 2022-04-11 16:45:56.036490: done, saving took 4.43 seconds 2022-04-11 16:45:56.056274: done 2022-04-11 16:45:56.202153: saving checkpoint... 2022-04-11 16:46:03.614984: done, saving took 7.56 seconds 2022-04-11 16:46:03.865997: This epoch took 135.374526 s 2022-04-11 16:46:03.965185: svmem(total=99932160000, available=77751353344, percent=22.2, used=18198671360, free=48387104768, active=42529529856, inactive=7247630336, buffers=11571200, cached=33334812672, shared=2167402496, slab=762642432) 2022-04-11 16:46:04.010477: epoch: 7 2022-04-11 16:47:53.524016: train loss : 0.3073 2022-04-11 16:48:04.754957: validation loss: 0.2666 2022-04-11 16:48:04.763534: Average global foreground Dice: [0.4036, 0.761, 0.6714, 0.0] 2022-04-11 16:48:04.765142: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:48:05.333871: lr: 0.009856 2022-04-11 16:48:05.334802: saving scheduled checkpoint file... 2022-04-11 16:48:05.472577: saving checkpoint... 2022-04-11 16:48:09.976815: done, saving took 4.64 seconds 2022-04-11 16:48:09.999807: done 2022-04-11 16:48:10.146307: saving checkpoint... 2022-04-11 16:48:16.255838: done, saving took 6.26 seconds 2022-04-11 16:48:16.596744: This epoch took 132.585328 s 2022-04-11 16:48:16.721293: svmem(total=99932160000, available=77643030528, percent=22.3, used=18306801664, free=48209190912, active=42840756224, inactive=7114399744, buffers=11624448, cached=33404542976, shared=2167574528, slab=762114048) 2022-04-11 16:48:16.855319: epoch: 8 2022-04-11 16:50:07.363999: train loss : 0.2678 2022-04-11 16:50:18.839669: validation loss: 0.2520 2022-04-11 16:50:18.841459: Average global foreground Dice: [0.3419, 0.7445, 0.6772, 0.0] 2022-04-11 16:50:18.842312: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:50:19.733616: lr: 0.009838 2022-04-11 16:50:19.734505: saving scheduled checkpoint file... 2022-04-11 16:50:19.893071: saving checkpoint... 2022-04-11 16:50:24.325037: done, saving took 4.59 seconds 2022-04-11 16:50:24.339194: done 2022-04-11 16:50:24.386045: saving checkpoint... 2022-04-11 16:50:31.362978: done, saving took 7.02 seconds 2022-04-11 16:50:31.685981: This epoch took 134.705672 s 2022-04-11 16:50:31.829115: svmem(total=99932160000, available=77328289792, percent=22.6, used=18621513728, free=47877652480, active=43189293056, inactive=7097532416, buffers=11706368, cached=33421287424, shared=2167402496, slab=761847808) 2022-04-11 16:50:31.953880: epoch: 9 2022-04-11 16:52:22.369337: train loss : 0.3229 2022-04-11 16:52:34.712620: validation loss: 0.3093 2022-04-11 16:52:34.724728: Average global foreground Dice: [0.4322, 0.7237, 0.6133, 0.0] 2022-04-11 16:52:34.725698: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:52:35.354867: lr: 0.00982 2022-04-11 16:52:35.355686: saving scheduled checkpoint file... 2022-04-11 16:52:35.491887: saving checkpoint... 2022-04-11 16:52:39.974528: done, saving took 4.62 seconds 2022-04-11 16:52:39.998956: done 2022-04-11 16:52:40.145467: saving checkpoint... 2022-04-11 16:52:47.964552: done, saving took 7.96 seconds 2022-04-11 16:52:48.064613: This epoch took 135.985580 s 2022-04-11 16:52:48.067281: svmem(total=99932160000, available=77517856768, percent=22.4, used=18430763008, free=48174850048, active=42968227840, inactive=7029219328, buffers=11735040, cached=33314811904, shared=2168479744, slab=758796288) 2022-04-11 16:52:48.069630: epoch: 10 2022-04-11 16:54:38.868746: train loss : 0.2433 2022-04-11 16:55:14.116186: validation loss: 0.1782 2022-04-11 16:55:14.342495: Average global foreground Dice: [0.4913, 0.7512, 0.6977, 0.0] 2022-04-11 16:55:14.483775: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:55:15.223697: lr: 0.009802 2022-04-11 16:55:15.362524: saving scheduled checkpoint file... 2022-04-11 16:55:15.582086: saving checkpoint... 2022-04-11 16:55:21.161453: done, saving took 5.71 seconds 2022-04-11 16:55:21.349660: done 2022-04-11 16:55:21.477989: saving checkpoint... 2022-04-11 16:55:26.306029: done, saving took 4.87 seconds 2022-04-11 16:55:26.494997: This epoch took 158.422720 s 2022-04-11 16:55:26.590449: svmem(total=99932160000, available=77521539072, percent=22.4, used=18461323264, free=70646587392, active=18916814848, inactive=8647897088, buffers=9699328, cached=10814550016, shared=2157191168, slab=707051520) 2022-04-11 16:55:26.668200: epoch: 11 2022-04-11 16:57:35.743304: train loss : 0.2729 2022-04-11 16:57:47.711825: validation loss: 0.2165 2022-04-11 16:57:47.881756: Average global foreground Dice: [0.3671, 0.7432, 0.7334, 0.0] 2022-04-11 16:57:48.068168: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 16:57:48.963813: lr: 0.009784 2022-04-11 16:57:49.090764: saving scheduled checkpoint file... 2022-04-11 16:57:49.222327: saving checkpoint... 2022-04-11 16:57:57.105774: done, saving took 7.99 seconds 2022-04-11 16:57:57.316907: done 2022-04-11 16:57:57.403543: saving checkpoint... 2022-04-11 16:58:01.808026: done, saving took 4.49 seconds 2022-04-11 16:58:01.869588: This epoch took 155.106220 s 2022-04-11 16:58:01.880452: svmem(total=99932160000, available=77146738688, percent=22.8, used=18758938624, free=53446291456, active=27862151168, inactive=16800768000, buffers=12423168, cached=27714506752, shared=2182279168, slab=813367296) 2022-04-11 16:58:01.892465: epoch: 12 2022-04-11 16:59:50.101643: train loss : 0.2209 2022-04-11 17:00:01.328221: validation loss: 0.1268 2022-04-11 17:00:01.460840: Average global foreground Dice: [0.4882, 0.7176, 0.7613, 0.0] 2022-04-11 17:00:01.610789: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:00:02.364417: lr: 0.009766 2022-04-11 17:00:02.365245: saving scheduled checkpoint file... 2022-04-11 17:00:02.464657: saving checkpoint... 2022-04-11 17:00:10.081944: done, saving took 7.72 seconds 2022-04-11 17:00:10.099645: done 2022-04-11 17:00:10.198093: saving checkpoint... 2022-04-11 17:00:14.615566: done, saving took 4.51 seconds 2022-04-11 17:00:14.650177: This epoch took 132.742052 s 2022-04-11 17:00:14.660040: svmem(total=99932160000, available=77075820544, percent=22.9, used=18852298752, free=50628014080, active=34481356800, inactive=13033644032, buffers=12754944, cached=30439092224, shared=2173009920, slab=783015936) 2022-04-11 17:00:14.669689: epoch: 13 2022-04-11 17:03:20.922797: train loss : 0.2013 2022-04-11 17:03:36.569040: validation loss: 0.1359 2022-04-11 17:03:36.580674: Average global foreground Dice: [0.4641, 0.7654, 0.7673, 0.0] 2022-04-11 17:03:36.594134: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:03:37.476409: lr: 0.009748 2022-04-11 17:03:37.481766: saving scheduled checkpoint file... 2022-04-11 17:03:37.602555: saving checkpoint... 2022-04-11 17:03:45.263969: done, saving took 7.78 seconds 2022-04-11 17:03:45.542935: done 2022-04-11 17:03:45.683384: saving checkpoint... 2022-04-11 17:03:53.487238: done, saving took 7.91 seconds 2022-04-11 17:03:53.604701: This epoch took 218.927191 s 2022-04-11 17:03:53.605752: svmem(total=99932160000, available=76855439360, percent=23.1, used=18995363840, free=44694675456, active=26556596224, inactive=26717966336, buffers=11247616, cached=36230873088, shared=2167574528, slab=961146880) 2022-04-11 17:03:53.606452: epoch: 14 2022-04-11 17:05:42.313968: train loss : 0.1156 2022-04-11 17:05:53.801457: validation loss: 0.0764 2022-04-11 17:05:53.803222: Average global foreground Dice: [0.5364, 0.7974, 0.7532, 0.0] 2022-04-11 17:05:53.808153: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:05:54.745148: lr: 0.00973 2022-04-11 17:05:54.746207: saving scheduled checkpoint file... 2022-04-11 17:05:54.888463: saving checkpoint... 2022-04-11 17:06:02.709853: done, saving took 7.96 seconds 2022-04-11 17:06:02.975100: done 2022-04-11 17:06:03.109805: saving checkpoint... 2022-04-11 17:06:11.043163: done, saving took 8.06 seconds 2022-04-11 17:06:11.063088: This epoch took 137.455676 s 2022-04-11 17:06:11.064121: svmem(total=99932160000, available=76930818048, percent=23.0, used=18860027904, free=35567996928, active=33504792576, inactive=28770922496, buffers=15073280, cached=45489061888, shared=2167488512, slab=1081483264) 2022-04-11 17:06:11.065984: epoch: 15 2022-04-11 17:08:00.801039: train loss : 0.1587 2022-04-11 17:08:12.270982: validation loss: 0.1271 2022-04-11 17:08:12.272566: Average global foreground Dice: [0.5127, 0.7759, 0.6467, 0.0] 2022-04-11 17:08:12.273431: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:08:13.167598: lr: 0.009712 2022-04-11 17:08:13.169552: saving scheduled checkpoint file... 2022-04-11 17:08:13.328503: saving checkpoint... 2022-04-11 17:08:21.200635: done, saving took 8.03 seconds 2022-04-11 17:08:21.382574: done 2022-04-11 17:08:21.529212: saving checkpoint... 2022-04-11 17:08:29.435560: done, saving took 8.05 seconds 2022-04-11 17:08:29.459288: This epoch took 138.392398 s 2022-04-11 17:08:29.462042: svmem(total=99932160000, available=76823461888, percent=23.1, used=18904858624, free=29566803968, active=37558099968, inactive=30593200128, buffers=20115456, cached=51440381952, shared=2167402496, slab=1206026240) 2022-04-11 17:08:29.464949: epoch: 16 2022-04-11 17:10:19.348781: train loss : 0.2112 2022-04-11 17:10:31.551675: validation loss: 0.1301 2022-04-11 17:10:31.553269: Average global foreground Dice: [0.4756, 0.7947, 0.7436, 0.0] 2022-04-11 17:10:31.554453: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:10:32.539065: lr: 0.009693 2022-04-11 17:10:32.672308: saving scheduled checkpoint file... 2022-04-11 17:10:32.921371: saving checkpoint... 2022-04-11 17:10:39.463129: done, saving took 6.67 seconds 2022-04-11 17:10:39.777255: done 2022-04-11 17:10:40.040882: saving checkpoint... 2022-04-11 17:10:46.436624: done, saving took 6.53 seconds 2022-04-11 17:10:46.455337: This epoch took 136.987898 s 2022-04-11 17:10:46.456391: svmem(total=99932160000, available=76568563712, percent=23.4, used=19105689600, free=31398723584, active=39902212096, inactive=26327392256, buffers=31354880, cached=49396391936, shared=2181750784, slab=1289826304) 2022-04-11 17:10:46.457172: epoch: 17 2022-04-11 17:12:36.304682: train loss : 0.1291 2022-04-11 17:12:48.429106: validation loss: 0.0624 2022-04-11 17:12:48.564577: Average global foreground Dice: [0.5052, 0.7972, 0.8238, 0.0] 2022-04-11 17:12:48.706331: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:12:49.771737: lr: 0.009675 2022-04-11 17:12:49.904739: saving scheduled checkpoint file... 2022-04-11 17:12:50.144974: saving checkpoint... 2022-04-11 17:12:56.677570: done, saving took 6.62 seconds 2022-04-11 17:12:56.992048: done 2022-04-11 17:12:57.171761: saving checkpoint... 2022-04-11 17:13:01.988251: done, saving took 4.87 seconds 2022-04-11 17:13:02.005133: This epoch took 135.547423 s 2022-04-11 17:13:02.007538: svmem(total=99932160000, available=76854681600, percent=23.1, used=18962075648, free=39992774656, active=41067216896, inactive=16862871552, buffers=31879168, cached=40945430528, shared=2181443584, slab=994889728) 2022-04-11 17:13:02.008292: epoch: 18 2022-04-11 17:14:48.930416: train loss : 0.0602 2022-04-11 17:15:00.247607: validation loss: 0.0118 2022-04-11 17:15:00.252620: Average global foreground Dice: [0.4841, 0.8121, 0.8187, 0.0] 2022-04-11 17:15:00.253699: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:15:00.974983: lr: 0.009657 2022-04-11 17:15:00.975831: saving scheduled checkpoint file... 2022-04-11 17:15:01.047321: saving checkpoint... 2022-04-11 17:15:08.860916: done, saving took 7.88 seconds 2022-04-11 17:15:09.050828: done 2022-04-11 17:15:09.104743: saving checkpoint... 2022-04-11 17:15:17.001781: done, saving took 7.95 seconds 2022-04-11 17:15:17.026085: This epoch took 135.017016 s 2022-04-11 17:15:17.027323: svmem(total=99932160000, available=76815351808, percent=23.1, used=19015663616, free=39812804608, active=41860083712, inactive=16243834880, buffers=31948800, cached=41071742976, shared=2167488512, slab=1008164864) 2022-04-11 17:15:17.028280: epoch: 19 2022-04-11 17:17:06.457116: train loss : 0.1218 2022-04-11 17:17:17.836277: validation loss: 0.0028 2022-04-11 17:17:17.845658: Average global foreground Dice: [0.3808, 0.849, 0.7976, 0.0] 2022-04-11 17:17:17.854763: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:17:18.695648: lr: 0.009639 2022-04-11 17:17:18.838097: saving scheduled checkpoint file... 2022-04-11 17:17:19.043925: saving checkpoint... 2022-04-11 17:17:25.725860: done, saving took 6.77 seconds 2022-04-11 17:17:26.047734: done 2022-04-11 17:17:26.236989: saving checkpoint... 2022-04-11 17:17:32.528752: done, saving took 6.35 seconds 2022-04-11 17:17:32.546407: This epoch took 135.517211 s 2022-04-11 17:17:32.547534: svmem(total=99932160000, available=76828180480, percent=23.1, used=18977562624, free=39739912192, active=42035957760, inactive=16147562496, buffers=32038912, cached=41182646272, shared=2192543744, slab=998735872) 2022-04-11 17:17:32.548294: epoch: 20 2022-04-11 17:20:36.857340: train loss : 0.0591 2022-04-11 17:20:52.988164: validation loss: 0.0281 2022-04-11 17:20:53.143356: Average global foreground Dice: [0.4674, 0.8251, 0.8142, 0.0] 2022-04-11 17:20:53.284159: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:20:54.047944: lr: 0.009621 2022-04-11 17:20:54.200026: saving scheduled checkpoint file... 2022-04-11 17:20:54.458643: saving checkpoint... 2022-04-11 17:21:01.188427: done, saving took 6.80 seconds 2022-04-11 17:21:01.492092: done 2022-04-11 17:21:01.733520: saving checkpoint... 2022-04-11 17:21:06.967103: done, saving took 5.32 seconds 2022-04-11 17:21:07.137641: This epoch took 214.588603 s 2022-04-11 17:21:07.170529: svmem(total=99932160000, available=76898873344, percent=23.0, used=18960420864, free=45539332096, active=25528233984, inactive=26910154752, buffers=11554816, cached=35420852224, shared=2160635904, slab=947474432) 2022-04-11 17:21:07.205782: epoch: 21 2022-04-11 17:22:57.511436: train loss : 0.0525 2022-04-11 17:23:09.442933: validation loss: -0.0650 2022-04-11 17:23:09.590028: Average global foreground Dice: [0.5601, 0.8328, 0.8014, 0.005] 2022-04-11 17:23:09.723425: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:23:10.476020: lr: 0.009603 2022-04-11 17:23:10.644885: saving scheduled checkpoint file... 2022-04-11 17:23:10.879967: saving checkpoint... 2022-04-11 17:23:18.651596: done, saving took 7.86 seconds 2022-04-11 17:23:18.929905: done 2022-04-11 17:23:19.077554: saving checkpoint... 2022-04-11 17:23:23.458756: done, saving took 4.48 seconds 2022-04-11 17:23:23.497975: This epoch took 136.263662 s 2022-04-11 17:23:23.508060: svmem(total=99932160000, available=76667715584, percent=23.3, used=19100979200, free=35205627904, active=33374683136, inactive=29264457728, buffers=15724544, cached=45609828352, shared=2187423744, slab=1074421760) 2022-04-11 17:23:23.517119: epoch: 22 2022-04-11 17:25:11.768223: train loss : -0.0165 2022-04-11 17:25:22.959995: validation loss: 0.0780 2022-04-11 17:25:23.094501: Average global foreground Dice: [0.5335, 0.8113, 0.7273, 0.0265] 2022-04-11 17:25:23.210669: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:25:24.294548: lr: 0.009585 2022-04-11 17:25:24.436633: saving scheduled checkpoint file... 2022-04-11 17:25:24.660996: saving checkpoint... 2022-04-11 17:25:32.610865: done, saving took 8.06 seconds 2022-04-11 17:25:32.785597: done 2022-04-11 17:25:32.909978: saving checkpoint... 2022-04-11 17:25:37.288647: done, saving took 4.50 seconds 2022-04-11 17:25:37.315162: This epoch took 133.788981 s 2022-04-11 17:25:37.318019: svmem(total=99932160000, available=76813709312, percent=23.1, used=18895257600, free=29383618560, active=37119664128, inactive=31211872256, buffers=21528576, cached=51631755264, shared=2181562368, slab=1210580992) 2022-04-11 17:25:37.319882: epoch: 23 2022-04-11 17:27:27.230602: train loss : -0.0229 2022-04-11 17:27:39.072095: validation loss: -0.0805 2022-04-11 17:27:39.211119: Average global foreground Dice: [0.5366, 0.8664, 0.8641, 0.0075] 2022-04-11 17:27:39.362977: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:27:40.305686: lr: 0.009567 2022-04-11 17:27:40.452306: saving scheduled checkpoint file... 2022-04-11 17:27:40.697182: saving checkpoint... 2022-04-11 17:27:47.258685: done, saving took 6.64 seconds 2022-04-11 17:27:47.572791: done 2022-04-11 17:27:47.747242: saving checkpoint... 2022-04-11 17:27:52.643563: done, saving took 4.95 seconds 2022-04-11 17:27:52.663881: This epoch took 135.341550 s 2022-04-11 17:27:52.665053: svmem(total=99932160000, available=76737929216, percent=23.2, used=18949652480, free=28684599296, active=39418302464, inactive=29547159552, buffers=31465472, cached=52266442752, shared=2181566464, slab=1265815552) 2022-04-11 17:27:52.666143: epoch: 24 2022-04-11 17:29:41.078652: train loss : 0.0698 2022-04-11 17:29:53.166107: validation loss: 0.0631 2022-04-11 17:29:53.322389: Average global foreground Dice: [0.5226, 0.8276, 0.7564, 0.016] 2022-04-11 17:29:53.473614: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:29:54.484140: lr: 0.009549 2022-04-11 17:29:54.634823: saving scheduled checkpoint file... 2022-04-11 17:29:54.851095: saving checkpoint... 2022-04-11 17:30:02.752741: done, saving took 7.97 seconds 2022-04-11 17:30:02.889590: done 2022-04-11 17:30:02.977924: saving checkpoint... 2022-04-11 17:30:07.372389: done, saving took 4.48 seconds 2022-04-11 17:30:07.388252: This epoch took 134.720969 s 2022-04-11 17:30:07.389286: svmem(total=99932160000, available=72932356096, percent=27.0, used=22722695168, free=27215011840, active=44325756928, inactive=25985929216, buffers=32251904, cached=49962201088, shared=2206511104, slab=1288085504) 2022-04-11 17:30:07.389857: epoch: 25 2022-04-11 17:31:55.991778: train loss : 0.0889 2022-04-11 17:32:08.446542: validation loss: 0.0651 2022-04-11 17:32:08.582205: Average global foreground Dice: [0.3863, 0.818, 0.8034, 0.021] 2022-04-11 17:32:08.697647: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:32:09.474613: lr: 0.009531 2022-04-11 17:32:09.618596: saving scheduled checkpoint file... 2022-04-11 17:32:09.885855: saving checkpoint... 2022-04-11 17:32:15.138513: done, saving took 5.35 seconds 2022-04-11 17:32:15.155736: done 2022-04-11 17:32:15.256586: saving checkpoint... 2022-04-11 17:32:19.655155: done, saving took 4.50 seconds 2022-04-11 17:32:19.669260: This epoch took 132.278794 s 2022-04-11 17:32:19.670406: svmem(total=99932160000, available=72298602496, percent=27.7, used=23369486336, free=26635075584, active=45975248896, inactive=24930684928, buffers=32325632, cached=49895272448, shared=2196652032, slab=1282596864) 2022-04-11 17:32:19.671095: epoch: 26 2022-04-11 17:34:08.166528: train loss : -0.0072 2022-04-11 17:34:20.204141: validation loss: 0.0932 2022-04-11 17:34:20.346336: Average global foreground Dice: [0.5879, 0.7524, 0.7894, 0.0139] 2022-04-11 17:34:20.514685: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:34:21.463390: lr: 0.009513 2022-04-11 17:34:21.578839: saving scheduled checkpoint file... 2022-04-11 17:34:21.804707: saving checkpoint... 2022-04-11 17:34:26.326853: done, saving took 4.65 seconds 2022-04-11 17:34:26.346435: done 2022-04-11 17:34:26.454523: saving checkpoint... 2022-04-11 17:34:30.857126: done, saving took 4.51 seconds 2022-04-11 17:34:30.877714: This epoch took 131.205953 s 2022-04-11 17:34:30.880423: svmem(total=99932160000, available=72462307328, percent=27.5, used=23207006208, free=26731958272, active=46264946688, inactive=24538902528, buffers=32411648, cached=49960783872, shared=2195939328, slab=1279795200) 2022-04-11 17:34:30.882012: epoch: 27 2022-04-11 17:36:19.020243: train loss : 0.0305 2022-04-11 17:36:31.528234: validation loss: -0.0637 2022-04-11 17:36:31.657694: Average global foreground Dice: [0.5951, 0.8352, 0.8428, 0.0014] 2022-04-11 17:36:31.781433: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:36:32.749321: lr: 0.009495 2022-04-11 17:36:32.873554: saving scheduled checkpoint file... 2022-04-11 17:36:33.121269: saving checkpoint... 2022-04-11 17:36:37.706429: done, saving took 4.71 seconds 2022-04-11 17:36:37.730033: done 2022-04-11 17:36:37.851513: saving checkpoint... 2022-04-11 17:36:42.249770: done, saving took 4.52 seconds 2022-04-11 17:36:42.270466: This epoch took 131.386000 s 2022-04-11 17:36:42.271780: svmem(total=99932160000, available=72432979968, percent=27.5, used=23231381504, free=26613932032, active=46633807872, inactive=24290820096, buffers=32505856, cached=50054340608, shared=2200702976, slab=1280544768) 2022-04-11 17:36:42.272744: epoch: 28 2022-04-11 17:38:32.238307: train loss : -0.0748 2022-04-11 17:38:44.447716: validation loss: -0.1557 2022-04-11 17:38:44.450632: Average global foreground Dice: [0.5367, 0.8855, 0.8604, 0.0214] 2022-04-11 17:38:44.456017: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:38:45.228815: lr: 0.009476 2022-04-11 17:38:45.230930: saving scheduled checkpoint file... 2022-04-11 17:38:45.361336: saving checkpoint... 2022-04-11 17:38:49.754275: done, saving took 4.52 seconds 2022-04-11 17:38:49.778291: done 2022-04-11 17:38:49.925256: saving checkpoint... 2022-04-11 17:38:54.807739: done, saving took 5.03 seconds 2022-04-11 17:38:54.828807: This epoch took 132.555363 s 2022-04-11 17:38:54.829859: svmem(total=99932160000, available=72474292224, percent=27.5, used=23194394624, free=26614751232, active=46831382528, inactive=24088461312, buffers=32571392, cached=50090442752, shared=2196422656, slab=1281507328) 2022-04-11 17:38:54.830470: epoch: 29 2022-04-11 17:40:45.137248: train loss : -0.1154 2022-04-11 17:40:56.979772: validation loss: -0.1575 2022-04-11 17:40:56.999481: Average global foreground Dice: [0.6329, 0.844, 0.8829, 0.0239] 2022-04-11 17:40:57.000529: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:40:57.704509: lr: 0.009458 2022-04-11 17:40:57.705439: saving scheduled checkpoint file... 2022-04-11 17:40:57.812261: saving checkpoint... 2022-04-11 17:41:02.257481: done, saving took 4.55 seconds 2022-04-11 17:41:02.270090: done 2022-04-11 17:41:02.317665: saving checkpoint... 2022-04-11 17:41:06.676270: done, saving took 4.41 seconds 2022-04-11 17:41:06.688688: This epoch took 131.857586 s 2022-04-11 17:41:06.690009: svmem(total=99932160000, available=72330706944, percent=27.6, used=23331897344, free=26445443072, active=47089537024, inactive=23999860736, buffers=32649216, cached=50122170368, shared=2202484736, slab=1278595072) 2022-04-11 17:41:06.690892: epoch: 30 2022-04-11 17:42:55.432215: train loss : -0.1316 2022-04-11 17:43:06.665865: validation loss: -0.2061 2022-04-11 17:43:06.677973: Average global foreground Dice: [0.59, 0.8963, 0.9068, 0.0162] 2022-04-11 17:43:06.679091: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:43:07.642874: lr: 0.00944 2022-04-11 17:43:07.644026: saving scheduled checkpoint file... 2022-04-11 17:43:07.752332: saving checkpoint... 2022-04-11 17:43:12.210185: done, saving took 4.57 seconds 2022-04-11 17:43:12.222409: done 2022-04-11 17:43:12.270891: saving checkpoint... 2022-04-11 17:43:16.721822: done, saving took 4.50 seconds 2022-04-11 17:43:16.743314: This epoch took 130.051474 s 2022-04-11 17:43:16.744464: svmem(total=99932160000, available=72271040512, percent=27.7, used=23397638144, free=26364940288, active=47273754624, inactive=23902461952, buffers=32702464, cached=50136879104, shared=2196299776, slab=1279606784) 2022-04-11 17:43:16.745202: epoch: 31 2022-04-11 17:45:05.632795: train loss : -0.0999 2022-04-11 17:45:17.298147: validation loss: -0.1626 2022-04-11 17:45:17.302248: Average global foreground Dice: [0.6362, 0.8596, 0.8912, 0.0188] 2022-04-11 17:45:17.303353: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:45:18.150275: lr: 0.009422 2022-04-11 17:45:18.151235: saving scheduled checkpoint file... 2022-04-11 17:45:18.246635: saving checkpoint... 2022-04-11 17:45:22.617038: done, saving took 4.47 seconds 2022-04-11 17:45:22.634251: done 2022-04-11 17:45:22.717204: saving checkpoint... 2022-04-11 17:45:27.170432: done, saving took 4.53 seconds 2022-04-11 17:45:27.216937: This epoch took 130.470773 s 2022-04-11 17:45:27.218275: svmem(total=99932160000, available=72440938496, percent=27.5, used=23241093120, free=26515226624, active=47209594880, inactive=23805075456, buffers=32825344, cached=50143014912, shared=2182811648, slab=1281830912) 2022-04-11 17:45:27.219156: epoch: 32 2022-04-11 17:47:14.168341: train loss : -0.0958 2022-04-11 17:47:26.314749: validation loss: -0.1057 2022-04-11 17:47:26.319976: Average global foreground Dice: [0.3938, 0.8748, 0.8584, 0.0256] 2022-04-11 17:47:26.320944: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:47:27.288996: lr: 0.009404 2022-04-11 17:47:27.290061: saving scheduled checkpoint file... 2022-04-11 17:47:27.402238: saving checkpoint... 2022-04-11 17:47:31.913971: done, saving took 4.62 seconds 2022-04-11 17:47:31.927410: done 2022-04-11 17:47:32.032854: saving checkpoint... 2022-04-11 17:47:36.611765: done, saving took 4.68 seconds 2022-04-11 17:47:36.630317: This epoch took 129.410509 s 2022-04-11 17:47:36.631740: svmem(total=99932160000, available=72417837056, percent=27.5, used=23250591744, free=26479845376, active=47271235584, inactive=23780933632, buffers=32886784, cached=50168836096, shared=2196467712, slab=1281404928) 2022-04-11 17:47:36.632481: epoch: 33 2022-04-11 17:49:26.007892: train loss : -0.0778 2022-04-11 17:49:37.711319: validation loss: -0.1240 2022-04-11 17:49:37.719561: Average global foreground Dice: [0.6235, 0.8471, 0.8756, 0.0112] 2022-04-11 17:49:37.720392: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:49:38.621251: lr: 0.009386 2022-04-11 17:49:38.622435: saving scheduled checkpoint file... 2022-04-11 17:49:38.757352: saving checkpoint... 2022-04-11 17:49:43.325178: done, saving took 4.70 seconds 2022-04-11 17:49:43.345081: done 2022-04-11 17:49:43.457422: saving checkpoint... 2022-04-11 17:49:47.893450: done, saving took 4.55 seconds 2022-04-11 17:49:47.908579: This epoch took 131.275294 s 2022-04-11 17:49:47.909889: svmem(total=99932160000, available=72273768448, percent=27.7, used=23398498304, free=26332446720, active=47458050048, inactive=23747047424, buffers=32952320, cached=50168262656, shared=2192584704, slab=1280397312) 2022-04-11 17:49:47.910815: epoch: 34 2022-04-11 17:51:38.277248: train loss : -0.1305 2022-04-11 17:51:50.039416: validation loss: -0.1136 2022-04-11 17:51:50.041248: Average global foreground Dice: [0.5884, 0.8608, 0.8736, 0.0256] 2022-04-11 17:51:50.042149: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:51:50.946452: lr: 0.009368 2022-04-11 17:51:50.947466: saving scheduled checkpoint file... 2022-04-11 17:51:51.089380: saving checkpoint... 2022-04-11 17:51:55.472192: done, saving took 4.52 seconds 2022-04-11 17:51:55.488812: done 2022-04-11 17:51:55.602172: saving checkpoint... 2022-04-11 17:52:00.086072: done, saving took 4.60 seconds 2022-04-11 17:52:00.106764: This epoch took 132.195037 s 2022-04-11 17:52:00.107875: svmem(total=99932160000, available=72258674688, percent=27.7, used=23406387200, free=26306433024, active=47502774272, inactive=23724605440, buffers=33030144, cached=50186309632, shared=2199789568, slab=1280835584) 2022-04-11 17:52:00.108536: epoch: 35 2022-04-11 17:53:49.761665: train loss : -0.0590 2022-04-11 17:54:03.317049: validation loss: -0.1534 2022-04-11 17:54:03.321555: Average global foreground Dice: [0.537, 0.8665, 0.891, 0.0142] 2022-04-11 17:54:03.322482: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:54:04.337432: lr: 0.00935 2022-04-11 17:54:04.338646: saving scheduled checkpoint file... 2022-04-11 17:54:04.440357: saving checkpoint... 2022-04-11 17:54:08.791962: done, saving took 4.45 seconds 2022-04-11 17:54:08.804449: done 2022-04-11 17:54:08.852715: saving checkpoint... 2022-04-11 17:54:13.165883: done, saving took 4.36 seconds 2022-04-11 17:54:13.181028: This epoch took 133.071824 s 2022-04-11 17:54:13.182296: svmem(total=99932160000, available=71739998208, percent=28.2, used=23935205376, free=26398711808, active=48075960320, inactive=23071145984, buffers=33124352, cached=49565118464, shared=2196328448, slab=1267662848) 2022-04-11 17:54:13.183276: epoch: 36 2022-04-11 17:56:02.490326: train loss : -0.1375 2022-04-11 17:56:13.845899: validation loss: -0.1888 2022-04-11 17:56:13.847872: Average global foreground Dice: [0.6432, 0.8811, 0.8982, 0.0739] 2022-04-11 17:56:13.848897: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:56:14.591457: lr: 0.009331 2022-04-11 17:56:14.592431: saving scheduled checkpoint file... 2022-04-11 17:56:14.690326: saving checkpoint... 2022-04-11 17:56:19.070134: done, saving took 4.48 seconds 2022-04-11 17:56:19.085006: done 2022-04-11 17:56:19.134292: saving checkpoint... 2022-04-11 17:56:23.534753: done, saving took 4.45 seconds 2022-04-11 17:56:23.547383: This epoch took 130.363103 s 2022-04-11 17:56:23.548627: svmem(total=99932160000, available=71719325696, percent=28.2, used=23943278592, free=26374664192, active=48110198784, inactive=23064551424, buffers=33230848, cached=49580986368, shared=2209042432, slab=1267322880) 2022-04-11 17:56:23.549314: epoch: 37 2022-04-11 17:58:11.392571: train loss : -0.1443 2022-04-11 17:58:22.623236: validation loss: -0.1371 2022-04-11 17:58:22.640039: Average global foreground Dice: [0.5978, 0.8792, 0.872, 0.0849] 2022-04-11 17:58:22.643057: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 17:58:23.585157: lr: 0.009313 2022-04-11 17:58:23.586090: saving scheduled checkpoint file... 2022-04-11 17:58:23.655379: saving checkpoint... 2022-04-11 17:58:29.212261: done, saving took 5.63 seconds 2022-04-11 17:58:29.274075: done 2022-04-11 17:58:29.322792: saving checkpoint... 2022-04-11 17:58:33.724001: done, saving took 4.45 seconds 2022-04-11 17:58:33.735642: This epoch took 130.185640 s 2022-04-11 17:58:33.736850: svmem(total=99932160000, available=71651250176, percent=28.3, used=24032735232, free=26304081920, active=48209223680, inactive=23030784000, buffers=33325056, cached=49562017792, shared=2187587584, slab=1267073024) 2022-04-11 17:58:33.737611: epoch: 38 2022-04-11 18:00:20.824373: train loss : -0.2167 2022-04-11 18:00:32.326185: validation loss: -0.2416 2022-04-11 18:00:32.328282: Average global foreground Dice: [0.5982, 0.8855, 0.9178, 0.1381] 2022-04-11 18:00:32.329329: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:00:33.131086: lr: 0.009295 2022-04-11 18:00:33.132333: saving scheduled checkpoint file... 2022-04-11 18:00:33.208500: saving checkpoint... 2022-04-11 18:00:37.804237: done, saving took 4.67 seconds 2022-04-11 18:00:37.817295: done 2022-04-11 18:00:37.893476: saving checkpoint... 2022-04-11 18:00:42.315624: done, saving took 4.50 seconds 2022-04-11 18:00:42.332703: This epoch took 128.594442 s 2022-04-11 18:00:42.333832: svmem(total=99932160000, available=71622725632, percent=28.3, used=24052944896, free=26274721792, active=48245870592, inactive=23030632448, buffers=33398784, cached=49571094528, shared=2195951616, slab=1268023296) 2022-04-11 18:00:42.334507: epoch: 39 2022-04-11 18:02:31.646031: train loss : -0.2059 2022-04-11 18:02:43.817834: validation loss: -0.2119 2022-04-11 18:02:43.827016: Average global foreground Dice: [0.5349, 0.8682, 0.9165, 0.0884] 2022-04-11 18:02:43.828526: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:02:44.802001: lr: 0.009277 2022-04-11 18:02:44.803174: saving scheduled checkpoint file... 2022-04-11 18:02:44.908625: saving checkpoint... 2022-04-11 18:02:49.445687: done, saving took 4.64 seconds 2022-04-11 18:02:49.462111: done 2022-04-11 18:02:49.553678: saving checkpoint... 2022-04-11 18:02:54.002457: done, saving took 4.54 seconds 2022-04-11 18:02:54.020047: This epoch took 131.684909 s 2022-04-11 18:02:54.021290: svmem(total=99932160000, available=71714545664, percent=28.2, used=23950290944, free=26365902848, active=48149045248, inactive=23035019264, buffers=33464320, cached=49582501888, shared=2206650368, slab=1267326976) 2022-04-11 18:02:54.022468: epoch: 40 2022-04-11 18:04:42.080630: train loss : -0.2175 2022-04-11 18:04:53.175014: validation loss: -0.2190 2022-04-11 18:04:53.185430: Average global foreground Dice: [0.627, 0.8837, 0.8947, 0.1225] 2022-04-11 18:04:53.186129: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:04:54.082279: lr: 0.009259 2022-04-11 18:04:54.083680: saving scheduled checkpoint file... 2022-04-11 18:04:54.216966: saving checkpoint... 2022-04-11 18:04:58.626768: done, saving took 4.54 seconds 2022-04-11 18:04:58.646199: done 2022-04-11 18:04:58.751786: saving checkpoint... 2022-04-11 18:05:03.195846: done, saving took 4.55 seconds 2022-04-11 18:05:03.215326: This epoch took 129.192206 s 2022-04-11 18:05:03.216729: svmem(total=99932160000, available=71722696704, percent=28.2, used=23966785536, free=26372366336, active=48165584896, inactive=23009280000, buffers=33542144, cached=49559465984, shared=2182168576, slab=1268523008) 2022-04-11 18:05:03.217360: epoch: 41 2022-04-11 18:06:50.279085: train loss : -0.1161 2022-04-11 18:07:02.347428: validation loss: -0.1560 2022-04-11 18:07:02.349132: Average global foreground Dice: [0.5239, 0.8881, 0.8695, 0.1187] 2022-04-11 18:07:02.349972: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:07:03.355535: lr: 0.009241 2022-04-11 18:07:03.356619: saving scheduled checkpoint file... 2022-04-11 18:07:03.465160: saving checkpoint... 2022-04-11 18:07:07.823272: done, saving took 4.47 seconds 2022-04-11 18:07:07.855080: done 2022-04-11 18:07:07.902170: saving checkpoint... 2022-04-11 18:07:12.333320: done, saving took 4.48 seconds 2022-04-11 18:07:12.345584: This epoch took 129.127521 s 2022-04-11 18:07:12.347083: svmem(total=99932160000, available=71878737920, percent=28.1, used=23773011968, free=26527186944, active=47977308160, inactive=23043612672, buffers=33628160, cached=49598332928, shared=2219802624, slab=1267703808) 2022-04-11 18:07:12.347960: epoch: 42 2022-04-11 18:09:00.591041: train loss : -0.1613 2022-04-11 18:09:12.721704: validation loss: -0.2325 2022-04-11 18:09:12.723747: Average global foreground Dice: [0.641, 0.8826, 0.9031, 0.1544] 2022-04-11 18:09:12.731688: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:09:13.641747: lr: 0.009223 2022-04-11 18:09:13.642617: saving scheduled checkpoint file... 2022-04-11 18:09:13.716553: saving checkpoint... 2022-04-11 18:09:18.167527: done, saving took 4.52 seconds 2022-04-11 18:09:18.180972: done 2022-04-11 18:09:18.262023: saving checkpoint... 2022-04-11 18:09:22.767869: done, saving took 4.59 seconds 2022-04-11 18:09:22.786571: This epoch took 130.437916 s 2022-04-11 18:09:22.788022: svmem(total=99932160000, available=71776227328, percent=28.2, used=23892692992, free=26423320576, active=48098082816, inactive=23025647616, buffers=33693696, cached=49582452736, shared=2202677248, slab=1269174272) 2022-04-11 18:09:22.788797: epoch: 43 2022-04-11 18:11:11.440221: train loss : -0.0195 2022-04-11 18:11:22.831479: validation loss: -0.2327 2022-04-11 18:11:22.833513: Average global foreground Dice: [0.6121, 0.8826, 0.8996, 0.1729] 2022-04-11 18:11:22.834573: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:11:23.850249: lr: 0.009204 2022-04-11 18:11:23.851140: saving scheduled checkpoint file... 2022-04-11 18:11:23.965666: saving checkpoint... 2022-04-11 18:11:28.349880: done, saving took 4.50 seconds 2022-04-11 18:11:28.368956: done 2022-04-11 18:11:28.469144: saving checkpoint... 2022-04-11 18:11:32.935390: done, saving took 4.57 seconds 2022-04-11 18:11:32.987597: This epoch took 130.197761 s 2022-04-11 18:11:32.988889: svmem(total=99932160000, available=71799296000, percent=28.2, used=23867494400, free=26445348864, active=48080449536, inactive=23021596672, buffers=33787904, cached=49585528832, shared=2204635136, slab=1268170752) 2022-04-11 18:11:32.989992: epoch: 44 2022-04-11 18:13:22.127722: train loss : -0.1719 2022-04-11 18:13:34.339249: validation loss: -0.2528 2022-04-11 18:13:34.346928: Average global foreground Dice: [0.6035, 0.8801, 0.8954, 0.1809] 2022-04-11 18:13:34.348213: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:13:35.046270: lr: 0.009186 2022-04-11 18:13:35.047257: saving scheduled checkpoint file... 2022-04-11 18:13:35.156972: saving checkpoint... 2022-04-11 18:13:39.696310: done, saving took 4.65 seconds 2022-04-11 18:13:39.731782: done 2022-04-11 18:13:39.840226: saving checkpoint... 2022-04-11 18:13:44.299734: done, saving took 4.57 seconds 2022-04-11 18:13:44.326713: This epoch took 131.335958 s 2022-04-11 18:13:44.327991: svmem(total=99932160000, available=71844941824, percent=28.1, used=23824355328, free=26488463360, active=48043339776, inactive=23013957632, buffers=33853440, cached=49585487872, shared=2202189824, slab=1266073600) 2022-04-11 18:13:44.328715: epoch: 45 2022-04-11 18:15:34.338158: train loss : -0.2477 2022-04-11 18:15:46.704884: validation loss: -0.2903 2022-04-11 18:15:46.706598: Average global foreground Dice: [0.6803, 0.8827, 0.9332, 0.1799] 2022-04-11 18:15:46.707465: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:15:47.393175: lr: 0.009168 2022-04-11 18:15:47.394212: saving scheduled checkpoint file... 2022-04-11 18:15:47.540426: saving checkpoint... 2022-04-11 18:15:52.121005: done, saving took 4.73 seconds 2022-04-11 18:15:52.168411: done 2022-04-11 18:15:52.298913: saving checkpoint... 2022-04-11 18:15:56.744429: done, saving took 4.57 seconds 2022-04-11 18:15:56.766161: This epoch took 132.436845 s 2022-04-11 18:15:56.767342: svmem(total=99932160000, available=72033021952, percent=27.9, used=23642566656, free=26756820992, active=47874568192, inactive=22919655424, buffers=33964032, cached=49498808320, shared=2196455424, slab=1265205248) 2022-04-11 18:15:56.768199: epoch: 46 2022-04-11 18:17:46.248167: train loss : -0.2215 2022-04-11 18:17:57.958086: validation loss: -0.2323 2022-04-11 18:17:57.972025: Average global foreground Dice: [0.5444, 0.8834, 0.9294, 0.1571] 2022-04-11 18:17:57.972885: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:17:58.859997: lr: 0.00915 2022-04-11 18:17:58.860829: saving scheduled checkpoint file... 2022-04-11 18:17:59.008427: saving checkpoint... 2022-04-11 18:18:03.548383: done, saving took 4.69 seconds 2022-04-11 18:18:03.606616: done 2022-04-11 18:18:03.737969: saving checkpoint... 2022-04-11 18:18:08.271446: done, saving took 4.66 seconds 2022-04-11 18:18:08.294059: This epoch took 131.525197 s 2022-04-11 18:18:08.295414: svmem(total=99932160000, available=71776071680, percent=28.2, used=23900766208, free=26446954496, active=48133853184, inactive=22964473856, buffers=36216832, cached=49548222464, shared=2194448384, slab=1268092928) 2022-04-11 18:18:08.296205: epoch: 47 2022-04-11 18:19:58.679751: train loss : -0.2548 2022-04-11 18:20:10.921225: validation loss: -0.3146 2022-04-11 18:20:10.929522: Average global foreground Dice: [0.6653, 0.8893, 0.9386, 0.2053] 2022-04-11 18:20:10.930283: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:20:11.983274: lr: 0.009132 2022-04-11 18:20:11.984384: saving scheduled checkpoint file... 2022-04-11 18:20:12.069046: saving checkpoint... 2022-04-11 18:20:16.446507: done, saving took 4.46 seconds 2022-04-11 18:20:16.460459: done 2022-04-11 18:20:16.508301: saving checkpoint... 2022-04-11 18:20:20.827060: done, saving took 4.37 seconds 2022-04-11 18:20:20.839935: This epoch took 132.543006 s 2022-04-11 18:20:20.841426: svmem(total=99932160000, available=71600074752, percent=28.4, used=24064401408, free=26270265344, active=48293453824, inactive=22977576960, buffers=36286464, cached=49561206784, shared=2206752768, slab=1268768768) 2022-04-11 18:20:20.842263: epoch: 48 2022-04-11 18:22:08.218605: train loss : -0.3094 2022-04-11 18:22:19.891774: validation loss: -0.3106 2022-04-11 18:22:19.893555: Average global foreground Dice: [0.6679, 0.8978, 0.9362, 0.1887] 2022-04-11 18:22:19.895326: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:22:20.807310: lr: 0.009114 2022-04-11 18:22:20.808614: saving scheduled checkpoint file... 2022-04-11 18:22:20.861458: saving checkpoint... 2022-04-11 18:22:25.294114: done, saving took 4.48 seconds 2022-04-11 18:22:25.305105: done 2022-04-11 18:22:25.352593: saving checkpoint... 2022-04-11 18:22:29.902286: done, saving took 4.60 seconds 2022-04-11 18:22:29.934473: This epoch took 129.091425 s 2022-04-11 18:22:29.935825: svmem(total=99932160000, available=71658811392, percent=28.3, used=24016060416, free=26327121920, active=48256372736, inactive=22964604928, buffers=36384768, cached=49552592896, shared=2196328448, slab=1268178944) 2022-04-11 18:22:29.936532: epoch: 49 2022-04-11 18:24:17.565849: train loss : -0.2241 2022-04-11 18:24:29.801204: validation loss: -0.3253 2022-04-11 18:24:29.804579: Average global foreground Dice: [0.6697, 0.8997, 0.9342, 0.1831] 2022-04-11 18:24:29.805672: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:24:30.712063: lr: 0.009095 2022-04-11 18:24:30.713348: saving scheduled checkpoint file... 2022-04-11 18:24:30.784617: saving checkpoint... 2022-04-11 18:24:35.379048: done, saving took 4.66 seconds 2022-04-11 18:24:35.393460: done 2022-04-11 18:24:35.440846: saving checkpoint... 2022-04-11 18:24:39.896839: done, saving took 4.50 seconds 2022-04-11 18:24:39.905081: This epoch took 129.967837 s 2022-04-11 18:24:39.906527: svmem(total=99932160000, available=71727529984, percent=28.2, used=23947833344, free=26394963968, active=48186417152, inactive=22963634176, buffers=36462592, cached=49552900096, shared=2195857408, slab=1269506048) 2022-04-11 18:24:39.907413: epoch: 50 2022-04-11 18:26:29.576477: train loss : -0.2727 2022-04-11 18:26:41.941293: validation loss: -0.3098 2022-04-11 18:26:41.945210: Average global foreground Dice: [0.6907, 0.8988, 0.9142, 0.226] 2022-04-11 18:26:41.946212: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:26:42.679753: lr: 0.009077 2022-04-11 18:26:42.681051: saving scheduled checkpoint file... 2022-04-11 18:26:42.743543: saving checkpoint... 2022-04-11 18:26:48.246994: done, saving took 5.57 seconds 2022-04-11 18:26:48.274025: done 2022-04-11 18:26:48.357218: saving checkpoint... 2022-04-11 18:26:52.880959: done, saving took 4.60 seconds 2022-04-11 18:26:52.899680: This epoch took 132.991257 s 2022-04-11 18:26:52.901900: svmem(total=99932160000, available=71610380288, percent=28.3, used=24050700288, free=26282868736, active=48292466688, inactive=22973239296, buffers=36569088, cached=49562021888, shared=2210033664, slab=1269805056) 2022-04-11 18:26:52.903130: epoch: 51 2022-04-11 18:28:41.574866: train loss : -0.2963 2022-04-11 18:28:53.436087: validation loss: -0.3555 2022-04-11 18:28:53.448000: Average global foreground Dice: [0.6683, 0.9094, 0.9465, 0.2167] 2022-04-11 18:28:53.451240: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:28:54.263865: lr: 0.009059 2022-04-11 18:28:54.265044: saving scheduled checkpoint file... 2022-04-11 18:28:54.383684: saving checkpoint... 2022-04-11 18:28:58.955178: done, saving took 4.69 seconds 2022-04-11 18:28:58.973727: done 2022-04-11 18:28:59.075499: saving checkpoint... 2022-04-11 18:29:03.541825: done, saving took 4.57 seconds 2022-04-11 18:29:03.592101: This epoch took 130.688019 s 2022-04-11 18:29:03.593433: svmem(total=99932160000, available=71462690816, percent=28.5, used=24206098432, free=26135486464, active=48443273216, inactive=22965579776, buffers=36638720, cached=49553936384, shared=2202361856, slab=1268740096) 2022-04-11 18:29:03.594243: epoch: 52 2022-04-11 18:30:50.411127: train loss : -0.3107 2022-04-11 18:31:02.445080: validation loss: -0.3850 2022-04-11 18:31:02.455705: Average global foreground Dice: [0.7039, 0.9088, 0.9377, 0.2216] 2022-04-11 18:31:02.456815: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:31:03.298851: lr: 0.009041 2022-04-11 18:31:03.300134: saving scheduled checkpoint file... 2022-04-11 18:31:03.428711: saving checkpoint... 2022-04-11 18:31:08.526881: done, saving took 5.23 seconds 2022-04-11 18:31:08.546822: done 2022-04-11 18:31:08.655996: saving checkpoint... 2022-04-11 18:31:13.151336: done, saving took 4.60 seconds 2022-04-11 18:31:13.193506: This epoch took 129.598294 s 2022-04-11 18:31:13.194961: svmem(total=99932160000, available=71598776320, percent=28.4, used=24075669504, free=26272243712, active=48313520128, inactive=22959153152, buffers=36712448, cached=49547534336, shared=2196566016, slab=1268355072) 2022-04-11 18:31:13.195661: epoch: 53 2022-04-11 18:33:03.117377: train loss : -0.2961 2022-04-11 18:33:15.356652: validation loss: -0.2525 2022-04-11 18:33:15.372297: Average global foreground Dice: [0.5791, 0.8862, 0.9048, 0.2262] 2022-04-11 18:33:15.373276: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:33:16.240678: lr: 0.009023 2022-04-11 18:33:16.242624: saving scheduled checkpoint file... 2022-04-11 18:33:16.345733: saving checkpoint... 2022-04-11 18:33:22.947997: done, saving took 6.70 seconds 2022-04-11 18:33:22.980865: done 2022-04-11 18:33:23.032306: saving checkpoint... 2022-04-11 18:33:27.526361: done, saving took 4.54 seconds 2022-04-11 18:33:27.540892: This epoch took 134.344487 s 2022-04-11 18:33:27.542404: svmem(total=99932160000, available=71634329600, percent=28.3, used=24039342080, free=26301202432, active=48321163264, inactive=22918643712, buffers=38002688, cached=49553612800, shared=2197114880, slab=1270038528) 2022-04-11 18:33:27.543278: epoch: 54 2022-04-11 18:35:16.712683: train loss : -0.2925 2022-04-11 18:35:28.525511: validation loss: -0.2981 2022-04-11 18:35:28.535470: Average global foreground Dice: [0.6177, 0.8855, 0.9181, 0.2165] 2022-04-11 18:35:28.536556: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:35:29.347002: lr: 0.009004 2022-04-11 18:35:29.348503: saving scheduled checkpoint file... 2022-04-11 18:35:29.405802: saving checkpoint... 2022-04-11 18:35:34.004719: done, saving took 4.65 seconds 2022-04-11 18:35:34.025125: done 2022-04-11 18:35:34.109118: saving checkpoint... 2022-04-11 18:35:39.554148: done, saving took 5.53 seconds 2022-04-11 18:35:39.577658: This epoch took 132.033724 s 2022-04-11 18:35:39.579067: svmem(total=99932160000, available=71571398656, percent=28.4, used=24092934144, free=26314756096, active=48371830784, inactive=22853844992, buffers=38096896, cached=49486372864, shared=2207285248, slab=1268477952) 2022-04-11 18:35:39.579776: epoch: 55 2022-04-11 18:37:28.640073: train loss : -0.3035 2022-04-11 18:37:40.367881: validation loss: -0.3285 2022-04-11 18:37:40.370768: Average global foreground Dice: [0.6782, 0.8817, 0.9342, 0.2234] 2022-04-11 18:37:40.372050: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:37:41.330021: lr: 0.008986 2022-04-11 18:37:41.331450: saving scheduled checkpoint file... 2022-04-11 18:37:41.441792: saving checkpoint... 2022-04-11 18:37:45.998539: done, saving took 4.67 seconds 2022-04-11 18:37:46.017558: done 2022-04-11 18:37:46.118798: saving checkpoint... 2022-04-11 18:37:50.666002: done, saving took 4.65 seconds 2022-04-11 18:37:50.683707: This epoch took 131.103276 s 2022-04-11 18:37:50.684757: svmem(total=99932160000, available=71464701952, percent=28.5, used=24210788352, free=26207236096, active=48492294144, inactive=22842175488, buffers=38162432, cached=49475973120, shared=2196156416, slab=1270177792) 2022-04-11 18:37:50.685725: epoch: 56 2022-04-11 18:39:39.478338: train loss : -0.3213 2022-04-11 18:39:51.126627: validation loss: -0.3554 2022-04-11 18:39:51.141716: Average global foreground Dice: [0.6691, 0.9184, 0.9323, 0.2547] 2022-04-11 18:39:51.142672: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:39:52.054994: lr: 0.008968 2022-04-11 18:39:52.056038: saving scheduled checkpoint file... 2022-04-11 18:39:52.198312: saving checkpoint... 2022-04-11 18:39:56.634041: done, saving took 4.58 seconds 2022-04-11 18:39:56.666616: done 2022-04-11 18:39:56.777284: saving checkpoint... 2022-04-11 18:40:01.281710: done, saving took 4.61 seconds 2022-04-11 18:40:01.314371: This epoch took 130.627754 s 2022-04-11 18:40:01.315439: svmem(total=99932160000, available=71716265984, percent=28.2, used=23957630976, free=26385530880, active=48230268928, inactive=22928240640, buffers=38260736, cached=49550737408, shared=2196234240, slab=1271246848) 2022-04-11 18:40:01.316538: epoch: 57 2022-04-11 18:41:48.912946: train loss : -0.2849 2022-04-11 18:42:01.114998: validation loss: -0.3127 2022-04-11 18:42:01.116895: Average global foreground Dice: [0.6739, 0.9058, 0.911, 0.237] 2022-04-11 18:42:01.117906: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:42:01.873206: lr: 0.00895 2022-04-11 18:42:01.874615: saving scheduled checkpoint file... 2022-04-11 18:42:01.992149: saving checkpoint... 2022-04-11 18:42:06.449503: done, saving took 4.57 seconds 2022-04-11 18:42:06.471187: done 2022-04-11 18:42:06.598241: saving checkpoint... 2022-04-11 18:42:11.103557: done, saving took 4.63 seconds 2022-04-11 18:42:11.148494: This epoch took 129.831289 s 2022-04-11 18:42:11.149966: svmem(total=99932160000, available=71661916160, percent=28.3, used=24009187328, free=26332950528, active=48348536832, inactive=22863003648, buffers=38338560, cached=49551683584, shared=2199019520, slab=1271382016) 2022-04-11 18:42:11.150712: epoch: 58 2022-04-11 18:44:01.092751: train loss : -0.2992 2022-04-11 18:44:13.966063: validation loss: -0.3487 2022-04-11 18:44:13.981417: Average global foreground Dice: [0.6501, 0.9098, 0.9259, 0.2558] 2022-04-11 18:44:13.982708: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:44:14.832553: lr: 0.008931 2022-04-11 18:44:14.834147: saving scheduled checkpoint file... 2022-04-11 18:44:14.962589: saving checkpoint... 2022-04-11 18:44:19.512757: done, saving took 4.68 seconds 2022-04-11 18:44:19.530703: done 2022-04-11 18:44:19.680928: saving checkpoint... 2022-04-11 18:44:24.164651: done, saving took 4.63 seconds 2022-04-11 18:44:24.193014: This epoch took 133.041609 s 2022-04-11 18:44:24.194316: svmem(total=99932160000, available=71489613824, percent=28.5, used=24178991104, free=26159955968, active=48658714624, inactive=22723260416, buffers=38391808, cached=49554821120, shared=2201608192, slab=1269768192) 2022-04-11 18:44:24.195017: epoch: 59 2022-04-11 18:46:13.492661: train loss : -0.2966 2022-04-11 18:46:25.605897: validation loss: -0.3373 2022-04-11 18:46:25.636439: Average global foreground Dice: [0.6833, 0.9041, 0.93, 0.2571] 2022-04-11 18:46:25.637327: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:46:26.444637: lr: 0.008913 2022-04-11 18:46:26.446006: saving scheduled checkpoint file... 2022-04-11 18:46:26.539464: saving checkpoint... 2022-04-11 18:46:30.972082: done, saving took 4.53 seconds 2022-04-11 18:46:30.985836: done 2022-04-11 18:46:31.033545: saving checkpoint... 2022-04-11 18:46:35.570113: done, saving took 4.58 seconds 2022-04-11 18:46:35.584309: This epoch took 131.388350 s 2022-04-11 18:46:35.585581: svmem(total=99932160000, available=71599202304, percent=28.4, used=24060768256, free=26269483008, active=48601710592, inactive=22674087936, buffers=38502400, cached=49563406336, shared=2210119680, slab=1270198272) 2022-04-11 18:46:35.586292: epoch: 60 2022-04-11 18:48:23.693566: train loss : -0.3323 2022-04-11 18:48:35.661390: validation loss: -0.3909 2022-04-11 18:48:35.673328: Average global foreground Dice: [0.7004, 0.9109, 0.9373, 0.2695] 2022-04-11 18:48:35.674351: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:48:36.497207: lr: 0.008895 2022-04-11 18:48:36.498325: saving scheduled checkpoint file... 2022-04-11 18:48:36.576968: saving checkpoint... 2022-04-11 18:48:41.134439: done, saving took 4.64 seconds 2022-04-11 18:48:41.147012: done 2022-04-11 18:48:41.195865: saving checkpoint... 2022-04-11 18:48:45.818881: done, saving took 4.67 seconds 2022-04-11 18:48:45.832528: This epoch took 130.245295 s 2022-04-11 18:48:45.833772: svmem(total=99932160000, available=71706546176, percent=28.2, used=23967260672, free=26376970240, active=48516661248, inactive=22650830848, buffers=38572032, cached=49549357056, shared=2196344832, slab=1269071872) 2022-04-11 18:48:45.834639: epoch: 61 2022-04-11 18:50:33.130478: train loss : -0.3591 2022-04-11 18:50:44.740417: validation loss: -0.2618 2022-04-11 18:50:44.751950: Average global foreground Dice: [0.6659, 0.8925, 0.8478, 0.2566] 2022-04-11 18:50:44.752941: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:50:45.582565: lr: 0.008877 2022-04-11 18:50:45.583765: saving scheduled checkpoint file... 2022-04-11 18:50:45.653071: saving checkpoint... 2022-04-11 18:50:50.399377: done, saving took 4.81 seconds 2022-04-11 18:50:50.434525: done 2022-04-11 18:50:50.435855: This epoch took 124.600477 s 2022-04-11 18:50:50.436903: svmem(total=99932160000, available=71607853056, percent=28.3, used=24066535424, free=26320703488, active=48613801984, inactive=22608355328, buffers=38645760, cached=49506275328, shared=2196320256, slab=1268695040) 2022-04-11 18:50:50.437686: epoch: 62 2022-04-11 18:52:39.714893: train loss : -0.1674 2022-04-11 18:52:50.808251: validation loss: -0.1829 2022-04-11 18:52:50.816185: Average global foreground Dice: [0.5343, 0.8841, 0.89, 0.2216] 2022-04-11 18:52:50.817350: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:52:51.875927: lr: 0.008859 2022-04-11 18:52:51.877342: saving scheduled checkpoint file... 2022-04-11 18:52:51.984458: saving checkpoint... 2022-04-11 18:52:56.509390: done, saving took 4.63 seconds 2022-04-11 18:52:56.530168: done 2022-04-11 18:52:56.531462: This epoch took 126.092995 s 2022-04-11 18:52:56.532508: svmem(total=99932160000, available=71563534336, percent=28.4, used=24104026112, free=26275471360, active=48653348864, inactive=22611988480, buffers=38690816, cached=49513971712, shared=2203095040, slab=1269813248) 2022-04-11 18:52:56.533659: epoch: 63 2022-04-11 18:54:44.118885: train loss : -0.2588 2022-04-11 18:54:55.911335: validation loss: -0.3048 2022-04-11 18:54:55.913478: Average global foreground Dice: [0.6674, 0.8902, 0.9209, 0.2411] 2022-04-11 18:54:55.914257: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:54:56.757299: lr: 0.00884 2022-04-11 18:54:56.758605: saving scheduled checkpoint file... 2022-04-11 18:54:56.894438: saving checkpoint... 2022-04-11 18:55:01.417926: done, saving took 4.66 seconds 2022-04-11 18:55:01.438566: done 2022-04-11 18:55:01.439763: This epoch took 124.904944 s 2022-04-11 18:55:01.440919: svmem(total=99932160000, available=71699283968, percent=28.3, used=23975256064, free=26410704896, active=48526008320, inactive=22609354752, buffers=38785024, cached=49507414016, shared=2195988480, slab=1268707328) 2022-04-11 18:55:01.441984: epoch: 64 2022-04-11 18:56:50.547596: train loss : -0.2805 2022-04-11 18:57:02.843235: validation loss: -0.3899 2022-04-11 18:57:02.845118: Average global foreground Dice: [0.6932, 0.9063, 0.9469, 0.2723] 2022-04-11 18:57:02.846009: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:57:03.635328: lr: 0.008822 2022-04-11 18:57:03.636464: saving scheduled checkpoint file... 2022-04-11 18:57:03.758488: saving checkpoint... 2022-04-11 18:57:08.720810: done, saving took 5.08 seconds 2022-04-11 18:57:08.783103: done 2022-04-11 18:57:08.917992: saving checkpoint... 2022-04-11 18:57:13.480803: done, saving took 4.70 seconds 2022-04-11 18:57:13.500603: This epoch took 132.057818 s 2022-04-11 18:57:13.502007: svmem(total=99932160000, available=71836090368, percent=28.1, used=23838359552, free=26546499584, active=48386560000, inactive=22610198528, buffers=38871040, cached=49508429824, shared=2196037632, slab=1270181888) 2022-04-11 18:57:13.502861: epoch: 65 2022-04-11 18:59:03.428882: train loss : -0.3293 2022-04-11 18:59:15.966074: validation loss: -0.3740 2022-04-11 18:59:15.971840: Average global foreground Dice: [0.6972, 0.9212, 0.9508, 0.2618] 2022-04-11 18:59:15.972989: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 18:59:17.021910: lr: 0.008804 2022-04-11 18:59:17.023200: saving scheduled checkpoint file... 2022-04-11 18:59:17.129821: saving checkpoint... 2022-04-11 18:59:21.575229: done, saving took 4.55 seconds 2022-04-11 18:59:21.610252: done 2022-04-11 18:59:21.658234: saving checkpoint... 2022-04-11 18:59:25.974214: done, saving took 4.36 seconds 2022-04-11 18:59:26.003019: This epoch took 132.499499 s 2022-04-11 18:59:26.004272: svmem(total=99932160000, available=71514935296, percent=28.4, used=24153792512, free=26224365568, active=48698785792, inactive=22616752128, buffers=38952960, cached=49515048960, shared=2201825280, slab=1268744192) 2022-04-11 18:59:26.005050: epoch: 66 2022-04-11 19:01:14.840733: train loss : -0.3584 2022-04-11 19:01:26.341200: validation loss: -0.3831 2022-04-11 19:01:26.343409: Average global foreground Dice: [0.6624, 0.9126, 0.9412, 0.2851] 2022-04-11 19:01:26.344562: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:01:27.248709: lr: 0.008785 2022-04-11 19:01:27.250056: saving scheduled checkpoint file... 2022-04-11 19:01:27.298142: saving checkpoint... 2022-04-11 19:01:31.812107: done, saving took 4.56 seconds 2022-04-11 19:01:31.838436: done 2022-04-11 19:01:31.888686: saving checkpoint... 2022-04-11 19:01:36.518113: done, saving took 4.68 seconds 2022-04-11 19:01:36.531741: This epoch took 130.525735 s 2022-04-11 19:01:36.533341: svmem(total=99932160000, available=71662645248, percent=28.3, used=23998423040, free=26371342336, active=48547401728, inactive=22624346112, buffers=39038976, cached=49523355648, shared=2209427456, slab=1270165504) 2022-04-11 19:01:36.534223: epoch: 67 2022-04-11 19:03:25.414249: train loss : -0.3177 2022-04-11 19:03:37.140378: validation loss: -0.3463 2022-04-11 19:03:37.155571: Average global foreground Dice: [0.626, 0.9139, 0.9226, 0.27] 2022-04-11 19:03:37.156792: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:03:37.989008: lr: 0.008767 2022-04-11 19:03:37.990121: saving scheduled checkpoint file... 2022-04-11 19:03:38.070198: saving checkpoint... 2022-04-11 19:03:42.565090: done, saving took 4.57 seconds 2022-04-11 19:03:42.580556: done 2022-04-11 19:03:42.628563: saving checkpoint... 2022-04-11 19:03:47.070929: done, saving took 4.49 seconds 2022-04-11 19:03:47.084941: This epoch took 130.549881 s 2022-04-11 19:03:47.086272: svmem(total=99932160000, available=71560445952, percent=28.4, used=24110952448, free=26254589952, active=48660709376, inactive=22624129024, buffers=39120896, cached=49527496704, shared=2196131840, slab=1275097088) 2022-04-11 19:03:47.087021: epoch: 68 2022-04-11 19:05:35.931736: train loss : -0.2977 2022-04-11 19:05:46.232391: validation loss: -0.3668 2022-04-11 19:05:46.234485: Average global foreground Dice: [0.6824, 0.9128, 0.9342, 0.2553] 2022-04-11 19:05:46.238213: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:05:47.371220: lr: 0.008749 2022-04-11 19:05:47.389216: saving scheduled checkpoint file... 2022-04-11 19:05:47.569403: saving checkpoint... 2022-04-11 19:05:52.743104: done, saving took 5.35 seconds 2022-04-11 19:05:52.760787: done 2022-04-11 19:05:52.844453: saving checkpoint... 2022-04-11 19:05:57.410658: done, saving took 4.65 seconds 2022-04-11 19:05:57.458231: This epoch took 130.370266 s 2022-04-11 19:05:57.459531: svmem(total=99932160000, available=71542263808, percent=28.4, used=24128946176, free=26237870080, active=48178348032, inactive=23120572416, buffers=39202816, cached=49526140928, shared=2196267008, slab=1274306560) 2022-04-11 19:05:57.460680: epoch: 69 2022-04-11 19:07:45.677128: train loss : -0.3652 2022-04-11 19:07:56.922598: validation loss: -0.3936 2022-04-11 19:07:56.948858: Average global foreground Dice: [0.6856, 0.9133, 0.9419, 0.2734] 2022-04-11 19:07:56.949567: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:07:57.532381: lr: 0.008731 2022-04-11 19:07:57.533941: saving scheduled checkpoint file... 2022-04-11 19:07:57.651454: saving checkpoint... 2022-04-11 19:08:02.199742: done, saving took 4.66 seconds 2022-04-11 19:08:02.218259: done 2022-04-11 19:08:02.323250: saving checkpoint... 2022-04-11 19:08:06.740102: done, saving took 4.52 seconds 2022-04-11 19:08:06.792221: This epoch took 129.330678 s 2022-04-11 19:08:06.793323: svmem(total=99932160000, available=71588765696, percent=28.4, used=24082755584, free=26283499520, active=48303321088, inactive=22951907328, buffers=39288832, cached=49526616064, shared=2195816448, slab=1275293696) 2022-04-11 19:08:06.794385: epoch: 70 2022-04-11 19:09:55.998854: train loss : -0.3721 2022-04-11 19:10:08.037819: validation loss: -0.3848 2022-04-11 19:10:08.040195: Average global foreground Dice: [0.7221, 0.8994, 0.9396, 0.2956] 2022-04-11 19:10:08.041448: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:10:08.940151: lr: 0.008712 2022-04-11 19:10:08.941236: saving scheduled checkpoint file... 2022-04-11 19:10:09.069282: saving checkpoint... 2022-04-11 19:10:13.528571: done, saving took 4.59 seconds 2022-04-11 19:10:13.548584: done 2022-04-11 19:10:13.662460: saving checkpoint... 2022-04-11 19:10:18.178811: done, saving took 4.63 seconds 2022-04-11 19:10:18.199568: This epoch took 131.404381 s 2022-04-11 19:10:18.200948: svmem(total=99932160000, available=71644168192, percent=28.3, used=24015114240, free=26338082816, active=48316346368, inactive=22887129088, buffers=39362560, cached=49539600384, shared=2208051200, slab=1275105280) 2022-04-11 19:10:18.201761: epoch: 71 2022-04-11 19:12:05.877136: train loss : -0.3600 2022-04-11 19:12:17.342470: validation loss: -0.4041 2022-04-11 19:12:17.356085: Average global foreground Dice: [0.6847, 0.9179, 0.9394, 0.3339] 2022-04-11 19:12:17.357818: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:12:18.472327: lr: 0.008694 2022-04-11 19:12:18.473532: saving scheduled checkpoint file... 2022-04-11 19:12:18.562549: saving checkpoint... 2022-04-11 19:12:23.044789: done, saving took 4.57 seconds 2022-04-11 19:12:23.059174: done 2022-04-11 19:12:23.106300: saving checkpoint... 2022-04-11 19:12:27.551095: done, saving took 4.49 seconds 2022-04-11 19:12:27.565723: This epoch took 129.361476 s 2022-04-11 19:12:27.566680: svmem(total=99932160000, available=71665942528, percent=28.3, used=24004931584, free=26358951936, active=48312045568, inactive=22869209088, buffers=39440384, cached=49528836096, shared=2196586496, slab=1275899904) 2022-04-11 19:12:27.567436: epoch: 72 2022-04-11 19:14:15.358081: train loss : -0.3710 2022-04-11 19:14:27.388683: validation loss: -0.4027 2022-04-11 19:14:27.394666: Average global foreground Dice: [0.6831, 0.9276, 0.9403, 0.2638] 2022-04-11 19:14:27.395876: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:14:28.130869: lr: 0.008676 2022-04-11 19:14:28.132074: saving scheduled checkpoint file... 2022-04-11 19:14:28.186796: saving checkpoint... 2022-04-11 19:14:32.211374: done, saving took 4.08 seconds 2022-04-11 19:14:32.262525: done 2022-04-11 19:14:32.347119: saving checkpoint... 2022-04-11 19:14:36.889698: done, saving took 4.63 seconds 2022-04-11 19:14:36.919523: This epoch took 129.351362 s 2022-04-11 19:14:36.938778: svmem(total=99932160000, available=71564451840, percent=28.4, used=24106618880, free=26256531456, active=48418197504, inactive=22864703488, buffers=39518208, cached=49529491456, shared=2196279296, slab=1275445248) 2022-04-11 19:14:36.940225: epoch: 73 2022-04-11 19:16:24.732725: train loss : -0.4040 2022-04-11 19:16:37.376808: validation loss: -0.4257 2022-04-11 19:16:37.378488: Average global foreground Dice: [0.6775, 0.9142, 0.9592, 0.3157] 2022-04-11 19:16:37.379475: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:16:38.279783: lr: 0.008658 2022-04-11 19:16:38.280822: saving scheduled checkpoint file... 2022-04-11 19:16:38.411131: saving checkpoint... 2022-04-11 19:16:42.910377: done, saving took 4.63 seconds 2022-04-11 19:16:42.929317: done 2022-04-11 19:16:43.034139: saving checkpoint... 2022-04-11 19:16:47.557591: done, saving took 4.63 seconds 2022-04-11 19:16:47.576306: This epoch took 130.635303 s 2022-04-11 19:16:47.577266: svmem(total=99932160000, available=71728316416, percent=28.2, used=23939100672, free=26419535872, active=48270594048, inactive=22851735552, buffers=39616512, cached=49533906944, shared=2199924736, slab=1275011072) 2022-04-11 19:16:47.577808: epoch: 74 2022-04-11 19:18:35.549463: train loss : -0.3739 2022-04-11 19:18:46.773409: validation loss: -0.4119 2022-04-11 19:18:46.787871: Average global foreground Dice: [0.7213, 0.9266, 0.9495, 0.2868] 2022-04-11 19:18:46.788670: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:18:47.654905: lr: 0.008639 2022-04-11 19:18:47.656188: saving scheduled checkpoint file... 2022-04-11 19:18:47.774129: saving checkpoint... 2022-04-11 19:18:52.139045: done, saving took 4.48 seconds 2022-04-11 19:18:52.159067: done 2022-04-11 19:18:52.270658: saving checkpoint... 2022-04-11 19:18:56.956924: done, saving took 4.80 seconds 2022-04-11 19:18:56.978259: This epoch took 129.399818 s 2022-04-11 19:18:56.979737: svmem(total=99932160000, available=71490621440, percent=28.5, used=24173600768, free=26181021696, active=48506781696, inactive=22854483968, buffers=39702528, cached=49537835008, shared=2203095040, slab=1275387904) 2022-04-11 19:18:56.980553: epoch: 75 2022-04-11 19:20:45.059828: train loss : -0.3980 2022-04-11 19:20:56.364429: validation loss: -0.3892 2022-04-11 19:20:56.367148: Average global foreground Dice: [0.7088, 0.9086, 0.9452, 0.2595] 2022-04-11 19:20:56.368699: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:20:57.389331: lr: 0.008621 2022-04-11 19:20:57.390827: saving scheduled checkpoint file... 2022-04-11 19:20:57.543863: saving checkpoint... 2022-04-11 19:21:01.998310: done, saving took 4.61 seconds 2022-04-11 19:21:02.017835: done 2022-04-11 19:21:02.155776: saving checkpoint... 2022-04-11 19:21:06.653823: done, saving took 4.63 seconds 2022-04-11 19:21:06.690271: This epoch took 129.708904 s 2022-04-11 19:21:06.691647: svmem(total=99932160000, available=71511789568, percent=28.4, used=24145567744, free=26201030656, active=48478085120, inactive=22860414976, buffers=39776256, cached=49545785344, shared=2209845248, slab=1277542400) 2022-04-11 19:21:06.692479: epoch: 76 2022-04-11 19:22:57.045855: train loss : -0.4106 2022-04-11 19:23:09.314528: validation loss: -0.3869 2022-04-11 19:23:09.324072: Average global foreground Dice: [0.7323, 0.9, 0.9126, 0.3204] 2022-04-11 19:23:09.325611: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:23:09.893554: lr: 0.008603 2022-04-11 19:23:09.894955: saving scheduled checkpoint file... 2022-04-11 19:23:10.022842: saving checkpoint... 2022-04-11 19:23:14.486649: done, saving took 4.59 seconds 2022-04-11 19:23:14.502433: done 2022-04-11 19:23:14.573675: saving checkpoint... 2022-04-11 19:23:19.023079: done, saving took 4.52 seconds 2022-04-11 19:23:19.047829: This epoch took 132.354639 s 2022-04-11 19:23:19.049392: svmem(total=99932160000, available=71549927424, percent=28.4, used=24121094144, free=26237427712, active=48454230016, inactive=22847504384, buffers=39854080, cached=49533784064, shared=2196242432, slab=1275908096) 2022-04-11 19:23:19.050087: epoch: 77 2022-04-11 19:25:10.135635: train loss : -0.3733 2022-04-11 19:25:22.228729: validation loss: -0.3860 2022-04-11 19:25:23.247606: Average global foreground Dice: [0.6869, 0.9111, 0.9394, 0.3179] 2022-04-11 19:25:23.266178: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:25:24.048889: lr: 0.008584 2022-04-11 19:25:24.050146: saving scheduled checkpoint file... 2022-04-11 19:25:24.145679: saving checkpoint... 2022-04-11 19:25:28.631785: done, saving took 4.58 seconds 2022-04-11 19:25:28.652988: done 2022-04-11 19:25:28.700644: saving checkpoint... 2022-04-11 19:25:33.204587: done, saving took 4.55 seconds 2022-04-11 19:25:33.213783: This epoch took 134.162761 s 2022-04-11 19:25:33.214681: svmem(total=99932160000, available=71545466880, percent=28.4, used=24124325888, free=26225819648, active=48493047808, inactive=22814990336, buffers=39997440, cached=49542017024, shared=2196156416, slab=1278791680) 2022-04-11 19:25:33.215349: epoch: 78 2022-04-11 19:27:22.134333: train loss : -0.3630 2022-04-11 19:27:34.991335: validation loss: -0.4033 2022-04-11 19:27:35.000250: Average global foreground Dice: [0.6944, 0.9162, 0.9373, 0.3037] 2022-04-11 19:27:35.001916: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:27:35.870211: lr: 0.008566 2022-04-11 19:27:35.871379: saving scheduled checkpoint file... 2022-04-11 19:27:35.940581: saving checkpoint... 2022-04-11 19:27:40.481856: done, saving took 4.61 seconds 2022-04-11 19:27:40.494423: done 2022-04-11 19:27:40.543094: saving checkpoint... 2022-04-11 19:27:45.136357: done, saving took 4.64 seconds 2022-04-11 19:27:45.149646: This epoch took 131.933634 s 2022-04-11 19:27:45.150763: svmem(total=99932160000, available=71461920768, percent=28.5, used=24201187328, free=26140938240, active=48576544768, inactive=22816088064, buffers=40067072, cached=49549967360, shared=2203455488, slab=1278943232) 2022-04-11 19:27:45.151317: epoch: 79 2022-04-11 19:29:32.733871: train loss : -0.3868 2022-04-11 19:29:44.901370: validation loss: -0.4496 2022-04-11 19:29:44.905062: Average global foreground Dice: [0.7104, 0.9221, 0.9576, 0.3175] 2022-04-11 19:29:44.906243: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:29:45.619288: lr: 0.008548 2022-04-11 19:29:45.620360: saving scheduled checkpoint file... 2022-04-11 19:29:45.685131: saving checkpoint... 2022-04-11 19:29:50.368643: done, saving took 4.75 seconds 2022-04-11 19:29:50.405892: done 2022-04-11 19:29:50.491840: saving checkpoint... 2022-04-11 19:29:54.969457: done, saving took 4.56 seconds 2022-04-11 19:29:54.989386: This epoch took 129.837383 s 2022-04-11 19:29:54.990787: svmem(total=99932160000, available=71632769024, percent=28.3, used=24029773824, free=26311634944, active=48401375232, inactive=22824128512, buffers=40128512, cached=49550622720, shared=2203402240, slab=1276809216) 2022-04-11 19:29:54.991602: epoch: 80 2022-04-11 19:31:44.080250: train loss : -0.4448 2022-04-11 19:31:56.323915: validation loss: -0.4552 2022-04-11 19:31:56.331892: Average global foreground Dice: [0.7535, 0.9262, 0.952, 0.3037] 2022-04-11 19:31:56.340147: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:31:57.196690: lr: 0.008529 2022-04-11 19:31:57.199056: saving scheduled checkpoint file... 2022-04-11 19:31:57.309347: saving checkpoint... 2022-04-11 19:32:01.742598: done, saving took 4.54 seconds 2022-04-11 19:32:01.781747: done 2022-04-11 19:32:01.885534: saving checkpoint... 2022-04-11 19:32:06.962238: done, saving took 5.18 seconds 2022-04-11 19:32:06.980601: This epoch took 131.987883 s 2022-04-11 19:32:06.982064: svmem(total=99932160000, available=71473950720, percent=28.5, used=24195764224, free=26151849984, active=48565063680, inactive=22815821824, buffers=40202240, cached=49544343552, shared=2196242432, slab=1279438848) 2022-04-11 19:32:06.982964: epoch: 81 2022-04-11 19:33:57.050493: train loss : -0.4096 2022-04-11 19:34:09.154996: validation loss: -0.4130 2022-04-11 19:34:09.176945: Average global foreground Dice: [0.6526, 0.925, 0.939, 0.3011] 2022-04-11 19:34:09.178322: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:34:09.994476: lr: 0.008511 2022-04-11 19:34:09.995870: saving scheduled checkpoint file... 2022-04-11 19:34:10.087293: saving checkpoint... 2022-04-11 19:34:14.725005: done, saving took 4.73 seconds 2022-04-11 19:34:14.749810: done 2022-04-11 19:34:14.750645: This epoch took 127.767024 s 2022-04-11 19:34:14.751624: svmem(total=99932160000, available=71704023040, percent=28.2, used=23955218432, free=26381119488, active=48326811648, inactive=22827290624, buffers=40275968, cached=49555546112, shared=2206662656, slab=1277960192) 2022-04-11 19:34:14.753192: epoch: 82 2022-04-11 19:36:04.310863: train loss : -0.3612 2022-04-11 19:36:16.091063: validation loss: -0.4410 2022-04-11 19:36:16.093875: Average global foreground Dice: [0.708, 0.9254, 0.9527, 0.3222] 2022-04-11 19:36:16.095651: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:36:16.974387: lr: 0.008493 2022-04-11 19:36:16.975826: saving scheduled checkpoint file... 2022-04-11 19:36:17.129396: saving checkpoint... 2022-04-11 19:36:21.600360: done, saving took 4.62 seconds 2022-04-11 19:36:21.641728: done 2022-04-11 19:36:21.777829: saving checkpoint... 2022-04-11 19:36:26.254807: done, saving took 4.61 seconds 2022-04-11 19:36:26.317581: This epoch took 131.563540 s 2022-04-11 19:36:26.319089: svmem(total=99932160000, available=71642103808, percent=28.3, used=24027541504, free=26318311424, active=48400392192, inactive=22817390592, buffers=40390656, cached=49545916416, shared=2196303872, slab=1276420096) 2022-04-11 19:36:26.320107: epoch: 83 2022-04-11 19:38:16.808902: train loss : -0.4104 2022-04-11 19:38:29.730065: validation loss: -0.4627 2022-04-11 19:38:29.740315: Average global foreground Dice: [0.7331, 0.9322, 0.9507, 0.3072] 2022-04-11 19:38:29.741242: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:38:30.458202: lr: 0.008474 2022-04-11 19:38:30.459808: saving scheduled checkpoint file... 2022-04-11 19:38:30.563952: saving checkpoint... 2022-04-11 19:38:34.996957: done, saving took 4.54 seconds 2022-04-11 19:38:35.035689: done 2022-04-11 19:38:35.106128: saving checkpoint... 2022-04-11 19:38:39.511912: done, saving took 4.48 seconds 2022-04-11 19:38:39.528036: This epoch took 133.207195 s 2022-04-11 19:38:39.529825: svmem(total=99932160000, available=71572623360, percent=28.4, used=24097067008, free=26247806976, active=48470913024, inactive=22818189312, buffers=40460288, cached=49546825728, shared=2196242432, slab=1278382080) 2022-04-11 19:38:39.531132: epoch: 84 2022-04-11 19:40:27.280923: train loss : -0.4155 2022-04-11 19:40:39.516195: validation loss: -0.4461 2022-04-11 19:40:39.524880: Average global foreground Dice: [0.7325, 0.9233, 0.9504, 0.3176] 2022-04-11 19:40:39.526098: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:40:40.140849: lr: 0.008456 2022-04-11 19:40:40.142279: saving scheduled checkpoint file... 2022-04-11 19:40:40.236061: saving checkpoint... 2022-04-11 19:40:44.794121: done, saving took 4.65 seconds 2022-04-11 19:40:44.811843: done 2022-04-11 19:40:44.895412: saving checkpoint... 2022-04-11 19:40:49.456858: done, saving took 4.64 seconds 2022-04-11 19:40:49.499838: This epoch took 129.967942 s 2022-04-11 19:40:49.501038: svmem(total=99932160000, available=71792365568, percent=28.2, used=23877062656, free=26466758656, active=48250900480, inactive=22818889728, buffers=40534016, cached=49547804672, shared=2196287488, slab=1278128128) 2022-04-11 19:40:49.501954: epoch: 85 2022-04-11 19:42:37.287330: train loss : -0.4172 2022-04-11 19:42:47.758431: validation loss: -0.3974 2022-04-11 19:42:47.771971: Average global foreground Dice: [0.7191, 0.9063, 0.9319, 0.2878] 2022-04-11 19:42:47.773105: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:42:48.525892: lr: 0.008438 2022-04-11 19:42:48.527438: saving scheduled checkpoint file... 2022-04-11 19:42:48.640390: saving checkpoint... 2022-04-11 19:42:53.211063: done, saving took 4.68 seconds 2022-04-11 19:42:53.226194: done 2022-04-11 19:42:53.227362: This epoch took 123.724557 s 2022-04-11 19:42:53.228231: svmem(total=99932160000, available=71718502400, percent=28.2, used=23950983168, free=26392104960, active=48326352896, inactive=22819643392, buffers=40595456, cached=49548476416, shared=2196328448, slab=1277681664) 2022-04-11 19:42:53.228983: epoch: 86 2022-04-11 19:44:41.781550: train loss : -0.2495 2022-04-11 19:44:53.812826: validation loss: -0.3352 2022-04-11 19:44:53.815717: Average global foreground Dice: [0.6904, 0.8935, 0.9038, 0.302] 2022-04-11 19:44:53.816701: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:44:54.725741: lr: 0.008419 2022-04-11 19:44:54.727298: saving scheduled checkpoint file... 2022-04-11 19:44:54.858162: saving checkpoint... 2022-04-11 19:44:59.216913: done, saving took 4.49 seconds 2022-04-11 19:44:59.295094: done 2022-04-11 19:44:59.296379: This epoch took 126.066712 s 2022-04-11 19:44:59.297391: svmem(total=99932160000, available=71777910784, percent=28.2, used=23892172800, free=26451034112, active=48270041088, inactive=22819479552, buffers=40660992, cached=49548292096, shared=2195808256, slab=1277435904) 2022-04-11 19:44:59.298231: epoch: 87 2022-04-11 19:46:46.828775: train loss : -0.3341 2022-04-11 19:46:58.114400: validation loss: -0.4187 2022-04-11 19:46:58.117058: Average global foreground Dice: [0.7142, 0.9207, 0.9327, 0.3331] 2022-04-11 19:46:58.118254: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:46:58.941216: lr: 0.008401 2022-04-11 19:46:58.942459: saving scheduled checkpoint file... 2022-04-11 19:46:59.093091: saving checkpoint... 2022-04-11 19:47:03.194418: done, saving took 4.25 seconds 2022-04-11 19:47:03.265430: done 2022-04-11 19:47:03.266385: This epoch took 123.966906 s 2022-04-11 19:47:03.267894: svmem(total=99932160000, available=71446822912, percent=28.5, used=24212312064, free=26119385088, active=48585580544, inactive=22830682112, buffers=40771584, cached=49559691264, shared=2206605312, slab=1277857792) 2022-04-11 19:47:03.268692: epoch: 88 2022-04-11 19:48:53.258987: train loss : -0.3861 2022-04-11 19:49:04.995120: validation loss: -0.4253 2022-04-11 19:49:05.010531: Average global foreground Dice: [0.7077, 0.9261, 0.9448, 0.3048] 2022-04-11 19:49:05.019721: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:49:05.906399: lr: 0.008383 2022-04-11 19:49:05.916491: saving scheduled checkpoint file... 2022-04-11 19:49:06.044207: saving checkpoint... 2022-04-11 19:49:11.840894: done, saving took 5.92 seconds 2022-04-11 19:49:11.865506: done 2022-04-11 19:49:12.010241: saving checkpoint... 2022-04-11 19:49:16.517218: done, saving took 4.65 seconds 2022-04-11 19:49:16.548550: This epoch took 133.279072 s 2022-04-11 19:49:16.551445: svmem(total=99932160000, available=71779344384, percent=28.2, used=23890124800, free=26452123648, active=48291885056, inactive=22794977280, buffers=40869888, cached=49549041664, shared=2196000768, slab=1277292544) 2022-04-11 19:49:16.552618: epoch: 89 2022-04-11 19:51:06.886498: train loss : -0.4039 2022-04-11 19:51:19.480164: validation loss: -0.4300 2022-04-11 19:51:19.487956: Average global foreground Dice: [0.7002, 0.9214, 0.951, 0.3194] 2022-04-11 19:51:19.488727: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:51:20.347590: lr: 0.008364 2022-04-11 19:51:20.349231: saving scheduled checkpoint file... 2022-04-11 19:51:20.444454: saving checkpoint... 2022-04-11 19:51:24.942250: done, saving took 4.59 seconds 2022-04-11 19:51:24.955611: done 2022-04-11 19:51:25.002809: saving checkpoint... 2022-04-11 19:51:29.539914: done, saving took 4.58 seconds 2022-04-11 19:51:29.590024: This epoch took 133.036464 s 2022-04-11 19:51:29.621548: svmem(total=99932160000, available=71504486400, percent=28.4, used=24164515840, free=26177101824, active=48566865920, inactive=22794919936, buffers=40955904, cached=49549586432, shared=2196500480, slab=1276252160) 2022-04-11 19:51:29.622849: epoch: 90 2022-04-11 19:53:17.859895: train loss : -0.4090 2022-04-11 19:53:30.224419: validation loss: -0.3744 2022-04-11 19:53:30.230760: Average global foreground Dice: [0.6051, 0.9238, 0.9362, 0.307] 2022-04-11 19:53:30.231589: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:53:31.112099: lr: 0.008346 2022-04-11 19:53:31.113342: saving scheduled checkpoint file... 2022-04-11 19:53:31.190931: saving checkpoint... 2022-04-11 19:53:36.088503: done, saving took 4.97 seconds 2022-04-11 19:53:36.130822: done 2022-04-11 19:53:36.133983: This epoch took 126.510064 s 2022-04-11 19:53:36.140381: svmem(total=99932160000, available=71334043648, percent=28.6, used=24335409152, free=26006441984, active=48736628736, inactive=22794616832, buffers=41025536, cached=49549283328, shared=2195918848, slab=1278242816) 2022-04-11 19:53:36.158503: epoch: 91 2022-04-11 19:55:26.028605: train loss : -0.3494 2022-04-11 19:55:38.603153: validation loss: -0.3939 2022-04-11 19:55:38.614536: Average global foreground Dice: [0.6356, 0.9163, 0.9411, 0.2884] 2022-04-11 19:55:38.615720: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:55:39.523292: lr: 0.008328 2022-04-11 19:55:39.524574: saving scheduled checkpoint file... 2022-04-11 19:55:39.589492: saving checkpoint... 2022-04-11 19:55:44.172227: done, saving took 4.65 seconds 2022-04-11 19:55:44.192266: done 2022-04-11 19:55:44.193638: This epoch took 128.033464 s 2022-04-11 19:55:44.195230: svmem(total=99932160000, available=71599108096, percent=28.4, used=24070664192, free=26271375360, active=48467746816, inactive=22793396224, buffers=41140224, cached=49548980224, shared=2195660800, slab=1278136320) 2022-04-11 19:55:44.196153: epoch: 92 2022-04-11 19:57:32.550651: train loss : -0.3844 2022-04-11 19:57:44.453231: validation loss: -0.4091 2022-04-11 19:57:44.460835: Average global foreground Dice: [0.7103, 0.9314, 0.9398, 0.3069] 2022-04-11 19:57:44.462112: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:57:45.288307: lr: 0.008309 2022-04-11 19:57:45.289788: saving scheduled checkpoint file... 2022-04-11 19:57:45.414749: saving checkpoint... 2022-04-11 19:57:50.038796: done, saving took 4.75 seconds 2022-04-11 19:57:50.059015: done 2022-04-11 19:57:50.060594: This epoch took 125.863561 s 2022-04-11 19:57:50.061672: svmem(total=99932160000, available=71562199040, percent=28.4, used=24094920704, free=26234097664, active=48494854144, inactive=22806233088, buffers=41185280, cached=49561956352, shared=2208313344, slab=1279070208) 2022-04-11 19:57:50.062662: epoch: 93 2022-04-11 19:59:39.512677: train loss : -0.3988 2022-04-11 19:59:51.909078: validation loss: -0.3881 2022-04-11 19:59:51.915817: Average global foreground Dice: [0.6768, 0.9189, 0.9306, 0.2951] 2022-04-11 19:59:51.916928: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 19:59:52.921978: lr: 0.008291 2022-04-11 19:59:52.924116: saving scheduled checkpoint file... 2022-04-11 19:59:53.058712: saving checkpoint... 2022-04-11 19:59:57.526218: done, saving took 4.60 seconds 2022-04-11 19:59:57.562885: done 2022-04-11 19:59:57.564087: This epoch took 127.500573 s 2022-04-11 19:59:57.565258: svmem(total=99932160000, available=71498047488, percent=28.5, used=24170582016, free=26169032704, active=48571752448, inactive=22794813440, buffers=41254912, cached=49551290368, shared=2196713472, slab=1280438272) 2022-04-11 19:59:57.566265: epoch: 94 2022-04-11 20:01:45.532434: train loss : -0.4101 2022-04-11 20:01:56.811867: validation loss: -0.4102 2022-04-11 20:01:56.816440: Average global foreground Dice: [0.7112, 0.9199, 0.9415, 0.2965] 2022-04-11 20:01:56.818262: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:01:57.685260: lr: 0.008272 2022-04-11 20:01:57.686611: saving scheduled checkpoint file... 2022-04-11 20:01:57.843076: saving checkpoint... 2022-04-11 20:02:04.976481: done, saving took 7.29 seconds 2022-04-11 20:02:05.001539: done 2022-04-11 20:02:05.005023: This epoch took 127.438080 s 2022-04-11 20:02:05.008065: svmem(total=99932160000, available=71580782592, percent=28.4, used=24080785408, free=26251612160, active=48483209216, inactive=22802276352, buffers=41324544, cached=49558437888, shared=2203754496, slab=1278554112) 2022-04-11 20:02:05.010296: epoch: 95 2022-04-11 20:03:54.922662: train loss : -0.4451 2022-04-11 20:04:07.544830: validation loss: -0.4631 2022-04-11 20:04:07.561880: Average global foreground Dice: [0.7417, 0.933, 0.9546, 0.3365] 2022-04-11 20:04:07.564678: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:04:08.217170: lr: 0.008254 2022-04-11 20:04:08.221091: saving scheduled checkpoint file... 2022-04-11 20:04:08.316764: saving checkpoint... 2022-04-11 20:04:13.718117: done, saving took 5.49 seconds 2022-04-11 20:04:13.737742: done 2022-04-11 20:04:13.810866: saving checkpoint... 2022-04-11 20:04:19.794370: done, saving took 6.05 seconds 2022-04-11 20:04:19.831259: This epoch took 134.818317 s 2022-04-11 20:04:19.832886: svmem(total=99932160000, available=71486857216, percent=28.5, used=24182255616, free=26163916800, active=48585396224, inactive=22788456448, buffers=41390080, cached=49544597504, shared=2196414464, slab=1277833216) 2022-04-11 20:04:19.833987: epoch: 96 2022-04-11 20:06:08.505653: train loss : -0.4678 2022-04-11 20:06:21.058418: validation loss: -0.5052 2022-04-11 20:06:21.065786: Average global foreground Dice: [0.7379, 0.9344, 0.9628, 0.3574] 2022-04-11 20:06:21.067712: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:06:21.728539: lr: 0.008236 2022-04-11 20:06:21.731200: saving scheduled checkpoint file... 2022-04-11 20:06:21.792707: saving checkpoint... 2022-04-11 20:06:26.228246: done, saving took 4.49 seconds 2022-04-11 20:06:26.246071: done 2022-04-11 20:06:26.331560: saving checkpoint... 2022-04-11 20:06:30.967130: done, saving took 4.72 seconds 2022-04-11 20:06:30.985806: This epoch took 131.151072 s 2022-04-11 20:06:30.987307: svmem(total=99932160000, available=71685881856, percent=28.3, used=23970283520, free=26362081280, active=48368939008, inactive=22802169856, buffers=41517056, cached=49558278144, shared=2209263616, slab=1278648320) 2022-04-11 20:06:30.988430: epoch: 97 2022-04-11 20:08:18.873165: train loss : -0.4957 2022-04-11 20:08:31.175355: validation loss: -0.4790 2022-04-11 20:08:31.178476: Average global foreground Dice: [0.7468, 0.9328, 0.9575, 0.3329] 2022-04-11 20:08:31.179358: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:08:31.954386: lr: 0.008217 2022-04-11 20:08:31.955762: saving scheduled checkpoint file... 2022-04-11 20:08:32.050080: saving checkpoint... 2022-04-11 20:08:36.614290: done, saving took 4.66 seconds 2022-04-11 20:08:36.634291: done 2022-04-11 20:08:36.739122: saving checkpoint... 2022-04-11 20:08:41.292898: done, saving took 4.66 seconds 2022-04-11 20:08:41.312366: This epoch took 130.323237 s 2022-04-11 20:08:41.313479: svmem(total=99932160000, available=71641665536, percent=28.3, used=24028078080, free=26316906496, active=48428969984, inactive=22789369856, buffers=41582592, cached=49545592832, shared=2195689472, slab=1277657088) 2022-04-11 20:08:41.314800: epoch: 98 2022-04-11 20:10:30.738041: train loss : -0.4486 2022-04-11 20:10:42.721202: validation loss: -0.4343 2022-04-11 20:10:42.723540: Average global foreground Dice: [0.7039, 0.911, 0.9376, 0.345] 2022-04-11 20:10:42.724750: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:10:43.704793: lr: 0.008199 2022-04-11 20:10:43.705981: saving scheduled checkpoint file... 2022-04-11 20:10:43.816886: saving checkpoint... 2022-04-11 20:10:48.301856: done, saving took 4.59 seconds 2022-04-11 20:10:48.324047: done 2022-04-11 20:10:48.437472: saving checkpoint... 2022-04-11 20:10:53.383820: done, saving took 5.06 seconds 2022-04-11 20:10:53.406040: This epoch took 132.090137 s 2022-04-11 20:10:53.407817: svmem(total=99932160000, available=71553040384, percent=28.4, used=24116072448, free=26227576832, active=48515874816, inactive=22790676480, buffers=41652224, cached=49546858496, shared=2196320256, slab=1278169088) 2022-04-11 20:10:53.408884: epoch: 99 2022-04-11 20:12:42.420721: train loss : -0.4516 2022-04-11 20:12:55.109425: validation loss: -0.4551 2022-04-11 20:12:55.119593: Average global foreground Dice: [0.7611, 0.9394, 0.9411, 0.3364] 2022-04-11 20:12:55.120852: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:12:55.917330: lr: 0.008181 2022-04-11 20:12:55.918569: saving scheduled checkpoint file... 2022-04-11 20:12:56.043505: saving checkpoint... 2022-04-11 20:13:00.468083: done, saving took 4.55 seconds 2022-04-11 20:13:00.490549: done 2022-04-11 20:13:00.621349: saving checkpoint... 2022-04-11 20:13:05.721027: done, saving took 5.23 seconds 2022-04-11 20:13:05.743281: This epoch took 132.333481 s 2022-04-11 20:13:05.744608: svmem(total=99932160000, available=71816601600, percent=28.1, used=23851589632, free=26513813504, active=48272150528, inactive=22744645632, buffers=41750528, cached=49525006336, shared=2196344832, slab=1279602688) 2022-04-11 20:13:05.745945: epoch: 100 2022-04-11 20:14:56.195963: train loss : -0.4645 2022-04-11 20:15:08.488445: validation loss: -0.4160 2022-04-11 20:15:08.501152: Average global foreground Dice: [0.7077, 0.9116, 0.936, 0.329] 2022-04-11 20:15:08.501919: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:15:09.096054: lr: 0.008162 2022-04-11 20:15:09.097476: saving scheduled checkpoint file... 2022-04-11 20:15:09.251769: saving checkpoint... 2022-04-11 20:15:14.430736: done, saving took 5.33 seconds 2022-04-11 20:15:14.453781: done 2022-04-11 20:15:14.455074: This epoch took 128.708363 s 2022-04-11 20:15:14.455999: svmem(total=99932160000, available=71485218816, percent=28.5, used=24175575040, free=26181513216, active=48599830528, inactive=22752804864, buffers=41832448, cached=49533239296, shared=2203676672, slab=1279549440) 2022-04-11 20:15:14.456686: epoch: 101 2022-04-11 20:17:04.998591: train loss : -0.4521 2022-04-11 20:17:17.096439: validation loss: -0.4857 2022-04-11 20:17:17.099216: Average global foreground Dice: [0.699, 0.9346, 0.9476, 0.3503] 2022-04-11 20:17:17.100582: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:17:17.915027: lr: 0.008144 2022-04-11 20:17:17.916163: saving scheduled checkpoint file... 2022-04-11 20:17:18.062271: saving checkpoint... 2022-04-11 20:17:22.633396: done, saving took 4.72 seconds 2022-04-11 20:17:22.721775: done 2022-04-11 20:17:22.821905: saving checkpoint... 2022-04-11 20:17:27.281541: done, saving took 4.56 seconds 2022-04-11 20:17:27.325760: This epoch took 132.868373 s 2022-04-11 20:17:27.328588: svmem(total=99932160000, available=71608823808, percent=28.3, used=24045182976, free=26304249856, active=48467001344, inactive=22760456192, buffers=41951232, cached=49540775936, shared=2210529280, slab=1280602112) 2022-04-11 20:17:27.329716: epoch: 102 2022-04-11 20:19:17.936508: train loss : -0.4576 2022-04-11 20:19:29.663710: validation loss: -0.4930 2022-04-11 20:19:29.674119: Average global foreground Dice: [0.7442, 0.9316, 0.9677, 0.3661] 2022-04-11 20:19:29.675188: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:19:30.691706: lr: 0.008125 2022-04-11 20:19:30.692932: saving scheduled checkpoint file... 2022-04-11 20:19:30.825495: saving checkpoint... 2022-04-11 20:19:35.713714: done, saving took 5.02 seconds 2022-04-11 20:19:35.795087: done 2022-04-11 20:19:35.916148: saving checkpoint... 2022-04-11 20:19:40.409943: done, saving took 4.61 seconds 2022-04-11 20:19:40.426676: This epoch took 133.095828 s 2022-04-11 20:19:40.427814: svmem(total=99932160000, available=71770451968, percent=28.2, used=23897620480, free=26465071104, active=48329924608, inactive=22746869760, buffers=42016768, cached=49527451648, shared=2196328448, slab=1280344064) 2022-04-11 20:19:40.428474: epoch: 103 2022-04-11 20:21:29.340058: train loss : -0.4612 2022-04-11 20:21:40.578488: validation loss: -0.5059 2022-04-11 20:21:40.592074: Average global foreground Dice: [0.7585, 0.9304, 0.9595, 0.3399] 2022-04-11 20:21:40.593158: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:21:41.491587: lr: 0.008107 2022-04-11 20:21:41.492725: saving scheduled checkpoint file... 2022-04-11 20:21:41.618787: saving checkpoint... 2022-04-11 20:21:46.149225: done, saving took 4.66 seconds 2022-04-11 20:21:46.177276: done 2022-04-11 20:21:46.308350: saving checkpoint... 2022-04-11 20:21:50.801956: done, saving took 4.62 seconds 2022-04-11 20:21:50.840865: This epoch took 130.411697 s 2022-04-11 20:21:50.842482: svmem(total=99932160000, available=71770218496, percent=28.2, used=23891636224, free=26463850496, active=48317366272, inactive=22753787904, buffers=42106880, cached=49534566400, shared=2202443776, slab=1280811008) 2022-04-11 20:21:50.843196: epoch: 104 2022-04-11 20:23:41.511911: train loss : -0.4514 2022-04-11 20:23:53.965476: validation loss: -0.4984 2022-04-11 20:23:53.967864: Average global foreground Dice: [0.7503, 0.9337, 0.9696, 0.3598] 2022-04-11 20:23:53.975402: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:23:54.576617: lr: 0.008088 2022-04-11 20:23:54.577920: saving scheduled checkpoint file... 2022-04-11 20:23:54.703010: saving checkpoint... 2022-04-11 20:23:59.326612: done, saving took 4.75 seconds 2022-04-11 20:23:59.398695: done 2022-04-11 20:23:59.527324: saving checkpoint... 2022-04-11 20:24:04.263837: done, saving took 4.86 seconds 2022-04-11 20:24:04.283570: This epoch took 133.439637 s 2022-04-11 20:24:04.285895: svmem(total=99932160000, available=71561621504, percent=28.4, used=24113909760, free=26399571968, active=48543694848, inactive=22596849664, buffers=42188800, cached=49376489472, shared=2190508032, slab=1276825600) 2022-04-11 20:24:04.288259: epoch: 105 2022-04-11 20:25:53.602247: train loss : -0.4811 2022-04-11 20:26:06.104987: validation loss: -0.5258 2022-04-11 20:26:06.107019: Average global foreground Dice: [0.7517, 0.9432, 0.9657, 0.3588] 2022-04-11 20:26:06.108363: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:26:06.995820: lr: 0.00807 2022-04-11 20:26:06.997037: saving scheduled checkpoint file... 2022-04-11 20:26:07.127224: saving checkpoint... 2022-04-11 20:26:11.556513: done, saving took 4.56 seconds 2022-04-11 20:26:11.651157: done 2022-04-11 20:26:11.778628: saving checkpoint... 2022-04-11 20:26:16.450765: done, saving took 4.80 seconds 2022-04-11 20:26:16.471760: This epoch took 132.181047 s 2022-04-11 20:26:16.473241: svmem(total=99932160000, available=71688019968, percent=28.3, used=23982043136, free=26525179904, active=48409153536, inactive=22603247616, buffers=42299392, cached=49382637568, shared=2195824640, slab=1276936192) 2022-04-11 20:26:16.474195: epoch: 106 2022-04-11 20:28:06.988083: train loss : -0.4999 2022-04-11 20:28:19.364996: validation loss: -0.4940 2022-04-11 20:28:19.367201: Average global foreground Dice: [0.74, 0.9209, 0.9685, 0.3589] 2022-04-11 20:28:19.368468: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:28:19.928209: lr: 0.008052 2022-04-11 20:28:19.929478: saving scheduled checkpoint file... 2022-04-11 20:28:20.054199: saving checkpoint... 2022-04-11 20:28:24.787191: done, saving took 4.86 seconds 2022-04-11 20:28:24.806780: done 2022-04-11 20:28:24.927441: saving checkpoint... 2022-04-11 20:28:29.668549: done, saving took 4.86 seconds 2022-04-11 20:28:29.720890: This epoch took 133.246068 s 2022-04-11 20:28:29.722485: svmem(total=99932160000, available=71579664384, percent=28.4, used=24076611584, free=26415943680, active=48504676352, inactive=22617403392, buffers=42377216, cached=49397227520, shared=2209701888, slab=1278115840) 2022-04-11 20:28:29.723379: epoch: 107 2022-04-11 20:30:20.241791: train loss : -0.4942 2022-04-11 20:30:32.810187: validation loss: -0.5018 2022-04-11 20:30:32.815027: Average global foreground Dice: [0.7499, 0.9332, 0.9612, 0.3646] 2022-04-11 20:30:32.816326: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:30:33.458142: lr: 0.008033 2022-04-11 20:30:33.459257: saving scheduled checkpoint file... 2022-04-11 20:30:33.540352: saving checkpoint... 2022-04-11 20:30:38.026315: done, saving took 4.57 seconds 2022-04-11 20:30:38.035348: done 2022-04-11 20:30:38.084193: saving checkpoint... 2022-04-11 20:30:42.623448: done, saving took 4.59 seconds 2022-04-11 20:30:42.636658: This epoch took 132.912562 s 2022-04-11 20:30:42.638050: svmem(total=99932160000, available=71360491520, percent=28.6, used=24309030912, free=26158702592, active=48746426368, inactive=22633402368, buffers=44007424, cached=49420419072, shared=2196406272, slab=1278582784) 2022-04-11 20:30:42.638866: epoch: 108 2022-04-11 20:32:30.756383: train loss : -0.4511 2022-04-11 20:32:41.968943: validation loss: -0.5204 2022-04-11 20:32:41.978425: Average global foreground Dice: [0.7882, 0.9351, 0.9631, 0.3711] 2022-04-11 20:32:41.979875: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:32:42.875715: lr: 0.008015 2022-04-11 20:32:42.877131: saving scheduled checkpoint file... 2022-04-11 20:32:42.961479: saving checkpoint... 2022-04-11 20:32:47.577471: done, saving took 4.70 seconds 2022-04-11 20:32:47.591960: done 2022-04-11 20:32:47.639570: saving checkpoint... 2022-04-11 20:32:52.175008: done, saving took 4.58 seconds 2022-04-11 20:32:52.199984: This epoch took 129.560052 s 2022-04-11 20:32:52.201468: svmem(total=99932160000, available=71602102272, percent=28.3, used=24067551232, free=26399387648, active=48502231040, inactive=22633943040, buffers=44064768, cached=49421156352, shared=2196217856, slab=1278066688) 2022-04-11 20:32:52.202617: epoch: 109 2022-04-11 20:34:40.496202: train loss : -0.4357 2022-04-11 20:34:51.716140: validation loss: -0.4914 2022-04-11 20:34:51.727563: Average global foreground Dice: [0.7148, 0.9408, 0.9587, 0.3688] 2022-04-11 20:34:51.728486: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:34:52.604900: lr: 0.007996 2022-04-11 20:34:52.606414: saving scheduled checkpoint file... 2022-04-11 20:34:52.696777: saving checkpoint... 2022-04-11 20:34:57.391325: done, saving took 4.78 seconds 2022-04-11 20:34:57.462245: done 2022-04-11 20:34:57.540239: saving checkpoint... 2022-04-11 20:35:02.185758: done, saving took 4.72 seconds 2022-04-11 20:35:02.225140: This epoch took 130.021441 s 2022-04-11 20:35:02.226631: svmem(total=99932160000, available=71500288000, percent=28.5, used=24167092224, free=26178920448, active=48617910272, inactive=22736760832, buffers=44158976, cached=49541988352, shared=2196328448, slab=1282330624) 2022-04-11 20:35:02.227632: epoch: 110 2022-04-11 20:36:50.791797: train loss : -0.4443 2022-04-11 20:37:03.194500: validation loss: -0.4974 2022-04-11 20:37:03.196758: Average global foreground Dice: [0.7541, 0.929, 0.9575, 0.3355] 2022-04-11 20:37:03.197642: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:37:03.998946: lr: 0.007978 2022-04-11 20:37:04.000338: saving scheduled checkpoint file... 2022-04-11 20:37:04.104456: saving checkpoint... 2022-04-11 20:37:08.718914: done, saving took 4.72 seconds 2022-04-11 20:37:08.737015: done 2022-04-11 20:37:08.826524: saving checkpoint... 2022-04-11 20:37:13.424412: done, saving took 4.69 seconds 2022-04-11 20:37:13.442696: This epoch took 131.214236 s 2022-04-11 20:37:13.444055: svmem(total=99932160000, available=71533813760, percent=28.4, used=24133390336, free=26211758080, active=48580460544, inactive=22737571840, buffers=44257280, cached=49542754304, shared=2196566016, slab=1282232320) 2022-04-11 20:37:13.444873: epoch: 111 2022-04-11 20:39:03.132386: train loss : -0.4888 2022-04-11 20:39:14.808929: validation loss: -0.5397 2022-04-11 20:39:14.811372: Average global foreground Dice: [0.7727, 0.9364, 0.9687, 0.3715] 2022-04-11 20:39:14.812959: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:39:15.735355: lr: 0.007959 2022-04-11 20:39:15.736679: saving scheduled checkpoint file... 2022-04-11 20:39:15.866861: saving checkpoint... 2022-04-11 20:39:20.470369: done, saving took 4.73 seconds 2022-04-11 20:39:20.491230: done 2022-04-11 20:39:20.601852: saving checkpoint... 2022-04-11 20:39:25.089212: done, saving took 4.60 seconds 2022-04-11 20:39:25.104016: This epoch took 131.658387 s 2022-04-11 20:39:25.105429: svmem(total=99932160000, available=71575076864, percent=28.4, used=24092364800, free=26252144640, active=48538165248, inactive=22738075648, buffers=44318720, cached=49543331840, shared=2196242432, slab=1281966080) 2022-04-11 20:39:25.106640: epoch: 112 2022-04-11 20:41:15.325743: train loss : -0.4982 2022-04-11 20:41:27.693238: validation loss: -0.5079 2022-04-11 20:41:27.701053: Average global foreground Dice: [0.758, 0.934, 0.9625, 0.3343] 2022-04-11 20:41:27.701878: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:41:28.373895: lr: 0.007941 2022-04-11 20:41:28.375158: saving scheduled checkpoint file... 2022-04-11 20:41:28.488580: saving checkpoint... 2022-04-11 20:41:32.972994: done, saving took 4.60 seconds 2022-04-11 20:41:33.009865: done 2022-04-11 20:41:33.135340: saving checkpoint... 2022-04-11 20:41:38.171882: done, saving took 5.16 seconds 2022-04-11 20:41:38.193897: This epoch took 133.086399 s 2022-04-11 20:41:38.195734: svmem(total=99932160000, available=71432192000, percent=28.5, used=24249028608, free=26108280832, active=48703283200, inactive=22725001216, buffers=44425216, cached=49530425344, shared=2182365184, slab=1282584576) 2022-04-11 20:41:38.197101: epoch: 113 2022-04-11 20:43:28.383124: train loss : -0.5033 2022-04-11 20:43:40.667806: validation loss: -0.5462 2022-04-11 20:43:40.674002: Average global foreground Dice: [0.7885, 0.9461, 0.9726, 0.3925] 2022-04-11 20:43:40.675089: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:43:41.498100: lr: 0.007922 2022-04-11 20:43:41.499372: saving scheduled checkpoint file... 2022-04-11 20:43:41.585967: saving checkpoint... 2022-04-11 20:43:46.025632: done, saving took 4.53 seconds 2022-04-11 20:43:46.035135: done 2022-04-11 20:43:46.086790: saving checkpoint... 2022-04-11 20:43:50.804548: done, saving took 4.77 seconds 2022-04-11 20:43:50.818855: This epoch took 132.621022 s 2022-04-11 20:43:50.820559: svmem(total=99932160000, available=71531352064, percent=28.4, used=24128094208, free=26206629888, active=48572653568, inactive=22748168192, buffers=44490752, cached=49552945152, shared=2204282880, slab=1282281472) 2022-04-11 20:43:50.821542: epoch: 114 2022-04-11 20:45:39.150093: train loss : -0.4967 2022-04-11 20:45:50.800540: validation loss: -0.5080 2022-04-11 20:45:50.808002: Average global foreground Dice: [0.7736, 0.937, 0.9609, 0.3472] 2022-04-11 20:45:50.808852: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:45:51.685588: lr: 0.007904 2022-04-11 20:45:51.687043: saving scheduled checkpoint file... 2022-04-11 20:45:51.739559: saving checkpoint... 2022-04-11 20:45:56.643764: done, saving took 4.96 seconds 2022-04-11 20:45:56.716421: done 2022-04-11 20:45:56.764868: saving checkpoint... 2022-04-11 20:46:01.296583: done, saving took 4.58 seconds 2022-04-11 20:46:01.310400: This epoch took 130.488071 s 2022-04-11 20:46:01.311899: svmem(total=99932160000, available=71363710976, percent=28.6, used=24297353216, free=26038124544, active=48747757568, inactive=22746796032, buffers=44589056, cached=49552093184, shared=2202599424, slab=1283997696) 2022-04-11 20:46:01.312666: epoch: 115 2022-04-11 20:47:50.298677: train loss : -0.4961 2022-04-11 20:48:02.200628: validation loss: -0.5090 2022-04-11 20:48:02.202569: Average global foreground Dice: [0.7406, 0.9262, 0.9635, 0.3661] 2022-04-11 20:48:02.203327: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:48:03.084578: lr: 0.007885 2022-04-11 20:48:03.086150: saving scheduled checkpoint file... 2022-04-11 20:48:03.167540: saving checkpoint... 2022-04-11 20:48:07.862492: done, saving took 4.78 seconds 2022-04-11 20:48:07.876712: done 2022-04-11 20:48:07.925076: saving checkpoint... 2022-04-11 20:48:12.456106: done, saving took 4.58 seconds 2022-04-11 20:48:12.538113: This epoch took 131.224723 s 2022-04-11 20:48:12.539563: svmem(total=99932160000, available=71400656896, percent=28.6, used=24266498048, free=26074300416, active=48712679424, inactive=22741311488, buffers=44662784, cached=49546698752, shared=2196516864, slab=1283354624) 2022-04-11 20:48:12.540326: epoch: 116 2022-04-11 20:50:01.739797: train loss : -0.4472 2022-04-11 20:50:13.034916: validation loss: -0.4780 2022-04-11 20:50:13.044065: Average global foreground Dice: [0.7495, 0.9272, 0.9565, 0.36] 2022-04-11 20:50:13.045420: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:50:13.940781: lr: 0.007867 2022-04-11 20:50:13.942320: saving scheduled checkpoint file... 2022-04-11 20:50:14.032833: saving checkpoint... 2022-04-11 20:50:18.484193: done, saving took 4.54 seconds 2022-04-11 20:50:18.503971: done 2022-04-11 20:50:18.589034: saving checkpoint... 2022-04-11 20:50:23.017781: done, saving took 4.51 seconds 2022-04-11 20:50:23.035271: This epoch took 130.494208 s 2022-04-11 20:50:23.036261: svmem(total=99932160000, available=71406559232, percent=28.5, used=24260956160, free=26079232000, active=48701444096, inactive=22741966848, buffers=44736512, cached=49547235328, shared=2196201472, slab=1285165056) 2022-04-11 20:50:23.037033: epoch: 117 2022-04-11 20:52:11.634071: train loss : -0.4991 2022-04-11 20:52:23.155825: validation loss: -0.5351 2022-04-11 20:52:23.157988: Average global foreground Dice: [0.7823, 0.9405, 0.971, 0.3487] 2022-04-11 20:52:23.159238: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:52:23.972203: lr: 0.007848 2022-04-11 20:52:23.973593: saving scheduled checkpoint file... 2022-04-11 20:52:24.076973: saving checkpoint... 2022-04-11 20:52:28.670659: done, saving took 4.70 seconds 2022-04-11 20:52:28.731766: done 2022-04-11 20:52:28.822244: saving checkpoint... 2022-04-11 20:52:33.379268: done, saving took 4.65 seconds 2022-04-11 20:52:33.398202: This epoch took 130.360418 s 2022-04-11 20:52:33.399790: svmem(total=99932160000, available=71527739392, percent=28.4, used=24139563008, free=26199658496, active=48589959168, inactive=22742925312, buffers=44814336, cached=49548124160, shared=2196336640, slab=1282805760) 2022-04-11 20:52:33.400899: epoch: 118 2022-04-11 20:54:22.709612: train loss : -0.5313 2022-04-11 20:54:34.509798: validation loss: -0.5371 2022-04-11 20:54:34.954707: Average global foreground Dice: [0.7793, 0.9431, 0.9711, 0.3525] 2022-04-11 20:54:34.956528: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:54:35.849663: lr: 0.00783 2022-04-11 20:54:35.851203: saving scheduled checkpoint file... 2022-04-11 20:54:35.998555: saving checkpoint... 2022-04-11 20:54:40.476017: done, saving took 4.62 seconds 2022-04-11 20:54:40.649938: done 2022-04-11 20:54:40.748663: saving checkpoint... 2022-04-11 20:54:45.312202: done, saving took 4.66 seconds 2022-04-11 20:54:45.331309: This epoch took 131.929562 s 2022-04-11 20:54:45.332896: svmem(total=99932160000, available=71737749504, percent=28.2, used=24021905408, free=26408759296, active=48471277568, inactive=22651420672, buffers=44908544, cached=49456586752, shared=2104012800, slab=1282383872) 2022-04-11 20:54:45.334042: epoch: 119 2022-04-11 20:56:35.123924: train loss : -0.4977 2022-04-11 20:56:47.585820: validation loss: -0.4929 2022-04-11 20:56:47.587729: Average global foreground Dice: [0.7529, 0.9376, 0.9517, 0.3569] 2022-04-11 20:56:47.588460: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:56:48.357608: lr: 0.007811 2022-04-11 20:56:48.359051: saving scheduled checkpoint file... 2022-04-11 20:56:48.434129: saving checkpoint... 2022-04-11 20:56:53.047014: done, saving took 4.69 seconds 2022-04-11 20:56:53.060870: done 2022-04-11 20:56:53.061818: This epoch took 127.726782 s 2022-04-11 20:56:53.062921: svmem(total=99932160000, available=71808618496, percent=28.1, used=23926857728, free=26478735360, active=48378707968, inactive=22676705280, buffers=44998656, cached=49481568256, shared=2128084992, slab=1282727936) 2022-04-11 20:56:53.063952: epoch: 120 2022-04-11 20:58:41.449838: train loss : -0.4643 2022-04-11 20:58:52.959724: validation loss: -0.4841 2022-04-11 20:58:52.961855: Average global foreground Dice: [0.7983, 0.9327, 0.9483, 0.3349] 2022-04-11 20:58:52.962939: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 20:58:53.720813: lr: 0.007793 2022-04-11 20:58:53.722180: saving scheduled checkpoint file... 2022-04-11 20:58:53.793349: saving checkpoint... 2022-04-11 20:58:58.588391: done, saving took 4.87 seconds 2022-04-11 20:58:58.597467: done 2022-04-11 20:58:58.645837: saving checkpoint... 2022-04-11 20:59:05.110307: done, saving took 6.51 seconds 2022-04-11 20:59:05.465977: This epoch took 132.400465 s 2022-04-11 20:59:05.618596: svmem(total=99932160000, available=71633309696, percent=28.3, used=24046157824, free=26302525440, active=48524152832, inactive=22685532160, buffers=45088768, cached=49538387968, shared=2182168576, slab=1300893696) 2022-04-11 20:59:05.743124: epoch: 121 2022-04-11 21:00:53.324363: train loss : -0.4797 2022-04-11 21:01:05.565430: validation loss: -0.4981 2022-04-11 21:01:05.571661: Average global foreground Dice: [0.7359, 0.9366, 0.9545, 0.3556] 2022-04-11 21:01:05.573367: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:01:06.499115: lr: 0.007774 2022-04-11 21:01:06.504952: saving scheduled checkpoint file... 2022-04-11 21:01:06.557934: saving checkpoint... 2022-04-11 21:01:12.273233: done, saving took 5.77 seconds 2022-04-11 21:01:12.596451: done 2022-04-11 21:01:12.739293: This epoch took 126.853364 s 2022-04-11 21:01:12.882019: svmem(total=99932160000, available=71513214976, percent=28.4, used=24166113280, free=26181672960, active=48649388032, inactive=22686257152, buffers=45178880, cached=49539194880, shared=2182254592, slab=1298399232) 2022-04-11 21:01:13.024831: epoch: 122 2022-04-11 21:03:01.334276: train loss : -0.4066 2022-04-11 21:03:12.879042: validation loss: -0.3258 2022-04-11 21:03:12.881087: Average global foreground Dice: [0.6738, 0.8992, 0.8993, 0.2978] 2022-04-11 21:03:12.881941: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:03:13.471447: lr: 0.007756 2022-04-11 21:03:13.472469: saving scheduled checkpoint file... 2022-04-11 21:03:13.532572: saving checkpoint... 2022-04-11 21:03:18.962527: done, saving took 5.49 seconds 2022-04-11 21:03:19.337911: done 2022-04-11 21:03:19.531558: This epoch took 126.373337 s 2022-04-11 21:03:19.642967: svmem(total=99932160000, available=71420674048, percent=28.5, used=24258781184, free=26087956480, active=48739545088, inactive=22687031296, buffers=45244416, cached=49540177920, shared=2182082560, slab=1300496384) 2022-04-11 21:03:19.767153: epoch: 123 2022-04-11 21:05:08.164434: train loss : -0.3202 2022-04-11 21:05:20.498875: validation loss: -0.3237 2022-04-11 21:05:20.508986: Average global foreground Dice: [0.7696, 0.8744, 0.8832, 0.313] 2022-04-11 21:05:20.511767: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:05:21.200130: lr: 0.007737 2022-04-11 21:05:21.203750: saving scheduled checkpoint file... 2022-04-11 21:05:21.265701: saving checkpoint... 2022-04-11 21:05:26.318138: done, saving took 5.11 seconds 2022-04-11 21:05:26.693615: done 2022-04-11 21:05:26.836792: This epoch took 126.926397 s 2022-04-11 21:05:26.962069: svmem(total=99932160000, available=71547920384, percent=28.4, used=24131371008, free=26214465536, active=48612122624, inactive=22687875072, buffers=45359104, cached=49540964352, shared=2182254592, slab=1300107264) 2022-04-11 21:05:27.095620: epoch: 124 2022-04-11 21:07:17.209111: train loss : -0.3462 2022-04-11 21:07:29.225857: validation loss: -0.4797 2022-04-11 21:07:29.227763: Average global foreground Dice: [0.7169, 0.9241, 0.9608, 0.3652] 2022-04-11 21:07:29.230015: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:07:30.092305: lr: 0.007719 2022-04-11 21:07:30.095172: saving scheduled checkpoint file... 2022-04-11 21:07:30.214952: saving checkpoint... 2022-04-11 21:07:35.748187: done, saving took 5.65 seconds 2022-04-11 21:07:36.061402: done 2022-04-11 21:07:36.186584: This epoch took 128.974563 s 2022-04-11 21:07:36.320929: svmem(total=99932160000, available=71470723072, percent=28.5, used=24208437248, free=26136297472, active=48686940160, inactive=22688686080, buffers=45428736, cached=49541996544, shared=2182340608, slab=1300307968) 2022-04-11 21:07:36.495480: epoch: 125 2022-04-11 21:09:24.211351: train loss : -0.4727 2022-04-11 21:09:34.992473: validation loss: -0.5230 2022-04-11 21:09:34.995878: Average global foreground Dice: [0.7645, 0.9364, 0.9629, 0.3882] 2022-04-11 21:09:34.997800: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:09:35.934085: lr: 0.0077 2022-04-11 21:09:35.937507: saving scheduled checkpoint file... 2022-04-11 21:09:36.016027: saving checkpoint... 2022-04-11 21:09:40.461227: done, saving took 4.52 seconds 2022-04-11 21:09:40.476542: done 2022-04-11 21:09:40.479877: This epoch took 123.850332 s 2022-04-11 21:09:40.482517: svmem(total=99932160000, available=71799394304, percent=28.2, used=23925542912, free=26464403456, active=48423735296, inactive=22643544064, buffers=45473792, cached=49496739840, shared=2136502272, slab=1287102464) 2022-04-11 21:09:40.484801: epoch: 126 2022-04-11 21:11:28.689002: train loss : -0.5046 2022-04-11 21:11:40.444675: validation loss: -0.5206 2022-04-11 21:11:40.446529: Average global foreground Dice: [0.795, 0.9346, 0.9571, 0.3707] 2022-04-11 21:11:40.447247: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:11:41.428104: lr: 0.007682 2022-04-11 21:11:41.429506: saving scheduled checkpoint file... 2022-04-11 21:11:41.514819: saving checkpoint... 2022-04-11 21:11:46.102963: done, saving took 4.67 seconds 2022-04-11 21:11:46.115965: done 2022-04-11 21:11:46.116821: This epoch took 125.629576 s 2022-04-11 21:11:46.117804: svmem(total=99932160000, available=71435665408, percent=28.5, used=24237342720, free=26400657408, active=48728834048, inactive=22395109376, buffers=45547520, cached=49248612352, shared=2188201984, slab=1290838016) 2022-04-11 21:11:46.118653: epoch: 127 2022-04-11 21:13:34.846267: train loss : -0.4902 2022-04-11 21:13:46.429121: validation loss: -0.4642 2022-04-11 21:13:46.431244: Average global foreground Dice: [0.7521, 0.916, 0.9471, 0.3548] 2022-04-11 21:13:46.432477: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:13:47.317438: lr: 0.007663 2022-04-11 21:13:47.318952: saving scheduled checkpoint file... 2022-04-11 21:13:47.384932: saving checkpoint... 2022-04-11 21:13:51.895094: done, saving took 4.58 seconds 2022-04-11 21:13:51.907834: done 2022-04-11 21:13:51.908853: This epoch took 125.789570 s 2022-04-11 21:13:51.909784: svmem(total=99932160000, available=71657091072, percent=28.3, used=24020467712, free=26321739776, active=48728432640, inactive=22480097280, buffers=45613056, cached=49544339456, shared=2183962624, slab=1286524928) 2022-04-11 21:13:51.910636: epoch: 128 2022-04-11 21:15:39.769767: train loss : -0.4706 2022-04-11 21:15:51.606474: validation loss: -0.4961 2022-04-11 21:15:51.608751: Average global foreground Dice: [0.7325, 0.9382, 0.9666, 0.3726] 2022-04-11 21:15:51.610186: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:15:52.231090: lr: 0.007645 2022-04-11 21:15:52.232182: saving scheduled checkpoint file... 2022-04-11 21:15:52.302459: saving checkpoint... 2022-04-11 21:15:57.027533: done, saving took 4.79 seconds 2022-04-11 21:15:57.086274: done 2022-04-11 21:15:57.088881: This epoch took 125.176955 s 2022-04-11 21:15:57.091747: svmem(total=99932160000, available=71663448064, percent=28.3, used=24050442240, free=26327949312, active=48771002368, inactive=22428123136, buffers=45723648, cached=49508044800, shared=2147549184, slab=1286549504) 2022-04-11 21:15:57.093914: epoch: 129 2022-04-11 21:17:45.219061: train loss : -0.4518 2022-04-11 21:17:56.241577: validation loss: -0.4382 2022-04-11 21:17:56.244029: Average global foreground Dice: [0.7607, 0.9215, 0.9402, 0.3294] 2022-04-11 21:17:56.245098: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:17:57.011192: lr: 0.007626 2022-04-11 21:17:57.012546: saving scheduled checkpoint file... 2022-04-11 21:17:57.067991: saving checkpoint... 2022-04-11 21:18:01.509411: done, saving took 4.50 seconds 2022-04-11 21:18:01.527175: done 2022-04-11 21:18:01.528328: This epoch took 124.431842 s 2022-04-11 21:18:01.529233: svmem(total=99932160000, available=71744299008, percent=28.2, used=23944065024, free=26410844160, active=48726151168, inactive=22393004032, buffers=45805568, cached=49531445248, shared=2172907520, slab=1287000064) 2022-04-11 21:18:01.529924: epoch: 130 2022-04-11 21:19:50.139485: train loss : -0.3585 2022-04-11 21:20:01.642350: validation loss: -0.4440 2022-04-11 21:20:01.655993: Average global foreground Dice: [0.7154, 0.9298, 0.9434, 0.3513] 2022-04-11 21:20:01.656873: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:20:02.615697: lr: 0.007608 2022-04-11 21:20:02.617102: saving scheduled checkpoint file... 2022-04-11 21:20:02.739905: saving checkpoint... 2022-04-11 21:20:07.437067: done, saving took 4.82 seconds 2022-04-11 21:20:07.459151: done 2022-04-11 21:20:07.460590: This epoch took 125.929845 s 2022-04-11 21:20:07.462311: svmem(total=99932160000, available=71605755904, percent=28.3, used=24059695104, free=26271686656, active=48919588864, inactive=22335176704, buffers=45879296, cached=49554898944, shared=2195857408, slab=1287933952) 2022-04-11 21:20:07.464198: epoch: 131 2022-04-11 21:21:56.191498: train loss : -0.4011 2022-04-11 21:22:08.369645: validation loss: -0.4343 2022-04-11 21:22:08.375894: Average global foreground Dice: [0.7668, 0.9053, 0.9073, 0.3338] 2022-04-11 21:22:08.380619: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:22:09.291994: lr: 0.007589 2022-04-11 21:22:09.293335: saving scheduled checkpoint file... 2022-04-11 21:22:09.401995: saving checkpoint... 2022-04-11 21:22:13.903756: done, saving took 4.61 seconds 2022-04-11 21:22:13.923104: done 2022-04-11 21:22:13.924016: This epoch took 126.458810 s 2022-04-11 21:22:13.925065: svmem(total=99932160000, available=71701299200, percent=28.3, used=23963447296, free=26458255360, active=48917831680, inactive=22155694080, buffers=45993984, cached=49464463360, shared=2196234240, slab=1286799360) 2022-04-11 21:22:13.925782: epoch: 132 2022-04-11 21:24:03.360335: train loss : -0.3872 2022-04-11 21:24:15.694845: validation loss: -0.4362 2022-04-11 21:24:15.696737: Average global foreground Dice: [0.728, 0.9224, 0.9435, 0.3525] 2022-04-11 21:24:15.698012: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:24:16.596948: lr: 0.007571 2022-04-11 21:24:16.598384: saving scheduled checkpoint file... 2022-04-11 21:24:16.709918: saving checkpoint... 2022-04-11 21:24:21.109638: done, saving took 4.51 seconds 2022-04-11 21:24:21.153953: done 2022-04-11 21:24:21.155039: This epoch took 127.228594 s 2022-04-11 21:24:21.156112: svmem(total=99932160000, available=71598727168, percent=28.4, used=24063791104, free=26354692096, active=49026830336, inactive=22148329472, buffers=46051328, cached=49467625472, shared=2198605824, slab=1287274496) 2022-04-11 21:24:21.156838: epoch: 133 2022-04-11 21:26:11.643248: train loss : -0.3804 2022-04-11 21:26:23.905130: validation loss: -0.4570 2022-04-11 21:26:23.907252: Average global foreground Dice: [0.7113, 0.9338, 0.9597, 0.3288] 2022-04-11 21:26:23.908004: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:26:24.747441: lr: 0.007552 2022-04-11 21:26:24.750331: saving scheduled checkpoint file... 2022-04-11 21:26:24.882780: saving checkpoint... 2022-04-11 21:26:29.474678: done, saving took 4.72 seconds 2022-04-11 21:26:29.523367: done 2022-04-11 21:26:29.530738: This epoch took 128.373222 s 2022-04-11 21:26:29.532128: svmem(total=99932160000, available=71791185920, percent=28.2, used=23887474688, free=26546352128, active=48849268736, inactive=22131539968, buffers=46166016, cached=49452167168, shared=2182381568, slab=1288372224) 2022-04-11 21:26:29.533150: epoch: 134 2022-04-11 21:28:18.724395: train loss : -0.4077 2022-04-11 21:28:30.152279: validation loss: -0.4884 2022-04-11 21:28:30.154431: Average global foreground Dice: [0.7636, 0.9345, 0.9656, 0.3432] 2022-04-11 21:28:30.155305: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:28:31.078814: lr: 0.007533 2022-04-11 21:28:31.080456: saving scheduled checkpoint file... 2022-04-11 21:28:31.232168: saving checkpoint... 2022-04-11 21:28:35.713189: done, saving took 4.63 seconds 2022-04-11 21:28:35.734410: done 2022-04-11 21:28:35.735725: This epoch took 126.201524 s 2022-04-11 21:28:35.736660: svmem(total=99932160000, available=71606644736, percent=28.3, used=24058331136, free=26360934400, active=49021472768, inactive=22145998848, buffers=46235648, cached=49466658816, shared=2196045824, slab=1289572352) 2022-04-11 21:28:35.737807: epoch: 135 2022-04-11 21:30:24.845080: train loss : -0.4912 2022-04-11 21:30:36.705928: validation loss: -0.5219 2022-04-11 21:30:36.753349: Average global foreground Dice: [0.8028, 0.9339, 0.9677, 0.3815] 2022-04-11 21:30:36.757735: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:30:37.672909: lr: 0.007515 2022-04-11 21:30:37.674735: saving scheduled checkpoint file... 2022-04-11 21:30:37.815853: saving checkpoint... 2022-04-11 21:30:42.390109: done, saving took 4.71 seconds 2022-04-11 21:30:42.418473: done 2022-04-11 21:30:42.421866: This epoch took 126.683041 s 2022-04-11 21:30:42.424982: svmem(total=99932160000, available=71867265024, percent=28.1, used=23808831488, free=26620678144, active=48769126400, inactive=22135549952, buffers=46292992, cached=49456357376, shared=2184921088, slab=1288192000) 2022-04-11 21:30:42.426925: epoch: 136 2022-04-11 21:32:33.061990: train loss : -0.4999 2022-04-11 21:32:46.187416: validation loss: -0.5254 2022-04-11 21:32:46.189274: Average global foreground Dice: [0.7212, 0.941, 0.9652, 0.3838] 2022-04-11 21:32:46.195471: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:32:46.790799: lr: 0.007496 2022-04-11 21:32:46.792028: saving scheduled checkpoint file... 2022-04-11 21:32:46.933290: saving checkpoint... 2022-04-11 21:32:51.700369: done, saving took 4.91 seconds 2022-04-11 21:32:51.718014: done 2022-04-11 21:32:51.719445: This epoch took 129.289884 s 2022-04-11 21:32:51.720677: svmem(total=99932160000, available=71591264256, percent=28.4, used=24073236480, free=26344083456, active=49034936320, inactive=22147928064, buffers=46354432, cached=49468485632, shared=2196312064, slab=1287155712) 2022-04-11 21:32:51.721407: epoch: 137 2022-04-11 21:34:41.272393: train loss : -0.5228 2022-04-11 21:34:53.948812: validation loss: -0.5152 2022-04-11 21:34:53.960463: Average global foreground Dice: [0.7841, 0.9447, 0.9565, 0.3604] 2022-04-11 21:34:53.961439: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:34:54.697314: lr: 0.007478 2022-04-11 21:34:54.698653: saving scheduled checkpoint file... 2022-04-11 21:34:54.789046: saving checkpoint... 2022-04-11 21:34:59.150285: done, saving took 4.45 seconds 2022-04-11 21:34:59.163858: done 2022-04-11 21:34:59.165116: This epoch took 127.443010 s 2022-04-11 21:34:59.166034: svmem(total=99932160000, available=71278292992, percent=28.7, used=24385925120, free=26030321664, active=49350832128, inactive=22148124672, buffers=46432256, cached=49469480960, shared=2196705280, slab=1289248768) 2022-04-11 21:34:59.166774: epoch: 138 2022-04-11 21:36:47.858060: train loss : -0.4592 2022-04-11 21:36:59.076761: validation loss: -0.3906 2022-04-11 21:36:59.079260: Average global foreground Dice: [0.7439, 0.8981, 0.9001, 0.3217] 2022-04-11 21:36:59.082628: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:36:59.976686: lr: 0.007459 2022-04-11 21:36:59.977943: saving scheduled checkpoint file... 2022-04-11 21:37:00.047591: saving checkpoint... 2022-04-11 21:37:04.560623: done, saving took 4.58 seconds 2022-04-11 21:37:04.573920: done 2022-04-11 21:37:04.574810: This epoch took 125.407273 s 2022-04-11 21:37:04.575795: svmem(total=99932160000, available=71300005888, percent=28.7, used=24364826624, free=26051239936, active=49326649344, inactive=22149021696, buffers=46526464, cached=49469566976, shared=2195984384, slab=1288228864) 2022-04-11 21:37:04.577011: epoch: 139 2022-04-11 21:38:52.630307: train loss : -0.4324 2022-04-11 21:39:04.193380: validation loss: -0.4536 2022-04-11 21:39:04.199696: Average global foreground Dice: [0.7198, 0.9356, 0.9293, 0.3433] 2022-04-11 21:39:04.202009: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:39:05.102186: lr: 0.00744 2022-04-11 21:39:05.103860: saving scheduled checkpoint file... 2022-04-11 21:39:05.180574: saving checkpoint... 2022-04-11 21:39:09.790612: done, saving took 4.69 seconds 2022-04-11 21:39:09.837700: done 2022-04-11 21:39:09.839073: This epoch took 125.261352 s 2022-04-11 21:39:09.840387: svmem(total=99932160000, available=71581007872, percent=28.4, used=24072994816, free=26352574464, active=48985976832, inactive=22185201664, buffers=46592000, cached=49459998720, shared=2206871552, slab=1286864896) 2022-04-11 21:39:09.841597: epoch: 140 2022-04-11 21:40:58.304897: train loss : -0.4310 2022-04-11 21:41:09.812293: validation loss: -0.5194 2022-04-11 21:41:09.826372: Average global foreground Dice: [0.7636, 0.9428, 0.9638, 0.3656] 2022-04-11 21:41:09.827327: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:41:10.638604: lr: 0.007422 2022-04-11 21:41:10.640195: saving scheduled checkpoint file... 2022-04-11 21:41:10.699572: saving checkpoint... 2022-04-11 21:41:15.350855: done, saving took 4.71 seconds 2022-04-11 21:41:15.365423: done 2022-04-11 21:41:15.366584: This epoch took 125.524170 s 2022-04-11 21:41:15.368814: svmem(total=99932160000, available=71322103808, percent=28.6, used=24342622208, free=26092851200, active=49347747840, inactive=22084669440, buffers=46665728, cached=49450020864, shared=2196123648, slab=1288802304) 2022-04-11 21:41:15.370193: epoch: 141 2022-04-11 21:43:03.861988: train loss : -0.4578 2022-04-11 21:43:14.880005: validation loss: -0.4517 2022-04-11 21:43:14.888685: Average global foreground Dice: [0.7152, 0.92, 0.9537, 0.3445] 2022-04-11 21:43:14.890182: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:43:15.760156: lr: 0.007403 2022-04-11 21:43:15.761518: saving scheduled checkpoint file... 2022-04-11 21:43:15.823246: saving checkpoint... 2022-04-11 21:43:20.485269: done, saving took 4.72 seconds 2022-04-11 21:43:20.501761: done 2022-04-11 21:43:20.503344: This epoch took 125.131948 s 2022-04-11 21:43:20.504708: svmem(total=99932160000, available=71431192576, percent=28.5, used=24233299968, free=26201001984, active=49242161152, inactive=22083248128, buffers=46743552, cached=49451114496, shared=2196328448, slab=1288019968) 2022-04-11 21:43:20.505703: epoch: 142 2022-04-11 21:45:09.925915: train loss : -0.4917 2022-04-11 21:45:21.610883: validation loss: -0.5209 2022-04-11 21:45:21.613576: Average global foreground Dice: [0.7977, 0.9277, 0.958, 0.3619] 2022-04-11 21:45:21.614674: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:45:22.491059: lr: 0.007385 2022-04-11 21:45:22.492471: saving scheduled checkpoint file... 2022-04-11 21:45:22.601190: saving checkpoint... 2022-04-11 21:45:27.214319: done, saving took 4.72 seconds 2022-04-11 21:45:27.285851: done 2022-04-11 21:45:27.287208: This epoch took 126.780018 s 2022-04-11 21:45:27.288538: svmem(total=99932160000, available=71483101184, percent=28.5, used=24175341568, free=26231885824, active=49224458240, inactive=22068748288, buffers=46866432, cached=49478066176, shared=2201653248, slab=1289646080) 2022-04-11 21:45:27.289700: epoch: 143 2022-04-11 21:47:17.132447: train loss : -0.5127 2022-04-11 21:47:29.905917: validation loss: -0.5123 2022-04-11 21:47:29.931383: Average global foreground Dice: [0.768, 0.9397, 0.9614, 0.3318] 2022-04-11 21:47:29.940257: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:47:30.872971: lr: 0.007366 2022-04-11 21:47:30.874056: saving scheduled checkpoint file... 2022-04-11 21:47:30.959352: saving checkpoint... 2022-04-11 21:47:35.533337: done, saving took 4.66 seconds 2022-04-11 21:47:35.561791: done 2022-04-11 21:47:35.563263: This epoch took 128.272845 s 2022-04-11 21:47:35.564812: svmem(total=99932160000, available=71307980800, percent=28.6, used=24355504128, free=26055831552, active=49421004800, inactive=22047657984, buffers=46931968, cached=49473892352, shared=2196500480, slab=1290350592) 2022-04-11 21:47:35.565619: epoch: 144 2022-04-11 21:49:25.957982: train loss : -0.5119 2022-04-11 21:49:37.534587: validation loss: -0.5202 2022-04-11 21:49:37.536766: Average global foreground Dice: [0.7655, 0.9409, 0.9566, 0.3834] 2022-04-11 21:49:37.538025: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:49:38.288177: lr: 0.007347 2022-04-11 21:49:38.289892: saving scheduled checkpoint file... 2022-04-11 21:49:38.404146: saving checkpoint... 2022-04-11 21:49:43.158175: done, saving took 4.87 seconds 2022-04-11 21:49:43.172491: done 2022-04-11 21:49:43.173556: This epoch took 127.607298 s 2022-04-11 21:49:43.174498: svmem(total=99932160000, available=71515000832, percent=28.4, used=24148905984, free=26262085632, active=49213927424, inactive=22048169984, buffers=47009792, cached=49474158592, shared=2196131840, slab=1289314304) 2022-04-11 21:49:43.175188: epoch: 145 2022-04-11 21:51:31.825324: train loss : -0.4016 2022-04-11 21:51:43.199258: validation loss: -0.4373 2022-04-11 21:51:43.223073: Average global foreground Dice: [0.7612, 0.9203, 0.9349, 0.3385] 2022-04-11 21:51:43.228732: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:51:44.143753: lr: 0.007329 2022-04-11 21:51:44.145442: saving scheduled checkpoint file... 2022-04-11 21:51:44.256852: saving checkpoint... 2022-04-11 21:51:49.059534: done, saving took 4.91 seconds 2022-04-11 21:51:49.075052: done 2022-04-11 21:51:49.075929: This epoch took 125.899887 s 2022-04-11 21:51:49.077118: svmem(total=99932160000, available=71626911744, percent=28.3, used=24036888576, free=26372997120, active=49103110144, inactive=22048972800, buffers=47083520, cached=49475190784, shared=2196156416, slab=1290285056) 2022-04-11 21:51:49.078300: epoch: 146 2022-04-11 21:53:37.549516: train loss : -0.5002 2022-04-11 21:53:49.455386: validation loss: -0.4893 2022-04-11 21:53:49.469780: Average global foreground Dice: [0.7544, 0.9311, 0.949, 0.3561] 2022-04-11 21:53:49.471444: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:53:50.416968: lr: 0.00731 2022-04-11 21:53:50.418371: saving scheduled checkpoint file... 2022-04-11 21:53:50.560769: saving checkpoint... 2022-04-11 21:53:55.274596: done, saving took 4.86 seconds 2022-04-11 21:53:55.294743: done 2022-04-11 21:53:55.296142: This epoch took 126.216898 s 2022-04-11 21:53:55.297086: svmem(total=99932160000, available=71530450944, percent=28.4, used=24130428928, free=26275721216, active=49199464448, inactive=22049656832, buffers=47132672, cached=49478877184, shared=2199044096, slab=1289265152) 2022-04-11 21:53:55.297751: epoch: 147 2022-04-11 21:55:44.316264: train loss : -0.5068 2022-04-11 21:55:55.455196: validation loss: -0.5015 2022-04-11 21:55:55.471030: Average global foreground Dice: [0.7585, 0.9342, 0.9604, 0.3283] 2022-04-11 21:55:55.474419: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:55:56.273436: lr: 0.007291 2022-04-11 21:55:56.275737: saving scheduled checkpoint file... 2022-04-11 21:55:56.402932: saving checkpoint... 2022-04-11 21:56:01.030525: done, saving took 4.75 seconds 2022-04-11 21:56:01.053512: done 2022-04-11 21:56:01.054872: This epoch took 125.756165 s 2022-04-11 21:56:01.056217: svmem(total=99932160000, available=71405301760, percent=28.5, used=24257945600, free=26149871616, active=49324015616, inactive=22050320384, buffers=47247360, cached=49477095424, shared=2196652032, slab=1290973184) 2022-04-11 21:56:01.057114: epoch: 148 2022-04-11 21:57:51.665854: train loss : -0.5079 2022-04-11 21:58:04.761430: validation loss: -0.5540 2022-04-11 21:58:04.767724: Average global foreground Dice: [0.764, 0.9433, 0.9695, 0.3957] 2022-04-11 21:58:04.769230: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 21:58:05.635436: lr: 0.007273 2022-04-11 21:58:05.636759: saving scheduled checkpoint file... 2022-04-11 21:58:05.759344: saving checkpoint... 2022-04-11 21:58:10.344963: done, saving took 4.71 seconds 2022-04-11 21:58:10.367360: done 2022-04-11 21:58:10.368556: This epoch took 129.310660 s 2022-04-11 21:58:10.369431: svmem(total=99932160000, available=71372173312, percent=28.6, used=24294756352, free=26115899392, active=49357230080, inactive=22047715328, buffers=47312896, cached=49474191360, shared=2192859136, slab=1291214848) 2022-04-11 21:58:10.370112: epoch: 149 2022-04-11 21:59:59.851480: train loss : -0.5435 2022-04-11 22:00:12.048220: validation loss: -0.5897 2022-04-11 22:00:12.055022: Average global foreground Dice: [0.8308, 0.9424, 0.9755, 0.3936] 2022-04-11 22:00:12.055786: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:00:13.004845: lr: 0.007254 2022-04-11 22:00:13.006007: saving scheduled checkpoint file... 2022-04-11 22:00:13.097589: saving checkpoint... 2022-04-11 22:00:17.675284: done, saving took 4.67 seconds 2022-04-11 22:00:17.689868: done 2022-04-11 22:00:17.737983: saving checkpoint... 2022-04-11 22:00:25.383244: done, saving took 7.69 seconds 2022-04-11 22:00:25.392506: This epoch took 135.021562 s 2022-04-11 22:00:25.393721: svmem(total=99932160000, available=71423037440, percent=28.5, used=24240517120, free=26165858304, active=49300238336, inactive=22052270080, buffers=47386624, cached=49478397952, shared=2196287488, slab=1291599872) 2022-04-11 22:00:25.394666: epoch: 150 2022-04-11 22:02:14.055742: train loss : -0.5424 2022-04-11 22:02:26.214996: validation loss: -0.5772 2022-04-11 22:02:26.223927: Average global foreground Dice: [0.7985, 0.951, 0.9688, 0.401] 2022-04-11 22:02:26.224747: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:02:28.957553: lr: 0.007236 2022-04-11 22:02:28.959307: saving scheduled checkpoint file... 2022-04-11 22:02:29.009022: saving checkpoint... 2022-04-11 22:02:33.707117: done, saving took 4.75 seconds 2022-04-11 22:02:33.716698: done 2022-04-11 22:02:33.763630: saving checkpoint... 2022-04-11 22:02:38.192837: done, saving took 4.48 seconds 2022-04-11 22:02:38.201554: This epoch took 132.806275 s 2022-04-11 22:02:38.203044: svmem(total=99932160000, available=71345897472, percent=28.6, used=24312213504, free=26087763968, active=49379270656, inactive=22058164224, buffers=47472640, cached=49484709888, shared=2201686016, slab=1290592256) 2022-04-11 22:02:38.203721: epoch: 151 2022-04-11 22:04:26.667949: train loss : -0.5351 2022-04-11 22:04:37.384340: validation loss: -0.5664 2022-04-11 22:04:37.388844: Average global foreground Dice: [0.8038, 0.9501, 0.9682, 0.3935] 2022-04-11 22:04:37.391752: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:04:38.247349: lr: 0.007217 2022-04-11 22:04:38.250721: saving scheduled checkpoint file... 2022-04-11 22:04:38.333552: saving checkpoint... 2022-04-11 22:04:42.919307: done, saving took 4.67 seconds 2022-04-11 22:04:42.989988: done 2022-04-11 22:04:43.062066: saving checkpoint... 2022-04-11 22:04:47.489032: done, saving took 4.50 seconds 2022-04-11 22:04:47.526006: This epoch took 129.321632 s 2022-04-11 22:04:47.527195: svmem(total=99932160000, available=71534305280, percent=28.4, used=24121151488, free=26275479552, active=49192730624, inactive=22054391808, buffers=47534080, cached=49487994880, shared=2204225536, slab=1291595776) 2022-04-11 22:04:47.528017: epoch: 152 2022-04-11 22:06:36.585693: train loss : -0.5384 2022-04-11 22:06:47.840569: validation loss: -0.5321 2022-04-11 22:06:47.850632: Average global foreground Dice: [0.7802, 0.9507, 0.968, 0.3437] 2022-04-11 22:06:47.852555: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:06:48.752624: lr: 0.007198 2022-04-11 22:06:48.754166: saving scheduled checkpoint file... 2022-04-11 22:06:48.851327: saving checkpoint... 2022-04-11 22:06:53.402404: done, saving took 4.65 seconds 2022-04-11 22:06:53.421900: done 2022-04-11 22:06:53.508003: saving checkpoint... 2022-04-11 22:06:58.171940: done, saving took 4.75 seconds 2022-04-11 22:06:58.189030: This epoch took 130.660124 s 2022-04-11 22:06:58.190241: svmem(total=99932160000, available=71350161408, percent=28.6, used=24313196544, free=26090389504, active=49377579008, inactive=22053969920, buffers=47628288, cached=49480945664, shared=2196131840, slab=1291341824) 2022-04-11 22:06:58.191013: epoch: 153 2022-04-11 22:08:46.568740: train loss : -0.5476 2022-04-11 22:08:58.909715: validation loss: -0.5625 2022-04-11 22:08:58.915268: Average global foreground Dice: [0.7955, 0.9415, 0.97, 0.3758] 2022-04-11 22:08:58.916289: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:08:59.699301: lr: 0.00718 2022-04-11 22:08:59.700720: saving scheduled checkpoint file... 2022-04-11 22:08:59.803689: saving checkpoint... 2022-04-11 22:09:04.378109: done, saving took 4.68 seconds 2022-04-11 22:09:04.418203: done 2022-04-11 22:09:04.530844: saving checkpoint... 2022-04-11 22:09:09.073309: done, saving took 4.65 seconds 2022-04-11 22:09:09.093317: This epoch took 130.901149 s 2022-04-11 22:09:09.094538: svmem(total=99932160000, available=71606984704, percent=28.3, used=24055222272, free=26348814336, active=49134182400, inactive=22037774336, buffers=47734784, cached=49480388608, shared=2196430848, slab=1292652544) 2022-04-11 22:09:09.095731: epoch: 154 2022-04-11 22:10:59.184684: train loss : -0.5276 2022-04-11 22:11:10.901246: validation loss: -0.5257 2022-04-11 22:11:10.903084: Average global foreground Dice: [0.7627, 0.9425, 0.9644, 0.3686] 2022-04-11 22:11:10.904574: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:11:11.819569: lr: 0.007161 2022-04-11 22:11:11.820667: saving scheduled checkpoint file... 2022-04-11 22:11:11.938011: saving checkpoint... 2022-04-11 22:11:16.446460: done, saving took 4.62 seconds 2022-04-11 22:11:16.466582: done 2022-04-11 22:11:16.582795: saving checkpoint... 2022-04-11 22:11:21.268857: done, saving took 4.80 seconds 2022-04-11 22:11:21.326552: This epoch took 132.230231 s 2022-04-11 22:11:21.328528: svmem(total=99932160000, available=71639953408, percent=28.3, used=24020488192, free=26380828672, active=49096032256, inactive=22040502272, buffers=47808512, cached=49483034624, shared=2198216704, slab=1293840384) 2022-04-11 22:11:21.329439: epoch: 155 2022-04-11 22:13:11.329284: train loss : -0.5377 2022-04-11 22:13:23.651336: validation loss: -0.5718 2022-04-11 22:13:23.653283: Average global foreground Dice: [0.7972, 0.9399, 0.9756, 0.4014] 2022-04-11 22:13:23.655259: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:13:24.452626: lr: 0.007142 2022-04-11 22:13:24.454056: saving scheduled checkpoint file... 2022-04-11 22:13:24.523495: saving checkpoint... 2022-04-11 22:13:28.940061: done, saving took 4.48 seconds 2022-04-11 22:13:28.991146: done 2022-04-11 22:13:29.039950: saving checkpoint... 2022-04-11 22:13:33.407747: done, saving took 4.42 seconds 2022-04-11 22:13:33.473275: This epoch took 132.143014 s 2022-04-11 22:13:33.474862: svmem(total=99932160000, available=71520776192, percent=28.4, used=24141819904, free=26260979712, active=49225998336, inactive=22039142400, buffers=47886336, cached=49481474048, shared=2195849216, slab=1294143488) 2022-04-11 22:13:33.475641: epoch: 156 2022-04-11 22:15:21.924293: train loss : -0.5251 2022-04-11 22:15:32.959450: validation loss: -0.4906 2022-04-11 22:15:32.961631: Average global foreground Dice: [0.741, 0.938, 0.9499, 0.3803] 2022-04-11 22:15:32.962704: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:15:33.942507: lr: 0.007123 2022-04-11 22:15:33.944140: saving scheduled checkpoint file... 2022-04-11 22:15:34.009982: saving checkpoint... 2022-04-11 22:15:38.688634: done, saving took 4.74 seconds 2022-04-11 22:15:38.703566: done 2022-04-11 22:15:38.704987: This epoch took 125.228641 s 2022-04-11 22:15:38.706578: svmem(total=99932160000, available=71427633152, percent=28.5, used=24224845824, free=26166640640, active=49302118400, inactive=22049906688, buffers=47996928, cached=49492676608, shared=2206068736, slab=1293570048) 2022-04-11 22:15:38.707842: epoch: 157 2022-04-11 22:17:27.387596: train loss : -0.5293 2022-04-11 22:17:39.281544: validation loss: -0.5615 2022-04-11 22:17:39.297430: Average global foreground Dice: [0.7793, 0.9457, 0.9693, 0.3712] 2022-04-11 22:17:39.298616: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:17:40.039292: lr: 0.007105 2022-04-11 22:17:40.041215: saving scheduled checkpoint file... 2022-04-11 22:17:40.091806: saving checkpoint... 2022-04-11 22:17:44.711745: done, saving took 4.67 seconds 2022-04-11 22:17:44.760533: done 2022-04-11 22:17:44.761767: This epoch took 126.053129 s 2022-04-11 22:17:44.762917: svmem(total=99932160000, available=71527387136, percent=28.4, used=24134868992, free=26265513984, active=49215963136, inactive=22040862720, buffers=48082944, cached=49483694080, shared=2196234240, slab=1292935168) 2022-04-11 22:17:44.763512: epoch: 158 2022-04-11 22:19:34.370193: train loss : -0.5037 2022-04-11 22:19:46.140502: validation loss: -0.5550 2022-04-11 22:19:46.155415: Average global foreground Dice: [0.7884, 0.937, 0.971, 0.3783] 2022-04-11 22:19:46.156762: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:19:46.942959: lr: 0.007086 2022-04-11 22:19:46.944102: saving scheduled checkpoint file... 2022-04-11 22:19:47.005048: saving checkpoint... 2022-04-11 22:19:51.442613: done, saving took 4.50 seconds 2022-04-11 22:19:51.459669: done 2022-04-11 22:19:51.530848: saving checkpoint... 2022-04-11 22:19:56.379827: done, saving took 4.92 seconds 2022-04-11 22:19:56.396572: This epoch took 131.632487 s 2022-04-11 22:19:56.398248: svmem(total=99932160000, available=71481466880, percent=28.5, used=24194408448, free=26218733568, active=49274761216, inactive=22027882496, buffers=48148480, cached=49470869504, shared=2182561792, slab=1294102528) 2022-04-11 22:19:56.399206: epoch: 159 2022-04-11 22:21:45.696800: train loss : -0.4990 2022-04-11 22:21:57.072953: validation loss: -0.5313 2022-04-11 22:21:57.084860: Average global foreground Dice: [0.8007, 0.9394, 0.9624, 0.356] 2022-04-11 22:21:57.087019: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:21:58.026277: lr: 0.007067 2022-04-11 22:21:58.027708: saving scheduled checkpoint file... 2022-04-11 22:21:58.131895: saving checkpoint... 2022-04-11 22:22:02.842380: done, saving took 4.81 seconds 2022-04-11 22:22:02.860848: done 2022-04-11 22:22:02.951158: saving checkpoint... 2022-04-11 22:22:07.580188: done, saving took 4.72 seconds 2022-04-11 22:22:07.700595: This epoch took 131.300452 s 2022-04-11 22:22:07.702782: svmem(total=99932160000, available=71624343552, percent=28.3, used=24037863424, free=26360774656, active=49118113792, inactive=22042316800, buffers=48230400, cached=49485291520, shared=2196242432, slab=1292656640) 2022-04-11 22:22:07.704064: epoch: 160 2022-04-11 22:23:56.074580: train loss : -0.5208 2022-04-11 22:24:08.157710: validation loss: -0.5281 2022-04-11 22:24:08.159623: Average global foreground Dice: [0.7897, 0.9414, 0.9563, 0.3314] 2022-04-11 22:24:08.160350: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:24:09.126529: lr: 0.007049 2022-04-11 22:24:09.127670: saving scheduled checkpoint file... 2022-04-11 22:24:09.237605: saving checkpoint... 2022-04-11 22:24:14.125058: done, saving took 5.00 seconds 2022-04-11 22:24:14.189442: done 2022-04-11 22:24:14.191009: This epoch took 126.485803 s 2022-04-11 22:24:14.192893: svmem(total=99932160000, available=71494733824, percent=28.5, used=24167260160, free=26230358016, active=49244848128, inactive=22043467776, buffers=48300032, cached=49486241792, shared=2196533248, slab=1293078528) 2022-04-11 22:24:14.194078: epoch: 161 2022-04-11 22:26:04.323475: train loss : -0.5337 2022-04-11 22:26:16.717368: validation loss: -0.4362 2022-04-11 22:26:16.722245: Average global foreground Dice: [0.738, 0.9269, 0.9295, 0.3606] 2022-04-11 22:26:16.723532: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:26:17.419883: lr: 0.00703 2022-04-11 22:26:17.421361: saving scheduled checkpoint file... 2022-04-11 22:26:17.543489: saving checkpoint... 2022-04-11 22:26:22.513080: done, saving took 5.09 seconds 2022-04-11 22:26:22.526298: done 2022-04-11 22:26:22.527177: This epoch took 128.332361 s 2022-04-11 22:26:22.528165: svmem(total=99932160000, available=71430844416, percent=28.5, used=24206999552, free=26165460992, active=49286598656, inactive=22068281344, buffers=48386048, cached=49511313408, shared=2220613632, slab=1291284480) 2022-04-11 22:26:22.529160: epoch: 162 2022-04-11 22:28:11.018749: train loss : -0.4737 2022-04-11 22:28:23.066616: validation loss: -0.5116 2022-04-11 22:28:23.071740: Average global foreground Dice: [0.768, 0.9399, 0.9574, 0.3698] 2022-04-11 22:28:23.072875: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:28:24.009221: lr: 0.007011 2022-04-11 22:28:24.010745: saving scheduled checkpoint file... 2022-04-11 22:28:24.074523: saving checkpoint... 2022-04-11 22:28:28.677193: done, saving took 4.67 seconds 2022-04-11 22:28:28.695149: done 2022-04-11 22:28:28.696338: This epoch took 126.166539 s 2022-04-11 22:28:28.697317: svmem(total=99932160000, available=71440207872, percent=28.5, used=24215760896, free=26174210048, active=49297137664, inactive=22050689024, buffers=48467968, cached=49493721088, shared=2202456064, slab=1292357632) 2022-04-11 22:28:28.698025: epoch: 163 2022-04-11 22:30:18.602605: train loss : -0.4370 2022-04-11 22:30:30.850356: validation loss: -0.4668 2022-04-11 22:30:30.859949: Average global foreground Dice: [0.7591, 0.9285, 0.9579, 0.3576] 2022-04-11 22:30:30.861495: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:30:31.613494: lr: 0.006992 2022-04-11 22:30:31.616280: saving scheduled checkpoint file... 2022-04-11 22:30:31.714235: saving checkpoint... 2022-04-11 22:30:36.269245: done, saving took 4.65 seconds 2022-04-11 22:30:36.285929: done 2022-04-11 22:30:36.287762: This epoch took 127.588660 s 2022-04-11 22:30:36.288958: svmem(total=99932160000, available=71437070336, percent=28.5, used=24225112064, free=26170298368, active=49308442624, inactive=22045204480, buffers=48541696, cached=49488207872, shared=2196242432, slab=1291071488) 2022-04-11 22:30:36.290239: epoch: 164 2022-04-11 22:32:25.016086: train loss : -0.4154 2022-04-11 22:32:36.996747: validation loss: -0.4836 2022-04-11 22:32:37.000556: Average global foreground Dice: [0.7687, 0.926, 0.9522, 0.3879] 2022-04-11 22:32:37.002555: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:32:37.836233: lr: 0.006974 2022-04-11 22:32:37.839107: saving scheduled checkpoint file... 2022-04-11 22:32:37.979143: saving checkpoint... 2022-04-11 22:32:42.651986: done, saving took 4.81 seconds 2022-04-11 22:32:42.673786: done 2022-04-11 22:32:42.675293: This epoch took 126.383823 s 2022-04-11 22:32:42.676671: svmem(total=99932160000, available=71533383680, percent=28.4, used=24134434816, free=26258132992, active=49226305536, inactive=22035853312, buffers=48627712, cached=49490964480, shared=2189217792, slab=1295028224) 2022-04-11 22:32:42.677827: epoch: 165 2022-04-11 22:34:32.381273: train loss : -0.4809 2022-04-11 22:34:43.854579: validation loss: -0.4546 2022-04-11 22:34:43.858413: Average global foreground Dice: [0.7708, 0.9274, 0.9388, 0.3662] 2022-04-11 22:34:43.861322: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:34:44.731946: lr: 0.006955 2022-04-11 22:34:44.734792: saving scheduled checkpoint file... 2022-04-11 22:34:44.890500: saving checkpoint... 2022-04-11 22:34:49.483136: done, saving took 4.74 seconds 2022-04-11 22:34:49.511285: done 2022-04-11 22:34:49.512799: This epoch took 126.834094 s 2022-04-11 22:34:49.514478: svmem(total=99932160000, available=71692795904, percent=28.3, used=23967375360, free=26416738304, active=49062092800, inactive=22043987968, buffers=48701440, cached=49499344896, shared=2196770816, slab=1294413824) 2022-04-11 22:34:49.515237: epoch: 166 2022-04-11 22:36:39.241726: train loss : -0.4924 2022-04-11 22:36:51.428609: validation loss: -0.5437 2022-04-11 22:36:51.443005: Average global foreground Dice: [0.7943, 0.9437, 0.9741, 0.373] 2022-04-11 22:36:51.445126: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:36:52.057246: lr: 0.006936 2022-04-11 22:36:52.058903: saving scheduled checkpoint file... 2022-04-11 22:36:52.189170: saving checkpoint... 2022-04-11 22:36:57.172444: done, saving took 5.11 seconds 2022-04-11 22:36:57.190980: done 2022-04-11 22:36:57.192420: This epoch took 127.676470 s 2022-04-11 22:36:57.193524: svmem(total=99932160000, available=71613538304, percent=28.3, used=24047276032, free=26336514048, active=49136717824, inactive=22044143616, buffers=48799744, cached=49499570176, shared=2196045824, slab=1296195584) 2022-04-11 22:36:57.194886: epoch: 167 2022-04-11 22:38:47.141430: train loss : -0.5265 2022-04-11 22:38:58.899292: validation loss: -0.5354 2022-04-11 22:38:58.906906: Average global foreground Dice: [0.7614, 0.9469, 0.9393, 0.3741] 2022-04-11 22:38:58.912765: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:38:59.731173: lr: 0.006918 2022-04-11 22:38:59.732532: saving scheduled checkpoint file... 2022-04-11 22:38:59.815310: saving checkpoint... 2022-04-11 22:39:05.054644: done, saving took 5.32 seconds 2022-04-11 22:39:05.069976: done 2022-04-11 22:39:05.073321: This epoch took 127.877633 s 2022-04-11 22:39:05.076156: svmem(total=99932160000, available=71563923456, percent=28.4, used=24102678528, free=26286174208, active=49189883904, inactive=22039080960, buffers=48857088, cached=49494450176, shared=2190295040, slab=1296650240) 2022-04-11 22:39:05.078301: epoch: 168 2022-04-11 22:40:53.892748: train loss : -0.5307 2022-04-11 22:41:05.238857: validation loss: -0.5378 2022-04-11 22:41:05.242835: Average global foreground Dice: [0.8044, 0.9506, 0.9553, 0.3588] 2022-04-11 22:41:05.243692: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:41:06.215317: lr: 0.006899 2022-04-11 22:41:06.216703: saving scheduled checkpoint file... 2022-04-11 22:41:06.287720: saving checkpoint... 2022-04-11 22:41:11.048988: done, saving took 4.83 seconds 2022-04-11 22:41:11.092333: done 2022-04-11 22:41:11.115057: This epoch took 126.034235 s 2022-04-11 22:41:11.117531: svmem(total=99932160000, available=71401803776, percent=28.5, used=24252010496, free=26123227136, active=49341382656, inactive=22052757504, buffers=48943104, cached=49507979264, shared=2203160576, slab=1297117184) 2022-04-11 22:41:11.118979: epoch: 169 2022-04-11 22:42:58.872872: train loss : -0.5539 2022-04-11 22:43:11.068019: validation loss: -0.5402 2022-04-11 22:43:11.075861: Average global foreground Dice: [0.7842, 0.95, 0.965, 0.3426] 2022-04-11 22:43:11.157216: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:43:11.809348: lr: 0.00688 2022-04-11 22:43:11.811144: saving scheduled checkpoint file... 2022-04-11 22:43:11.881415: saving checkpoint... 2022-04-11 22:43:16.545919: done, saving took 4.73 seconds 2022-04-11 22:43:16.569887: done 2022-04-11 22:43:16.571306: This epoch took 125.449304 s 2022-04-11 22:43:16.572257: svmem(total=99932160000, available=71322583040, percent=28.6, used=24337911808, free=26043113472, active=49431887872, inactive=22046781440, buffers=48996352, cached=49502138368, shared=2196492288, slab=1295994880) 2022-04-11 22:43:16.573381: epoch: 170 2022-04-11 22:45:04.292118: train loss : -0.5693 2022-04-11 22:45:15.899836: validation loss: -0.6014 2022-04-11 22:45:15.902483: Average global foreground Dice: [0.8098, 0.9538, 0.973, 0.4214] 2022-04-11 22:45:15.904219: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:45:16.751973: lr: 0.006861 2022-04-11 22:45:16.754801: saving scheduled checkpoint file... 2022-04-11 22:45:16.859558: saving checkpoint... 2022-04-11 22:45:21.574228: done, saving took 4.81 seconds 2022-04-11 22:45:21.594261: done 2022-04-11 22:45:21.680315: saving checkpoint... 2022-04-11 22:45:27.362983: done, saving took 5.77 seconds 2022-04-11 22:45:27.391844: This epoch took 130.817614 s 2022-04-11 22:45:27.393561: svmem(total=99932160000, available=71277637632, percent=28.7, used=24383143936, free=25997352960, active=49470689280, inactive=22047260672, buffers=49111040, cached=49502552064, shared=2196156416, slab=1295859712) 2022-04-11 22:45:27.394285: epoch: 171 2022-04-11 22:47:15.787155: train loss : -0.5618 2022-04-11 22:47:27.335303: validation loss: -0.5541 2022-04-11 22:47:27.341085: Average global foreground Dice: [0.8031, 0.9416, 0.9608, 0.3618] 2022-04-11 22:47:27.342728: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:47:28.205829: lr: 0.006842 2022-04-11 22:47:28.207394: saving scheduled checkpoint file... 2022-04-11 22:47:28.331251: saving checkpoint... 2022-04-11 22:47:33.064548: done, saving took 4.86 seconds 2022-04-11 22:47:33.085462: done 2022-04-11 22:47:33.190159: saving checkpoint... 2022-04-11 22:47:37.741014: done, saving took 4.65 seconds 2022-04-11 22:47:37.758918: This epoch took 130.363765 s 2022-04-11 22:47:37.760040: svmem(total=99932160000, available=71629742080, percent=28.3, used=24015728640, free=26348511232, active=49106018304, inactive=22058778624, buffers=49188864, cached=49518731264, shared=2211348480, slab=1296613376) 2022-04-11 22:47:37.760854: epoch: 172 2022-04-11 22:49:26.273606: train loss : -0.5547 2022-04-11 22:49:38.003091: validation loss: -0.5790 2022-04-11 22:49:38.005212: Average global foreground Dice: [0.7964, 0.9498, 0.972, 0.394] 2022-04-11 22:49:38.006213: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:49:38.846660: lr: 0.006824 2022-04-11 22:49:38.847709: saving scheduled checkpoint file... 2022-04-11 22:49:38.986649: saving checkpoint... 2022-04-11 22:49:43.448985: done, saving took 4.60 seconds 2022-04-11 22:49:43.471700: done 2022-04-11 22:49:43.587008: saving checkpoint... 2022-04-11 22:49:48.030132: done, saving took 4.56 seconds 2022-04-11 22:49:48.083898: This epoch took 130.322220 s 2022-04-11 22:49:48.085929: svmem(total=99932160000, available=71405813760, percent=28.5, used=24252637184, free=26123837440, active=49340620800, inactive=22050791424, buffers=49254400, cached=49506430976, shared=2198310912, slab=1296052224) 2022-04-11 22:49:48.087441: epoch: 173 2022-04-11 22:51:38.666980: train loss : -0.5572 2022-04-11 22:51:50.220605: validation loss: -0.5536 2022-04-11 22:51:50.231950: Average global foreground Dice: [0.7905, 0.948, 0.9685, 0.4003] 2022-04-11 22:51:50.233674: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:51:51.497107: lr: 0.006805 2022-04-11 22:51:51.498711: saving scheduled checkpoint file... 2022-04-11 22:51:51.589246: saving checkpoint... 2022-04-11 22:51:56.115327: done, saving took 4.62 seconds 2022-04-11 22:51:56.154354: done 2022-04-11 22:51:56.206372: saving checkpoint... 2022-04-11 22:52:00.538390: done, saving took 4.38 seconds 2022-04-11 22:52:00.553311: This epoch took 132.464685 s 2022-04-11 22:52:00.556805: svmem(total=99932160000, available=71267364864, percent=28.7, used=24379686912, free=25984434176, active=49469173760, inactive=22063296512, buffers=49315840, cached=49518723072, shared=2209845248, slab=1297764352) 2022-04-11 22:52:00.559977: epoch: 174 2022-04-11 22:53:48.044883: train loss : -0.5557 2022-04-11 22:53:59.126019: validation loss: -0.5669 2022-04-11 22:53:59.130353: Average global foreground Dice: [0.815, 0.9473, 0.967, 0.3938] 2022-04-11 22:53:59.131739: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:54:00.022103: lr: 0.006786 2022-04-11 22:54:00.023289: saving scheduled checkpoint file... 2022-04-11 22:54:00.075288: saving checkpoint... 2022-04-11 22:54:04.714266: done, saving took 4.69 seconds 2022-04-11 22:54:04.729321: done 2022-04-11 22:54:04.777162: saving checkpoint... 2022-04-11 22:54:09.246367: done, saving took 4.52 seconds 2022-04-11 22:54:09.260918: This epoch took 128.698265 s 2022-04-11 22:54:09.262072: svmem(total=99932160000, available=71183765504, percent=28.8, used=24462131200, free=25900126208, active=49556971520, inactive=22065385472, buffers=49381376, cached=49520521216, shared=2210836480, slab=1297092608) 2022-04-11 22:54:09.263017: epoch: 175 2022-04-11 22:55:57.329911: train loss : -0.5660 2022-04-11 22:56:08.990237: validation loss: -0.5723 2022-04-11 22:56:08.992416: Average global foreground Dice: [0.7997, 0.9475, 0.9736, 0.3518] 2022-04-11 22:56:08.993763: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:56:10.005531: lr: 0.006767 2022-04-11 22:56:10.006881: saving scheduled checkpoint file... 2022-04-11 22:56:10.098121: saving checkpoint... 2022-04-11 22:56:14.802351: done, saving took 4.79 seconds 2022-04-11 22:56:14.815570: done 2022-04-11 22:56:14.863726: saving checkpoint... 2022-04-11 22:56:19.628225: done, saving took 4.81 seconds 2022-04-11 22:56:19.698341: This epoch took 130.434580 s 2022-04-11 22:56:19.699841: svmem(total=99932160000, available=71446495232, percent=28.5, used=24215138304, free=26138492928, active=49310941184, inactive=22061850624, buffers=49504256, cached=49529024512, shared=2192961536, slab=1301745664) 2022-04-11 22:56:19.700970: epoch: 176 2022-04-11 22:58:07.346754: train loss : -0.5402 2022-04-11 22:58:18.629181: validation loss: -0.5228 2022-04-11 22:58:18.647479: Average global foreground Dice: [0.8052, 0.9369, 0.9489, 0.4021] 2022-04-11 22:58:18.694755: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 22:58:19.452799: lr: 0.006749 2022-04-11 22:58:19.454455: saving scheduled checkpoint file... 2022-04-11 22:58:19.522311: saving checkpoint... 2022-04-11 22:58:24.487228: done, saving took 5.03 seconds 2022-04-11 22:58:24.503460: done 2022-04-11 22:58:24.576214: saving checkpoint... 2022-04-11 22:58:29.370116: done, saving took 4.87 seconds 2022-04-11 22:58:29.386434: This epoch took 129.684621 s 2022-04-11 22:58:29.388144: svmem(total=99932160000, available=71319187456, percent=28.6, used=24328826880, free=26010386432, active=49423314944, inactive=22077218816, buffers=49586176, cached=49543360512, shared=2206539776, slab=1300070400) 2022-04-11 22:58:29.389051: epoch: 177 2022-04-11 23:00:16.917126: train loss : -0.5095 2022-04-11 23:00:27.720070: validation loss: -0.5422 2022-04-11 23:00:27.742824: Average global foreground Dice: [0.7902, 0.9422, 0.9695, 0.3793] 2022-04-11 23:00:27.743772: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:00:28.621238: lr: 0.00673 2022-04-11 23:00:28.623163: saving scheduled checkpoint file... 2022-04-11 23:00:28.706959: saving checkpoint... 2022-04-11 23:00:34.162616: done, saving took 5.54 seconds 2022-04-11 23:00:34.176739: done 2022-04-11 23:00:34.265186: saving checkpoint... 2022-04-11 23:00:39.013086: done, saving took 4.84 seconds 2022-04-11 23:00:39.076344: This epoch took 129.686412 s 2022-04-11 23:00:39.077788: svmem(total=99932160000, available=71540576256, percent=28.4, used=24114569216, free=26230910976, active=49209933824, inactive=22070636544, buffers=49664000, cached=49537015808, shared=2199363584, slab=1300406272) 2022-04-11 23:00:39.078598: epoch: 178 2022-04-11 23:02:27.622537: train loss : -0.5344 2022-04-11 23:02:38.765084: validation loss: -0.5859 2022-04-11 23:02:38.826669: Average global foreground Dice: [0.8112, 0.9481, 0.9759, 0.4046] 2022-04-11 23:02:38.827841: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:02:39.767718: lr: 0.006711 2022-04-11 23:02:39.768787: saving scheduled checkpoint file... 2022-04-11 23:02:39.889368: saving checkpoint... 2022-04-11 23:02:44.399822: done, saving took 4.63 seconds 2022-04-11 23:02:44.419363: done 2022-04-11 23:02:44.517025: saving checkpoint... 2022-04-11 23:02:49.059657: done, saving took 4.64 seconds 2022-04-11 23:02:49.080106: This epoch took 130.000835 s 2022-04-11 23:02:49.081195: svmem(total=99932160000, available=71514546176, percent=28.4, used=24143507456, free=26204110848, active=49241427968, inactive=22068453376, buffers=49725440, cached=49534816256, shared=2196439040, slab=1299673088) 2022-04-11 23:02:49.081912: epoch: 179 2022-04-11 23:04:36.967186: train loss : -0.5598 2022-04-11 23:04:48.397052: validation loss: -0.5875 2022-04-11 23:04:48.405705: Average global foreground Dice: [0.7977, 0.9516, 0.9744, 0.4069] 2022-04-11 23:04:48.406500: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:04:49.445356: lr: 0.006692 2022-04-11 23:04:49.446586: saving scheduled checkpoint file... 2022-04-11 23:04:49.547882: saving checkpoint... 2022-04-11 23:04:53.918122: done, saving took 4.47 seconds 2022-04-11 23:04:53.960164: done 2022-04-11 23:04:54.008993: saving checkpoint... 2022-04-11 23:04:58.885913: done, saving took 4.92 seconds 2022-04-11 23:04:58.901098: This epoch took 129.818467 s 2022-04-11 23:04:58.902620: svmem(total=99932160000, available=71479345152, percent=28.5, used=24178208768, free=26167951360, active=49268125696, inactive=22070198272, buffers=49790976, cached=49536208896, shared=2196840448, slab=1301184512) 2022-04-11 23:04:58.903587: epoch: 180 2022-04-11 23:06:48.156289: train loss : -0.5670 2022-04-11 23:06:59.329191: validation loss: -0.5893 2022-04-11 23:06:59.341276: Average global foreground Dice: [0.843, 0.9456, 0.9725, 0.3841] 2022-04-11 23:06:59.343210: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:07:00.386766: lr: 0.006673 2022-04-11 23:07:00.388131: saving scheduled checkpoint file... 2022-04-11 23:07:00.482726: saving checkpoint... 2022-04-11 23:07:04.981523: done, saving took 4.59 seconds 2022-04-11 23:07:05.035386: done 2022-04-11 23:07:05.108665: saving checkpoint... 2022-04-11 23:07:09.754038: done, saving took 4.72 seconds 2022-04-11 23:07:09.771762: This epoch took 130.867395 s 2022-04-11 23:07:09.773378: svmem(total=99932160000, available=71703461888, percent=28.2, used=23954169856, free=26391322624, active=49053622272, inactive=22069936128, buffers=49901568, cached=49536765952, shared=2196844544, slab=1300111360) 2022-04-11 23:07:09.774836: epoch: 181 2022-04-11 23:08:58.806967: train loss : -0.5477 2022-04-11 23:09:10.804768: validation loss: -0.5963 2022-04-11 23:09:10.813606: Average global foreground Dice: [0.8191, 0.952, 0.9766, 0.3946] 2022-04-11 23:09:10.814656: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:09:11.840185: lr: 0.006654 2022-04-11 23:09:11.843537: saving scheduled checkpoint file... 2022-04-11 23:09:11.927800: saving checkpoint... 2022-04-11 23:09:17.965403: done, saving took 6.12 seconds 2022-04-11 23:09:17.978639: done 2022-04-11 23:09:18.064729: saving checkpoint... 2022-04-11 23:09:22.538087: done, saving took 4.56 seconds 2022-04-11 23:09:22.558886: This epoch took 132.782740 s 2022-04-11 23:09:22.560160: svmem(total=99932160000, available=71569944576, percent=28.4, used=24115314688, free=26259685376, active=49119027200, inactive=22132625408, buffers=49967104, cached=49507192832, shared=2169155584, slab=1301725184) 2022-04-11 23:09:22.561545: epoch: 182 2022-04-11 23:11:12.312366: train loss : -0.5442 2022-04-11 23:11:24.770422: validation loss: -0.5887 2022-04-11 23:11:24.779910: Average global foreground Dice: [0.7825, 0.9542, 0.9762, 0.3933] 2022-04-11 23:11:24.781561: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:11:25.667617: lr: 0.006636 2022-04-11 23:11:25.668998: saving scheduled checkpoint file... 2022-04-11 23:11:25.791707: saving checkpoint... 2022-04-11 23:11:30.255088: done, saving took 4.59 seconds 2022-04-11 23:11:30.274948: done 2022-04-11 23:11:30.374282: saving checkpoint... 2022-04-11 23:11:36.440591: done, saving took 6.16 seconds 2022-04-11 23:11:36.501373: This epoch took 133.938961 s 2022-04-11 23:11:36.509506: svmem(total=99932160000, available=71590252544, percent=28.4, used=24067760128, free=26435694592, active=49138991104, inactive=21930737664, buffers=50044928, cached=49378660352, shared=2196131840, slab=1310044160) 2022-04-11 23:11:36.543693: epoch: 183 2022-04-11 23:13:25.963457: train loss : -0.5729 2022-04-11 23:13:37.609607: validation loss: -0.5575 2022-04-11 23:13:37.633454: Average global foreground Dice: [0.7886, 0.9516, 0.9719, 0.4111] 2022-04-11 23:13:37.646656: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:13:38.547854: lr: 0.006617 2022-04-11 23:13:38.553843: saving scheduled checkpoint file... 2022-04-11 23:13:38.692045: saving checkpoint... 2022-04-11 23:13:43.422045: done, saving took 4.84 seconds 2022-04-11 23:13:43.444634: done 2022-04-11 23:13:43.559203: saving checkpoint... 2022-04-11 23:13:48.648466: done, saving took 5.20 seconds 2022-04-11 23:13:48.998129: This epoch took 132.418905 s 2022-04-11 23:13:49.140730: svmem(total=99932160000, available=71608360960, percent=28.3, used=24049954816, free=26296504320, active=49141686272, inactive=22061764608, buffers=50122752, cached=49535578112, shared=2196045824, slab=1314623488) 2022-04-11 23:13:49.274374: epoch: 184 2022-04-11 23:15:37.840236: train loss : -0.5680 2022-04-11 23:15:49.850575: validation loss: -0.5847 2022-04-11 23:15:49.859977: Average global foreground Dice: [0.8192, 0.9532, 0.9783, 0.3796] 2022-04-11 23:15:49.861396: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:15:50.954557: lr: 0.006598 2022-04-11 23:15:50.956526: saving scheduled checkpoint file... 2022-04-11 23:15:51.106994: saving checkpoint... 2022-04-11 23:15:55.670997: done, saving took 4.71 seconds 2022-04-11 23:15:55.692907: done 2022-04-11 23:15:55.821743: saving checkpoint... 2022-04-11 23:16:00.467258: done, saving took 4.77 seconds 2022-04-11 23:16:00.480075: This epoch took 131.062825 s 2022-04-11 23:16:00.481039: svmem(total=99932160000, available=71603900416, percent=28.3, used=24068280320, free=26291216384, active=49175457792, inactive=22048739328, buffers=50208768, cached=49522454528, shared=2182254592, slab=1301188608) 2022-04-11 23:16:00.481729: epoch: 185 2022-04-11 23:17:49.649509: train loss : -0.5682 2022-04-11 23:18:02.074713: validation loss: -0.5756 2022-04-11 23:18:02.076947: Average global foreground Dice: [0.808, 0.951, 0.9713, 0.3847] 2022-04-11 23:18:02.078427: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:18:03.043654: lr: 0.006579 2022-04-11 23:18:03.044932: saving scheduled checkpoint file... 2022-04-11 23:18:03.130232: saving checkpoint... 2022-04-11 23:18:07.978118: done, saving took 4.93 seconds 2022-04-11 23:18:07.991354: done 2022-04-11 23:18:08.039043: saving checkpoint... 2022-04-11 23:18:12.515351: done, saving took 4.52 seconds 2022-04-11 23:18:12.529911: This epoch took 132.047575 s 2022-04-11 23:18:12.531611: svmem(total=99932160000, available=71691837440, percent=28.3, used=23972376576, free=26682298368, active=49102594048, inactive=21724053504, buffers=50315264, cached=49227169792, shared=2187706368, slab=1310240768) 2022-04-11 23:18:12.533136: epoch: 186 2022-04-11 23:20:00.498020: train loss : -0.4616 2022-04-11 23:20:12.456322: validation loss: -0.5274 2022-04-11 23:20:12.460519: Average global foreground Dice: [0.7918, 0.9371, 0.9534, 0.3636] 2022-04-11 23:20:12.461811: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:20:13.355328: lr: 0.00656 2022-04-11 23:20:13.356919: saving scheduled checkpoint file... 2022-04-11 23:20:13.419522: saving checkpoint... 2022-04-11 23:20:18.038286: done, saving took 4.68 seconds 2022-04-11 23:20:18.051675: done 2022-04-11 23:20:18.052982: This epoch took 125.517803 s 2022-04-11 23:20:18.054033: svmem(total=99932160000, available=71732342784, percent=28.2, used=23934930944, free=26467106816, active=49066135552, inactive=21977546752, buffers=50397184, cached=49479725056, shared=2185621504, slab=1305141248) 2022-04-11 23:20:18.055399: epoch: 187 2022-04-11 23:22:06.493130: train loss : -0.4454 2022-04-11 23:22:17.537011: validation loss: -0.5046 2022-04-11 23:22:17.540166: Average global foreground Dice: [0.766, 0.9375, 0.9608, 0.3361] 2022-04-11 23:22:17.541478: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:22:18.429049: lr: 0.006541 2022-04-11 23:22:18.430500: saving scheduled checkpoint file... 2022-04-11 23:22:18.489552: saving checkpoint... 2022-04-11 23:22:23.207406: done, saving took 4.78 seconds 2022-04-11 23:22:23.224898: done 2022-04-11 23:22:23.226280: This epoch took 125.170201 s 2022-04-11 23:22:23.227441: svmem(total=99932160000, available=71623270400, percent=28.3, used=24031563776, free=26357186560, active=49160933376, inactive=21990805504, buffers=50483200, cached=49492926464, shared=2198167552, slab=1304637440) 2022-04-11 23:22:23.228082: epoch: 188 2022-04-11 23:24:11.302475: train loss : -0.5126 2022-04-11 23:24:22.216161: validation loss: -0.5343 2022-04-11 23:24:22.218536: Average global foreground Dice: [0.791, 0.9375, 0.9634, 0.358] 2022-04-11 23:24:22.219403: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:24:23.139895: lr: 0.006522 2022-04-11 23:24:23.141471: saving scheduled checkpoint file... 2022-04-11 23:24:23.250593: saving checkpoint... 2022-04-11 23:24:27.768664: done, saving took 4.63 seconds 2022-04-11 23:24:27.844485: done 2022-04-11 23:24:27.846998: This epoch took 124.618328 s 2022-04-11 23:24:27.849698: svmem(total=99932160000, available=71619178496, percent=28.3, used=24031297536, free=26352410624, active=49159712768, inactive=21995700224, buffers=50556928, cached=49497894912, shared=2202476544, slab=1305432064) 2022-04-11 23:24:27.852050: epoch: 189 2022-04-11 23:26:16.568242: train loss : -0.5178 2022-04-11 23:26:28.318815: validation loss: -0.5628 2022-04-11 23:26:28.321031: Average global foreground Dice: [0.8112, 0.9472, 0.9669, 0.366] 2022-04-11 23:26:28.322034: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:26:29.248315: lr: 0.006504 2022-04-11 23:26:29.249570: saving scheduled checkpoint file... 2022-04-11 23:26:29.362233: saving checkpoint... 2022-04-11 23:26:33.969587: done, saving took 4.72 seconds 2022-04-11 23:26:34.009187: done 2022-04-11 23:26:34.011515: This epoch took 126.156916 s 2022-04-11 23:26:34.014218: svmem(total=99932160000, available=71551926272, percent=28.4, used=24104796160, free=26284318720, active=49235369984, inactive=21990580224, buffers=50659328, cached=49492385792, shared=2196156416, slab=1303433216) 2022-04-11 23:26:34.016772: epoch: 190 2022-04-11 23:28:22.651257: train loss : -0.5150 2022-04-11 23:28:34.573579: validation loss: -0.5605 2022-04-11 23:28:34.576021: Average global foreground Dice: [0.7915, 0.9471, 0.9733, 0.3965] 2022-04-11 23:28:34.577549: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:28:35.446943: lr: 0.006485 2022-04-11 23:28:35.448523: saving scheduled checkpoint file... 2022-04-11 23:28:35.590227: saving checkpoint... 2022-04-11 23:28:40.095286: done, saving took 4.65 seconds 2022-04-11 23:28:40.126028: done 2022-04-11 23:28:40.129398: This epoch took 126.110528 s 2022-04-11 23:28:40.132107: svmem(total=99932160000, available=71558217728, percent=28.4, used=24098525184, free=26289762304, active=49230962688, inactive=21990875136, buffers=50724864, cached=49493147648, shared=2196156416, slab=1303932928) 2022-04-11 23:28:40.134456: epoch: 191 2022-04-11 23:30:30.278197: train loss : -0.4418 2022-04-11 23:30:42.929509: validation loss: -0.4698 2022-04-11 23:30:42.944012: Average global foreground Dice: [0.7215, 0.9334, 0.9518, 0.369] 2022-04-11 23:30:42.945615: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:30:43.873767: lr: 0.006466 2022-04-11 23:30:43.875271: saving scheduled checkpoint file... 2022-04-11 23:30:43.956515: saving checkpoint... 2022-04-11 23:30:48.610618: done, saving took 4.73 seconds 2022-04-11 23:30:48.623841: done 2022-04-11 23:30:48.625213: This epoch took 128.487874 s 2022-04-11 23:30:48.626474: svmem(total=99932160000, available=71455313920, percent=28.5, used=24197087232, free=26186035200, active=49329340416, inactive=21995888640, buffers=50794496, cached=49498243072, shared=2200481792, slab=1303777280) 2022-04-11 23:30:48.627840: epoch: 192 2022-04-11 23:32:38.201353: train loss : -0.5010 2022-04-11 23:32:50.522733: validation loss: -0.5625 2022-04-11 23:32:50.536280: Average global foreground Dice: [0.7977, 0.9451, 0.9708, 0.3791] 2022-04-11 23:32:50.537659: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:32:51.492918: lr: 0.006447 2022-04-11 23:32:51.494323: saving scheduled checkpoint file... 2022-04-11 23:32:51.572997: saving checkpoint... 2022-04-11 23:32:57.235440: done, saving took 5.74 seconds 2022-04-11 23:32:57.264256: done 2022-04-11 23:32:57.265826: This epoch took 128.637119 s 2022-04-11 23:32:57.268238: svmem(total=99932160000, available=71633371136, percent=28.3, used=24023003136, free=26363240448, active=49154662400, inactive=21992587264, buffers=50851840, cached=49495064576, shared=2196439040, slab=1303760896) 2022-04-11 23:32:57.274210: epoch: 193 2022-04-11 23:34:45.044466: train loss : -0.5484 2022-04-11 23:34:57.081589: validation loss: -0.5713 2022-04-11 23:34:57.098448: Average global foreground Dice: [0.8026, 0.9497, 0.9739, 0.3884] 2022-04-11 23:34:57.116783: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:34:57.965169: lr: 0.006428 2022-04-11 23:34:57.972763: saving scheduled checkpoint file... 2022-04-11 23:34:58.089843: saving checkpoint... 2022-04-11 23:35:04.912441: done, saving took 6.92 seconds 2022-04-11 23:35:05.006479: done 2022-04-11 23:35:05.040956: This epoch took 127.755693 s 2022-04-11 23:35:05.066542: svmem(total=99932160000, available=71460069376, percent=28.5, used=24196923392, free=26189197312, active=49331838976, inactive=21992775680, buffers=50937856, cached=49495101440, shared=2195853312, slab=1303633920) 2022-04-11 23:35:05.077284: epoch: 194 2022-04-11 23:36:53.874780: train loss : -0.5725 2022-04-11 23:37:05.356876: validation loss: -0.6058 2022-04-11 23:37:05.374483: Average global foreground Dice: [0.8305, 0.9559, 0.9773, 0.3951] 2022-04-11 23:37:05.379439: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:37:06.279421: lr: 0.006409 2022-04-11 23:37:06.298180: saving scheduled checkpoint file... 2022-04-11 23:37:06.431216: saving checkpoint... 2022-04-11 23:37:14.353669: done, saving took 8.04 seconds 2022-04-11 23:37:14.391726: done 2022-04-11 23:37:14.408850: This epoch took 129.322767 s 2022-04-11 23:37:14.419592: svmem(total=99932160000, available=71357620224, percent=28.6, used=24307556352, free=26085933056, active=49440894976, inactive=21985054720, buffers=51027968, cached=49487642624, shared=2187579392, slab=1303642112) 2022-04-11 23:37:14.421757: epoch: 195 2022-04-11 23:39:01.998292: train loss : -0.5864 2022-04-11 23:39:14.135542: validation loss: -0.6028 2022-04-11 23:39:14.146768: Average global foreground Dice: [0.8127, 0.9464, 0.9779, 0.4077] 2022-04-11 23:39:14.156142: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:39:14.814563: lr: 0.00639 2022-04-11 23:39:14.825589: saving scheduled checkpoint file... 2022-04-11 23:39:14.947562: saving checkpoint... 2022-04-11 23:39:20.059216: done, saving took 5.22 seconds 2022-04-11 23:39:20.079623: done 2022-04-11 23:39:20.080875: This epoch took 125.657710 s 2022-04-11 23:39:20.081863: svmem(total=99932160000, available=71510966272, percent=28.4, used=24136192000, free=26238312448, active=49267019776, inactive=22003982336, buffers=51089408, cached=49506566144, shared=2205564928, slab=1304735744) 2022-04-11 23:39:20.082728: epoch: 196 2022-04-11 23:41:08.622478: train loss : -0.5782 2022-04-11 23:41:21.409830: validation loss: -0.5985 2022-04-11 23:41:21.411750: Average global foreground Dice: [0.8273, 0.943, 0.9732, 0.4012] 2022-04-11 23:41:21.412583: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:41:22.280900: lr: 0.006371 2022-04-11 23:41:22.284394: saving scheduled checkpoint file... 2022-04-11 23:41:22.413949: saving checkpoint... 2022-04-11 23:41:26.893626: done, saving took 4.61 seconds 2022-04-11 23:41:26.917365: done 2022-04-11 23:41:26.918727: This epoch took 126.835037 s 2022-04-11 23:41:26.920274: svmem(total=99932160000, available=71627546624, percent=28.3, used=24028061696, free=26356543488, active=49144180736, inactive=22008193024, buffers=51216384, cached=49496338432, shared=2195984384, slab=1305497600) 2022-04-11 23:41:26.921593: epoch: 197 2022-04-11 23:43:15.263619: train loss : -0.5894 2022-04-11 23:43:27.844250: validation loss: -0.5810 2022-04-11 23:43:27.855291: Average global foreground Dice: [0.8302, 0.9391, 0.9701, 0.4077] 2022-04-11 23:43:27.856453: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:43:28.789282: lr: 0.006352 2022-04-11 23:43:28.790735: saving scheduled checkpoint file... 2022-04-11 23:43:28.871690: saving checkpoint... 2022-04-11 23:43:33.864538: done, saving took 5.07 seconds 2022-04-11 23:43:33.877155: done 2022-04-11 23:43:33.925892: saving checkpoint... 2022-04-11 23:43:38.569558: done, saving took 4.69 seconds 2022-04-11 23:43:38.583462: This epoch took 131.660932 s 2022-04-11 23:43:38.585053: svmem(total=99932160000, available=71508660224, percent=28.4, used=24146968576, free=26294464512, active=49259782144, inactive=21956157440, buffers=51286016, cached=49439440896, shared=2195972096, slab=1304240128) 2022-04-11 23:43:38.585710: epoch: 198 2022-04-11 23:45:26.368699: train loss : -0.5785 2022-04-11 23:45:38.289040: validation loss: -0.5850 2022-04-11 23:45:38.299747: Average global foreground Dice: [0.8125, 0.9456, 0.9748, 0.4077] 2022-04-11 23:45:38.301341: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:45:39.269583: lr: 0.006333 2022-04-11 23:45:39.271178: saving scheduled checkpoint file... 2022-04-11 23:45:39.341625: saving checkpoint... 2022-04-11 23:45:43.940580: done, saving took 4.67 seconds 2022-04-11 23:45:43.954372: done 2022-04-11 23:45:44.002382: saving checkpoint... 2022-04-11 23:45:49.161393: done, saving took 5.21 seconds 2022-04-11 23:45:49.175183: This epoch took 130.588706 s 2022-04-11 23:45:49.176815: svmem(total=99932160000, available=71468863488, percent=28.5, used=24186433536, free=26253692928, active=49296982016, inactive=21957480448, buffers=51380224, cached=49440653312, shared=2196406272, slab=1306988544) 2022-04-11 23:45:49.178338: epoch: 199 2022-04-11 23:47:37.975613: train loss : -0.5308 2022-04-11 23:47:49.569875: validation loss: -0.4733 2022-04-11 23:47:49.571432: Average global foreground Dice: [0.7676, 0.9175, 0.9439, 0.3808] 2022-04-11 23:47:49.572603: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:47:50.406225: lr: 0.006314 2022-04-11 23:47:50.407579: saving scheduled checkpoint file... 2022-04-11 23:47:50.471753: saving checkpoint... 2022-04-11 23:47:55.292608: done, saving took 4.88 seconds 2022-04-11 23:47:55.309445: done 2022-04-11 23:47:55.311227: This epoch took 126.131964 s 2022-04-11 23:47:55.312354: svmem(total=99932160000, available=71447670784, percent=28.5, used=24201539584, free=26231738368, active=49311535104, inactive=21961998336, buffers=51445760, cached=49447436288, shared=2202537984, slab=1306808320) 2022-04-11 23:47:55.313660: epoch: 200 2022-04-11 23:49:42.858270: train loss : -0.3790 2022-04-11 23:49:55.065723: validation loss: -0.3756 2022-04-11 23:49:55.088206: Average global foreground Dice: [0.7338, 0.8994, 0.9024, 0.3676] 2022-04-11 23:49:55.089985: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:49:55.892270: lr: 0.006296 2022-04-11 23:49:55.893391: saving scheduled checkpoint file... 2022-04-11 23:49:55.998282: saving checkpoint... 2022-04-11 23:50:00.825312: done, saving took 4.93 seconds 2022-04-11 23:50:00.913922: done 2022-04-11 23:50:00.918812: This epoch took 125.604345 s 2022-04-11 23:50:00.924155: svmem(total=99932160000, available=71527297024, percent=28.4, used=24128401408, free=26310557696, active=49238196224, inactive=21958668288, buffers=51503104, cached=49441697792, shared=2196013056, slab=1306411008) 2022-04-11 23:50:00.928158: epoch: 201 2022-04-11 23:51:49.345018: train loss : -0.4777 2022-04-11 23:52:01.185303: validation loss: -0.5323 2022-04-11 23:52:01.200701: Average global foreground Dice: [0.7737, 0.9404, 0.9658, 0.3875] 2022-04-11 23:52:01.203396: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:52:01.803684: lr: 0.006277 2022-04-11 23:52:01.805481: saving scheduled checkpoint file... 2022-04-11 23:52:01.919847: saving checkpoint... 2022-04-11 23:52:06.706054: done, saving took 4.90 seconds 2022-04-11 23:52:06.725842: done 2022-04-11 23:52:06.727026: This epoch took 125.786095 s 2022-04-11 23:52:06.728031: svmem(total=99932160000, available=71394897920, percent=28.6, used=24250109952, free=26177490944, active=49360322560, inactive=21970460672, buffers=51564544, cached=49452994560, shared=2206760960, slab=1305649152) 2022-04-11 23:52:06.729353: epoch: 202 2022-04-11 23:53:55.677371: train loss : -0.5285 2022-04-11 23:54:07.419103: validation loss: -0.5590 2022-04-11 23:54:07.428293: Average global foreground Dice: [0.7874, 0.9452, 0.9612, 0.3957] 2022-04-11 23:54:07.430913: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:54:08.181644: lr: 0.006258 2022-04-11 23:54:08.183684: saving scheduled checkpoint file... 2022-04-11 23:54:08.304898: saving checkpoint... 2022-04-11 23:54:12.928516: done, saving took 4.74 seconds 2022-04-11 23:54:12.951464: done 2022-04-11 23:54:12.953254: This epoch took 126.223114 s 2022-04-11 23:54:12.954709: svmem(total=99932160000, available=71450181632, percent=28.5, used=24205381632, free=26231775232, active=49316098048, inactive=21960732672, buffers=51638272, cached=49443364864, shared=2196131840, slab=1306161152) 2022-04-11 23:54:12.956135: epoch: 203 2022-04-11 23:56:02.594640: train loss : -0.5683 2022-04-11 23:56:14.677110: validation loss: -0.5852 2022-04-11 23:56:14.684475: Average global foreground Dice: [0.8029, 0.9515, 0.9678, 0.3936] 2022-04-11 23:56:14.685706: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:56:15.409227: lr: 0.006239 2022-04-11 23:56:15.410583: saving scheduled checkpoint file... 2022-04-11 23:56:15.504337: saving checkpoint... 2022-04-11 23:56:20.472440: done, saving took 5.06 seconds 2022-04-11 23:56:20.487703: done 2022-04-11 23:56:20.489253: This epoch took 127.531695 s 2022-04-11 23:56:20.490507: svmem(total=99932160000, available=71286235136, percent=28.7, used=24375013376, free=26067005440, active=49483845632, inactive=21955194880, buffers=51744768, cached=49438396416, shared=2190450688, slab=1307840512) 2022-04-11 23:56:20.491493: epoch: 204 2022-04-11 23:58:09.203394: train loss : -0.5464 2022-04-11 23:58:21.899477: validation loss: -0.5576 2022-04-11 23:58:21.902180: Average global foreground Dice: [0.8197, 0.9466, 0.9678, 0.3647] 2022-04-11 23:58:21.903331: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-11 23:58:22.479319: lr: 0.00622 2022-04-11 23:58:22.480684: saving scheduled checkpoint file... 2022-04-11 23:58:22.533098: saving checkpoint... 2022-04-11 23:58:27.194516: done, saving took 4.71 seconds 2022-04-11 23:58:27.283173: done 2022-04-11 23:58:27.286155: This epoch took 126.793419 s 2022-04-11 23:58:27.339880: svmem(total=99932160000, available=71569924096, percent=28.4, used=24085602304, free=26349838336, active=49195991040, inactive=21961519104, buffers=51822592, cached=49444896768, shared=2196070400, slab=1306484736) 2022-04-11 23:58:27.341425: epoch: 205 2022-04-12 00:00:16.849570: train loss : -0.5553 2022-04-12 00:00:28.520794: validation loss: -0.5659 2022-04-12 00:00:28.522858: Average global foreground Dice: [0.807, 0.9461, 0.9621, 0.3844] 2022-04-12 00:00:28.524190: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:00:29.354629: lr: 0.006201 2022-04-12 00:00:29.356412: saving scheduled checkpoint file... 2022-04-12 00:00:29.448871: saving checkpoint... 2022-04-12 00:00:34.544808: done, saving took 5.19 seconds 2022-04-12 00:00:34.563816: done 2022-04-12 00:00:34.565872: This epoch took 127.223258 s 2022-04-12 00:00:34.567335: svmem(total=99932160000, available=71495307264, percent=28.5, used=24153550848, free=26257506304, active=49257234432, inactive=21963444224, buffers=60424192, cached=49460678656, shared=2196799488, slab=1317113856) 2022-04-12 00:00:34.568635: epoch: 206 2022-04-12 00:02:23.651049: train loss : -0.5625 2022-04-12 00:02:35.910854: validation loss: -0.5907 2022-04-12 00:02:35.920516: Average global foreground Dice: [0.8263, 0.9466, 0.9726, 0.4149] 2022-04-12 00:02:35.921621: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:02:36.795764: lr: 0.006182 2022-04-12 00:02:36.797961: saving scheduled checkpoint file... 2022-04-12 00:02:36.913270: saving checkpoint... 2022-04-12 00:02:41.734517: done, saving took 4.94 seconds 2022-04-12 00:02:41.854274: done 2022-04-12 00:02:41.855792: This epoch took 127.283262 s 2022-04-12 00:02:41.857102: svmem(total=99932160000, available=71502163968, percent=28.4, used=24147070976, free=26263523328, active=49269059584, inactive=21963636736, buffers=60522496, cached=49461043200, shared=2196463616, slab=1318154240) 2022-04-12 00:02:41.858536: epoch: 207 2022-04-12 00:04:30.088358: train loss : -0.5719 2022-04-12 00:04:41.865105: validation loss: -0.5671 2022-04-12 00:04:41.867513: Average global foreground Dice: [0.7659, 0.9473, 0.9737, 0.3769] 2022-04-12 00:04:41.868461: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:04:42.721268: lr: 0.006163 2022-04-12 00:04:42.722830: saving scheduled checkpoint file... 2022-04-12 00:04:42.857192: saving checkpoint... 2022-04-12 00:04:47.961555: done, saving took 5.24 seconds 2022-04-12 00:04:47.996764: done 2022-04-12 00:04:47.998007: This epoch took 126.138627 s 2022-04-12 00:04:47.999198: svmem(total=99932160000, available=71607545856, percent=28.3, used=24040869888, free=26337009664, active=49122123776, inactive=22034624512, buffers=60612608, cached=49493667840, shared=2196770816, slab=1319784448) 2022-04-12 00:04:47.999892: epoch: 208 2022-04-12 00:06:39.809022: train loss : -0.5526 2022-04-12 00:06:51.948307: validation loss: -0.5962 2022-04-12 00:06:51.995415: Average global foreground Dice: [0.8224, 0.9487, 0.9738, 0.3819] 2022-04-12 00:06:52.024738: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:06:52.655768: lr: 0.006144 2022-04-12 00:06:52.661407: saving scheduled checkpoint file... 2022-04-12 00:06:52.802141: saving checkpoint... 2022-04-12 00:06:57.464499: done, saving took 4.79 seconds 2022-04-12 00:06:57.524539: done 2022-04-12 00:06:57.525903: This epoch took 129.525314 s 2022-04-12 00:06:57.527695: svmem(total=99932160000, available=71374405632, percent=28.6, used=24271609856, free=26103828480, active=49353572352, inactive=22037372928, buffers=60723200, cached=49495998464, shared=2199130112, slab=1318830080) 2022-04-12 00:06:57.528560: epoch: 209 2022-04-12 00:08:47.161606: train loss : -0.5813 2022-04-12 00:08:59.205855: validation loss: -0.5978 2022-04-12 00:08:59.216663: Average global foreground Dice: [0.8013, 0.9514, 0.9738, 0.3892] 2022-04-12 00:08:59.217906: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:09:00.195509: lr: 0.006125 2022-04-12 00:09:00.196971: saving scheduled checkpoint file... 2022-04-12 00:09:00.287554: saving checkpoint... 2022-04-12 00:09:06.316589: done, saving took 6.12 seconds 2022-04-12 00:09:06.330208: done 2022-04-12 00:09:06.331316: This epoch took 128.801716 s 2022-04-12 00:09:06.332580: svmem(total=99932160000, available=71404998656, percent=28.5, used=24243879936, free=26134020096, active=49327034368, inactive=22034984960, buffers=60776448, cached=49493483520, shared=2196242432, slab=1319239680) 2022-04-12 00:09:06.333337: epoch: 210 2022-04-12 00:10:55.655658: train loss : -0.5826 2022-04-12 00:11:07.770568: validation loss: -0.6058 2022-04-12 00:11:07.781260: Average global foreground Dice: [0.8192, 0.96, 0.9733, 0.3999] 2022-04-12 00:11:07.792164: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:11:08.628080: lr: 0.006106 2022-04-12 00:11:08.629581: saving scheduled checkpoint file... 2022-04-12 00:11:08.677684: saving checkpoint... 2022-04-12 00:11:13.270053: done, saving took 4.64 seconds 2022-04-12 00:11:13.296898: done 2022-04-12 00:11:13.297687: This epoch took 126.963629 s 2022-04-12 00:11:13.299165: svmem(total=99932160000, available=71363641344, percent=28.6, used=24280715264, free=26092347392, active=49363234816, inactive=22039912448, buffers=60841984, cached=49498255360, shared=2200711168, slab=1318711296) 2022-04-12 00:11:13.300421: epoch: 211 2022-04-12 00:13:02.881676: train loss : -0.5833 2022-04-12 00:13:14.493219: validation loss: -0.6115 2022-04-12 00:13:14.503027: Average global foreground Dice: [0.813, 0.9598, 0.973, 0.4109] 2022-04-12 00:13:14.504198: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:13:15.173269: lr: 0.006087 2022-04-12 00:13:15.174999: saving scheduled checkpoint file... 2022-04-12 00:13:15.258151: saving checkpoint... 2022-04-12 00:13:20.046667: done, saving took 4.87 seconds 2022-04-12 00:13:20.063818: done 2022-04-12 00:13:20.064590: This epoch took 126.762970 s 2022-04-12 00:13:20.065805: svmem(total=99932160000, available=71274029056, percent=28.7, used=24374906880, free=26040139776, active=49348153344, inactive=22104309760, buffers=60907520, cached=49456205824, shared=2196156416, slab=1319739392) 2022-04-12 00:13:20.066508: epoch: 212 2022-04-12 00:15:07.603534: train loss : -0.6015 2022-04-12 00:15:19.756866: validation loss: -0.5916 2022-04-12 00:15:19.768316: Average global foreground Dice: [0.845, 0.9489, 0.9716, 0.4115] 2022-04-12 00:15:19.769608: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:15:20.580528: lr: 0.006068 2022-04-12 00:15:20.582363: saving scheduled checkpoint file... 2022-04-12 00:15:20.687343: saving checkpoint... 2022-04-12 00:15:25.334848: done, saving took 4.75 seconds 2022-04-12 00:15:25.415548: done 2022-04-12 00:15:25.501426: saving checkpoint... 2022-04-12 00:15:30.339054: done, saving took 4.92 seconds 2022-04-12 00:15:30.361355: This epoch took 130.294192 s 2022-04-12 00:15:30.365001: svmem(total=99932160000, available=71500140544, percent=28.5, used=24148652032, free=26266165248, active=49201803264, inactive=22027825152, buffers=61022208, cached=49456320512, shared=2196320256, slab=1318780928) 2022-04-12 00:15:30.367320: epoch: 213 2022-04-12 00:17:18.729697: train loss : -0.5937 2022-04-12 00:17:31.166932: validation loss: -0.6442 2022-04-12 00:17:31.168895: Average global foreground Dice: [0.872, 0.9613, 0.9809, 0.4088] 2022-04-12 00:17:31.170151: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:17:31.912173: lr: 0.006049 2022-04-12 00:17:31.913501: saving scheduled checkpoint file... 2022-04-12 00:17:32.020097: saving checkpoint... 2022-04-12 00:17:36.668280: done, saving took 4.75 seconds 2022-04-12 00:17:36.688930: done 2022-04-12 00:17:36.788832: saving checkpoint... 2022-04-12 00:17:41.359009: done, saving took 4.67 seconds 2022-04-12 00:17:41.379403: This epoch took 131.009409 s 2022-04-12 00:17:41.381586: svmem(total=99932160000, available=71520800768, percent=28.4, used=24127975424, free=26286149632, active=49195868160, inactive=22012956672, buffers=61100032, cached=49456934912, shared=2196242432, slab=1320488960) 2022-04-12 00:17:41.384157: epoch: 214 2022-04-12 00:19:30.827939: train loss : -0.6088 2022-04-12 00:19:42.609203: validation loss: -0.6239 2022-04-12 00:19:42.624580: Average global foreground Dice: [0.841, 0.9554, 0.9755, 0.3843] 2022-04-12 00:19:42.625633: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:19:43.479377: lr: 0.00603 2022-04-12 00:19:43.480483: saving scheduled checkpoint file... 2022-04-12 00:19:43.617302: saving checkpoint... 2022-04-12 00:19:48.223141: done, saving took 4.74 seconds 2022-04-12 00:19:48.240229: done 2022-04-12 00:19:48.356641: saving checkpoint... 2022-04-12 00:19:52.960809: done, saving took 4.72 seconds 2022-04-12 00:19:52.982932: This epoch took 131.595906 s 2022-04-12 00:19:52.985075: svmem(total=99932160000, available=71541870592, percent=28.4, used=24107036672, free=26307043328, active=49177817088, inactive=22007033856, buffers=61177856, cached=49456902144, shared=2196242432, slab=1319317504) 2022-04-12 00:19:52.986168: epoch: 215 2022-04-12 00:21:42.960775: train loss : -0.6097 2022-04-12 00:21:56.171867: validation loss: -0.6184 2022-04-12 00:21:56.176207: Average global foreground Dice: [0.8439, 0.9535, 0.9757, 0.41] 2022-04-12 00:21:56.177433: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:21:56.962819: lr: 0.006011 2022-04-12 00:21:56.963857: saving scheduled checkpoint file... 2022-04-12 00:21:57.038014: saving checkpoint... 2022-04-12 00:22:01.803203: done, saving took 4.84 seconds 2022-04-12 00:22:01.818319: done 2022-04-12 00:22:01.866761: saving checkpoint... 2022-04-12 00:22:06.619815: done, saving took 4.80 seconds 2022-04-12 00:22:06.676184: This epoch took 133.688873 s 2022-04-12 00:22:06.677985: svmem(total=99932160000, available=71234928640, percent=28.7, used=24393990144, free=25999388672, active=49468354560, inactive=22026960896, buffers=61259776, cached=49477521408, shared=2215981056, slab=1319182336) 2022-04-12 00:22:06.678840: epoch: 216 2022-04-12 00:23:56.134044: train loss : -0.6078 2022-04-12 00:24:07.238861: validation loss: -0.6320 2022-04-12 00:24:07.242796: Average global foreground Dice: [0.8287, 0.9605, 0.975, 0.4209] 2022-04-12 00:24:07.244891: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:24:08.124540: lr: 0.005991 2022-04-12 00:24:08.125874: saving scheduled checkpoint file... 2022-04-12 00:24:08.206135: saving checkpoint... 2022-04-12 00:24:12.859175: done, saving took 4.73 seconds 2022-04-12 00:24:12.871338: done 2022-04-12 00:24:12.919560: saving checkpoint... 2022-04-12 00:24:17.430706: done, saving took 4.56 seconds 2022-04-12 00:24:17.448100: This epoch took 130.768448 s 2022-04-12 00:24:17.449987: svmem(total=99932160000, available=71497801728, percent=28.5, used=24146194432, free=26263703552, active=49220104192, inactive=22009634816, buffers=61325312, cached=49460936704, shared=2201198592, slab=1318240256) 2022-04-12 00:24:17.451004: epoch: 217 2022-04-12 00:26:05.000580: train loss : -0.6048 2022-04-12 00:26:17.497805: validation loss: -0.6097 2022-04-12 00:26:17.504134: Average global foreground Dice: [0.8106, 0.957, 0.9728, 0.4089] 2022-04-12 00:26:17.505597: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:26:18.374242: lr: 0.005972 2022-04-12 00:26:18.375950: saving scheduled checkpoint file... 2022-04-12 00:26:18.462103: saving checkpoint... 2022-04-12 00:26:23.473512: done, saving took 5.10 seconds 2022-04-12 00:26:23.491749: done 2022-04-12 00:26:23.578396: saving checkpoint... 2022-04-12 00:26:28.326580: done, saving took 4.83 seconds 2022-04-12 00:26:28.365046: This epoch took 130.913106 s 2022-04-12 00:26:28.366410: svmem(total=99932160000, available=71362265088, percent=28.6, used=24286265344, free=26127065088, active=49357950976, inactive=22006112256, buffers=61427712, cached=49457401856, shared=2196729856, slab=1318621184) 2022-04-12 00:26:28.367645: epoch: 218 2022-04-12 00:28:17.970857: train loss : -0.5655 2022-04-12 00:28:29.701576: validation loss: -0.4724 2022-04-12 00:28:29.703505: Average global foreground Dice: [0.7974, 0.9306, 0.9363, 0.3806] 2022-04-12 00:28:29.704553: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:28:30.555091: lr: 0.005953 2022-04-12 00:28:30.557055: saving scheduled checkpoint file... 2022-04-12 00:28:30.663857: saving checkpoint... 2022-04-12 00:28:35.166541: done, saving took 4.61 seconds 2022-04-12 00:28:35.186325: done 2022-04-12 00:28:35.187990: This epoch took 126.819267 s 2022-04-12 00:28:35.188971: svmem(total=99932160000, available=71543857152, percent=28.4, used=24105816064, free=26358173696, active=49165406208, inactive=21976047616, buffers=61534208, cached=49406636032, shared=2195550208, slab=1319923712) 2022-04-12 00:28:35.189791: epoch: 219 2022-04-12 00:30:24.837927: train loss : -0.5204 2022-04-12 00:30:36.624363: validation loss: -0.5802 2022-04-12 00:30:36.626235: Average global foreground Dice: [0.8195, 0.9494, 0.9687, 0.3859] 2022-04-12 00:30:36.627408: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:30:37.330137: lr: 0.005934 2022-04-12 00:30:37.331450: saving scheduled checkpoint file... 2022-04-12 00:30:37.443523: saving checkpoint... 2022-04-12 00:30:41.838429: done, saving took 4.51 seconds 2022-04-12 00:30:41.857821: done 2022-04-12 00:30:41.859061: This epoch took 126.668545 s 2022-04-12 00:30:41.860547: svmem(total=99932160000, available=71583518720, percent=28.4, used=24052678656, free=26396884992, active=49138937856, inactive=21962158080, buffers=61665280, cached=49420931072, shared=2209054720, slab=1318682624) 2022-04-12 00:30:41.861492: epoch: 220 2022-04-12 00:32:31.685643: train loss : -0.3334 2022-04-12 00:32:44.344051: validation loss: -0.3641 2022-04-12 00:32:44.355817: Average global foreground Dice: [0.7404, 0.9046, 0.8869, 0.3627] 2022-04-12 00:32:44.356694: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:32:45.168700: lr: 0.005915 2022-04-12 00:32:45.170603: saving scheduled checkpoint file... 2022-04-12 00:32:45.328563: saving checkpoint... 2022-04-12 00:32:50.634475: done, saving took 5.46 seconds 2022-04-12 00:32:50.657343: done 2022-04-12 00:32:50.658710: This epoch took 128.796296 s 2022-04-12 00:32:50.660032: svmem(total=99932160000, available=71505043456, percent=28.4, used=24147451904, free=26317467648, active=49233154048, inactive=21946646528, buffers=61751296, cached=49405489152, shared=2192654336, slab=1321705472) 2022-04-12 00:32:50.660828: epoch: 221 2022-04-12 00:34:40.854730: train loss : -0.3997 2022-04-12 00:34:53.240738: validation loss: -0.4696 2022-04-12 00:34:53.247768: Average global foreground Dice: [0.732, 0.9215, 0.9555, 0.3623] 2022-04-12 00:34:53.251806: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:34:54.284529: lr: 0.005896 2022-04-12 00:34:54.286357: saving scheduled checkpoint file... 2022-04-12 00:34:54.376500: saving checkpoint... 2022-04-12 00:34:59.663831: done, saving took 5.38 seconds 2022-04-12 00:34:59.684034: done 2022-04-12 00:34:59.687693: This epoch took 129.026129 s 2022-04-12 00:34:59.689625: svmem(total=99932160000, available=71357538304, percent=28.6, used=24277860352, free=26169151488, active=49366110208, inactive=21959553024, buffers=61816832, cached=49423331328, shared=2209726464, slab=1321897984) 2022-04-12 00:34:59.693787: epoch: 222 2022-04-12 00:36:48.164782: train loss : -0.5211 2022-04-12 00:36:59.392601: validation loss: -0.5763 2022-04-12 00:36:59.394923: Average global foreground Dice: [0.8375, 0.9468, 0.9698, 0.4066] 2022-04-12 00:36:59.396412: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:37:00.420668: lr: 0.005877 2022-04-12 00:37:00.437390: saving scheduled checkpoint file... 2022-04-12 00:37:00.522285: saving checkpoint... 2022-04-12 00:37:06.988816: done, saving took 6.53 seconds 2022-04-12 00:37:07.108195: done 2022-04-12 00:37:07.131418: This epoch took 127.407994 s 2022-04-12 00:37:07.135195: svmem(total=99932160000, available=71280967680, percent=28.7, used=24367652864, free=26091786240, active=49451384832, inactive=21949816832, buffers=61911040, cached=49410809856, shared=2196414464, slab=1321910272) 2022-04-12 00:37:07.160468: epoch: 223 2022-04-12 00:38:55.023641: train loss : -0.5733 2022-04-12 00:39:06.121627: validation loss: -0.5909 2022-04-12 00:39:06.124219: Average global foreground Dice: [0.8173, 0.9558, 0.9762, 0.3913] 2022-04-12 00:39:06.125291: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:39:07.120584: lr: 0.005858 2022-04-12 00:39:07.186065: saving scheduled checkpoint file... 2022-04-12 00:39:07.253241: saving checkpoint... 2022-04-12 00:39:14.288985: done, saving took 7.10 seconds 2022-04-12 00:39:14.334164: done 2022-04-12 00:39:14.335988: This epoch took 127.118740 s 2022-04-12 00:39:14.338246: svmem(total=99932160000, available=71315668992, percent=28.6, used=24333144064, free=26125705216, active=49415315456, inactive=21949575168, buffers=61976576, cached=49411334144, shared=2196242432, slab=1321607168) 2022-04-12 00:39:14.340308: epoch: 224 2022-04-12 00:41:03.052464: train loss : -0.5918 2022-04-12 00:41:15.178594: validation loss: -0.6059 2022-04-12 00:41:15.184650: Average global foreground Dice: [0.8454, 0.9492, 0.9729, 0.4131] 2022-04-12 00:41:15.186087: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:41:15.884117: lr: 0.005839 2022-04-12 00:41:15.889019: saving scheduled checkpoint file... 2022-04-12 00:41:15.995661: saving checkpoint... 2022-04-12 00:41:20.889116: done, saving took 4.99 seconds 2022-04-12 00:41:20.900722: done 2022-04-12 00:41:20.902123: This epoch took 126.552873 s 2022-04-12 00:41:20.902971: svmem(total=99932160000, available=71344381952, percent=28.6, used=24304861184, free=26153381888, active=49397440512, inactive=21947252736, buffers=62054400, cached=49411862528, shared=2195869696, slab=1320968192) 2022-04-12 00:41:20.903713: epoch: 225 2022-04-12 00:43:09.375358: train loss : -0.5825 2022-04-12 00:43:21.915281: validation loss: -0.5850 2022-04-12 00:43:21.926531: Average global foreground Dice: [0.8099, 0.9512, 0.9695, 0.3717] 2022-04-12 00:43:21.942973: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:43:22.748935: lr: 0.00582 2022-04-12 00:43:22.792147: saving scheduled checkpoint file... 2022-04-12 00:43:22.940953: saving checkpoint... 2022-04-12 00:43:28.705631: done, saving took 5.89 seconds 2022-04-12 00:43:28.727030: done 2022-04-12 00:43:28.728015: This epoch took 127.823083 s 2022-04-12 00:43:28.728985: svmem(total=99932160000, available=71419854848, percent=28.5, used=24228872192, free=26227920896, active=49313296384, inactive=21948162048, buffers=62136320, cached=49413230592, shared=2196185088, slab=1320026112) 2022-04-12 00:43:28.729790: epoch: 226 2022-04-12 00:45:16.433523: train loss : -0.5862 2022-04-12 00:45:27.752280: validation loss: -0.5989 2022-04-12 00:45:27.768127: Average global foreground Dice: [0.8296, 0.952, 0.9713, 0.4037] 2022-04-12 00:45:27.793574: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:45:28.719740: lr: 0.005801 2022-04-12 00:45:28.721607: saving scheduled checkpoint file... 2022-04-12 00:45:28.833944: saving checkpoint... 2022-04-12 00:45:33.621282: done, saving took 4.90 seconds 2022-04-12 00:45:33.637952: done 2022-04-12 00:45:33.638897: This epoch took 124.908369 s 2022-04-12 00:45:33.639826: svmem(total=99932160000, available=71237283840, percent=28.7, used=24411725824, free=26044579840, active=49503391744, inactive=21948964864, buffers=62214144, cached=49413640192, shared=2195972096, slab=1319571456) 2022-04-12 00:45:33.641065: epoch: 227 2022-04-12 00:47:24.057903: train loss : -0.5814 2022-04-12 00:47:36.395188: validation loss: -0.5603 2022-04-12 00:47:36.403930: Average global foreground Dice: [0.8397, 0.9531, 0.9687, 0.3819] 2022-04-12 00:47:36.405648: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:47:37.477935: lr: 0.005781 2022-04-12 00:47:37.479265: saving scheduled checkpoint file... 2022-04-12 00:47:37.584649: saving checkpoint... 2022-04-12 00:47:42.273088: done, saving took 4.79 seconds 2022-04-12 00:47:42.327619: done 2022-04-12 00:47:42.328922: This epoch took 128.686994 s 2022-04-12 00:47:42.330180: svmem(total=99932160000, available=71358742528, percent=28.6, used=24289574912, free=26165198848, active=49382641664, inactive=21947375616, buffers=62296064, cached=49415090176, shared=2196602880, slab=1320886272) 2022-04-12 00:47:42.331383: epoch: 228 2022-04-12 00:49:30.867507: train loss : -0.5902 2022-04-12 00:49:42.424906: validation loss: -0.5655 2022-04-12 00:49:42.435350: Average global foreground Dice: [0.8362, 0.9505, 0.9562, 0.3982] 2022-04-12 00:49:42.436610: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:49:43.317171: lr: 0.005762 2022-04-12 00:49:43.319124: saving scheduled checkpoint file... 2022-04-12 00:49:43.386347: saving checkpoint... 2022-04-12 00:49:48.108331: done, saving took 4.79 seconds 2022-04-12 00:49:48.165859: done 2022-04-12 00:49:48.167496: This epoch took 125.835005 s 2022-04-12 00:49:48.169045: svmem(total=99932160000, available=71405264896, percent=28.5, used=24241008640, free=26211000320, active=49329315840, inactive=21950201856, buffers=62361600, cached=49417789440, shared=2198634496, slab=1321041920) 2022-04-12 00:49:48.169986: epoch: 229 2022-04-12 00:51:37.004005: train loss : -0.5362 2022-04-12 00:51:49.599803: validation loss: -0.5564 2022-04-12 00:51:49.607241: Average global foreground Dice: [0.7982, 0.9429, 0.9611, 0.3919] 2022-04-12 00:51:49.608698: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:51:50.316437: lr: 0.005743 2022-04-12 00:51:50.318407: saving scheduled checkpoint file... 2022-04-12 00:51:50.437171: saving checkpoint... 2022-04-12 00:51:55.816105: done, saving took 5.48 seconds 2022-04-12 00:51:55.956213: done 2022-04-12 00:51:55.960891: This epoch took 127.789962 s 2022-04-12 00:51:55.967047: svmem(total=99932160000, available=71419170816, percent=28.5, used=24221433856, free=26179477504, active=49301565440, inactive=22010228736, buffers=62459904, cached=49468788736, shared=2203070464, slab=1322401792) 2022-04-12 00:51:55.991128: epoch: 230 2022-04-12 00:53:45.660192: train loss : -0.5818 2022-04-12 00:53:58.042864: validation loss: -0.6108 2022-04-12 00:53:58.068361: Average global foreground Dice: [0.8332, 0.9578, 0.9757, 0.4072] 2022-04-12 00:53:58.085255: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:53:58.779177: lr: 0.005724 2022-04-12 00:53:58.789838: saving scheduled checkpoint file... 2022-04-12 00:53:58.903444: saving checkpoint... 2022-04-12 00:54:03.884835: done, saving took 5.08 seconds 2022-04-12 00:54:03.957911: done 2022-04-12 00:54:03.959072: This epoch took 127.926799 s 2022-04-12 00:54:03.960345: svmem(total=99932160000, available=71486230528, percent=28.5, used=24161218560, free=26277789696, active=49017159680, inactive=22191583232, buffers=62537728, cached=49430614016, shared=2196242432, slab=1323155456) 2022-04-12 00:54:03.961040: epoch: 231 2022-04-12 00:55:53.413075: train loss : -0.5970 2022-04-12 00:56:05.601800: validation loss: -0.6085 2022-04-12 00:56:05.604021: Average global foreground Dice: [0.8349, 0.9538, 0.9725, 0.4057] 2022-04-12 00:56:05.604714: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:56:06.519893: lr: 0.005705 2022-04-12 00:56:06.521695: saving scheduled checkpoint file... 2022-04-12 00:56:06.641790: saving checkpoint... 2022-04-12 00:56:11.278926: done, saving took 4.76 seconds 2022-04-12 00:56:11.298620: done 2022-04-12 00:56:11.300048: This epoch took 127.338043 s 2022-04-12 00:56:11.301054: svmem(total=99932160000, available=71335563264, percent=28.6, used=24301744128, free=26171199488, active=49247510528, inactive=22076395520, buffers=62652416, cached=49396563968, shared=2206248960, slab=1322467328) 2022-04-12 00:56:11.301891: epoch: 232 2022-04-12 00:58:01.182516: train loss : -0.6021 2022-04-12 00:58:12.897132: validation loss: -0.5856 2022-04-12 00:58:12.906037: Average global foreground Dice: [0.8322, 0.9608, 0.9775, 0.3973] 2022-04-12 00:58:12.907490: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 00:58:13.742226: lr: 0.005686 2022-04-12 00:58:13.755536: saving scheduled checkpoint file... 2022-04-12 00:58:13.898750: saving checkpoint... 2022-04-12 00:58:18.363076: done, saving took 4.60 seconds 2022-04-12 00:58:18.381422: done 2022-04-12 00:58:18.382890: This epoch took 127.080183 s 2022-04-12 00:58:18.383840: svmem(total=99932160000, available=71456206848, percent=28.5, used=24186568704, free=26226176000, active=49208475648, inactive=22060154880, buffers=62717952, cached=49456697344, shared=2200784896, slab=1320550400) 2022-04-12 00:58:18.384611: epoch: 233 2022-04-12 01:00:09.605939: train loss : -0.6129 2022-04-12 01:00:22.122394: validation loss: -0.6325 2022-04-12 01:00:22.131558: Average global foreground Dice: [0.8506, 0.9561, 0.9822, 0.4325] 2022-04-12 01:00:22.132595: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:00:23.155045: lr: 0.005667 2022-04-12 01:00:23.160785: saving scheduled checkpoint file... 2022-04-12 01:00:23.260902: saving checkpoint... 2022-04-12 01:00:30.809220: done, saving took 7.64 seconds 2022-04-12 01:00:31.009133: done 2022-04-12 01:00:31.077469: saving checkpoint... 2022-04-12 01:00:36.989205: done, saving took 5.96 seconds 2022-04-12 01:00:37.005400: This epoch took 138.619855 s 2022-04-12 01:00:37.011008: svmem(total=99932160000, available=71532478464, percent=28.4, used=24114864128, free=26301800448, active=49150930944, inactive=22039351296, buffers=62820352, cached=49452675072, shared=2196242432, slab=1321836544) 2022-04-12 01:00:37.014466: epoch: 234 2022-04-12 01:02:24.740292: train loss : -0.6114 2022-04-12 01:02:35.699327: validation loss: -0.5992 2022-04-12 01:02:35.706120: Average global foreground Dice: [0.8481, 0.9584, 0.9738, 0.3928] 2022-04-12 01:02:35.707072: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:02:36.256965: lr: 0.005647 2022-04-12 01:02:36.258073: saving scheduled checkpoint file... 2022-04-12 01:02:36.309875: saving checkpoint... 2022-04-12 01:02:42.161604: done, saving took 5.90 seconds 2022-04-12 01:02:42.246682: done 2022-04-12 01:02:42.330126: saving checkpoint... 2022-04-12 01:02:47.521497: done, saving took 5.24 seconds 2022-04-12 01:02:47.537686: This epoch took 130.519912 s 2022-04-12 01:02:47.538715: svmem(total=99932160000, available=71477608448, percent=28.5, used=24169906176, free=26246840320, active=49210134528, inactive=22037184512, buffers=62877696, cached=49452535808, shared=2196054016, slab=1320579072) 2022-04-12 01:02:47.542520: epoch: 235 2022-04-12 01:04:35.814419: train loss : -0.6106 2022-04-12 01:04:48.425498: validation loss: -0.6200 2022-04-12 01:04:48.429395: Average global foreground Dice: [0.8244, 0.9567, 0.978, 0.415] 2022-04-12 01:04:48.430847: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:04:49.306162: lr: 0.005628 2022-04-12 01:04:49.307439: saving scheduled checkpoint file... 2022-04-12 01:04:49.403929: saving checkpoint... 2022-04-12 01:04:54.204575: done, saving took 4.90 seconds 2022-04-12 01:04:54.222028: done 2022-04-12 01:04:54.293078: saving checkpoint... 2022-04-12 01:04:59.115159: done, saving took 4.89 seconds 2022-04-12 01:04:59.130102: This epoch took 131.581067 s 2022-04-12 01:04:59.131426: svmem(total=99932160000, available=71499399168, percent=28.5, used=24151379968, free=26268667904, active=49220169728, inactive=22002651136, buffers=62939136, cached=49449172992, shared=2192662528, slab=1321525248) 2022-04-12 01:04:59.132783: epoch: 236 2022-04-12 01:06:47.757521: train loss : -0.6246 2022-04-12 01:06:58.986077: validation loss: -0.6386 2022-04-12 01:06:58.988523: Average global foreground Dice: [0.8464, 0.9604, 0.976, 0.4473] 2022-04-12 01:06:58.990211: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:06:59.993994: lr: 0.005609 2022-04-12 01:06:59.995566: saving scheduled checkpoint file... 2022-04-12 01:07:00.080980: saving checkpoint... 2022-04-12 01:07:04.823201: done, saving took 4.83 seconds 2022-04-12 01:07:04.890206: done 2022-04-12 01:07:04.983354: saving checkpoint... 2022-04-12 01:07:09.631839: done, saving took 4.74 seconds 2022-04-12 01:07:09.653554: This epoch took 130.519855 s 2022-04-12 01:07:09.654892: svmem(total=99932160000, available=71429865472, percent=28.5, used=24217841664, free=26200371200, active=49287749632, inactive=22001418240, buffers=63049728, cached=49450897408, shared=2195791872, slab=1321148416) 2022-04-12 01:07:09.655686: epoch: 237 2022-04-12 01:08:57.641980: train loss : -0.6158 2022-04-12 01:09:09.069754: validation loss: -0.6272 2022-04-12 01:09:09.103070: Average global foreground Dice: [0.8419, 0.958, 0.979, 0.4176] 2022-04-12 01:09:09.106674: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:09:10.130021: lr: 0.00559 2022-04-12 01:09:10.138679: saving scheduled checkpoint file... 2022-04-12 01:09:10.267815: saving checkpoint... 2022-04-12 01:09:15.734725: done, saving took 5.59 seconds 2022-04-12 01:09:15.784000: done 2022-04-12 01:09:15.887486: saving checkpoint... 2022-04-12 01:09:21.531169: done, saving took 5.74 seconds 2022-04-12 01:09:21.551011: This epoch took 131.894627 s 2022-04-12 01:09:21.552661: svmem(total=99932160000, available=71435337728, percent=28.5, used=24212176896, free=26251927552, active=49286426624, inactive=21956304896, buffers=63115264, cached=49404940288, shared=2196328448, slab=1321762816) 2022-04-12 01:09:21.553407: epoch: 238 2022-04-12 01:11:10.400894: train loss : -0.6169 2022-04-12 01:11:22.813085: validation loss: -0.6331 2022-04-12 01:11:22.815275: Average global foreground Dice: [0.8488, 0.9551, 0.9783, 0.4014] 2022-04-12 01:11:22.816159: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:11:23.702776: lr: 0.005571 2022-04-12 01:11:23.704917: saving scheduled checkpoint file... 2022-04-12 01:11:23.840249: saving checkpoint... 2022-04-12 01:11:28.433595: done, saving took 4.73 seconds 2022-04-12 01:11:28.456478: done 2022-04-12 01:11:28.580049: saving checkpoint... 2022-04-12 01:11:33.262393: done, saving took 4.80 seconds 2022-04-12 01:11:33.280550: This epoch took 131.725748 s 2022-04-12 01:11:33.282375: svmem(total=99932160000, available=71314501632, percent=28.6, used=24322187264, free=26130444288, active=49399156736, inactive=21961691136, buffers=63197184, cached=49416331264, shared=2207035392, slab=1322319872) 2022-04-12 01:11:33.283226: epoch: 239 2022-04-12 01:13:22.270558: train loss : -0.6308 2022-04-12 01:13:34.540074: validation loss: -0.6198 2022-04-12 01:13:34.542845: Average global foreground Dice: [0.8376, 0.9605, 0.9794, 0.4186] 2022-04-12 01:13:34.543757: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:13:35.411761: lr: 0.005551 2022-04-12 01:13:35.413124: saving scheduled checkpoint file... 2022-04-12 01:13:35.499939: saving checkpoint... 2022-04-12 01:13:40.173122: done, saving took 4.76 seconds 2022-04-12 01:13:40.201471: done 2022-04-12 01:13:40.252287: saving checkpoint... 2022-04-12 01:13:47.197344: done, saving took 6.99 seconds 2022-04-12 01:13:47.206465: This epoch took 133.922577 s 2022-04-12 01:13:47.207621: svmem(total=99932160000, available=76637761536, percent=23.3, used=19024142336, free=31460315136, active=44689600512, inactive=21468479488, buffers=63311872, cached=49384390656, shared=2181488640, slab=1312587776) 2022-04-12 01:13:47.208594: epoch: 240 2022-04-12 01:15:35.897511: train loss : -0.6298 2022-04-12 01:15:47.781368: validation loss: -0.6516 2022-04-12 01:15:47.820693: Average global foreground Dice: [0.8707, 0.9649, 0.9765, 0.42] 2022-04-12 01:15:47.859769: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:15:48.723939: lr: 0.005532 2022-04-12 01:15:48.725574: saving scheduled checkpoint file... 2022-04-12 01:15:48.774480: saving checkpoint... 2022-04-12 01:15:53.992865: done, saving took 5.27 seconds 2022-04-12 01:15:54.005500: done 2022-04-12 01:15:54.054067: saving checkpoint... 2022-04-12 01:16:00.205503: done, saving took 6.20 seconds 2022-04-12 01:16:00.265453: This epoch took 133.052104 s 2022-04-12 01:16:00.269103: svmem(total=99932160000, available=76913278976, percent=23.0, used=18948669440, free=36260061184, active=44471918592, inactive=17253339136, buffers=63578112, cached=44659851264, shared=2161823744, slab=944316416) 2022-04-12 01:16:00.271626: epoch: 241 2022-04-12 01:17:48.217424: train loss : -0.6068 2022-04-12 01:17:59.633359: validation loss: -0.5933 2022-04-12 01:17:59.635588: Average global foreground Dice: [0.8361, 0.9497, 0.9675, 0.3791] 2022-04-12 01:17:59.636895: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:18:00.519106: lr: 0.005513 2022-04-12 01:18:00.520478: saving scheduled checkpoint file... 2022-04-12 01:18:00.602296: saving checkpoint... 2022-04-12 01:18:06.181794: done, saving took 5.66 seconds 2022-04-12 01:18:06.254112: done 2022-04-12 01:18:06.297839: This epoch took 126.023822 s 2022-04-12 01:18:06.307686: svmem(total=99932160000, available=77068992512, percent=22.9, used=18846801920, free=36415336448, active=44368449536, inactive=17200041984, buffers=63647744, cached=44606373888, shared=2107977728, slab=943554560) 2022-04-12 01:18:06.350567: epoch: 242 2022-04-12 01:19:54.533334: train loss : -0.5557 2022-04-12 01:20:06.448721: validation loss: -0.5875 2022-04-12 01:20:06.456874: Average global foreground Dice: [0.8314, 0.9525, 0.974, 0.3837] 2022-04-12 01:20:06.461962: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:20:07.336693: lr: 0.005494 2022-04-12 01:20:07.341270: saving scheduled checkpoint file... 2022-04-12 01:20:07.419104: saving checkpoint... 2022-04-12 01:20:12.865053: done, saving took 5.52 seconds 2022-04-12 01:20:12.925894: done 2022-04-12 01:20:12.927208: This epoch took 126.558061 s 2022-04-12 01:20:12.929183: svmem(total=99932160000, available=77043802112, percent=22.9, used=18817667072, free=36389814272, active=44338212864, inactive=17254846464, buffers=63713280, cached=44660965376, shared=2162454528, slab=944861184) 2022-04-12 01:20:12.930763: epoch: 243 2022-04-12 01:22:01.916508: train loss : -0.5433 2022-04-12 01:22:14.633271: validation loss: -0.5835 2022-04-12 01:22:14.635546: Average global foreground Dice: [0.8295, 0.9546, 0.967, 0.3798] 2022-04-12 01:22:14.646744: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:22:15.386412: lr: 0.005474 2022-04-12 01:22:15.387925: saving scheduled checkpoint file... 2022-04-12 01:22:15.486894: saving checkpoint... 2022-04-12 01:22:20.217085: done, saving took 4.83 seconds 2022-04-12 01:22:20.233030: done 2022-04-12 01:22:20.234287: This epoch took 127.302450 s 2022-04-12 01:22:20.235461: svmem(total=99932160000, available=77055877120, percent=22.9, used=18853953536, free=36401381376, active=44375990272, inactive=17206734848, buffers=63766528, cached=44613058560, shared=2114056192, slab=943190016) 2022-04-12 01:22:20.236668: epoch: 244 2022-04-12 01:24:08.236623: train loss : -0.5908 2022-04-12 01:24:20.141863: validation loss: -0.5956 2022-04-12 01:24:20.145953: Average global foreground Dice: [0.8338, 0.9549, 0.9697, 0.3746] 2022-04-12 01:24:20.146989: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:24:21.108581: lr: 0.005455 2022-04-12 01:24:21.109928: saving scheduled checkpoint file... 2022-04-12 01:24:21.216496: saving checkpoint... 2022-04-12 01:24:25.927462: done, saving took 4.82 seconds 2022-04-12 01:24:25.947361: done 2022-04-12 01:24:25.948288: This epoch took 125.710771 s 2022-04-12 01:24:25.949274: svmem(total=99932160000, available=77024821248, percent=22.9, used=18866483200, free=36369932288, active=44390199296, inactive=17225764864, buffers=63827968, cached=44631916544, shared=2132615168, slab=942510080) 2022-04-12 01:24:25.950089: epoch: 245 2022-04-12 01:26:13.805465: train loss : -0.5913 2022-04-12 01:26:25.519262: validation loss: -0.5844 2022-04-12 01:26:25.549480: Average global foreground Dice: [0.8231, 0.9498, 0.9695, 0.3904] 2022-04-12 01:26:25.555518: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:26:26.603835: lr: 0.005436 2022-04-12 01:26:26.623756: saving scheduled checkpoint file... 2022-04-12 01:26:26.773043: saving checkpoint... 2022-04-12 01:26:34.815850: done, saving took 8.13 seconds 2022-04-12 01:26:34.830361: done 2022-04-12 01:26:34.832119: This epoch took 128.881277 s 2022-04-12 01:26:34.833296: svmem(total=99932160000, available=76899471360, percent=23.0, used=18946850816, free=36244115456, active=44471083008, inactive=17270910976, buffers=63926272, cached=44677267456, shared=2177536000, slab=944562176) 2022-04-12 01:26:34.834601: epoch: 246 2022-04-12 01:28:23.731966: train loss : -0.5792 2022-04-12 01:28:36.083181: validation loss: -0.6151 2022-04-12 01:28:36.085439: Average global foreground Dice: [0.8232, 0.9551, 0.9774, 0.3888] 2022-04-12 01:28:36.087147: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:28:37.016861: lr: 0.005417 2022-04-12 01:28:37.018711: saving scheduled checkpoint file... 2022-04-12 01:28:37.086956: saving checkpoint... 2022-04-12 01:28:42.199007: done, saving took 5.18 seconds 2022-04-12 01:28:42.220450: done 2022-04-12 01:28:42.224317: This epoch took 127.388774 s 2022-04-12 01:28:42.228280: svmem(total=99932160000, available=76752683008, percent=23.2, used=19089391616, free=36096802816, active=44613021696, inactive=17275551744, buffers=63971328, cached=44681994240, shared=2181750784, slab=944095232) 2022-04-12 01:28:42.230489: epoch: 247 2022-04-12 01:30:31.682312: train loss : -0.6099 2022-04-12 01:30:43.607497: validation loss: -0.6126 2022-04-12 01:30:43.611810: Average global foreground Dice: [0.8543, 0.9527, 0.9656, 0.4154] 2022-04-12 01:30:43.613235: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:30:44.538400: lr: 0.005397 2022-04-12 01:30:44.550548: saving scheduled checkpoint file... 2022-04-12 01:30:44.634400: saving checkpoint... 2022-04-12 01:30:50.396277: done, saving took 5.84 seconds 2022-04-12 01:30:50.415570: done 2022-04-12 01:30:50.416745: This epoch took 128.185066 s 2022-04-12 01:30:50.417969: svmem(total=99932160000, available=77037891584, percent=22.9, used=18853552128, free=36381716480, active=44375937024, inactive=17227169792, buffers=64024576, cached=44632866816, shared=2132426752, slab=942440448) 2022-04-12 01:30:50.419228: epoch: 248 2022-04-12 01:32:38.010616: train loss : -0.6056 2022-04-12 01:32:51.251490: validation loss: -0.5853 2022-04-12 01:32:51.257314: Average global foreground Dice: [0.8335, 0.9564, 0.9729, 0.3875] 2022-04-12 01:32:51.258480: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:32:52.147838: lr: 0.005378 2022-04-12 01:32:52.149697: saving scheduled checkpoint file... 2022-04-12 01:32:52.270148: saving checkpoint... 2022-04-12 01:32:56.985894: done, saving took 4.84 seconds 2022-04-12 01:32:57.000723: done 2022-04-12 01:32:57.001904: This epoch took 126.581947 s 2022-04-12 01:32:57.002791: svmem(total=99932160000, available=77111042048, percent=22.8, used=18853490688, free=36454404096, active=44370767872, inactive=17154097152, buffers=64081920, cached=44560183296, shared=2059374592, slab=942952448) 2022-04-12 01:32:57.003572: epoch: 249 2022-04-12 01:34:46.607684: train loss : -0.6002 2022-04-12 01:34:58.315486: validation loss: -0.6214 2022-04-12 01:34:58.354886: Average global foreground Dice: [0.8366, 0.9574, 0.9784, 0.4111] 2022-04-12 01:34:58.356454: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:34:59.166221: lr: 0.005359 2022-04-12 01:34:59.168047: saving scheduled checkpoint file... 2022-04-12 01:34:59.311465: saving checkpoint... 2022-04-12 01:35:04.015431: done, saving took 4.85 seconds 2022-04-12 01:35:04.038036: done 2022-04-12 01:35:04.039139: This epoch took 127.034700 s 2022-04-12 01:35:04.040163: svmem(total=99932160000, available=77048840192, percent=22.9, used=18831769600, free=36391682048, active=44357771264, inactive=17238167552, buffers=64135168, cached=44644573184, shared=2143203328, slab=943058944) 2022-04-12 01:35:04.041385: epoch: 250 2022-04-12 01:36:54.265495: train loss : -0.6339 2022-04-12 01:37:06.512813: validation loss: -0.6389 2022-04-12 01:37:06.562506: Average global foreground Dice: [0.8648, 0.9652, 0.9746, 0.3898] 2022-04-12 01:37:06.563687: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:37:07.327175: lr: 0.00534 2022-04-12 01:37:07.328826: saving scheduled checkpoint file... 2022-04-12 01:37:07.479991: saving checkpoint... 2022-04-12 01:37:12.033818: done, saving took 4.70 seconds 2022-04-12 01:37:12.056372: done 2022-04-12 01:37:12.057379: This epoch took 128.015191 s 2022-04-12 01:37:12.058530: svmem(total=99932160000, available=76959076352, percent=23.0, used=18882985984, free=36301598720, active=44410179584, inactive=17276829696, buffers=64204800, cached=44683370496, shared=2181672960, slab=943620096) 2022-04-12 01:37:12.059940: epoch: 251 2022-04-12 01:39:01.877570: train loss : -0.6320 2022-04-12 01:39:14.702594: validation loss: -0.6577 2022-04-12 01:39:14.708295: Average global foreground Dice: [0.8491, 0.9606, 0.9812, 0.4366] 2022-04-12 01:39:14.711216: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:39:15.545707: lr: 0.00532 2022-04-12 01:39:15.547628: saving scheduled checkpoint file... 2022-04-12 01:39:15.640346: saving checkpoint... 2022-04-12 01:39:20.794815: done, saving took 5.25 seconds 2022-04-12 01:39:20.809759: done 2022-04-12 01:39:20.811053: This epoch took 128.749569 s 2022-04-12 01:39:20.812507: svmem(total=99932160000, available=77012606976, percent=22.9, used=18893156352, free=36354543616, active=44413939712, inactive=17213837312, buffers=64262144, cached=44620197888, shared=2117976064, slab=945459200) 2022-04-12 01:39:20.813284: epoch: 252 2022-04-12 01:41:13.621181: train loss : -0.6425 2022-04-12 01:41:24.641301: validation loss: -0.6465 2022-04-12 01:41:24.657600: Average global foreground Dice: [0.858, 0.9613, 0.9816, 0.4147] 2022-04-12 01:41:24.661548: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:41:25.274323: lr: 0.005301 2022-04-12 01:41:25.275693: saving scheduled checkpoint file... 2022-04-12 01:41:25.329683: saving checkpoint... 2022-04-12 01:41:30.143096: done, saving took 4.87 seconds 2022-04-12 01:41:30.237496: done 2022-04-12 01:41:30.285833: saving checkpoint... 2022-04-12 01:41:38.186554: done, saving took 7.95 seconds 2022-04-12 01:41:38.559160: This epoch took 137.744715 s 2022-04-12 01:41:38.697421: svmem(total=99932160000, available=76819578880, percent=23.1, used=19036520448, free=36161200128, active=44542783488, inactive=17263427584, buffers=64331776, cached=44670107648, shared=2167488512, slab=957341696) 2022-04-12 01:41:38.833564: epoch: 253 2022-04-12 01:43:27.081374: train loss : -0.6296 2022-04-12 01:43:38.206067: validation loss: -0.6316 2022-04-12 01:43:38.239900: Average global foreground Dice: [0.8268, 0.9617, 0.9778, 0.4244] 2022-04-12 01:43:38.256480: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:43:39.269861: lr: 0.005282 2022-04-12 01:43:39.276110: saving scheduled checkpoint file... 2022-04-12 01:43:39.415543: saving checkpoint... 2022-04-12 01:43:55.018791: done, saving took 15.66 seconds 2022-04-12 01:43:55.237594: done 2022-04-12 01:43:55.313448: saving checkpoint... 2022-04-12 01:43:59.881056: done, saving took 4.64 seconds 2022-04-12 01:43:59.893608: This epoch took 140.840911 s 2022-04-12 01:43:59.894658: svmem(total=99932160000, available=76858470400, percent=23.1, used=18967076864, free=36199493632, active=44483760128, inactive=17294561280, buffers=64376832, cached=44701212672, shared=2198208512, slab=944955392) 2022-04-12 01:43:59.895500: epoch: 254 2022-04-12 01:45:48.957104: train loss : -0.6111 2022-04-12 01:46:01.241076: validation loss: -0.6423 2022-04-12 01:46:01.247795: Average global foreground Dice: [0.8695, 0.9574, 0.9789, 0.4041] 2022-04-12 01:46:01.250203: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:46:01.967247: lr: 0.005262 2022-04-12 01:46:01.968877: saving scheduled checkpoint file... 2022-04-12 01:46:02.056532: saving checkpoint... 2022-04-12 01:46:07.978221: done, saving took 6.01 seconds 2022-04-12 01:46:07.996765: done 2022-04-12 01:46:08.088886: saving checkpoint... 2022-04-12 01:46:14.595456: done, saving took 6.60 seconds 2022-04-12 01:46:14.613217: This epoch took 134.716652 s 2022-04-12 01:46:14.614606: svmem(total=99932160000, available=76559609856, percent=23.4, used=19282255872, free=35900227584, active=44802220032, inactive=17278324736, buffers=64466944, cached=44685209600, shared=2181971968, slab=943357952) 2022-04-12 01:46:14.615390: epoch: 255 2022-04-12 01:48:02.798169: train loss : -0.6152 2022-04-12 01:48:13.705709: validation loss: -0.6190 2022-04-12 01:48:13.734138: Average global foreground Dice: [0.8471, 0.9596, 0.977, 0.4134] 2022-04-12 01:48:13.794078: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:48:14.745360: lr: 0.005243 2022-04-12 01:48:14.746320: saving scheduled checkpoint file... 2022-04-12 01:48:14.848688: saving checkpoint... 2022-04-12 01:48:20.165033: done, saving took 5.41 seconds 2022-04-12 01:48:20.182575: done 2022-04-12 01:48:20.288830: saving checkpoint... 2022-04-12 01:48:25.117112: done, saving took 4.93 seconds 2022-04-12 01:48:25.146947: This epoch took 130.530757 s 2022-04-12 01:48:25.162403: svmem(total=99932160000, available=76719726592, percent=23.2, used=19122319360, free=36059840512, active=44644175872, inactive=17278844928, buffers=64512000, cached=44685488128, shared=2181689344, slab=943022080) 2022-04-12 01:48:25.185106: epoch: 256 2022-04-12 01:50:14.519092: train loss : -0.5789 2022-04-12 01:50:25.477107: validation loss: -0.5922 2022-04-12 01:50:25.520691: Average global foreground Dice: [0.8451, 0.9502, 0.9688, 0.4035] 2022-04-12 01:50:25.535793: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:50:26.671990: lr: 0.005224 2022-04-12 01:50:26.679545: saving scheduled checkpoint file... 2022-04-12 01:50:26.900864: saving checkpoint... 2022-04-12 01:50:37.248290: done, saving took 10.50 seconds 2022-04-12 01:50:37.437154: done 2022-04-12 01:50:37.442515: This epoch took 132.234963 s 2022-04-12 01:50:37.444320: svmem(total=99932160000, available=76710354944, percent=23.2, used=19130224640, free=36050063360, active=44653727744, inactive=17280765952, buffers=64581632, cached=44687290368, shared=2183237632, slab=943431680) 2022-04-12 01:50:37.445861: epoch: 257 2022-04-12 01:52:27.924444: train loss : -0.5698 2022-04-12 01:52:40.698794: validation loss: -0.5999 2022-04-12 01:52:40.703065: Average global foreground Dice: [0.8523, 0.9496, 0.9703, 0.3986] 2022-04-12 01:52:40.704374: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:52:41.712546: lr: 0.005204 2022-04-12 01:52:41.714760: saving scheduled checkpoint file... 2022-04-12 01:52:41.802148: saving checkpoint... 2022-04-12 01:52:46.298578: done, saving took 4.58 seconds 2022-04-12 01:52:46.313917: done 2022-04-12 01:52:46.315332: This epoch took 128.868127 s 2022-04-12 01:52:46.316407: svmem(total=99932160000, available=76826832896, percent=23.1, used=19027095552, free=36166062080, active=44550721536, inactive=17267728384, buffers=64638976, cached=44674363392, shared=2169794560, slab=943247360) 2022-04-12 01:52:46.317114: epoch: 258 2022-04-12 01:54:33.989061: train loss : -0.5631 2022-04-12 01:54:45.400652: validation loss: -0.5912 2022-04-12 01:54:45.531390: Average global foreground Dice: [0.83, 0.9442, 0.9646, 0.3856] 2022-04-12 01:54:45.673743: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:54:46.876993: lr: 0.005185 2022-04-12 01:54:46.939383: saving scheduled checkpoint file... 2022-04-12 01:54:46.998161: saving checkpoint... 2022-04-12 01:54:55.749303: done, saving took 8.81 seconds 2022-04-12 01:54:55.944254: done 2022-04-12 01:54:55.948133: This epoch took 129.630278 s 2022-04-12 01:54:55.951059: svmem(total=99932160000, available=76813004800, percent=23.1, used=19043213312, free=36151906304, active=44557860864, inactive=17265889280, buffers=64696320, cached=44672344064, shared=2167574528, slab=953556992) 2022-04-12 01:54:55.954140: epoch: 259 2022-04-12 01:56:44.922930: train loss : -0.5649 2022-04-12 01:56:56.122182: validation loss: -0.5912 2022-04-12 01:56:56.188681: Average global foreground Dice: [0.8339, 0.9512, 0.9708, 0.3929] 2022-04-12 01:56:56.267742: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:56:57.174097: lr: 0.005166 2022-04-12 01:56:57.507210: saving scheduled checkpoint file... 2022-04-12 01:56:57.557893: saving checkpoint... 2022-04-12 01:57:11.831667: done, saving took 14.32 seconds 2022-04-12 01:57:11.918174: done 2022-04-12 01:57:11.919671: This epoch took 135.964228 s 2022-04-12 01:57:11.920649: svmem(total=99932160000, available=76674375680, percent=23.3, used=19181789184, free=36012732416, active=44695535616, inactive=17266348032, buffers=64782336, cached=44672856064, shared=2167574528, slab=954920960) 2022-04-12 01:57:11.921983: epoch: 260 2022-04-12 01:58:59.235859: train loss : -0.6083 2022-04-12 01:59:11.747066: validation loss: -0.6208 2022-04-12 01:59:11.762987: Average global foreground Dice: [0.8407, 0.9588, 0.976, 0.409] 2022-04-12 01:59:11.781533: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 01:59:12.546397: lr: 0.005146 2022-04-12 01:59:12.558807: saving scheduled checkpoint file... 2022-04-12 01:59:12.690605: saving checkpoint... 2022-04-12 01:59:19.457295: done, saving took 6.83 seconds 2022-04-12 01:59:19.544507: done 2022-04-12 01:59:19.562896: This epoch took 127.640165 s 2022-04-12 01:59:19.564450: svmem(total=99932160000, available=76848353280, percent=23.1, used=19007574016, free=36581330944, active=44532846592, inactive=16871628800, buffers=64827392, cached=44278427648, shared=2167574528, slab=943017984) 2022-04-12 01:59:19.565330: epoch: 261 2022-04-12 02:01:07.032792: train loss : -0.6115 2022-04-12 02:01:18.831182: validation loss: -0.6141 2022-04-12 02:01:18.861901: Average global foreground Dice: [0.8399, 0.9586, 0.979, 0.4013] 2022-04-12 02:01:18.868266: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:01:19.747393: lr: 0.005127 2022-04-12 02:01:19.759928: saving scheduled checkpoint file... 2022-04-12 02:01:19.881463: saving checkpoint... 2022-04-12 02:01:25.085867: done, saving took 5.31 seconds 2022-04-12 02:01:25.106958: done 2022-04-12 02:01:25.110357: This epoch took 125.544288 s 2022-04-12 02:01:25.113307: svmem(total=99932160000, available=76864942080, percent=23.1, used=18972229632, free=36202442752, active=44493365248, inactive=17286447104, buffers=64880640, cached=44692606976, shared=2186620928, slab=943910912) 2022-04-12 02:01:25.116426: epoch: 262 2022-04-12 02:03:14.572006: train loss : -0.6276 2022-04-12 02:03:26.837449: validation loss: -0.6232 2022-04-12 02:03:26.870759: Average global foreground Dice: [0.8591, 0.9611, 0.9776, 0.4002] 2022-04-12 02:03:26.947444: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:03:27.687812: lr: 0.005107 2022-04-12 02:03:27.698831: saving scheduled checkpoint file... 2022-04-12 02:03:27.867587: saving checkpoint... 2022-04-12 02:03:37.781928: done, saving took 10.03 seconds 2022-04-12 02:03:37.801897: done 2022-04-12 02:03:37.805429: This epoch took 132.686358 s 2022-04-12 02:03:37.808320: svmem(total=99932160000, available=76612497408, percent=23.3, used=19219144704, free=35949629440, active=44741210112, inactive=17291816960, buffers=64942080, cached=44698443776, shared=2192015360, slab=942702592) 2022-04-12 02:03:37.810809: epoch: 263 2022-04-12 02:05:27.195181: train loss : -0.6205 2022-04-12 02:05:38.550275: validation loss: -0.6499 2022-04-12 02:05:38.572282: Average global foreground Dice: [0.8432, 0.9578, 0.9817, 0.4402] 2022-04-12 02:05:38.618189: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:05:39.785232: lr: 0.005088 2022-04-12 02:05:39.844577: saving scheduled checkpoint file... 2022-04-12 02:05:39.956466: saving checkpoint... 2022-04-12 02:05:45.798871: done, saving took 5.94 seconds 2022-04-12 02:05:45.814733: done 2022-04-12 02:05:45.817409: This epoch took 128.004884 s 2022-04-12 02:05:45.820051: svmem(total=99932160000, available=76675244032, percent=23.3, used=19161010176, free=36011831296, active=44678770688, inactive=17287581696, buffers=65032192, cached=44694286336, shared=2187423744, slab=944476160) 2022-04-12 02:05:45.822296: epoch: 264 2022-04-12 02:07:35.267246: train loss : -0.5800 2022-04-12 02:07:47.811112: validation loss: -0.5938 2022-04-12 02:07:47.822681: Average global foreground Dice: [0.8132, 0.9559, 0.969, 0.3939] 2022-04-12 02:07:47.830713: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:07:48.656784: lr: 0.005069 2022-04-12 02:07:48.718834: saving scheduled checkpoint file... 2022-04-12 02:07:48.800283: saving checkpoint... 2022-04-12 02:07:58.189404: done, saving took 9.45 seconds 2022-04-12 02:07:58.321455: done 2022-04-12 02:07:58.363629: This epoch took 132.538718 s 2022-04-12 02:07:58.408273: svmem(total=99932160000, available=76817686528, percent=23.1, used=19024334848, free=36153810944, active=44542664704, inactive=17282088960, buffers=65077248, cached=44688936960, shared=2181562368, slab=943951872) 2022-04-12 02:07:58.448591: epoch: 265 2022-04-12 02:09:46.703948: train loss : -0.5927 2022-04-12 02:09:58.908396: validation loss: -0.6092 2022-04-12 02:09:59.009927: Average global foreground Dice: [0.8179, 0.9543, 0.9707, 0.414] 2022-04-12 02:09:59.027498: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:10:00.066371: lr: 0.005049 2022-04-12 02:10:00.132065: saving scheduled checkpoint file... 2022-04-12 02:10:00.238826: saving checkpoint... 2022-04-12 02:10:07.676864: done, saving took 7.54 seconds 2022-04-12 02:10:07.748305: done 2022-04-12 02:10:07.783932: This epoch took 129.328996 s 2022-04-12 02:10:07.791396: svmem(total=99932160000, available=76532482048, percent=23.4, used=19298689024, free=35868065792, active=44821999616, inactive=17290969088, buffers=65142784, cached=44700262400, shared=2192465920, slab=945377280) 2022-04-12 02:10:07.794469: epoch: 266 2022-04-12 02:11:55.391842: train loss : -0.6111 2022-04-12 02:12:08.029537: validation loss: -0.6274 2022-04-12 02:12:08.040725: Average global foreground Dice: [0.8489, 0.9585, 0.9767, 0.4115] 2022-04-12 02:12:08.044722: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:12:08.732785: lr: 0.00503 2022-04-12 02:12:08.734987: saving scheduled checkpoint file... 2022-04-12 02:12:08.833264: saving checkpoint... 2022-04-12 02:12:14.888243: done, saving took 6.15 seconds 2022-04-12 02:12:14.971658: done 2022-04-12 02:12:14.977093: This epoch took 127.180740 s 2022-04-12 02:12:14.980052: svmem(total=99932160000, available=76645867520, percent=23.3, used=19196350464, free=35981127680, active=44720975872, inactive=17282818048, buffers=65187840, cached=44689494016, shared=2181476352, slab=943157248) 2022-04-12 02:12:14.982739: epoch: 267 2022-04-12 02:14:03.546651: train loss : -0.6282 2022-04-12 02:14:15.983954: validation loss: -0.6415 2022-04-12 02:14:15.996746: Average global foreground Dice: [0.8722, 0.9585, 0.9781, 0.4154] 2022-04-12 02:14:16.008319: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:14:16.982958: lr: 0.00501 2022-04-12 02:14:17.002347: saving scheduled checkpoint file... 2022-04-12 02:14:17.129885: saving checkpoint... 2022-04-12 02:14:28.114082: done, saving took 11.10 seconds 2022-04-12 02:14:28.166243: done 2022-04-12 02:14:28.167213: This epoch took 133.183491 s 2022-04-12 02:14:28.177422: svmem(total=99932160000, available=76809445376, percent=23.1, used=19034591232, free=36144214016, active=44554444800, inactive=17281380352, buffers=65249280, cached=44688105472, shared=2179600384, slab=944021504) 2022-04-12 02:14:28.186425: epoch: 268 2022-04-12 02:16:18.139015: train loss : -0.6407 2022-04-12 02:16:30.576874: validation loss: -0.6383 2022-04-12 02:16:30.598200: Average global foreground Dice: [0.8714, 0.9596, 0.9786, 0.4062] 2022-04-12 02:16:30.600591: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:16:31.262597: lr: 0.004991 2022-04-12 02:16:31.289053: saving scheduled checkpoint file... 2022-04-12 02:16:31.455513: saving checkpoint... 2022-04-12 02:16:38.212096: done, saving took 6.89 seconds 2022-04-12 02:16:38.234280: done 2022-04-12 02:16:38.342264: saving checkpoint... 2022-04-12 02:16:43.259517: done, saving took 5.02 seconds 2022-04-12 02:16:43.277684: This epoch took 135.076317 s 2022-04-12 02:16:43.279482: svmem(total=99932160000, available=76706201600, percent=23.2, used=19136237568, free=36040454144, active=44658044928, inactive=17283223552, buffers=65335296, cached=44690132992, shared=2181152768, slab=944521216) 2022-04-12 02:16:43.280294: epoch: 269 2022-04-12 02:18:33.180112: train loss : -0.6343 2022-04-12 02:18:45.548754: validation loss: -0.6428 2022-04-12 02:18:45.560430: Average global foreground Dice: [0.8608, 0.9662, 0.9809, 0.4174] 2022-04-12 02:18:45.561645: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:18:46.393933: lr: 0.004971 2022-04-12 02:18:46.404347: saving scheduled checkpoint file... 2022-04-12 02:18:46.495017: saving checkpoint... 2022-04-12 02:18:54.042099: done, saving took 7.61 seconds 2022-04-12 02:18:54.055089: done 2022-04-12 02:18:54.103338: saving checkpoint... 2022-04-12 02:19:00.943682: done, saving took 6.89 seconds 2022-04-12 02:19:00.958794: This epoch took 137.677731 s 2022-04-12 02:19:00.960218: svmem(total=99932160000, available=77021818880, percent=22.9, used=18830295040, free=36355526656, active=44357283840, inactive=17274732544, buffers=65400832, cached=44680937472, shared=2171564032, slab=943230976) 2022-04-12 02:19:00.961733: epoch: 270 2022-04-12 02:20:48.507937: train loss : -0.6501 2022-04-12 02:21:00.117087: validation loss: -0.6721 2022-04-12 02:21:00.365627: Average global foreground Dice: [0.8416, 0.9711, 0.9819, 0.4211] 2022-04-12 02:21:00.371677: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:21:01.242274: lr: 0.004952 2022-04-12 02:21:01.243954: saving scheduled checkpoint file... 2022-04-12 02:21:01.299931: saving checkpoint... 2022-04-12 02:21:05.853820: done, saving took 4.61 seconds 2022-04-12 02:21:05.869478: done 2022-04-12 02:21:05.916690: saving checkpoint... 2022-04-12 02:21:11.655916: done, saving took 5.79 seconds 2022-04-12 02:21:11.739506: This epoch took 130.776536 s 2022-04-12 02:21:11.741880: svmem(total=99932160000, available=76705214464, percent=23.2, used=19136638976, free=36038553600, active=44661960704, inactive=17285300224, buffers=65454080, cached=44691513344, shared=2181783552, slab=942325760) 2022-04-12 02:21:11.763103: epoch: 271 2022-04-12 02:23:00.577804: train loss : -0.6313 2022-04-12 02:23:12.905513: validation loss: -0.6570 2022-04-12 02:23:12.945687: Average global foreground Dice: [0.8472, 0.9612, 0.9818, 0.4289] 2022-04-12 02:23:12.949980: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:23:13.683079: lr: 0.004933 2022-04-12 02:23:13.687198: saving scheduled checkpoint file... 2022-04-12 02:23:13.742589: saving checkpoint... 2022-04-12 02:23:25.799479: done, saving took 12.11 seconds 2022-04-12 02:23:25.901421: done 2022-04-12 02:23:25.989584: saving checkpoint... 2022-04-12 02:23:33.019387: done, saving took 7.08 seconds 2022-04-12 02:23:33.037604: This epoch took 141.241416 s 2022-04-12 02:23:33.039751: svmem(total=99932160000, available=77157937152, percent=22.8, used=18802819072, free=36490756096, active=44329033728, inactive=17166467072, buffers=65548288, cached=44573036544, shared=2062872576, slab=943730688) 2022-04-12 02:23:33.040732: epoch: 272 2022-04-12 02:25:20.733759: train loss : -0.6373 2022-04-12 02:25:32.599561: validation loss: -0.6581 2022-04-12 02:25:32.619826: Average global foreground Dice: [0.8662, 0.9694, 0.9811, 0.4458] 2022-04-12 02:25:32.626386: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:25:33.484531: lr: 0.004913 2022-04-12 02:25:33.492193: saving scheduled checkpoint file... 2022-04-12 02:25:33.569610: saving checkpoint... 2022-04-12 02:25:42.146968: done, saving took 8.64 seconds 2022-04-12 02:25:42.209752: done 2022-04-12 02:25:42.271409: saving checkpoint... 2022-04-12 02:25:49.271709: done, saving took 7.05 seconds 2022-04-12 02:25:49.923676: This epoch took 136.881724 s 2022-04-12 02:25:50.172259: svmem(total=99932160000, available=76710219776, percent=23.2, used=19145879552, free=36042608640, active=44656128000, inactive=17271369728, buffers=65626112, cached=44678045696, shared=2167574528, slab=957001728) 2022-04-12 02:25:50.325724: epoch: 273 2022-04-12 02:27:37.978730: train loss : -0.6412 2022-04-12 02:27:50.012309: validation loss: -0.6128 2022-04-12 02:27:50.043577: Average global foreground Dice: [0.8573, 0.9567, 0.9689, 0.4009] 2022-04-12 02:27:50.049498: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:27:50.816961: lr: 0.004894 2022-04-12 02:27:50.818695: saving scheduled checkpoint file... 2022-04-12 02:27:50.908971: saving checkpoint... 2022-04-12 02:27:56.959393: done, saving took 6.14 seconds 2022-04-12 02:27:56.992786: done 2022-04-12 02:27:57.003291: This epoch took 126.489492 s 2022-04-12 02:27:57.013064: svmem(total=99932160000, available=77147222016, percent=22.8, used=18768048128, free=36479037440, active=44296785920, inactive=17212579840, buffers=65683456, cached=44619390976, shared=2108317696, slab=943136768) 2022-04-12 02:27:57.031638: epoch: 274 2022-04-12 02:29:45.971856: train loss : -0.6349 2022-04-12 02:29:57.908011: validation loss: -0.6331 2022-04-12 02:29:57.956107: Average global foreground Dice: [0.8543, 0.9613, 0.9762, 0.4075] 2022-04-12 02:29:57.969375: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:29:58.686916: lr: 0.004874 2022-04-12 02:29:58.712779: saving scheduled checkpoint file... 2022-04-12 02:29:58.848107: saving checkpoint... 2022-04-12 02:30:05.974624: done, saving took 7.22 seconds 2022-04-12 02:30:05.994901: done 2022-04-12 02:30:05.996437: This epoch took 128.958571 s 2022-04-12 02:30:05.997710: svmem(total=99932160000, available=76644376576, percent=23.3, used=19197964288, free=35975921664, active=44722311168, inactive=17285681152, buffers=65736704, cached=44692537344, shared=2181251072, slab=943710208) 2022-04-12 02:30:05.998406: epoch: 275 2022-04-12 02:31:54.553861: train loss : -0.6405 2022-04-12 02:32:06.746458: validation loss: -0.6542 2022-04-12 02:32:06.765022: Average global foreground Dice: [0.8736, 0.9607, 0.9781, 0.4441] 2022-04-12 02:32:06.772799: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:32:07.712830: lr: 0.004855 2022-04-12 02:32:07.736918: saving scheduled checkpoint file... 2022-04-12 02:32:07.880114: saving checkpoint... 2022-04-12 02:32:18.977828: done, saving took 11.19 seconds 2022-04-12 02:32:19.130403: done 2022-04-12 02:32:19.243484: saving checkpoint... 2022-04-12 02:32:26.256743: done, saving took 7.06 seconds 2022-04-12 02:32:26.343966: This epoch took 140.344581 s 2022-04-12 02:32:26.372036: svmem(total=99932160000, available=76913233920, percent=23.0, used=18918776832, free=36244193280, active=44436635648, inactive=17296883712, buffers=65798144, cached=44703391744, shared=2191499264, slab=944123904) 2022-04-12 02:32:26.407474: epoch: 276 2022-04-12 02:34:15.132350: train loss : -0.6394 2022-04-12 02:34:26.701434: validation loss: -0.6372 2022-04-12 02:34:26.787476: Average global foreground Dice: [0.8633, 0.9641, 0.9759, 0.3976] 2022-04-12 02:34:26.806890: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:34:27.533076: lr: 0.004835 2022-04-12 02:34:27.548212: saving scheduled checkpoint file... 2022-04-12 02:34:27.661573: saving checkpoint... 2022-04-12 02:34:39.640338: done, saving took 12.04 seconds 2022-04-12 02:34:39.819904: done 2022-04-12 02:34:39.902528: This epoch took 133.477752 s 2022-04-12 02:34:39.907685: svmem(total=99932160000, available=77139636224, percent=22.8, used=18839695360, free=36470120448, active=44360908800, inactive=17149841408, buffers=65855488, cached=44556488704, shared=2044235776, slab=944934912) 2022-04-12 02:34:39.910320: epoch: 277 2022-04-12 02:36:29.126730: train loss : -0.6308 2022-04-12 02:36:41.493155: validation loss: -0.6588 2022-04-12 02:36:41.519010: Average global foreground Dice: [0.846, 0.9658, 0.9779, 0.4332] 2022-04-12 02:36:41.556813: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:36:42.450440: lr: 0.004816 2022-04-12 02:36:42.493165: saving scheduled checkpoint file... 2022-04-12 02:36:42.581528: saving checkpoint... 2022-04-12 02:36:52.285376: done, saving took 9.79 seconds 2022-04-12 02:36:52.364835: done 2022-04-12 02:36:52.457022: saving checkpoint... 2022-04-12 02:37:01.742977: done, saving took 9.37 seconds 2022-04-12 02:37:01.872746: This epoch took 141.909690 s 2022-04-12 02:37:01.877120: svmem(total=99932160000, available=77190332416, percent=22.8, used=18802331648, free=36520366080, active=44323631104, inactive=17136922624, buffers=65929216, cached=44543533056, shared=2030886912, slab=943902720) 2022-04-12 02:37:01.880240: epoch: 278 2022-04-12 02:38:50.074991: train loss : -0.6332 2022-04-12 02:39:02.071730: validation loss: -0.6388 2022-04-12 02:39:02.079031: Average global foreground Dice: [0.8465, 0.9603, 0.9821, 0.4154] 2022-04-12 02:39:02.156984: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:39:03.041223: lr: 0.004796 2022-04-12 02:39:03.044933: saving scheduled checkpoint file... 2022-04-12 02:39:03.149700: saving checkpoint... 2022-04-12 02:39:11.285267: done, saving took 8.24 seconds 2022-04-12 02:39:11.307594: done 2022-04-12 02:39:11.310537: This epoch took 129.425152 s 2022-04-12 02:39:11.312490: svmem(total=99932160000, available=77051543552, percent=22.9, used=18812567552, free=36381122560, active=44338315264, inactive=17265537024, buffers=65974272, cached=44672495616, shared=2159476736, slab=943722496) 2022-04-12 02:39:11.322821: epoch: 279 2022-04-12 02:41:01.186114: train loss : -0.6334 2022-04-12 02:41:13.540961: validation loss: -0.6330 2022-04-12 02:41:13.578246: Average global foreground Dice: [0.8588, 0.963, 0.978, 0.4041] 2022-04-12 02:41:13.618741: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:41:14.480302: lr: 0.004776 2022-04-12 02:41:14.528486: saving scheduled checkpoint file... 2022-04-12 02:41:14.641391: saving checkpoint... 2022-04-12 02:41:23.481569: done, saving took 8.95 seconds 2022-04-12 02:41:23.661931: done 2022-04-12 02:41:23.711658: This epoch took 132.368828 s 2022-04-12 02:41:23.719358: svmem(total=99932160000, available=76691931136, percent=23.3, used=19163852800, free=36021059584, active=44690493440, inactive=17274081280, buffers=66039808, cached=44681207808, shared=2167660544, slab=941846528) 2022-04-12 02:41:23.779883: epoch: 280 2022-04-12 02:43:13.634085: train loss : -0.6349 2022-04-12 02:43:26.277254: validation loss: -0.6543 2022-04-12 02:43:26.356270: Average global foreground Dice: [0.8542, 0.9614, 0.9761, 0.421] 2022-04-12 02:43:26.425189: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:43:27.287503: lr: 0.004757 2022-04-12 02:43:27.330397: saving scheduled checkpoint file... 2022-04-12 02:43:27.514952: saving checkpoint... 2022-04-12 02:43:38.620499: done, saving took 11.25 seconds 2022-04-12 02:43:38.657251: done 2022-04-12 02:43:38.807940: saving checkpoint... 2022-04-12 02:43:54.138877: done, saving took 15.46 seconds 2022-04-12 02:43:54.408102: This epoch took 150.569750 s 2022-04-12 02:43:54.417764: svmem(total=99932160000, available=76760162304, percent=23.2, used=19081695232, free=36088799232, active=44608880640, inactive=17288507392, buffers=66101248, cached=44695564288, shared=2181640192, slab=944148480) 2022-04-12 02:43:54.477861: epoch: 281 2022-04-12 02:45:44.190732: train loss : -0.6228 2022-04-12 02:45:55.706424: validation loss: -0.6213 2022-04-12 02:45:58.188349: Average global foreground Dice: [0.8476, 0.9623, 0.9742, 0.38] 2022-04-12 02:45:58.743960: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:46:00.019916: lr: 0.004737 2022-04-12 02:46:00.375461: saving scheduled checkpoint file... 2022-04-12 02:46:00.637543: saving checkpoint... 2022-04-12 02:46:14.869859: done, saving took 14.31 seconds 2022-04-12 02:46:15.675388: done 2022-04-12 02:46:15.811999: This epoch took 141.260405 s 2022-04-12 02:46:16.008451: svmem(total=99932160000, available=78428864512, percent=21.5, used=18147495936, free=76078067712, active=18057691136, inactive=4097155072, buffers=7745536, cached=5698850816, shared=1571512320, slab=690417664) 2022-04-12 02:46:16.241458: epoch: 282 2022-04-12 02:52:24.916185: train loss : -0.5822 2022-04-12 02:53:08.784967: validation loss: -0.5689 2022-04-12 02:53:08.971264: Average global foreground Dice: [0.8447, 0.949, 0.9547, 0.3707] 2022-04-12 02:53:09.042171: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:53:09.814973: lr: 0.004718 2022-04-12 02:53:09.948712: saving scheduled checkpoint file... 2022-04-12 02:53:10.172275: saving checkpoint... 2022-04-12 02:53:22.561373: done, saving took 12.48 seconds 2022-04-12 02:53:23.040149: done 2022-04-12 02:53:23.190596: This epoch took 426.757593 s 2022-04-12 02:53:23.293976: svmem(total=99932160000, available=76763168768, percent=23.2, used=19473854464, free=57908277248, active=27424538624, inactive=12502241280, buffers=10305536, cached=22539722752, shared=1891643392, slab=755458048) 2022-04-12 02:53:23.440803: epoch: 283 2022-04-12 02:55:54.073271: train loss : -0.6054 2022-04-12 02:56:17.431466: validation loss: -0.6309 2022-04-12 02:56:17.577646: Average global foreground Dice: [0.8084, 0.957, 0.9791, 0.4225] 2022-04-12 02:56:17.667546: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:56:18.555138: lr: 0.004698 2022-04-12 02:56:18.769543: saving scheduled checkpoint file... 2022-04-12 02:56:19.157402: saving checkpoint... 2022-04-12 02:56:31.155256: done, saving took 12.05 seconds 2022-04-12 02:56:31.415486: done 2022-04-12 02:56:31.463705: This epoch took 187.870285 s 2022-04-12 02:56:31.508893: svmem(total=99932160000, available=76339081216, percent=23.6, used=19732758528, free=52461846528, active=34801414144, inactive=10554945536, buffers=10645504, cached=27726909440, shared=2051272704, slab=766316544) 2022-04-12 02:56:31.597409: epoch: 284 2022-04-12 02:58:20.334271: train loss : -0.6183 2022-04-12 02:58:33.834141: validation loss: -0.6462 2022-04-12 02:58:33.868306: Average global foreground Dice: [0.866, 0.9574, 0.9802, 0.428] 2022-04-12 02:58:33.922439: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 02:58:34.867951: lr: 0.004679 2022-04-12 02:58:34.873551: saving scheduled checkpoint file... 2022-04-12 02:58:34.972102: saving checkpoint... 2022-04-12 02:58:41.540411: done, saving took 6.65 seconds 2022-04-12 02:58:41.649330: done 2022-04-12 02:58:41.674862: This epoch took 130.069504 s 2022-04-12 02:58:41.726273: svmem(total=99932160000, available=76089602048, percent=23.9, used=19770830848, free=50506698752, active=38522363904, inactive=8784261120, buffers=10805248, cached=29643825152, shared=2260365312, slab=771825664) 2022-04-12 02:58:41.750762: epoch: 285 2022-04-12 03:00:30.444517: train loss : -0.6456 2022-04-12 03:00:43.061517: validation loss: -0.6350 2022-04-12 03:00:43.088009: Average global foreground Dice: [0.8176, 0.9597, 0.9783, 0.4172] 2022-04-12 03:00:43.097619: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:00:44.079392: lr: 0.004659 2022-04-12 03:00:44.081314: saving scheduled checkpoint file... 2022-04-12 03:00:44.199306: saving checkpoint... 2022-04-12 03:00:52.815217: done, saving took 8.73 seconds 2022-04-12 03:00:52.986931: done 2022-04-12 03:00:52.992086: This epoch took 131.201613 s 2022-04-12 03:00:53.021218: svmem(total=99932160000, available=75991994368, percent=24.0, used=19881426944, free=49330044928, active=40529092608, inactive=7953510400, buffers=10969088, cached=30709719040, shared=2246414336, slab=774176768) 2022-04-12 03:00:53.085411: epoch: 286 2022-04-12 03:02:41.183559: train loss : -0.6393 2022-04-12 03:02:53.085618: validation loss: -0.6673 2022-04-12 03:02:53.153911: Average global foreground Dice: [0.8873, 0.9571, 0.981, 0.3972] 2022-04-12 03:02:53.224760: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:02:54.172632: lr: 0.004639 2022-04-12 03:02:54.243409: saving scheduled checkpoint file... 2022-04-12 03:02:54.414179: saving checkpoint... 2022-04-12 03:03:02.586329: done, saving took 8.31 seconds 2022-04-12 03:03:02.652425: done 2022-04-12 03:03:02.653711: This epoch took 129.564057 s 2022-04-12 03:03:02.654638: svmem(total=99932160000, available=76054949888, percent=23.9, used=19808829440, free=48884629504, active=41613201408, inactive=7313276928, buffers=11124736, cached=31227576320, shared=2254856192, slab=773423104) 2022-04-12 03:03:02.655885: epoch: 287 2022-04-12 03:04:52.497993: train loss : -0.6543 2022-04-12 03:05:04.268939: validation loss: -0.6615 2022-04-12 03:05:04.301932: Average global foreground Dice: [0.8439, 0.9666, 0.9837, 0.459] 2022-04-12 03:05:04.359339: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:05:05.367740: lr: 0.00462 2022-04-12 03:05:05.427741: saving scheduled checkpoint file... 2022-04-12 03:05:05.540509: saving checkpoint... 2022-04-12 03:05:13.695773: done, saving took 8.26 seconds 2022-04-12 03:05:13.815089: done 2022-04-12 03:05:13.818038: This epoch took 131.161366 s 2022-04-12 03:05:13.860182: svmem(total=99932160000, available=75770544128, percent=24.2, used=20072951808, free=48406970368, active=42617917440, inactive=6784425984, buffers=11337728, cached=31440900096, shared=2275315712, slab=774852608) 2022-04-12 03:05:13.903860: epoch: 288 2022-04-12 03:07:02.282916: train loss : -0.6370 2022-04-12 03:07:14.278126: validation loss: -0.6506 2022-04-12 03:07:14.296790: Average global foreground Dice: [0.8462, 0.9657, 0.9806, 0.4233] 2022-04-12 03:07:14.366940: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:07:15.255696: lr: 0.0046 2022-04-12 03:07:15.271002: saving scheduled checkpoint file... 2022-04-12 03:07:15.379746: saving checkpoint... 2022-04-12 03:07:30.049838: done, saving took 14.72 seconds 2022-04-12 03:07:30.179854: done 2022-04-12 03:07:30.277207: This epoch took 136.342169 s 2022-04-12 03:07:30.316283: svmem(total=99932160000, available=75621167104, percent=24.3, used=20236185600, free=48152596480, active=43292917760, inactive=6364688384, buffers=11599872, cached=31531778048, shared=2260942848, slab=774688768) 2022-04-12 03:07:30.328092: epoch: 289 2022-04-12 03:09:19.068629: train loss : -0.6596 2022-04-12 03:09:29.518730: validation loss: -0.6525 2022-04-12 03:09:29.618348: Average global foreground Dice: [0.8737, 0.9666, 0.9809, 0.419] 2022-04-12 03:09:29.622772: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:09:30.619989: lr: 0.004581 2022-04-12 03:09:30.802604: saving scheduled checkpoint file... 2022-04-12 03:09:31.018830: saving checkpoint... 2022-04-12 03:09:43.453207: done, saving took 12.49 seconds 2022-04-12 03:09:43.551389: done 2022-04-12 03:09:43.616870: saving checkpoint... 2022-04-12 03:09:49.067727: done, saving took 5.50 seconds 2022-04-12 03:09:49.144919: This epoch took 138.797862 s 2022-04-12 03:09:49.160595: svmem(total=99932160000, available=75888099328, percent=24.1, used=19969277952, free=47878529024, active=43372814336, inactive=6557642752, buffers=11853824, cached=32072499200, shared=2261037056, slab=776839168) 2022-04-12 03:09:49.173909: epoch: 290 2022-04-12 03:11:37.479662: train loss : -0.6509 2022-04-12 03:11:49.514239: validation loss: -0.6380 2022-04-12 03:11:49.593907: Average global foreground Dice: [0.8551, 0.966, 0.9801, 0.4154] 2022-04-12 03:11:49.614702: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:11:50.359233: lr: 0.004561 2022-04-12 03:11:50.374329: saving scheduled checkpoint file... 2022-04-12 03:11:50.457911: saving checkpoint... 2022-04-12 03:12:02.481678: done, saving took 12.10 seconds 2022-04-12 03:12:02.500212: done 2022-04-12 03:12:02.571966: saving checkpoint... 2022-04-12 03:12:16.173878: done, saving took 13.67 seconds 2022-04-12 03:12:16.274682: This epoch took 147.059350 s 2022-04-12 03:12:16.276090: svmem(total=99932160000, available=76055650304, percent=23.9, used=19801378816, free=48166367232, active=43023380480, inactive=6617157632, buffers=12152832, cached=31952261120, shared=2260951040, slab=778244096) 2022-04-12 03:12:16.276832: epoch: 291 2022-04-12 03:14:05.003773: train loss : -0.6400 2022-04-12 03:14:17.256955: validation loss: -0.6514 2022-04-12 03:14:17.328186: Average global foreground Dice: [0.8677, 0.9631, 0.977, 0.4326] 2022-04-12 03:14:17.416846: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:14:18.249897: lr: 0.004541 2022-04-12 03:14:18.276461: saving scheduled checkpoint file... 2022-04-12 03:14:18.375837: saving checkpoint... 2022-04-12 03:14:24.598018: done, saving took 6.30 seconds 2022-04-12 03:14:24.689532: done 2022-04-12 03:14:24.786226: saving checkpoint... 2022-04-12 03:14:41.960076: done, saving took 17.26 seconds 2022-04-12 03:14:42.197441: This epoch took 145.919829 s 2022-04-12 03:14:42.215165: svmem(total=99932160000, available=75926716416, percent=24.0, used=19930361856, free=47988736000, active=43269611520, inactive=6546366464, buffers=12427264, cached=32000634880, shared=2260959232, slab=776744960) 2022-04-12 03:14:42.244041: epoch: 292 2022-04-12 03:16:30.887350: train loss : -0.6547 2022-04-12 03:16:42.542207: validation loss: -0.6478 2022-04-12 03:16:42.564255: Average global foreground Dice: [0.8765, 0.9611, 0.979, 0.3993] 2022-04-12 03:16:42.568437: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:16:43.578540: lr: 0.004522 2022-04-12 03:16:43.596298: saving scheduled checkpoint file... 2022-04-12 03:16:43.776148: saving checkpoint... 2022-04-12 03:16:54.410778: done, saving took 10.76 seconds 2022-04-12 03:16:54.447068: done 2022-04-12 03:16:54.553228: saving checkpoint... 2022-04-12 03:17:01.516792: done, saving took 7.06 seconds 2022-04-12 03:17:01.664735: This epoch took 139.354513 s 2022-04-12 03:17:01.670553: svmem(total=99932160000, available=76182953984, percent=23.8, used=19684835328, free=48332378112, active=43096309760, inactive=6378401792, buffers=12709888, cached=31902236672, shared=2250211328, slab=776388608) 2022-04-12 03:17:01.727354: epoch: 293 2022-04-12 03:18:50.702617: train loss : -0.5916 2022-04-12 03:19:01.649419: validation loss: -0.5626 2022-04-12 03:19:01.706363: Average global foreground Dice: [0.8376, 0.9483, 0.9622, 0.3708] 2022-04-12 03:19:01.712522: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:19:02.915454: lr: 0.004502 2022-04-12 03:19:02.979153: saving scheduled checkpoint file... 2022-04-12 03:19:03.096263: saving checkpoint... 2022-04-12 03:19:13.056554: done, saving took 10.05 seconds 2022-04-12 03:19:13.117426: done 2022-04-12 03:19:13.130890: This epoch took 131.399669 s 2022-04-12 03:19:13.153589: svmem(total=99932160000, available=76191129600, percent=23.8, used=19664797696, free=48289652736, active=43141812224, inactive=6378795008, buffers=12967936, cached=31964741632, shared=2262020096, slab=776433664) 2022-04-12 03:19:13.157052: epoch: 294 2022-04-12 03:21:02.839338: train loss : -0.5980 2022-04-12 03:21:15.009951: validation loss: -0.6557 2022-04-12 03:21:15.015075: Average global foreground Dice: [0.865, 0.9619, 0.9816, 0.4221] 2022-04-12 03:21:15.017584: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:21:15.987565: lr: 0.004482 2022-04-12 03:21:16.025602: saving scheduled checkpoint file... 2022-04-12 03:21:16.124528: saving checkpoint... 2022-04-12 03:21:26.785918: done, saving took 10.75 seconds 2022-04-12 03:21:26.867759: done 2022-04-12 03:21:26.899013: This epoch took 133.740354 s 2022-04-12 03:21:26.913152: svmem(total=99932160000, available=76368482304, percent=23.6, used=19488350208, free=48422002688, active=43016716288, inactive=6364405760, buffers=13234176, cached=32008572928, shared=2261098496, slab=777859072) 2022-04-12 03:21:26.939377: epoch: 295 2022-04-12 03:23:14.077502: train loss : -0.5959 2022-04-12 03:23:26.469349: validation loss: -0.5400 2022-04-12 03:23:26.497831: Average global foreground Dice: [0.8444, 0.9381, 0.935, 0.4115] 2022-04-12 03:23:26.585585: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:23:27.439023: lr: 0.004463 2022-04-12 03:23:27.443258: saving scheduled checkpoint file... 2022-04-12 03:23:27.600083: saving checkpoint... 2022-04-12 03:23:38.849432: done, saving took 11.34 seconds 2022-04-12 03:23:39.009346: done 2022-04-12 03:23:39.020474: This epoch took 132.065051 s 2022-04-12 03:23:39.038992: svmem(total=99932160000, available=76075921408, percent=23.9, used=19780902912, free=48106508288, active=43358142464, inactive=6340321280, buffers=13492224, cached=32031256576, shared=2261037056, slab=777048064) 2022-04-12 03:23:39.042207: epoch: 296 2022-04-12 03:25:27.727721: train loss : -0.5820 2022-04-12 03:25:39.027476: validation loss: -0.5828 2022-04-12 03:25:39.090601: Average global foreground Dice: [0.8491, 0.948, 0.9552, 0.3771] 2022-04-12 03:25:39.102664: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:25:40.026607: lr: 0.004443 2022-04-12 03:25:40.048340: saving scheduled checkpoint file... 2022-04-12 03:25:40.235579: saving checkpoint... 2022-04-12 03:25:56.407002: done, saving took 16.27 seconds 2022-04-12 03:25:56.639575: done 2022-04-12 03:25:56.801213: This epoch took 137.731906 s 2022-04-12 03:25:56.959564: svmem(total=99932160000, available=76266319872, percent=23.7, used=19590529024, free=48275091456, active=43208581120, inactive=6322176000, buffers=13778944, cached=32052760576, shared=2261061632, slab=777613312) 2022-04-12 03:25:57.018173: epoch: 297 2022-04-12 03:27:45.554088: train loss : -0.5965 2022-04-12 03:27:57.857512: validation loss: -0.6468 2022-04-12 03:27:57.914275: Average global foreground Dice: [0.8727, 0.9582, 0.9738, 0.4164] 2022-04-12 03:27:57.957509: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:27:58.971276: lr: 0.004423 2022-04-12 03:27:59.018987: saving scheduled checkpoint file... 2022-04-12 03:27:59.253958: saving checkpoint... 2022-04-12 03:28:12.866420: done, saving took 13.74 seconds 2022-04-12 03:28:13.011261: done 2022-04-12 03:28:13.033280: This epoch took 135.917491 s 2022-04-12 03:28:13.090826: svmem(total=99932160000, available=76291145728, percent=23.7, used=19628568576, free=48288591872, active=43276242944, inactive=6242234368, buffers=14024704, cached=32000974848, shared=2198102016, slab=778153984) 2022-04-12 03:28:13.155447: epoch: 298 2022-04-12 03:30:02.735960: train loss : -0.5634 2022-04-12 03:30:14.598382: validation loss: -0.6200 2022-04-12 03:30:14.657018: Average global foreground Dice: [0.861, 0.9597, 0.9724, 0.3944] 2022-04-12 03:30:14.719101: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:30:15.583117: lr: 0.004404 2022-04-12 03:30:15.593344: saving scheduled checkpoint file... 2022-04-12 03:30:15.816278: saving checkpoint... 2022-04-12 03:30:24.032621: done, saving took 8.37 seconds 2022-04-12 03:30:24.081134: done 2022-04-12 03:30:24.085405: This epoch took 130.902881 s 2022-04-12 03:30:24.089399: svmem(total=99932160000, available=76243951616, percent=23.7, used=19651518464, free=48223326208, active=43322531840, inactive=6257049600, buffers=14266368, cached=32043048960, shared=2222489600, slab=777568256) 2022-04-12 03:30:24.096621: epoch: 299 2022-04-12 03:32:14.204727: train loss : -0.5840 2022-04-12 03:32:26.154091: validation loss: -0.6021 2022-04-12 03:32:26.192187: Average global foreground Dice: [0.8473, 0.9503, 0.9698, 0.3837] 2022-04-12 03:32:26.203205: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:32:27.238427: lr: 0.004384 2022-04-12 03:32:27.242429: saving scheduled checkpoint file... 2022-04-12 03:32:27.399220: saving checkpoint... 2022-04-12 03:32:38.018583: done, saving took 10.71 seconds 2022-04-12 03:32:38.081032: done 2022-04-12 03:32:38.154241: This epoch took 134.045006 s 2022-04-12 03:32:38.159081: svmem(total=99932160000, available=76303609856, percent=23.6, used=19633741824, free=48284377088, active=43327270912, inactive=6198484992, buffers=14524416, cached=31999516672, shared=2180587520, slab=776765440) 2022-04-12 03:32:38.163389: epoch: 300 2022-04-12 03:34:26.639536: train loss : -0.5969 2022-04-12 03:34:37.873129: validation loss: -0.6162 2022-04-12 03:34:37.889896: Average global foreground Dice: [0.8498, 0.9511, 0.9711, 0.3976] 2022-04-12 03:34:37.935866: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:34:38.917185: lr: 0.004364 2022-04-12 03:34:38.977672: saving scheduled checkpoint file... 2022-04-12 03:34:39.114134: saving checkpoint... 2022-04-12 03:34:51.603039: done, saving took 12.55 seconds 2022-04-12 03:34:51.754642: done 2022-04-12 03:34:51.769324: This epoch took 133.555861 s 2022-04-12 03:34:51.786378: svmem(total=99932160000, available=76090859520, percent=23.9, used=19780534272, free=48314884096, active=43441881088, inactive=6045888512, buffers=14770176, cached=31821971456, shared=2246963200, slab=782745600) 2022-04-12 03:34:51.834855: epoch: 301 2022-04-12 03:36:41.296835: train loss : -0.6049 2022-04-12 03:36:52.629090: validation loss: -0.6540 2022-04-12 03:36:52.660741: Average global foreground Dice: [0.8624, 0.9605, 0.9803, 0.4203] 2022-04-12 03:36:52.673535: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:36:53.712488: lr: 0.004344 2022-04-12 03:36:53.715140: saving scheduled checkpoint file... 2022-04-12 03:36:53.778220: saving checkpoint... 2022-04-12 03:37:02.039585: done, saving took 8.32 seconds 2022-04-12 03:37:02.131542: done 2022-04-12 03:37:02.186311: This epoch took 130.299480 s 2022-04-12 03:37:02.273355: svmem(total=99932160000, available=76311314432, percent=23.6, used=19583991808, free=48310595584, active=43263123456, inactive=6238887936, buffers=15036416, cached=32022536192, shared=2222583808, slab=776151040) 2022-04-12 03:37:02.277508: epoch: 302 2022-04-12 03:38:50.198186: train loss : -0.6286 2022-04-12 03:39:02.254150: validation loss: -0.6149 2022-04-12 03:39:02.275516: Average global foreground Dice: [0.8371, 0.967, 0.9755, 0.393] 2022-04-12 03:39:02.289158: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:39:03.071361: lr: 0.004325 2022-04-12 03:39:03.163617: saving scheduled checkpoint file... 2022-04-12 03:39:03.288684: saving checkpoint... 2022-04-12 03:39:14.463680: done, saving took 11.28 seconds 2022-04-12 03:39:14.564792: done 2022-04-12 03:39:14.568854: This epoch took 132.258866 s 2022-04-12 03:39:14.575745: svmem(total=99932160000, available=76102541312, percent=23.8, used=19767934976, free=48510259200, active=43450589184, inactive=5846061056, buffers=15278080, cached=31638687744, shared=2246955008, slab=776159232) 2022-04-12 03:39:14.585432: epoch: 303 2022-04-12 03:41:02.048567: train loss : -0.6100 2022-04-12 03:41:13.501474: validation loss: -0.6550 2022-04-12 03:41:13.589858: Average global foreground Dice: [0.8622, 0.9624, 0.9791, 0.4109] 2022-04-12 03:41:13.592894: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:41:14.630572: lr: 0.004305 2022-04-12 03:41:14.704545: saving scheduled checkpoint file... 2022-04-12 03:41:14.838139: saving checkpoint... 2022-04-12 03:41:32.839470: done, saving took 18.11 seconds 2022-04-12 03:41:33.511326: done 2022-04-12 03:41:33.687238: This epoch took 139.077575 s 2022-04-12 03:41:33.951736: svmem(total=99932160000, available=76087771136, percent=23.9, used=19770519552, free=48076496896, active=43444310016, inactive=6274027520, buffers=15589376, cached=32069554176, shared=2259353600, slab=791216128) 2022-04-12 03:41:34.293018: epoch: 304 2022-04-12 03:43:23.176054: train loss : -0.6424 2022-04-12 03:43:35.304504: validation loss: -0.6309 2022-04-12 03:43:35.349354: Average global foreground Dice: [0.836, 0.9597, 0.979, 0.411] 2022-04-12 03:43:35.353020: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:43:36.282553: lr: 0.004285 2022-04-12 03:43:36.286103: saving scheduled checkpoint file... 2022-04-12 03:43:36.431080: saving checkpoint... 2022-04-12 03:43:45.069290: done, saving took 8.78 seconds 2022-04-12 03:43:45.608532: done 2022-04-12 03:43:45.898336: This epoch took 131.376526 s 2022-04-12 03:43:46.216240: svmem(total=99932160000, available=76197244928, percent=23.8, used=19662954496, free=48183214080, active=43342622720, inactive=6271533056, buffers=15851520, cached=32070139904, shared=2257371136, slab=789647360) 2022-04-12 03:43:46.428207: epoch: 305 2022-04-12 03:45:36.071055: train loss : -0.6423 2022-04-12 03:45:48.356485: validation loss: -0.6483 2022-04-12 03:45:48.385901: Average global foreground Dice: [0.8815, 0.9629, 0.9847, 0.3994] 2022-04-12 03:45:48.448389: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:45:49.491128: lr: 0.004265 2022-04-12 03:45:49.546427: saving scheduled checkpoint file... 2022-04-12 03:45:49.630366: saving checkpoint... 2022-04-12 03:46:05.260490: done, saving took 15.71 seconds 2022-04-12 03:46:05.933694: done 2022-04-12 03:46:06.289306: This epoch took 139.608653 s 2022-04-12 03:46:06.581454: svmem(total=99932160000, available=76176117760, percent=23.8, used=19694362624, free=48175529984, active=43363700736, inactive=6256865280, buffers=16138240, cached=32046129152, shared=2247135232, slab=790499328) 2022-04-12 03:46:06.869848: epoch: 306 2022-04-12 03:47:55.556710: train loss : -0.6349 2022-04-12 03:48:07.568209: validation loss: -0.6453 2022-04-12 03:48:07.575493: Average global foreground Dice: [0.8765, 0.966, 0.9762, 0.4067] 2022-04-12 03:48:07.598850: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:48:08.296758: lr: 0.004245 2022-04-12 03:48:08.299205: saving scheduled checkpoint file... 2022-04-12 03:48:08.351959: saving checkpoint... 2022-04-12 03:48:23.588632: done, saving took 15.29 seconds 2022-04-12 03:48:25.174911: done 2022-04-12 03:48:25.573784: This epoch took 138.307612 s 2022-04-12 03:48:25.952121: svmem(total=99932160000, available=76003766272, percent=23.9, used=19866669056, free=48016576512, active=43514343424, inactive=6259245056, buffers=16404480, cached=32032509952, shared=2247049216, slab=791764992) 2022-04-12 03:48:26.352546: epoch: 307 2022-04-12 03:50:15.067253: train loss : -0.6407 2022-04-12 03:50:26.336182: validation loss: -0.6465 2022-04-12 03:50:26.430165: Average global foreground Dice: [0.8418, 0.9657, 0.9772, 0.4003] 2022-04-12 03:50:26.675301: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:50:27.830518: lr: 0.004226 2022-04-12 03:50:27.859087: saving scheduled checkpoint file... 2022-04-12 03:50:27.941482: saving checkpoint... 2022-04-12 03:50:44.050150: done, saving took 16.19 seconds 2022-04-12 03:50:44.684598: done 2022-04-12 03:50:45.017990: This epoch took 138.158360 s 2022-04-12 03:50:45.329931: svmem(total=99932160000, available=75991535616, percent=24.0, used=19878809600, free=48018104320, active=43517190144, inactive=6257758208, buffers=16658432, cached=32018587648, shared=2247049216, slab=792506368) 2022-04-12 03:50:45.511101: epoch: 308 2022-04-12 03:52:34.142460: train loss : -0.6431 2022-04-12 03:52:45.877800: validation loss: -0.6359 2022-04-12 03:52:45.915812: Average global foreground Dice: [0.8505, 0.963, 0.9752, 0.4139] 2022-04-12 03:52:45.922551: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:52:47.008626: lr: 0.004206 2022-04-12 03:52:47.017734: saving scheduled checkpoint file... 2022-04-12 03:52:47.150296: saving checkpoint... 2022-04-12 03:53:02.777744: done, saving took 15.69 seconds 2022-04-12 03:53:04.672886: done 2022-04-12 03:53:05.435123: This epoch took 139.788465 s 2022-04-12 03:53:06.349118: svmem(total=99932160000, available=76070649856, percent=23.9, used=19790336000, free=48094371840, active=43431145472, inactive=6268243968, buffers=16908288, cached=32030543872, shared=2256379904, slab=790212608) 2022-04-12 03:53:06.694415: epoch: 309 2022-04-12 03:54:56.633520: train loss : -0.5908 2022-04-12 03:55:08.449440: validation loss: -0.6218 2022-04-12 03:55:08.510308: Average global foreground Dice: [0.8553, 0.9499, 0.9664, 0.4179] 2022-04-12 03:55:08.555208: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:55:09.741582: lr: 0.004186 2022-04-12 03:55:09.773665: saving scheduled checkpoint file... 2022-04-12 03:55:09.950459: saving checkpoint... 2022-04-12 03:55:23.012924: done, saving took 13.19 seconds 2022-04-12 03:55:24.128867: done 2022-04-12 03:55:24.770859: This epoch took 137.711405 s 2022-04-12 03:55:25.380582: svmem(total=99932160000, available=76042649600, percent=23.9, used=19827765248, free=48063463424, active=43471691776, inactive=6258647040, buffers=17195008, cached=32023736320, shared=2246963200, slab=790052864) 2022-04-12 03:55:26.206290: epoch: 310 2022-04-12 03:57:14.291541: train loss : -0.6332 2022-04-12 03:57:26.838507: validation loss: -0.6379 2022-04-12 03:57:26.880921: Average global foreground Dice: [0.8506, 0.9613, 0.9794, 0.4152] 2022-04-12 03:57:26.943733: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:57:27.827637: lr: 0.004166 2022-04-12 03:57:27.851849: saving scheduled checkpoint file... 2022-04-12 03:57:28.060713: saving checkpoint... 2022-04-12 03:57:47.448916: done, saving took 19.48 seconds 2022-04-12 03:57:49.270726: done 2022-04-12 03:57:49.894842: This epoch took 143.218369 s 2022-04-12 03:57:50.457114: svmem(total=99932160000, available=76045377536, percent=23.9, used=19817570304, free=48062455808, active=43467255808, inactive=6261723136, buffers=17485824, cached=32034648064, shared=2254397440, slab=791674880) 2022-04-12 03:57:51.023716: epoch: 311 2022-04-12 03:59:41.734255: train loss : -0.6140 2022-04-12 03:59:54.259806: validation loss: -0.6366 2022-04-12 03:59:54.300292: Average global foreground Dice: [0.8786, 0.9598, 0.9716, 0.3957] 2022-04-12 03:59:54.322352: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 03:59:55.311869: lr: 0.004146 2022-04-12 03:59:55.316523: saving scheduled checkpoint file... 2022-04-12 03:59:55.429995: saving checkpoint... 2022-04-12 04:00:04.251818: done, saving took 8.93 seconds 2022-04-12 04:00:04.282748: done 2022-04-12 04:00:04.303648: This epoch took 132.869585 s 2022-04-12 04:00:04.320627: svmem(total=99932160000, available=76266565632, percent=23.7, used=19653685248, free=48280465408, active=43314139136, inactive=6207275008, buffers=17735680, cached=31980273664, shared=2197000192, slab=778014720) 2022-04-12 04:00:04.400668: epoch: 312 2022-04-12 04:01:53.977247: train loss : -0.6381 2022-04-12 04:02:05.894031: validation loss: -0.6633 2022-04-12 04:02:05.918671: Average global foreground Dice: [0.8634, 0.9671, 0.9793, 0.4325] 2022-04-12 04:02:05.922458: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:02:06.569005: lr: 0.004127 2022-04-12 04:02:06.810542: saving scheduled checkpoint file... 2022-04-12 04:02:06.998322: saving checkpoint... 2022-04-12 04:02:15.062464: done, saving took 8.15 seconds 2022-04-12 04:02:15.290073: done 2022-04-12 04:02:15.364940: This epoch took 130.959587 s 2022-04-12 04:02:15.368815: svmem(total=99932160000, available=76343853056, percent=23.6, used=19561275392, free=48355069952, active=43223650304, inactive=6225186816, buffers=17973248, cached=31997841408, shared=2212081664, slab=778706944) 2022-04-12 04:02:15.375264: epoch: 313 2022-04-12 04:04:03.838073: train loss : -0.6457 2022-04-12 04:04:15.901180: validation loss: -0.6464 2022-04-12 04:04:15.967894: Average global foreground Dice: [0.8584, 0.9652, 0.9816, 0.3972] 2022-04-12 04:04:15.972814: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:04:16.810496: lr: 0.004107 2022-04-12 04:04:16.820198: saving scheduled checkpoint file... 2022-04-12 04:04:17.008117: saving checkpoint... 2022-04-12 04:04:28.776806: done, saving took 11.88 seconds 2022-04-12 04:04:28.838381: done 2022-04-12 04:04:28.846173: This epoch took 133.391173 s 2022-04-12 04:04:28.849494: svmem(total=99932160000, available=76336234496, percent=23.6, used=19615715328, free=48344334336, active=43281166336, inactive=6180241408, buffers=18231296, cached=31953879040, shared=2165248000, slab=778309632) 2022-04-12 04:04:28.912527: epoch: 314 2022-04-12 04:06:16.865567: train loss : -0.6014 2022-04-12 04:06:29.358238: validation loss: -0.6236 2022-04-12 04:06:29.386194: Average global foreground Dice: [0.8178, 0.9633, 0.9752, 0.4204] 2022-04-12 04:06:29.402331: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:06:30.249987: lr: 0.004087 2022-04-12 04:06:30.260431: saving scheduled checkpoint file... 2022-04-12 04:06:30.403209: saving checkpoint... 2022-04-12 04:06:41.208038: done, saving took 10.94 seconds 2022-04-12 04:06:41.345820: done 2022-04-12 04:06:41.351912: This epoch took 132.421935 s 2022-04-12 04:06:41.386443: svmem(total=99932160000, available=76536172544, percent=23.4, used=19446181888, free=48541663232, active=43116331008, inactive=6152273920, buffers=18505728, cached=31925809152, shared=2134867968, slab=779018240) 2022-04-12 04:06:41.423756: epoch: 315 2022-04-12 04:08:31.437722: train loss : -0.6069 2022-04-12 04:08:43.225207: validation loss: -0.5926 2022-04-12 04:08:43.258432: Average global foreground Dice: [0.8224, 0.95, 0.9606, 0.4117] 2022-04-12 04:08:43.268129: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:08:44.247417: lr: 0.004067 2022-04-12 04:08:44.295252: saving scheduled checkpoint file... 2022-04-12 04:08:44.505829: saving checkpoint... 2022-04-12 04:08:59.587994: done, saving took 15.23 seconds 2022-04-12 04:08:59.846800: done 2022-04-12 04:08:59.937688: This epoch took 138.482713 s 2022-04-12 04:09:00.029031: svmem(total=99932160000, available=76364505088, percent=23.6, used=19544829952, free=48367407104, active=43213520896, inactive=6226980864, buffers=18759680, cached=32001163264, shared=2207723520, slab=778559488) 2022-04-12 04:09:00.135700: epoch: 316 2022-04-12 04:10:50.702372: train loss : -0.5940 2022-04-12 04:11:01.212466: validation loss: -0.6302 2022-04-12 04:11:01.269181: Average global foreground Dice: [0.8508, 0.9627, 0.9782, 0.3984] 2022-04-12 04:11:01.319437: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:11:02.387180: lr: 0.004047 2022-04-12 04:11:02.393707: saving scheduled checkpoint file... 2022-04-12 04:11:02.649851: saving checkpoint... 2022-04-12 04:11:20.751804: done, saving took 18.23 seconds 2022-04-12 04:11:21.895417: done 2022-04-12 04:11:22.414015: This epoch took 142.198296 s 2022-04-12 04:11:22.927123: svmem(total=99932160000, available=76128333824, percent=23.8, used=19741708288, free=48128081920, active=43387273216, inactive=6269079552, buffers=19025920, cached=32043343872, shared=2247049216, slab=792858624) 2022-04-12 04:11:23.272161: epoch: 317 2022-04-12 04:13:13.407228: train loss : -0.6308 2022-04-12 04:13:25.066464: validation loss: -0.6653 2022-04-12 04:13:25.163847: Average global foreground Dice: [0.8621, 0.9665, 0.9795, 0.4292] 2022-04-12 04:13:25.239572: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:13:26.299143: lr: 0.004027 2022-04-12 04:13:26.330776: saving scheduled checkpoint file... 2022-04-12 04:13:26.477849: saving checkpoint... 2022-04-12 04:13:53.357888: done, saving took 26.96 seconds 2022-04-12 04:13:53.543230: done 2022-04-12 04:13:53.567033: This epoch took 149.870660 s 2022-04-12 04:13:53.643835: svmem(total=99932160000, available=76108963840, percent=23.8, used=19747385344, free=48105746432, active=43413319680, inactive=6285283328, buffers=19320832, cached=32059707392, shared=2260754432, slab=779849728) 2022-04-12 04:13:53.650445: epoch: 318 2022-04-12 04:15:41.325598: train loss : -0.6524 2022-04-12 04:15:53.440027: validation loss: -0.6455 2022-04-12 04:15:53.484379: Average global foreground Dice: [0.849, 0.9634, 0.9809, 0.4023] 2022-04-12 04:15:53.540499: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:15:54.449465: lr: 0.004007 2022-04-12 04:15:54.459409: saving scheduled checkpoint file... 2022-04-12 04:15:54.533806: saving checkpoint... 2022-04-12 04:16:15.560581: done, saving took 21.08 seconds 2022-04-12 04:16:15.720901: done 2022-04-12 04:16:15.800358: This epoch took 142.117258 s 2022-04-12 04:16:15.807476: svmem(total=99932160000, available=75993849856, percent=24.0, used=19862081536, free=47987716096, active=43529584640, inactive=6286807040, buffers=19603456, cached=32062758912, shared=2261114880, slab=779747328) 2022-04-12 04:16:15.881632: epoch: 319 2022-04-12 04:18:04.629237: train loss : -0.6102 2022-04-12 04:18:16.731087: validation loss: -0.5961 2022-04-12 04:18:16.791213: Average global foreground Dice: [0.8546, 0.9544, 0.9652, 0.36] 2022-04-12 04:18:16.815711: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:18:17.839132: lr: 0.003987 2022-04-12 04:18:18.310933: saving scheduled checkpoint file... 2022-04-12 04:18:18.905587: saving checkpoint... 2022-04-12 04:18:31.158600: done, saving took 12.34 seconds 2022-04-12 04:18:31.190465: done 2022-04-12 04:18:31.192106: This epoch took 135.244833 s 2022-04-12 04:18:31.193760: svmem(total=99932160000, available=76095852544, percent=23.9, used=19763929088, free=48086888448, active=43429568512, inactive=6285475840, buffers=19890176, cached=32061452288, shared=2257285120, slab=779022336) 2022-04-12 04:18:31.196919: epoch: 320 2022-04-12 04:20:18.288352: train loss : -0.6270 2022-04-12 04:20:29.929130: validation loss: -0.6511 2022-04-12 04:20:29.933257: Average global foreground Dice: [0.8689, 0.9621, 0.9728, 0.4359] 2022-04-12 04:20:29.934917: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:20:30.879683: lr: 0.003967 2022-04-12 04:20:30.894797: saving scheduled checkpoint file... 2022-04-12 04:20:31.028008: saving checkpoint... 2022-04-12 04:20:49.087522: done, saving took 18.16 seconds 2022-04-12 04:20:49.310786: done 2022-04-12 04:20:49.316121: This epoch took 138.113409 s 2022-04-12 04:20:49.325290: svmem(total=99932160000, available=76325830656, percent=23.6, used=19543773184, free=48314150912, active=43215364096, inactive=6277783552, buffers=20140032, cached=32054095872, shared=2247307264, slab=778833920) 2022-04-12 04:20:49.355433: epoch: 321 2022-04-12 04:22:37.567131: train loss : -0.6261 2022-04-12 04:22:49.124361: validation loss: -0.6663 2022-04-12 04:22:49.132245: Average global foreground Dice: [0.8488, 0.9631, 0.9785, 0.4421] 2022-04-12 04:22:49.133708: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:22:50.036217: lr: 0.003947 2022-04-12 04:22:50.046008: saving scheduled checkpoint file... 2022-04-12 04:22:50.199823: saving checkpoint... 2022-04-12 04:23:06.761889: done, saving took 16.67 seconds 2022-04-12 04:23:08.095574: done 2022-04-12 04:23:08.896453: This epoch took 139.521244 s 2022-04-12 04:23:09.498145: svmem(total=99932160000, available=75975770112, percent=24.0, used=19880333312, free=47961157632, active=43540045824, inactive=6293401600, buffers=20414464, cached=32070254592, shared=2260840448, slab=791777280) 2022-04-12 04:23:10.110947: epoch: 322 2022-04-12 04:24:59.620836: train loss : -0.6394 2022-04-12 04:25:10.737056: validation loss: -0.6192 2022-04-12 04:25:10.743238: Average global foreground Dice: [0.8464, 0.9665, 0.9718, 0.3993] 2022-04-12 04:25:10.816272: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:25:11.986131: lr: 0.003927 2022-04-12 04:25:12.055195: saving scheduled checkpoint file... 2022-04-12 04:25:12.203717: saving checkpoint... 2022-04-12 04:25:33.070659: done, saving took 21.00 seconds 2022-04-12 04:25:34.706797: done 2022-04-12 04:25:35.203454: This epoch took 144.385751 s 2022-04-12 04:25:35.764253: svmem(total=99932160000, available=76188786688, percent=23.8, used=19680866304, free=48171130880, active=43332694016, inactive=6282223616, buffers=20738048, cached=32059424768, shared=2247135232, slab=790888448) 2022-04-12 04:25:36.122877: epoch: 323 2022-04-12 04:27:26.289540: train loss : -0.6401 2022-04-12 04:27:38.818670: validation loss: -0.6467 2022-04-12 04:27:38.866156: Average global foreground Dice: [0.846, 0.965, 0.9776, 0.4177] 2022-04-12 04:27:38.890322: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:27:40.069643: lr: 0.003907 2022-04-12 04:27:40.508057: saving scheduled checkpoint file... 2022-04-12 04:27:41.060405: saving checkpoint... 2022-04-12 04:27:56.795522: done, saving took 15.83 seconds 2022-04-12 04:27:56.954167: done 2022-04-12 04:27:56.981960: This epoch took 140.515342 s 2022-04-12 04:27:57.012697: svmem(total=99932160000, available=76040740864, percent=23.9, used=19815043072, free=48020320256, active=43484401664, inactive=6298689536, buffers=21004288, cached=32075792384, shared=2261037056, slab=777924608) 2022-04-12 04:27:57.067185: epoch: 324 2022-04-12 04:29:46.733052: train loss : -0.6428 2022-04-12 04:29:58.841130: validation loss: -0.6544 2022-04-12 04:29:59.325590: Average global foreground Dice: [0.8561, 0.9662, 0.9769, 0.4073] 2022-04-12 04:30:00.296145: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:30:01.896559: lr: 0.003887 2022-04-12 04:30:02.325433: saving scheduled checkpoint file... 2022-04-12 04:30:02.924001: saving checkpoint... 2022-04-12 04:30:16.428634: done, saving took 13.56 seconds 2022-04-12 04:30:16.760889: done 2022-04-12 04:30:16.868211: This epoch took 139.761575 s 2022-04-12 04:30:16.976816: svmem(total=99932160000, available=76132343808, percent=23.8, used=19723235328, free=48101036032, active=43371757568, inactive=6327939072, buffers=22827008, cached=32085061632, shared=2261037056, slab=778420224) 2022-04-12 04:30:17.056584: epoch: 325 2022-04-12 04:32:04.965936: train loss : -0.6489 2022-04-12 04:32:16.172733: validation loss: -0.6753 2022-04-12 04:32:16.987140: Average global foreground Dice: [0.8929, 0.9641, 0.9826, 0.4149] 2022-04-12 04:32:17.768948: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:32:19.705084: lr: 0.003867 2022-04-12 04:32:20.565971: saving scheduled checkpoint file... 2022-04-12 04:32:21.177127: saving checkpoint... 2022-04-12 04:32:31.880084: done, saving took 10.75 seconds 2022-04-12 04:32:32.043913: done 2022-04-12 04:32:32.052113: This epoch took 134.922055 s 2022-04-12 04:32:32.097291: svmem(total=99932160000, available=75768627200, percent=24.2, used=20087545856, free=47734763520, active=43742208000, inactive=6329286656, buffers=23093248, cached=32086757376, shared=2260590592, slab=777277440) 2022-04-12 04:32:32.157553: epoch: 326 2022-04-12 04:34:21.414376: train loss : -0.6516 2022-04-12 04:34:33.117493: validation loss: -0.6208 2022-04-12 04:34:33.763494: Average global foreground Dice: [0.8607, 0.9579, 0.976, 0.4051] 2022-04-12 04:34:34.193455: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:34:36.816747: lr: 0.003847 2022-04-12 04:34:37.449757: saving scheduled checkpoint file... 2022-04-12 04:34:38.305501: saving checkpoint... 2022-04-12 04:34:47.603433: done, saving took 9.37 seconds 2022-04-12 04:34:47.655736: done 2022-04-12 04:34:47.675813: This epoch took 135.514959 s 2022-04-12 04:34:47.677631: svmem(total=99932160000, available=75984199680, percent=24.0, used=19871711232, free=47947530240, active=43526664192, inactive=6331801600, buffers=23347200, cached=32089571328, shared=2260627456, slab=779411456) 2022-04-12 04:34:47.680026: epoch: 327 2022-04-12 04:36:38.049680: train loss : -0.6473 2022-04-12 04:36:49.581259: validation loss: -0.6769 2022-04-12 04:36:49.929802: Average global foreground Dice: [0.8726, 0.9629, 0.9813, 0.4496] 2022-04-12 04:36:50.273599: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:36:51.960356: lr: 0.003827 2022-04-12 04:36:52.230865: saving scheduled checkpoint file... 2022-04-12 04:36:52.676193: saving checkpoint... 2022-04-12 04:37:02.358808: done, saving took 9.77 seconds 2022-04-12 04:37:02.425941: done 2022-04-12 04:37:02.427160: This epoch took 134.745685 s 2022-04-12 04:37:02.428094: svmem(total=99932160000, available=75842404352, percent=24.1, used=20008075264, free=47812513792, active=43652657152, inactive=6338461696, buffers=23629824, cached=32087941120, shared=2266079232, slab=779526144) 2022-04-12 04:37:02.429312: epoch: 328 2022-04-12 04:38:51.904847: train loss : -0.6550 2022-04-12 04:39:03.676805: validation loss: -0.6396 2022-04-12 04:39:03.699568: Average global foreground Dice: [0.8685, 0.9587, 0.9746, 0.417] 2022-04-12 04:39:03.701176: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:39:04.605192: lr: 0.003807 2022-04-12 04:39:04.667882: saving scheduled checkpoint file... 2022-04-12 04:39:04.798409: saving checkpoint... 2022-04-12 04:39:13.373579: done, saving took 8.70 seconds 2022-04-12 04:39:13.407480: done 2022-04-12 04:39:13.518614: saving checkpoint... 2022-04-12 04:39:19.020588: done, saving took 5.61 seconds 2022-04-12 04:39:19.081551: This epoch took 136.651580 s 2022-04-12 04:39:19.085551: svmem(total=99932160000, available=76085653504, percent=23.9, used=19769561088, free=48053207040, active=43415769088, inactive=6336106496, buffers=23883776, cached=32085508096, shared=2261286912, slab=778076160) 2022-04-12 04:39:19.088393: epoch: 329 2022-04-12 04:41:09.431150: train loss : -0.6303 2022-04-12 04:41:22.100940: validation loss: -0.6481 2022-04-12 04:41:22.137495: Average global foreground Dice: [0.8651, 0.9648, 0.9773, 0.4188] 2022-04-12 04:41:22.162992: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:41:23.214135: lr: 0.003787 2022-04-12 04:41:23.238726: saving scheduled checkpoint file... 2022-04-12 04:41:23.339400: saving checkpoint... 2022-04-12 04:41:33.452762: done, saving took 10.20 seconds 2022-04-12 04:41:33.559701: done 2022-04-12 04:41:33.723905: saving checkpoint... 2022-04-12 04:41:44.655027: done, saving took 11.00 seconds 2022-04-12 04:41:44.694403: This epoch took 145.603841 s 2022-04-12 04:41:44.743913: svmem(total=99932160000, available=76078252032, percent=23.9, used=19776716800, free=48042795008, active=43420372992, inactive=6338859008, buffers=24174592, cached=32088473600, shared=2261504000, slab=778977280) 2022-04-12 04:41:44.780295: epoch: 330 2022-04-12 04:43:35.370544: train loss : -0.6356 2022-04-12 04:43:47.511737: validation loss: -0.6503 2022-04-12 04:43:47.542791: Average global foreground Dice: [0.8729, 0.9661, 0.9778, 0.3947] 2022-04-12 04:43:47.592563: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:43:48.462322: lr: 0.003767 2022-04-12 04:43:48.517518: saving scheduled checkpoint file... 2022-04-12 04:43:48.714040: saving checkpoint... 2022-04-12 04:43:59.943009: done, saving took 11.34 seconds 2022-04-12 04:44:00.125144: done 2022-04-12 04:44:00.131716: This epoch took 135.279405 s 2022-04-12 04:44:00.222386: svmem(total=99932160000, available=76029661184, percent=23.9, used=19811520512, free=47991345152, active=43458138112, inactive=6355054592, buffers=24428544, cached=32104865792, shared=2275225600, slab=778842112) 2022-04-12 04:44:00.315020: epoch: 331 2022-04-12 04:45:49.635064: train loss : -0.6566 2022-04-12 04:46:01.422470: validation loss: -0.6590 2022-04-12 04:46:01.449261: Average global foreground Dice: [0.844, 0.9665, 0.9815, 0.4002] 2022-04-12 04:46:01.454223: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:46:02.506738: lr: 0.003747 2022-04-12 04:46:02.513133: saving scheduled checkpoint file... 2022-04-12 04:46:02.629974: saving checkpoint... 2022-04-12 04:46:09.379619: done, saving took 6.86 seconds 2022-04-12 04:46:09.511193: done 2022-04-12 04:46:09.512204: This epoch took 129.141990 s 2022-04-12 04:46:09.513565: svmem(total=99932160000, available=75982807040, percent=24.0, used=19870781440, free=47941980160, active=43519561728, inactive=6343372800, buffers=24702976, cached=32094695424, shared=2262732800, slab=778838016) 2022-04-12 04:46:09.515029: epoch: 332 2022-04-12 04:47:59.098371: train loss : -0.6562 2022-04-12 04:48:11.837068: validation loss: -0.6727 2022-04-12 04:48:11.861199: Average global foreground Dice: [0.8495, 0.9641, 0.9808, 0.439] 2022-04-12 04:48:11.875083: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:48:12.796021: lr: 0.003727 2022-04-12 04:48:12.843727: saving scheduled checkpoint file... 2022-04-12 04:48:12.978661: saving checkpoint... 2022-04-12 04:48:25.095439: done, saving took 12.25 seconds 2022-04-12 04:48:25.162483: done 2022-04-12 04:48:25.180443: This epoch took 135.664557 s 2022-04-12 04:48:25.186152: svmem(total=99932160000, available=76198137856, percent=23.8, used=19653902336, free=48154562560, active=43302707200, inactive=6341881856, buffers=24952832, cached=32098742272, shared=2264322048, slab=778723328) 2022-04-12 04:48:25.221652: epoch: 333 2022-04-12 04:50:15.946511: train loss : -0.6409 2022-04-12 04:50:28.085000: validation loss: -0.6304 2022-04-12 04:50:28.113277: Average global foreground Dice: [0.8457, 0.9677, 0.9721, 0.4203] 2022-04-12 04:50:28.127681: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:50:29.046556: lr: 0.003707 2022-04-12 04:50:29.052194: saving scheduled checkpoint file... 2022-04-12 04:50:29.202971: saving checkpoint... 2022-04-12 04:50:36.338073: done, saving took 7.28 seconds 2022-04-12 04:50:36.455501: done 2022-04-12 04:50:36.501021: This epoch took 131.277256 s 2022-04-12 04:50:36.504741: svmem(total=99932160000, available=75986075648, percent=24.0, used=19857457152, free=47940009984, active=43502362624, inactive=6359584768, buffers=25223168, cached=32109469696, shared=2272833536, slab=779509760) 2022-04-12 04:50:36.566859: epoch: 334 2022-04-12 04:52:25.202934: train loss : -0.6529 2022-04-12 04:52:37.548708: validation loss: -0.6666 2022-04-12 04:52:37.574100: Average global foreground Dice: [0.867, 0.9656, 0.9822, 0.4368] 2022-04-12 04:52:37.595193: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:52:38.568478: lr: 0.003687 2022-04-12 04:52:38.593439: saving scheduled checkpoint file... 2022-04-12 04:52:38.726415: saving checkpoint... 2022-04-12 04:52:47.799767: done, saving took 9.19 seconds 2022-04-12 04:52:48.114762: done 2022-04-12 04:52:48.346158: saving checkpoint... 2022-04-12 04:53:01.913398: done, saving took 13.71 seconds 2022-04-12 04:53:01.955848: This epoch took 145.387452 s 2022-04-12 04:53:01.960863: svmem(total=99932160000, available=75934294016, percent=24.0, used=19920539648, free=47885426688, active=43568955392, inactive=6350733312, buffers=25481216, cached=32100712448, shared=2261520384, slab=779915264) 2022-04-12 04:53:01.994081: epoch: 335 2022-04-12 04:54:50.443161: train loss : -0.6672 2022-04-12 04:55:03.154906: validation loss: -0.6659 2022-04-12 04:55:03.220060: Average global foreground Dice: [0.8647, 0.9628, 0.9805, 0.4366] 2022-04-12 04:55:03.232158: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:55:04.447439: lr: 0.003667 2022-04-12 04:55:04.534253: saving scheduled checkpoint file... 2022-04-12 04:55:04.692975: saving checkpoint... 2022-04-12 04:55:16.387594: done, saving took 11.79 seconds 2022-04-12 04:55:16.578408: done 2022-04-12 04:55:16.652894: saving checkpoint... 2022-04-12 04:55:23.456330: done, saving took 6.85 seconds 2022-04-12 04:55:23.585328: This epoch took 141.550380 s 2022-04-12 04:55:23.589662: svmem(total=99932160000, available=76053209088, percent=23.9, used=19801706496, free=48001630208, active=43449090048, inactive=6352932864, buffers=25784320, cached=32103038976, shared=2261356544, slab=780296192) 2022-04-12 04:55:23.595813: epoch: 336 2022-04-12 04:57:11.795903: train loss : -0.6481 2022-04-12 04:57:23.267132: validation loss: -0.6574 2022-04-12 04:57:23.299517: Average global foreground Dice: [0.8483, 0.9651, 0.9793, 0.416] 2022-04-12 04:57:23.303751: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:57:24.081366: lr: 0.003647 2022-04-12 04:57:24.132207: saving scheduled checkpoint file... 2022-04-12 04:57:24.186273: saving checkpoint... 2022-04-12 04:57:33.295627: done, saving took 9.16 seconds 2022-04-12 04:57:33.390817: done 2022-04-12 04:57:33.394089: This epoch took 129.762654 s 2022-04-12 04:57:33.399155: svmem(total=99932160000, available=75991584768, percent=24.0, used=19852910592, free=47937490944, active=43501481984, inactive=6365405184, buffers=26046464, cached=32115712000, shared=2271645696, slab=777662464) 2022-04-12 04:57:33.416382: epoch: 337 2022-04-12 04:59:21.441375: train loss : -0.6690 2022-04-12 04:59:32.930324: validation loss: -0.6612 2022-04-12 04:59:32.982836: Average global foreground Dice: [0.8704, 0.9699, 0.9792, 0.3978] 2022-04-12 04:59:33.049171: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 04:59:34.003750: lr: 0.003627 2022-04-12 04:59:34.093396: saving scheduled checkpoint file... 2022-04-12 04:59:34.278351: saving checkpoint... 2022-04-12 04:59:42.032853: done, saving took 7.85 seconds 2022-04-12 04:59:42.049195: done 2022-04-12 04:59:42.051935: This epoch took 128.628048 s 2022-04-12 04:59:42.053648: svmem(total=99932160000, available=75993604096, percent=24.0, used=19861168128, free=47937003520, active=43505790976, inactive=6357975040, buffers=26288128, cached=32107700224, shared=2261426176, slab=779677696) 2022-04-12 04:59:42.055329: epoch: 338 2022-04-12 05:01:31.635787: train loss : -0.6705 2022-04-12 05:01:42.752628: validation loss: -0.6738 2022-04-12 05:01:42.827544: Average global foreground Dice: [0.8569, 0.97, 0.981, 0.4086] 2022-04-12 05:01:42.874507: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:01:43.938749: lr: 0.003606 2022-04-12 05:01:44.060594: saving scheduled checkpoint file... 2022-04-12 05:01:44.263042: saving checkpoint... 2022-04-12 05:01:53.482524: done, saving took 9.31 seconds 2022-04-12 05:01:53.562433: done 2022-04-12 05:01:53.613473: This epoch took 131.556928 s 2022-04-12 05:01:53.618184: svmem(total=99932160000, available=76029870080, percent=23.9, used=19824799744, free=47970660352, active=43475054592, inactive=6360252416, buffers=26533888, cached=32110166016, shared=2261426176, slab=779259904) 2022-04-12 05:01:53.659598: epoch: 339 2022-04-12 05:03:42.483006: train loss : -0.6692 2022-04-12 05:03:54.035087: validation loss: -0.6669 2022-04-12 05:03:54.093143: Average global foreground Dice: [0.8718, 0.967, 0.9801, 0.4481] 2022-04-12 05:03:54.095741: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:03:54.940213: lr: 0.003586 2022-04-12 05:03:54.947051: saving scheduled checkpoint file... 2022-04-12 05:03:55.070763: saving checkpoint... 2022-04-12 05:04:05.056808: done, saving took 10.09 seconds 2022-04-12 05:04:05.238461: done 2022-04-12 05:04:05.358491: saving checkpoint... 2022-04-12 05:04:14.632427: done, saving took 9.38 seconds 2022-04-12 05:04:14.768278: This epoch took 141.103834 s 2022-04-12 05:04:14.771724: svmem(total=99932160000, available=75868839936, percent=24.1, used=19989618688, free=47806758912, active=43635507200, inactive=6358773760, buffers=26791936, cached=32108990464, shared=2257444864, slab=780771328) 2022-04-12 05:04:14.779822: epoch: 340 2022-04-12 05:06:02.604141: train loss : -0.6741 2022-04-12 05:06:14.411813: validation loss: -0.6759 2022-04-12 05:06:14.496130: Average global foreground Dice: [0.8722, 0.9689, 0.9822, 0.4141] 2022-04-12 05:06:14.502537: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:06:15.241879: lr: 0.003566 2022-04-12 05:06:15.323271: saving scheduled checkpoint file... 2022-04-12 05:06:15.502470: saving checkpoint... 2022-04-12 05:06:26.345210: done, saving took 10.96 seconds 2022-04-12 05:06:26.555794: done 2022-04-12 05:06:26.655981: saving checkpoint... 2022-04-12 05:06:36.553637: done, saving took 9.99 seconds 2022-04-12 05:06:36.725190: This epoch took 141.905696 s 2022-04-12 05:06:36.731389: svmem(total=99932160000, available=75950399488, percent=24.0, used=19893731328, free=47885570048, active=43542355968, inactive=6375124992, buffers=27099136, cached=32125759488, shared=2271834112, slab=780271616) 2022-04-12 05:06:36.733912: epoch: 341 2022-04-12 05:08:26.485278: train loss : -0.6668 2022-04-12 05:08:38.164451: validation loss: -0.6620 2022-04-12 05:08:38.172314: Average global foreground Dice: [0.8856, 0.9686, 0.9803, 0.3992] 2022-04-12 05:08:38.173721: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:08:39.105933: lr: 0.003546 2022-04-12 05:08:39.107870: saving scheduled checkpoint file... 2022-04-12 05:08:39.229524: saving checkpoint... 2022-04-12 05:08:49.184400: done, saving took 10.07 seconds 2022-04-12 05:08:49.204606: done 2022-04-12 05:08:49.298034: saving checkpoint... 2022-04-12 05:08:55.404357: done, saving took 6.20 seconds 2022-04-12 05:08:55.439947: This epoch took 138.688243 s 2022-04-12 05:08:55.452472: svmem(total=99932160000, available=76130611200, percent=23.8, used=19710013440, free=48063176704, active=43357642752, inactive=6381068288, buffers=27357184, cached=32131612672, shared=2275389440, slab=779931648) 2022-04-12 05:08:55.485661: epoch: 342 2022-04-12 05:10:45.468732: train loss : -0.6660 2022-04-12 05:10:57.913826: validation loss: -0.6579 2022-04-12 05:10:58.007265: Average global foreground Dice: [0.8851, 0.9639, 0.9809, 0.4219] 2022-04-12 05:10:58.036770: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:10:58.816262: lr: 0.003526 2022-04-12 05:10:58.875634: saving scheduled checkpoint file... 2022-04-12 05:10:59.001099: saving checkpoint... 2022-04-12 05:11:18.682751: done, saving took 19.80 seconds 2022-04-12 05:11:18.753314: done 2022-04-12 05:11:18.921743: saving checkpoint... 2022-04-12 05:11:34.863185: done, saving took 16.06 seconds 2022-04-12 05:11:35.128490: This epoch took 159.640680 s 2022-04-12 05:11:35.276644: svmem(total=99932160000, available=76354224128, percent=23.6, used=19500494848, free=48283435008, active=43150667776, inactive=6369787904, buffers=27676672, cached=32120553472, shared=2261164032, slab=780079104) 2022-04-12 05:11:35.353010: epoch: 343 2022-04-12 05:13:24.947558: train loss : -0.6602 2022-04-12 05:13:37.269313: validation loss: -0.6783 2022-04-12 05:13:37.286111: Average global foreground Dice: [0.894, 0.9678, 0.9826, 0.4489] 2022-04-12 05:13:37.294899: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:13:38.214037: lr: 0.003505 2022-04-12 05:13:38.216857: saving scheduled checkpoint file... 2022-04-12 05:13:38.329034: saving checkpoint... 2022-04-12 05:13:43.451927: done, saving took 5.23 seconds 2022-04-12 05:13:43.555770: done 2022-04-12 05:13:43.678747: saving checkpoint... 2022-04-12 05:13:50.515676: done, saving took 6.96 seconds 2022-04-12 05:13:50.613939: This epoch took 135.256569 s 2022-04-12 05:13:50.635360: svmem(total=99932160000, available=76237418496, percent=23.7, used=19630968832, free=48164040704, active=43276632064, inactive=6358740992, buffers=27918336, cached=32109232128, shared=2247385088, slab=781406208) 2022-04-12 05:13:50.705706: epoch: 344 2022-04-12 05:15:39.066425: train loss : -0.6639 2022-04-12 05:15:51.421017: validation loss: -0.6955 2022-04-12 05:15:51.439137: Average global foreground Dice: [0.9046, 0.9734, 0.9802, 0.3925] 2022-04-12 05:15:51.492176: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:15:52.402051: lr: 0.003485 2022-04-12 05:15:52.453373: saving scheduled checkpoint file... 2022-04-12 05:15:52.600998: saving checkpoint... 2022-04-12 05:15:59.686780: done, saving took 7.23 seconds 2022-04-12 05:15:59.866503: done 2022-04-12 05:16:00.033747: saving checkpoint... 2022-04-12 05:16:06.994907: done, saving took 7.09 seconds 2022-04-12 05:16:07.084537: This epoch took 136.310627 s 2022-04-12 05:16:07.116170: svmem(total=99932160000, available=76073340928, percent=23.9, used=19781099520, free=47997198336, active=43431739392, inactive=6374854656, buffers=28200960, cached=32125661184, shared=2261487616, slab=781959168) 2022-04-12 05:16:07.142478: epoch: 345 2022-04-12 05:17:57.354309: train loss : -0.6731 2022-04-12 05:18:09.130982: validation loss: -0.6765 2022-04-12 05:18:09.140022: Average global foreground Dice: [0.8723, 0.9658, 0.981, 0.4552] 2022-04-12 05:18:09.144991: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:18:09.891266: lr: 0.003465 2022-04-12 05:18:10.003434: saving scheduled checkpoint file... 2022-04-12 05:18:10.171183: saving checkpoint... 2022-04-12 05:18:22.155181: done, saving took 12.13 seconds 2022-04-12 05:18:22.220879: done 2022-04-12 05:18:22.362574: saving checkpoint... 2022-04-12 05:18:35.145247: done, saving took 12.91 seconds 2022-04-12 05:18:35.171818: This epoch took 147.951965 s 2022-04-12 05:18:35.182589: svmem(total=99932160000, available=76123615232, percent=23.8, used=19720192000, free=48044593152, active=43370782720, inactive=6387744768, buffers=28520448, cached=32138854400, shared=2271928320, slab=781193216) 2022-04-12 05:18:35.232770: epoch: 346 2022-04-12 05:20:24.765478: train loss : -0.6645 2022-04-12 05:20:37.367658: validation loss: -0.6809 2022-04-12 05:20:37.473025: Average global foreground Dice: [0.8896, 0.9696, 0.9825, 0.4236] 2022-04-12 05:20:37.595176: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:20:38.675980: lr: 0.003445 2022-04-12 05:20:38.752298: saving scheduled checkpoint file... 2022-04-12 05:20:38.940375: saving checkpoint... 2022-04-12 05:20:51.084178: done, saving took 12.27 seconds 2022-04-12 05:20:51.176898: done 2022-04-12 05:20:51.333257: saving checkpoint... 2022-04-12 05:20:58.947094: done, saving took 7.75 seconds 2022-04-12 05:20:59.088001: This epoch took 143.852924 s 2022-04-12 05:20:59.091768: svmem(total=99932160000, available=76047052800, percent=23.9, used=19807617024, free=47965548544, active=43458973696, inactive=6379216896, buffers=28786688, cached=32130207744, shared=2261073920, slab=781635584) 2022-04-12 05:20:59.093577: epoch: 347 2022-04-12 05:22:48.228425: train loss : -0.6818 2022-04-12 05:23:01.168249: validation loss: -0.6778 2022-04-12 05:23:01.224857: Average global foreground Dice: [0.8876, 0.9606, 0.9833, 0.4149] 2022-04-12 05:23:01.243354: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:23:02.294246: lr: 0.003424 2022-04-12 05:23:02.300331: saving scheduled checkpoint file... 2022-04-12 05:23:02.403321: saving checkpoint... 2022-04-12 05:23:12.873861: done, saving took 10.57 seconds 2022-04-12 05:23:13.077214: done 2022-04-12 05:23:13.156102: saving checkpoint... 2022-04-12 05:23:23.106620: done, saving took 10.00 seconds 2022-04-12 05:23:23.266395: This epoch took 144.169114 s 2022-04-12 05:23:23.275652: svmem(total=99932160000, available=76378800128, percent=23.6, used=19621154816, free=48294699008, active=43270094848, inactive=6237044736, buffers=29061120, cached=31987245056, shared=2115919872, slab=782622720) 2022-04-12 05:23:23.291135: epoch: 348 2022-04-12 05:25:12.516066: train loss : -0.6686 2022-04-12 05:25:24.014448: validation loss: -0.6813 2022-04-12 05:25:24.042263: Average global foreground Dice: [0.8921, 0.9692, 0.9836, 0.4225] 2022-04-12 05:25:24.055514: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:25:25.094881: lr: 0.003404 2022-04-12 05:25:25.097793: saving scheduled checkpoint file... 2022-04-12 05:25:25.189935: saving checkpoint... 2022-04-12 05:25:37.831753: done, saving took 12.72 seconds 2022-04-12 05:25:37.954642: done 2022-04-12 05:25:38.056086: saving checkpoint... 2022-04-12 05:25:54.996226: done, saving took 16.99 seconds 2022-04-12 05:25:55.900561: This epoch took 152.574084 s 2022-04-12 05:25:56.224576: svmem(total=99932160000, available=76178399232, percent=23.8, used=19690074112, free=48091688960, active=43331219456, inactive=6370373632, buffers=29388800, cached=32121008128, shared=2247274496, slab=794431488) 2022-04-12 05:25:56.458053: epoch: 349 2022-04-12 05:27:45.245538: train loss : -0.6598 2022-04-12 05:27:56.772701: validation loss: -0.6593 2022-04-12 05:27:56.776423: Average global foreground Dice: [0.8902, 0.9621, 0.9784, 0.4139] 2022-04-12 05:27:56.823610: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:27:57.613540: lr: 0.003384 2022-04-12 05:27:57.617774: saving scheduled checkpoint file... 2022-04-12 05:27:57.683423: saving checkpoint... 2022-04-12 05:28:07.043469: done, saving took 9.42 seconds 2022-04-12 05:28:07.165581: done 2022-04-12 05:28:07.168579: This epoch took 130.444265 s 2022-04-12 05:28:07.170779: svmem(total=99932160000, available=76190167040, percent=23.8, used=19667337216, free=48101126144, active=43321843712, inactive=6382911488, buffers=29634560, cached=32134062080, shared=2258264064, slab=781615104) 2022-04-12 05:28:07.177105: epoch: 350 2022-04-12 05:29:55.594508: train loss : -0.6649 2022-04-12 05:30:08.200893: validation loss: -0.6798 2022-04-12 05:30:08.653534: Average global foreground Dice: [0.8762, 0.9677, 0.9797, 0.4484] 2022-04-12 05:30:09.202022: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:30:10.242812: lr: 0.003364 2022-04-12 05:30:10.625462: saving scheduled checkpoint file... 2022-04-12 05:30:11.105744: saving checkpoint... 2022-04-12 05:30:29.099097: done, saving took 18.07 seconds 2022-04-12 05:30:29.167776: done 2022-04-12 05:30:29.266859: saving checkpoint... 2022-04-12 05:30:43.817839: done, saving took 14.64 seconds 2022-04-12 05:30:43.928219: This epoch took 156.745067 s 2022-04-12 05:30:44.004223: svmem(total=99932160000, available=76008955904, percent=23.9, used=19845443584, free=47917072384, active=43493924864, inactive=6388740096, buffers=29933568, cached=32139710464, shared=2261237760, slab=782475264) 2022-04-12 05:30:44.010624: epoch: 351 2022-04-12 05:32:32.345007: train loss : -0.6219 2022-04-12 05:32:44.864219: validation loss: -0.6417 2022-04-12 05:32:44.943458: Average global foreground Dice: [0.8692, 0.9609, 0.9715, 0.4426] 2022-04-12 05:32:45.097021: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:32:46.269436: lr: 0.003343 2022-04-12 05:32:46.348363: saving scheduled checkpoint file... 2022-04-12 05:32:46.546238: saving checkpoint... 2022-04-12 05:33:04.158898: done, saving took 17.72 seconds 2022-04-12 05:33:04.305711: done 2022-04-12 05:33:04.373352: This epoch took 140.341338 s 2022-04-12 05:33:04.452363: svmem(total=99932160000, available=76035452928, percent=23.9, used=19818778624, free=47941156864, active=43475877888, inactive=6390755328, buffers=30187520, cached=32142036992, shared=2261479424, slab=781631488) 2022-04-12 05:33:04.459209: epoch: 352 2022-04-12 05:34:52.695897: train loss : -0.6477 2022-04-12 05:35:04.849785: validation loss: -0.6646 2022-04-12 05:35:04.911952: Average global foreground Dice: [0.8716, 0.9673, 0.981, 0.4002] 2022-04-12 05:35:04.916841: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:35:05.831863: lr: 0.003323 2022-04-12 05:35:05.844200: saving scheduled checkpoint file... 2022-04-12 05:35:06.015931: saving checkpoint... 2022-04-12 05:35:15.590054: done, saving took 9.71 seconds 2022-04-12 05:35:15.619452: done 2022-04-12 05:35:15.620835: This epoch took 131.046390 s 2022-04-12 05:35:15.621866: svmem(total=99932160000, available=75929460736, percent=24.0, used=19924934656, free=47832727552, active=43578793984, inactive=6393307136, buffers=30453760, cached=32144044032, shared=2261348352, slab=781795328) 2022-04-12 05:35:15.622716: epoch: 353 2022-04-12 05:37:06.091404: train loss : -0.6251 2022-04-12 05:37:18.678916: validation loss: -0.6473 2022-04-12 05:37:18.699213: Average global foreground Dice: [0.8597, 0.9628, 0.9802, 0.4042] 2022-04-12 05:37:18.723348: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:37:19.667560: lr: 0.003303 2022-04-12 05:37:19.672138: saving scheduled checkpoint file... 2022-04-12 05:37:19.760292: saving checkpoint... 2022-04-12 05:37:31.204015: done, saving took 11.53 seconds 2022-04-12 05:37:31.300154: done 2022-04-12 05:37:31.328377: This epoch took 135.704216 s 2022-04-12 05:37:31.343285: svmem(total=99932160000, available=76008214528, percent=23.9, used=19845918720, free=47908999168, active=43501162496, inactive=6395359232, buffers=30732288, cached=32146509824, shared=2261348352, slab=781729792) 2022-04-12 05:37:31.361595: epoch: 354 2022-04-12 05:39:20.778295: train loss : -0.6402 2022-04-12 05:39:31.886701: validation loss: -0.6473 2022-04-12 05:39:31.907697: Average global foreground Dice: [0.8391, 0.9569, 0.9811, 0.4512] 2022-04-12 05:39:31.961480: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:39:32.914327: lr: 0.003282 2022-04-12 05:39:32.970431: saving scheduled checkpoint file... 2022-04-12 05:39:33.089848: saving checkpoint... 2022-04-12 05:39:43.114888: done, saving took 10.08 seconds 2022-04-12 05:39:43.292815: done 2022-04-12 05:39:43.312834: This epoch took 131.939048 s 2022-04-12 05:39:43.365546: svmem(total=99932160000, available=76148613120, percent=23.8, used=19705516032, free=48046993408, active=43359891456, inactive=6397222912, buffers=30973952, cached=32148676608, shared=2261479424, slab=781602816) 2022-04-12 05:39:43.371048: epoch: 355 2022-04-12 05:41:32.181107: train loss : -0.6592 2022-04-12 05:41:45.138706: validation loss: -0.6664 2022-04-12 05:41:45.900033: Average global foreground Dice: [0.8804, 0.9695, 0.9798, 0.4022] 2022-04-12 05:41:46.673366: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:41:48.440330: lr: 0.003262 2022-04-12 05:41:48.607803: saving scheduled checkpoint file... 2022-04-12 05:41:48.706399: saving checkpoint... 2022-04-12 05:41:58.163309: done, saving took 9.53 seconds 2022-04-12 05:41:58.407874: done 2022-04-12 05:41:58.529965: This epoch took 135.099445 s 2022-04-12 05:41:58.582972: svmem(total=99932160000, available=75901673472, percent=24.0, used=19952160768, free=47797559296, active=43607851008, inactive=6399508480, buffers=31232000, cached=32151207936, shared=2261688320, slab=782385152) 2022-04-12 05:41:58.713522: epoch: 356 2022-04-12 05:43:47.031577: train loss : -0.6562 2022-04-12 05:43:58.727968: validation loss: -0.6473 2022-04-12 05:43:59.098485: Average global foreground Dice: [0.8875, 0.9618, 0.9767, 0.4204] 2022-04-12 05:43:59.371495: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:44:00.171503: lr: 0.003241 2022-04-12 05:44:00.186548: saving scheduled checkpoint file... 2022-04-12 05:44:00.264770: saving checkpoint... 2022-04-12 05:44:22.243912: done, saving took 22.03 seconds 2022-04-12 05:44:22.337365: done 2022-04-12 05:44:22.343686: This epoch took 143.497946 s 2022-04-12 05:44:22.370156: svmem(total=99932160000, available=76149022720, percent=23.8, used=19705454592, free=48042594304, active=43357499392, inactive=6400978944, buffers=31494144, cached=32152616960, shared=2260996096, slab=783888384) 2022-04-12 05:44:22.374646: epoch: 357 2022-04-12 05:46:11.339071: train loss : -0.6528 2022-04-12 05:46:22.700041: validation loss: -0.6816 2022-04-12 05:46:22.800980: Average global foreground Dice: [0.8896, 0.968, 0.9801, 0.4111] 2022-04-12 05:46:22.863185: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:46:23.774957: lr: 0.003221 2022-04-12 05:46:23.777171: saving scheduled checkpoint file... 2022-04-12 05:46:23.878862: saving checkpoint... 2022-04-12 05:46:41.017667: done, saving took 17.24 seconds 2022-04-12 05:46:41.941414: done 2022-04-12 05:46:41.975598: This epoch took 139.559309 s 2022-04-12 05:46:42.030932: svmem(total=99932160000, available=76098834432, percent=23.8, used=19768881152, free=47989731328, active=43422023680, inactive=6390865920, buffers=31797248, cached=32141750272, shared=2247614464, slab=782782464) 2022-04-12 05:46:42.109789: epoch: 358 2022-04-12 05:48:30.399281: train loss : -0.6872 2022-04-12 05:48:41.884825: validation loss: -0.6759 2022-04-12 05:48:41.999119: Average global foreground Dice: [0.8901, 0.9623, 0.9811, 0.4335] 2022-04-12 05:48:42.065066: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:48:43.059279: lr: 0.003201 2022-04-12 05:48:43.222116: saving scheduled checkpoint file... 2022-04-12 05:48:43.426615: saving checkpoint... 2022-04-12 05:48:51.514491: done, saving took 8.18 seconds 2022-04-12 05:48:51.630441: done 2022-04-12 05:48:51.649048: This epoch took 129.476847 s 2022-04-12 05:48:51.713685: svmem(total=99932160000, available=76459307008, percent=23.5, used=19537547264, free=48347914240, active=43193016320, inactive=6263021568, buffers=32034816, cached=32014663680, shared=2118582272, slab=782704640) 2022-04-12 05:48:51.742273: epoch: 359 2022-04-12 05:50:40.792116: train loss : -0.6765 2022-04-12 05:50:52.570931: validation loss: -0.6707 2022-04-12 05:50:52.693725: Average global foreground Dice: [0.8753, 0.9718, 0.981, 0.4199] 2022-04-12 05:50:52.757187: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:50:53.869955: lr: 0.00318 2022-04-12 05:50:53.891971: saving scheduled checkpoint file... 2022-04-12 05:50:54.042198: saving checkpoint... 2022-04-12 05:51:03.446508: done, saving took 9.50 seconds 2022-04-12 05:51:03.731618: done 2022-04-12 05:51:03.757455: This epoch took 131.979396 s 2022-04-12 05:51:03.791670: svmem(total=99932160000, available=76094701568, percent=23.9, used=19772653568, free=48397443072, active=43424690176, inactive=5977620480, buffers=32284672, cached=31729778688, shared=2247614464, slab=780791808) 2022-04-12 05:51:03.796430: epoch: 360 2022-04-12 05:52:52.425937: train loss : -0.6553 2022-04-12 05:53:04.576068: validation loss: -0.6613 2022-04-12 05:53:04.647567: Average global foreground Dice: [0.883, 0.9638, 0.9811, 0.3648] 2022-04-12 05:53:04.703259: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:53:05.634429: lr: 0.00316 2022-04-12 05:53:05.697699: saving scheduled checkpoint file... 2022-04-12 05:53:05.883125: saving checkpoint... 2022-04-12 05:53:15.560608: done, saving took 9.80 seconds 2022-04-12 05:53:15.849104: done 2022-04-12 05:53:15.867600: This epoch took 132.024613 s 2022-04-12 05:53:15.919003: svmem(total=99932160000, available=76348039168, percent=23.6, used=19530051584, free=48232022016, active=43186786304, inactive=6385922048, buffers=32526336, cached=32137560064, shared=2237128704, slab=781852672) 2022-04-12 05:53:15.997624: epoch: 361 2022-04-12 05:55:05.898341: train loss : -0.6656 2022-04-12 05:55:18.809018: validation loss: -0.6713 2022-04-12 05:55:18.910905: Average global foreground Dice: [0.8988, 0.9656, 0.98, 0.4138] 2022-04-12 05:55:18.931808: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:55:19.690663: lr: 0.003139 2022-04-12 05:55:19.744539: saving scheduled checkpoint file... 2022-04-12 05:55:19.903516: saving checkpoint... 2022-04-12 05:55:37.005103: done, saving took 17.25 seconds 2022-04-12 05:55:37.076720: done 2022-04-12 05:55:37.145794: This epoch took 141.097648 s 2022-04-12 05:55:37.157348: svmem(total=99932160000, available=76089651200, percent=23.9, used=19757096960, free=47971102720, active=43415207936, inactive=6418964480, buffers=32849920, cached=32171110400, shared=2268454912, slab=782663680) 2022-04-12 05:55:37.328137: epoch: 362 2022-04-12 05:57:28.029638: train loss : -0.6779 2022-04-12 05:57:41.273528: validation loss: -0.6822 2022-04-12 05:57:41.343625: Average global foreground Dice: [0.8686, 0.9713, 0.9814, 0.4325] 2022-04-12 05:57:41.354779: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:57:42.190223: lr: 0.003119 2022-04-12 05:57:42.234809: saving scheduled checkpoint file... 2022-04-12 05:57:42.376272: saving checkpoint... 2022-04-12 05:57:51.675521: done, saving took 9.42 seconds 2022-04-12 05:57:51.808980: done 2022-04-12 05:57:51.885827: This epoch took 134.471801 s 2022-04-12 05:57:51.944428: svmem(total=99932160000, available=76256546816, percent=23.7, used=19599917056, free=48135942144, active=43258241024, inactive=6411149312, buffers=33091584, cached=32163209216, shared=2258792448, slab=781193216) 2022-04-12 05:57:51.970004: epoch: 363 2022-04-12 05:59:42.307796: train loss : -0.6498 2022-04-12 05:59:55.206014: validation loss: -0.6412 2022-04-12 05:59:55.215677: Average global foreground Dice: [0.8742, 0.9613, 0.9758, 0.404] 2022-04-12 05:59:55.269102: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 05:59:56.097541: lr: 0.003098 2022-04-12 05:59:56.120484: saving scheduled checkpoint file... 2022-04-12 05:59:56.285855: saving checkpoint... 2022-04-12 06:00:07.259086: done, saving took 11.09 seconds 2022-04-12 06:00:07.382606: done 2022-04-12 06:00:07.408283: This epoch took 135.332910 s 2022-04-12 06:00:07.460871: svmem(total=99932160000, available=76457127936, percent=23.5, used=19489329152, free=48333967360, active=43146989568, inactive=6323585024, buffers=33345536, cached=32075517952, shared=2168766464, slab=782106624) 2022-04-12 06:00:07.514372: epoch: 364 2022-04-12 06:01:57.192623: train loss : -0.6493 2022-04-12 06:02:09.382509: validation loss: -0.5845 2022-04-12 06:02:09.409490: Average global foreground Dice: [0.8661, 0.942, 0.9364, 0.4384] 2022-04-12 06:02:09.446332: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:02:10.323533: lr: 0.003078 2022-04-12 06:02:10.935210: saving scheduled checkpoint file... 2022-04-12 06:02:11.165303: saving checkpoint... 2022-04-12 06:02:22.392035: done, saving took 11.35 seconds 2022-04-12 06:02:22.538105: done 2022-04-12 06:02:22.597225: This epoch took 135.024442 s 2022-04-12 06:02:22.687736: svmem(total=99932160000, available=76068134912, percent=23.9, used=19785617408, free=47942684672, active=43437621248, inactive=6418178048, buffers=33587200, cached=32170270720, shared=2261516288, slab=781312000) 2022-04-12 06:02:22.733694: epoch: 365 2022-04-12 06:04:13.724101: train loss : -0.6365 2022-04-12 06:04:26.622988: validation loss: -0.6363 2022-04-12 06:04:26.701010: Average global foreground Dice: [0.8547, 0.9641, 0.9749, 0.4147] 2022-04-12 06:04:26.753365: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:04:27.845484: lr: 0.003057 2022-04-12 06:04:27.900619: saving scheduled checkpoint file... 2022-04-12 06:04:28.043599: saving checkpoint... 2022-04-12 06:04:37.902404: done, saving took 9.95 seconds 2022-04-12 06:04:38.035292: done 2022-04-12 06:04:38.074069: This epoch took 135.272089 s 2022-04-12 06:04:38.099540: svmem(total=99932160000, available=76087255040, percent=23.9, used=19766198272, free=47959363584, active=43418820608, inactive=6420389888, buffers=33837056, cached=32172761088, shared=2261688320, slab=782987264) 2022-04-12 06:04:38.123979: epoch: 366 2022-04-12 06:06:26.234931: train loss : -0.6475 2022-04-12 06:06:37.335589: validation loss: -0.6474 2022-04-12 06:06:37.450002: Average global foreground Dice: [0.8781, 0.9669, 0.9809, 0.41] 2022-04-12 06:06:37.534457: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:06:38.513105: lr: 0.003037 2022-04-12 06:06:38.692202: saving scheduled checkpoint file... 2022-04-12 06:06:38.814344: saving checkpoint... 2022-04-12 06:07:00.997627: done, saving took 22.26 seconds 2022-04-12 06:07:01.673225: done 2022-04-12 06:07:01.767669: This epoch took 143.579322 s 2022-04-12 06:07:01.806900: svmem(total=99932160000, available=75854159872, percent=24.1, used=19999428608, free=47723757568, active=43655286784, inactive=6422257664, buffers=34131968, cached=32174841856, shared=2261413888, slab=784179200) 2022-04-12 06:07:01.934433: epoch: 367 2022-04-12 06:08:50.069666: train loss : -0.6595 2022-04-12 06:09:02.014452: validation loss: -0.6925 2022-04-12 06:09:02.095202: Average global foreground Dice: [0.8785, 0.9698, 0.9811, 0.4391] 2022-04-12 06:09:02.154026: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:09:03.027901: lr: 0.003016 2022-04-12 06:09:03.252801: saving scheduled checkpoint file... 2022-04-12 06:09:03.535241: saving checkpoint... 2022-04-12 06:09:16.243021: done, saving took 12.78 seconds 2022-04-12 06:09:16.434330: done 2022-04-12 06:09:16.475455: This epoch took 134.528498 s 2022-04-12 06:09:16.531077: svmem(total=99932160000, available=75935862784, percent=24.0, used=19903647744, free=47803314176, active=43561738240, inactive=6438387712, buffers=34373632, cached=32190824448, shared=2275557376, slab=782409728) 2022-04-12 06:09:16.579539: epoch: 368 2022-04-12 06:11:04.508142: train loss : -0.6841 2022-04-12 06:11:16.227583: validation loss: -0.6882 2022-04-12 06:11:16.338485: Average global foreground Dice: [0.8892, 0.9679, 0.9818, 0.4212] 2022-04-12 06:11:16.412893: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:11:17.419137: lr: 0.002996 2022-04-12 06:11:17.537248: saving scheduled checkpoint file... 2022-04-12 06:11:17.772292: saving checkpoint... 2022-04-12 06:11:32.178140: done, saving took 14.52 seconds 2022-04-12 06:11:32.319502: done 2022-04-12 06:11:32.376557: This epoch took 135.769997 s 2022-04-12 06:11:32.435169: svmem(total=99932160000, available=75962380288, percent=24.0, used=19890814976, free=47827292160, active=43548815360, inactive=6426177536, buffers=34635776, cached=32179417088, shared=2261757952, slab=782426112) 2022-04-12 06:11:32.469501: epoch: 369 2022-04-12 06:13:21.427249: train loss : -0.6769 2022-04-12 06:13:31.796365: validation loss: -0.6867 2022-04-12 06:13:31.861183: Average global foreground Dice: [0.8947, 0.9672, 0.9818, 0.4427] 2022-04-12 06:13:31.977735: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:13:32.933680: lr: 0.002975 2022-04-12 06:13:32.968439: saving scheduled checkpoint file... 2022-04-12 06:13:33.102332: saving checkpoint... 2022-04-12 06:13:44.760167: done, saving took 11.75 seconds 2022-04-12 06:13:44.964097: done 2022-04-12 06:13:44.969299: This epoch took 132.451246 s 2022-04-12 06:13:44.986024: svmem(total=99932160000, available=75986956288, percent=24.0, used=19852619776, free=47849623552, active=43504648192, inactive=6442409984, buffers=34877440, cached=32195039232, shared=2275446784, slab=783597568) 2022-04-12 06:13:45.131298: epoch: 370 2022-04-12 06:15:34.024340: train loss : -0.6676 2022-04-12 06:15:46.273753: validation loss: -0.6941 2022-04-12 06:15:46.327993: Average global foreground Dice: [0.8841, 0.9692, 0.9822, 0.4267] 2022-04-12 06:15:46.386941: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:15:47.272483: lr: 0.002954 2022-04-12 06:15:47.399189: saving scheduled checkpoint file... 2022-04-12 06:15:47.553845: saving checkpoint... 2022-04-12 06:16:01.434481: done, saving took 14.02 seconds 2022-04-12 06:16:02.504875: done 2022-04-12 06:16:02.579503: This epoch took 137.436597 s 2022-04-12 06:16:02.637237: svmem(total=99932160000, available=76023046144, percent=23.9, used=19819585536, free=47883296768, active=43469557760, inactive=6441619456, buffers=35151872, cached=32194125824, shared=2272112640, slab=784166912) 2022-04-12 06:16:02.668210: epoch: 371 2022-04-12 06:17:52.928639: train loss : -0.6923 2022-04-12 06:18:03.893966: validation loss: -0.7002 2022-04-12 06:18:03.981241: Average global foreground Dice: [0.898, 0.9694, 0.9826, 0.4491] 2022-04-12 06:18:04.046074: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:18:05.168073: lr: 0.002934 2022-04-12 06:18:05.190372: saving scheduled checkpoint file... 2022-04-12 06:18:05.344792: saving checkpoint... 2022-04-12 06:18:14.406733: done, saving took 9.16 seconds 2022-04-12 06:18:14.478322: done 2022-04-12 06:18:14.542366: saving checkpoint... 2022-04-12 06:18:28.029356: done, saving took 13.54 seconds 2022-04-12 06:18:28.112434: This epoch took 145.344989 s 2022-04-12 06:18:28.152634: svmem(total=99932160000, available=75838451712, percent=24.1, used=20014829568, free=47696195584, active=43674259456, inactive=6432702464, buffers=35442688, cached=32185692160, shared=2261516288, slab=783536128) 2022-04-12 06:18:28.205752: epoch: 372 2022-04-12 06:20:18.158117: train loss : -0.6787 2022-04-12 06:20:29.508330: validation loss: -0.6742 2022-04-12 06:20:29.544833: Average global foreground Dice: [0.9133, 0.9702, 0.9756, 0.4177] 2022-04-12 06:20:29.592757: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:20:30.511932: lr: 0.002913 2022-04-12 06:20:30.542950: saving scheduled checkpoint file... 2022-04-12 06:20:30.665248: saving checkpoint... 2022-04-12 06:20:45.636599: done, saving took 15.04 seconds 2022-04-12 06:20:45.831517: done 2022-04-12 06:20:45.958557: saving checkpoint... 2022-04-12 06:21:01.456545: done, saving took 15.57 seconds 2022-04-12 06:21:01.584041: This epoch took 153.326877 s 2022-04-12 06:21:01.659297: svmem(total=99932160000, available=76241858560, percent=23.7, used=19761590272, free=48096845824, active=43417423872, inactive=6285344768, buffers=35729408, cached=32037994496, shared=2111479808, slab=784945152) 2022-04-12 06:21:01.737507: epoch: 373 2022-04-12 06:22:49.956306: train loss : -0.6791 2022-04-12 06:23:01.636941: validation loss: -0.6851 2022-04-12 06:23:01.752248: Average global foreground Dice: [0.8971, 0.9716, 0.9842, 0.4045] 2022-04-12 06:23:01.823099: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:23:02.931246: lr: 0.002892 2022-04-12 06:23:02.962126: saving scheduled checkpoint file... 2022-04-12 06:23:03.143432: saving checkpoint... 2022-04-12 06:23:17.740742: done, saving took 14.69 seconds 2022-04-12 06:23:17.844433: done 2022-04-12 06:23:17.940682: saving checkpoint... 2022-04-12 06:23:31.143053: done, saving took 13.29 seconds 2022-04-12 06:23:31.289445: This epoch took 149.535191 s 2022-04-12 06:23:31.348800: svmem(total=99932160000, available=76130701312, percent=23.8, used=19736350720, free=47983218688, active=43394994176, inactive=6424076288, buffers=36016128, cached=32176574464, shared=2247786496, slab=781950976) 2022-04-12 06:23:31.358484: epoch: 374 2022-04-12 06:25:20.083353: train loss : -0.6816 2022-04-12 06:25:31.932898: validation loss: -0.6949 2022-04-12 06:25:32.000736: Average global foreground Dice: [0.8885, 0.9701, 0.9814, 0.4274] 2022-04-12 06:25:32.054642: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:25:32.903319: lr: 0.002872 2022-04-12 06:25:32.966284: saving scheduled checkpoint file... 2022-04-12 06:25:33.092522: saving checkpoint... 2022-04-12 06:25:43.936969: done, saving took 10.95 seconds 2022-04-12 06:25:44.074132: done 2022-04-12 06:25:44.246988: saving checkpoint... 2022-04-12 06:25:54.837444: done, saving took 10.70 seconds 2022-04-12 06:25:54.982275: This epoch took 143.568981 s 2022-04-12 06:25:55.043617: svmem(total=99932160000, available=76469137408, percent=23.5, used=19485261824, free=48286212096, active=43151613952, inactive=6360518656, buffers=39260160, cached=32121425920, shared=2160316416, slab=783298560) 2022-04-12 06:25:55.088272: epoch: 375 2022-04-12 06:27:43.112982: train loss : -0.6931 2022-04-12 06:27:55.858935: validation loss: -0.6921 2022-04-12 06:27:55.917154: Average global foreground Dice: [0.8958, 0.9732, 0.9832, 0.4242] 2022-04-12 06:27:55.957017: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:27:57.047383: lr: 0.002851 2022-04-12 06:27:57.104162: saving scheduled checkpoint file... 2022-04-12 06:27:57.247452: saving checkpoint... 2022-04-12 06:28:14.558515: done, saving took 17.44 seconds 2022-04-12 06:28:14.875045: done 2022-04-12 06:28:15.086145: saving checkpoint... 2022-04-12 06:28:33.807597: done, saving took 18.85 seconds 2022-04-12 06:28:34.079730: This epoch took 158.951254 s 2022-04-12 06:28:34.170875: svmem(total=99932160000, available=76037435392, percent=23.9, used=19829698560, free=47852240896, active=43494408192, inactive=6450167808, buffers=39550976, cached=32210669568, shared=2247614464, slab=782831616) 2022-04-12 06:28:34.232138: epoch: 376 2022-04-12 06:30:25.001202: train loss : -0.6870 2022-04-12 06:30:36.448862: validation loss: -0.6937 2022-04-12 06:30:36.522129: Average global foreground Dice: [0.8952, 0.9718, 0.983, 0.4366] 2022-04-12 06:30:36.621015: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:30:38.090986: lr: 0.00283 2022-04-12 06:30:38.254953: saving scheduled checkpoint file... 2022-04-12 06:30:38.488892: saving checkpoint... 2022-04-12 06:30:58.461132: done, saving took 20.10 seconds 2022-04-12 06:30:58.670993: done 2022-04-12 06:30:58.865676: saving checkpoint... 2022-04-12 06:31:22.996386: done, saving took 24.27 seconds 2022-04-12 06:31:23.203120: This epoch took 168.915779 s 2022-04-12 06:31:23.276771: svmem(total=99932160000, available=76075782144, percent=23.9, used=19791306752, free=47887233024, active=43456159744, inactive=6453112832, buffers=39862272, cached=32213757952, shared=2247614464, slab=782327808) 2022-04-12 06:31:23.374680: epoch: 377 2022-04-12 06:33:12.698014: train loss : -0.6903 2022-04-12 06:33:24.393656: validation loss: -0.6999 2022-04-12 06:33:24.556930: Average global foreground Dice: [0.9113, 0.9738, 0.9859, 0.4319] 2022-04-12 06:33:24.711754: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:33:26.057100: lr: 0.00281 2022-04-12 06:33:26.214744: saving scheduled checkpoint file... 2022-04-12 06:33:26.357678: saving checkpoint... 2022-04-12 06:33:40.965616: done, saving took 14.68 seconds 2022-04-12 06:33:41.196345: done 2022-04-12 06:33:41.300846: saving checkpoint... 2022-04-12 06:33:59.883985: done, saving took 18.63 seconds 2022-04-12 06:34:00.113312: This epoch took 156.684819 s 2022-04-12 06:34:00.119214: svmem(total=99932160000, available=76017344512, percent=23.9, used=19838775296, free=47826161664, active=43513999360, inactive=6465851392, buffers=40148992, cached=32227074048, shared=2258518016, slab=782790656) 2022-04-12 06:34:00.139065: epoch: 378 2022-04-12 06:35:49.254816: train loss : -0.6896 2022-04-12 06:36:00.433165: validation loss: -0.6904 2022-04-12 06:36:00.474951: Average global foreground Dice: [0.8804, 0.971, 0.982, 0.4376] 2022-04-12 06:36:00.543332: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:36:01.435841: lr: 0.002789 2022-04-12 06:36:01.564043: saving scheduled checkpoint file... 2022-04-12 06:36:01.634629: saving checkpoint... 2022-04-12 06:36:35.579829: done, saving took 34.00 seconds 2022-04-12 06:36:35.866062: done 2022-04-12 06:36:35.990015: saving checkpoint... 2022-04-12 06:37:01.245186: done, saving took 25.32 seconds 2022-04-12 06:37:01.340139: This epoch took 181.143499 s 2022-04-12 06:37:01.393529: svmem(total=99932160000, available=76013391872, percent=23.9, used=19853545472, free=47818997760, active=43523612672, inactive=6458777600, buffers=40517632, cached=32219099136, shared=2247700480, slab=784187392) 2022-04-12 06:37:01.402165: epoch: 379 2022-04-12 06:38:50.008984: train loss : -0.6908 2022-04-12 06:39:01.649046: validation loss: -0.6678 2022-04-12 06:39:01.740703: Average global foreground Dice: [0.8961, 0.9656, 0.9822, 0.4096] 2022-04-12 06:39:01.801339: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:39:02.716670: lr: 0.002768 2022-04-12 06:39:02.756500: saving scheduled checkpoint file... 2022-04-12 06:39:02.817264: saving checkpoint... 2022-04-12 06:39:24.181504: done, saving took 21.42 seconds 2022-04-12 06:39:24.826856: done 2022-04-12 06:39:24.893922: This epoch took 143.380466 s 2022-04-12 06:39:24.955148: svmem(total=99932160000, available=75992055808, percent=24.0, used=19874885632, free=47795425280, active=43546124288, inactive=6460243968, buffers=40783872, cached=32221065216, shared=2247614464, slab=784400384) 2022-04-12 06:39:24.981175: epoch: 380 2022-04-12 06:41:13.681599: train loss : -0.6945 2022-04-12 06:41:25.643562: validation loss: -0.6920 2022-04-12 06:41:25.698095: Average global foreground Dice: [0.9091, 0.9721, 0.9813, 0.3962] 2022-04-12 06:41:25.757206: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:41:27.213616: lr: 0.002747 2022-04-12 06:41:28.041233: saving scheduled checkpoint file... 2022-04-12 06:41:28.892026: saving checkpoint... 2022-04-12 06:41:57.414382: done, saving took 28.58 seconds 2022-04-12 06:41:57.630955: done 2022-04-12 06:41:57.636092: This epoch took 152.515083 s 2022-04-12 06:41:57.683385: svmem(total=99932160000, available=76095610880, percent=23.9, used=19757355008, free=47898886144, active=43425460224, inactive=6473789440, buffers=41054208, cached=32234864640, shared=2261491712, slab=785854464) 2022-04-12 06:41:57.687148: epoch: 381 2022-04-12 06:43:47.570248: train loss : -0.6932 2022-04-12 06:43:59.912818: validation loss: -0.6880 2022-04-12 06:44:01.749159: Average global foreground Dice: [0.8912, 0.968, 0.9826, 0.4254] 2022-04-12 06:44:03.173429: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:44:05.888572: lr: 0.002727 2022-04-12 06:44:06.462264: saving scheduled checkpoint file... 2022-04-12 06:44:07.404556: saving checkpoint... 2022-04-12 06:44:37.529023: done, saving took 30.23 seconds 2022-04-12 06:44:37.776399: done 2022-04-12 06:44:37.790839: This epoch took 160.078421 s 2022-04-12 06:44:37.916501: svmem(total=99932160000, available=76278013952, percent=23.7, used=19573989376, free=48078639104, active=43243458560, inactive=6476603392, buffers=41345024, cached=32238186496, shared=2262347776, slab=785199104) 2022-04-12 06:44:38.018877: epoch: 382 2022-04-12 06:46:26.171303: train loss : -0.6878 2022-04-12 06:46:38.697454: validation loss: -0.6720 2022-04-12 06:46:40.036305: Average global foreground Dice: [0.8841, 0.967, 0.982, 0.4254] 2022-04-12 06:46:40.829752: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:46:43.201577: lr: 0.002706 2022-04-12 06:46:43.895059: saving scheduled checkpoint file... 2022-04-12 06:46:44.655111: saving checkpoint... 2022-04-12 06:47:07.564756: done, saving took 23.02 seconds 2022-04-12 06:47:07.861586: done 2022-04-12 06:47:07.871807: This epoch took 149.785493 s 2022-04-12 06:47:07.980397: svmem(total=99932160000, available=75913076736, percent=24.0, used=19925893120, free=47711248384, active=43593027584, inactive=6492307456, buffers=41635840, cached=32253382656, shared=2275479552, slab=784883712) 2022-04-12 06:47:07.991005: epoch: 383 2022-04-12 06:48:58.486601: train loss : -0.6928 2022-04-12 06:49:11.547407: validation loss: -0.6990 2022-04-12 06:49:12.315030: Average global foreground Dice: [0.8868, 0.9707, 0.9833, 0.4349] 2022-04-12 06:49:13.208024: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:49:15.306447: lr: 0.002685 2022-04-12 06:49:16.112236: saving scheduled checkpoint file... 2022-04-12 06:49:16.925956: saving checkpoint... 2022-04-12 06:49:41.592883: done, saving took 24.76 seconds 2022-04-12 06:49:43.600627: done 2022-04-12 06:49:44.171992: This epoch took 156.072765 s 2022-04-12 06:49:44.587141: svmem(total=99932160000, available=76125876224, percent=23.8, used=19726761984, free=47921369088, active=43386691584, inactive=6480838656, buffers=41926656, cached=32242102272, shared=2261729280, slab=795279360) 2022-04-12 06:49:45.167902: epoch: 384 2022-04-12 06:51:35.018931: train loss : -0.6892 2022-04-12 06:51:46.833852: validation loss: -0.7063 2022-04-12 06:51:47.372218: Average global foreground Dice: [0.8955, 0.972, 0.984, 0.452] 2022-04-12 06:51:47.924638: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:51:49.414274: lr: 0.002664 2022-04-12 06:51:49.527989: saving scheduled checkpoint file... 2022-04-12 06:51:49.800316: saving checkpoint... 2022-04-12 06:52:15.936859: done, saving took 26.19 seconds 2022-04-12 06:52:16.013254: done 2022-04-12 06:52:16.138352: saving checkpoint... 2022-04-12 06:52:34.153579: done, saving took 18.08 seconds 2022-04-12 06:52:40.425116: This epoch took 174.534710 s 2022-04-12 06:52:41.457612: svmem(total=99932160000, available=76065001472, percent=23.9, used=19801485312, free=47857582080, active=43454341120, inactive=6469574656, buffers=42254336, cached=32230838272, shared=2247856128, slab=799199232) 2022-04-12 06:52:42.266484: epoch: 385 2022-04-12 06:54:30.921387: train loss : -0.7092 2022-04-12 06:54:42.426071: validation loss: -0.6765 2022-04-12 06:54:42.506954: Average global foreground Dice: [0.8832, 0.9737, 0.9803, 0.4228] 2022-04-12 06:54:42.559724: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:54:43.625654: lr: 0.002643 2022-04-12 06:54:43.765429: saving scheduled checkpoint file... 2022-04-12 06:54:43.961767: saving checkpoint... 2022-04-12 06:54:56.548949: done, saving took 12.67 seconds 2022-04-12 06:54:56.810503: done 2022-04-12 06:54:56.931754: This epoch took 133.793738 s 2022-04-12 06:54:56.991634: svmem(total=99932160000, available=76245561344, percent=23.7, used=19606982656, free=48035815424, active=43276742656, inactive=6485995520, buffers=42500096, cached=32246861824, shared=2261757952, slab=784023552) 2022-04-12 06:54:57.050349: epoch: 386 2022-04-12 06:56:45.783870: train loss : -0.7068 2022-04-12 06:56:58.357751: validation loss: -0.6956 2022-04-12 06:56:58.473269: Average global foreground Dice: [0.9089, 0.9694, 0.9849, 0.4314] 2022-04-12 06:56:58.557915: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:56:59.526632: lr: 0.002622 2022-04-12 06:56:59.573147: saving scheduled checkpoint file... 2022-04-12 06:56:59.691561: saving checkpoint... 2022-04-12 06:57:16.391373: done, saving took 16.81 seconds 2022-04-12 06:57:16.531848: done 2022-04-12 06:57:16.697610: saving checkpoint... 2022-04-12 06:57:32.190414: done, saving took 15.60 seconds 2022-04-12 06:57:32.436574: This epoch took 155.328887 s 2022-04-12 06:57:32.501410: svmem(total=99932160000, available=76319215616, percent=23.6, used=19571421184, free=48056791040, active=43247104000, inactive=6496296960, buffers=43257856, cached=32260689920, shared=2223108096, slab=785059840) 2022-04-12 06:57:32.563852: epoch: 387 2022-04-12 06:59:22.074989: train loss : -0.6935 2022-04-12 06:59:33.516358: validation loss: -0.6958 2022-04-12 06:59:33.740260: Average global foreground Dice: [0.905, 0.9715, 0.9809, 0.4416] 2022-04-12 06:59:33.855659: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 06:59:34.803803: lr: 0.002601 2022-04-12 06:59:34.851678: saving scheduled checkpoint file... 2022-04-12 06:59:35.028328: saving checkpoint... 2022-04-12 06:59:50.485508: done, saving took 15.59 seconds 2022-04-12 06:59:50.658997: done 2022-04-12 06:59:50.828327: saving checkpoint... 2022-04-12 07:00:11.777652: done, saving took 21.07 seconds 2022-04-12 07:00:13.810747: This epoch took 161.192378 s 2022-04-12 07:00:14.197520: svmem(total=99932160000, available=76175015936, percent=23.8, used=19690713088, free=47909941248, active=43353657344, inactive=6522331136, buffers=43569152, cached=32287936512, shared=2247856128, slab=797356032) 2022-04-12 07:00:14.908543: epoch: 388 2022-04-12 07:02:05.935176: train loss : -0.6982 2022-04-12 07:02:17.888062: validation loss: -0.7093 2022-04-12 07:02:18.024570: Average global foreground Dice: [0.9074, 0.9737, 0.9843, 0.426] 2022-04-12 07:02:18.123264: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:02:18.868062: lr: 0.002581 2022-04-12 07:02:18.892507: saving scheduled checkpoint file... 2022-04-12 07:02:19.083882: saving checkpoint... 2022-04-12 07:02:33.758960: done, saving took 14.81 seconds 2022-04-12 07:02:34.594048: done 2022-04-12 07:02:35.017831: saving checkpoint... 2022-04-12 07:02:47.550426: done, saving took 12.65 seconds 2022-04-12 07:02:47.733677: This epoch took 152.094321 s 2022-04-12 07:02:47.790562: svmem(total=99932160000, available=75873357824, percent=24.1, used=19978235904, free=47605501952, active=43651948544, inactive=6538625024, buffers=43876352, cached=32304545792, shared=2261860352, slab=787165184) 2022-04-12 07:02:47.805889: epoch: 389 2022-04-12 07:04:37.481131: train loss : -0.6825 2022-04-12 07:04:49.255182: validation loss: -0.6913 2022-04-12 07:04:51.010360: Average global foreground Dice: [0.8998, 0.9701, 0.9818, 0.4526] 2022-04-12 07:04:52.090754: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:04:55.137411: lr: 0.00256 2022-04-12 07:04:55.674991: saving scheduled checkpoint file... 2022-04-12 07:04:56.761838: saving checkpoint... 2022-04-12 07:05:15.869943: done, saving took 19.19 seconds 2022-04-12 07:05:16.207855: done 2022-04-12 07:05:16.306206: saving checkpoint... 2022-04-12 07:05:32.124905: done, saving took 15.88 seconds 2022-04-12 07:05:32.269217: This epoch took 164.384693 s 2022-04-12 07:05:32.381202: svmem(total=99932160000, available=75874889728, percent=24.1, used=19976921088, free=47604363264, active=43652706304, inactive=6540730368, buffers=44220416, cached=32306655232, shared=2261671936, slab=786403328) 2022-04-12 07:05:32.450213: epoch: 390 2022-04-12 07:07:19.830864: train loss : -0.6896 2022-04-12 07:07:31.611951: validation loss: -0.6829 2022-04-12 07:07:32.585279: Average global foreground Dice: [0.8911, 0.9751, 0.9847, 0.428] 2022-04-12 07:07:33.534401: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:07:35.700839: lr: 0.002539 2022-04-12 07:07:36.554890: saving scheduled checkpoint file... 2022-04-12 07:07:36.980153: saving checkpoint... 2022-04-12 07:07:53.630683: done, saving took 16.70 seconds 2022-04-12 07:07:53.894972: done 2022-04-12 07:07:54.034049: saving checkpoint... 2022-04-12 07:08:10.399605: done, saving took 16.42 seconds 2022-04-12 07:08:10.664805: This epoch took 158.167681 s 2022-04-12 07:08:10.733611: svmem(total=99932160000, available=75904176128, percent=24.0, used=19937243136, free=47630954496, active=43616403456, inactive=6553313280, buffers=44531712, cached=32319430656, shared=2271944704, slab=786325504) 2022-04-12 07:08:10.796892: epoch: 391 2022-04-12 07:09:58.796343: train loss : -0.6934 2022-04-12 07:10:12.293234: validation loss: -0.6752 2022-04-12 07:10:13.251267: Average global foreground Dice: [0.8898, 0.9715, 0.9831, 0.4393] 2022-04-12 07:10:13.566762: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:10:14.565975: lr: 0.002518 2022-04-12 07:10:14.610516: saving scheduled checkpoint file... 2022-04-12 07:10:14.695722: saving checkpoint... 2022-04-12 07:10:35.786471: done, saving took 21.15 seconds 2022-04-12 07:10:35.942231: done 2022-04-12 07:10:36.107763: saving checkpoint... 2022-04-12 07:10:46.450167: done, saving took 10.41 seconds 2022-04-12 07:10:46.858191: This epoch took 155.960615 s 2022-04-12 07:10:46.925946: svmem(total=99932160000, available=75936182272, percent=24.0, used=19915251712, free=47660318720, active=43590127616, inactive=6545575936, buffers=44847104, cached=32311742464, shared=2261921792, slab=787165184) 2022-04-12 07:10:46.933282: epoch: 392 2022-04-12 07:12:36.744675: train loss : -0.6833 2022-04-12 07:12:49.415472: validation loss: -0.6925 2022-04-12 07:12:50.949137: Average global foreground Dice: [0.9099, 0.9729, 0.9831, 0.4241] 2022-04-12 07:12:52.010823: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:12:54.032191: lr: 0.002497 2022-04-12 07:12:54.452664: saving scheduled checkpoint file... 2022-04-12 07:12:54.901540: saving checkpoint... 2022-04-12 07:13:14.550739: done, saving took 19.71 seconds 2022-04-12 07:13:15.602521: done 2022-04-12 07:13:15.903647: saving checkpoint... 2022-04-12 07:13:27.419645: done, saving took 11.61 seconds 2022-04-12 07:13:27.636091: This epoch took 160.692914 s 2022-04-12 07:13:27.692883: svmem(total=99932160000, available=75953549312, percent=24.0, used=19897511936, free=47674982400, active=43571339264, inactive=6548267008, buffers=45162496, cached=32314503168, shared=2262282240, slab=788111360) 2022-04-12 07:13:27.766032: epoch: 393 2022-04-12 07:15:16.023799: train loss : -0.7021 2022-04-12 07:15:27.169770: validation loss: -0.6975 2022-04-12 07:15:27.486462: Average global foreground Dice: [0.9025, 0.9723, 0.9847, 0.4154] 2022-04-12 07:15:27.746557: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:15:28.958605: lr: 0.002476 2022-04-12 07:15:29.202849: saving scheduled checkpoint file... 2022-04-12 07:15:29.447055: saving checkpoint... 2022-04-12 07:15:51.181695: done, saving took 21.83 seconds 2022-04-12 07:15:52.592843: done 2022-04-12 07:15:52.634557: This epoch took 144.803482 s 2022-04-12 07:15:52.707887: svmem(total=99932160000, available=76252835840, percent=23.7, used=19612676096, free=47971889152, active=43293306880, inactive=6535868416, buffers=45469696, cached=32302125056, shared=2247770112, slab=787574784) 2022-04-12 07:15:52.764817: epoch: 394 2022-04-12 07:17:41.427097: train loss : -0.6871 2022-04-12 07:17:52.586895: validation loss: -0.6969 2022-04-12 07:17:52.938007: Average global foreground Dice: [0.9083, 0.9733, 0.9806, 0.4258] 2022-04-12 07:17:53.016834: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:17:53.871986: lr: 0.002455 2022-04-12 07:17:53.929024: saving scheduled checkpoint file... 2022-04-12 07:17:54.081040: saving checkpoint... 2022-04-12 07:18:14.242891: done, saving took 20.28 seconds 2022-04-12 07:18:14.710588: done 2022-04-12 07:18:14.924071: saving checkpoint... 2022-04-12 07:18:24.754941: done, saving took 9.95 seconds 2022-04-12 07:18:25.010335: This epoch took 151.927847 s 2022-04-12 07:18:25.084512: svmem(total=99932160000, available=75966783488, percent=24.0, used=19884580864, free=47683309568, active=43559268352, inactive=6551973888, buffers=45776896, cached=32318492672, shared=2261819392, slab=787513344) 2022-04-12 07:18:25.130379: epoch: 395 2022-04-12 07:20:16.369748: train loss : -0.7048 2022-04-12 07:20:30.719238: validation loss: -0.7041 2022-04-12 07:20:31.442675: Average global foreground Dice: [0.9046, 0.971, 0.9833, 0.4288] 2022-04-12 07:20:32.062500: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:20:33.207428: lr: 0.002434 2022-04-12 07:20:33.242364: saving scheduled checkpoint file... 2022-04-12 07:20:33.356919: saving checkpoint... 2022-04-12 07:20:43.993352: done, saving took 10.72 seconds 2022-04-12 07:20:44.219725: done 2022-04-12 07:20:44.402420: saving checkpoint... 2022-04-12 07:20:58.284770: done, saving took 13.95 seconds 2022-04-12 07:20:59.441975: This epoch took 154.206854 s 2022-04-12 07:20:59.489083: svmem(total=99932160000, available=76078043136, percent=23.9, used=19772338176, free=47788621824, active=43493167104, inactive=6513106944, buffers=46845952, cached=32324354048, shared=2262441984, slab=788832256) 2022-04-12 07:20:59.538944: epoch: 396 2022-04-12 07:22:47.787796: train loss : -0.6939 2022-04-12 07:22:59.761600: validation loss: -0.7033 2022-04-12 07:22:59.841737: Average global foreground Dice: [0.8922, 0.9689, 0.9828, 0.4293] 2022-04-12 07:22:59.919963: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:23:01.001147: lr: 0.002413 2022-04-12 07:23:01.052531: saving scheduled checkpoint file... 2022-04-12 07:23:01.169037: saving checkpoint... 2022-04-12 07:23:18.992598: done, saving took 17.92 seconds 2022-04-12 07:23:19.216300: done 2022-04-12 07:23:19.288435: This epoch took 139.687484 s 2022-04-12 07:23:19.356381: svmem(total=99932160000, available=76027965440, percent=23.9, used=19808423936, free=47736078336, active=43529330688, inactive=6529273856, buffers=47116288, cached=32340541440, shared=2276429824, slab=789217280) 2022-04-12 07:23:19.421048: epoch: 397 2022-04-12 07:25:08.606226: train loss : -0.6952 2022-04-12 07:25:20.598985: validation loss: -0.6772 2022-04-12 07:25:20.657096: Average global foreground Dice: [0.9008, 0.9698, 0.9843, 0.398] 2022-04-12 07:25:20.764042: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:25:22.141017: lr: 0.002391 2022-04-12 07:25:22.197330: saving scheduled checkpoint file... 2022-04-12 07:25:22.367005: saving checkpoint... 2022-04-12 07:25:47.054458: done, saving took 24.80 seconds 2022-04-12 07:25:47.683043: done 2022-04-12 07:25:47.739063: This epoch took 148.247913 s 2022-04-12 07:25:47.808056: svmem(total=99932160000, available=75894751232, percent=24.1, used=19956256768, free=47600418816, active=43678547968, inactive=6517149696, buffers=47452160, cached=32328032256, shared=2261798912, slab=788611072) 2022-04-12 07:25:47.855561: epoch: 398 2022-04-12 07:27:36.597250: train loss : -0.7069 2022-04-12 07:27:48.128013: validation loss: -0.7025 2022-04-12 07:27:48.176412: Average global foreground Dice: [0.8941, 0.9691, 0.9851, 0.4549] 2022-04-12 07:27:48.227232: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:27:49.273790: lr: 0.00237 2022-04-12 07:27:49.308401: saving scheduled checkpoint file... 2022-04-12 07:27:49.484710: saving checkpoint... 2022-04-12 07:28:12.588717: done, saving took 23.22 seconds 2022-04-12 07:28:12.926096: done 2022-04-12 07:28:13.167756: This epoch took 145.208327 s 2022-04-12 07:28:13.248460: svmem(total=99932160000, available=76000739328, percent=23.9, used=19845455872, free=47704084480, active=43567140864, inactive=6523371520, buffers=47710208, cached=32334909440, shared=2266509312, slab=787542016) 2022-04-12 07:28:13.298812: epoch: 399 2022-04-12 07:30:03.066450: train loss : -0.7066 2022-04-12 07:30:18.668978: validation loss: -0.7075 2022-04-12 07:30:18.842905: Average global foreground Dice: [0.8902, 0.9724, 0.9866, 0.4529] 2022-04-12 07:30:18.922023: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:30:20.051983: lr: 0.002349 2022-04-12 07:30:20.150557: saving scheduled checkpoint file... 2022-04-12 07:30:20.460497: saving checkpoint... 2022-04-12 07:30:35.732993: done, saving took 15.43 seconds 2022-04-12 07:30:35.985219: done 2022-04-12 07:30:36.205998: saving checkpoint... 2022-04-12 07:30:56.024114: done, saving took 19.95 seconds 2022-04-12 07:30:56.328981: This epoch took 162.978906 s 2022-04-12 07:30:56.387909: svmem(total=99932160000, available=75865714688, percent=24.1, used=19985240064, free=47566438400, active=43708121088, inactive=6520455168, buffers=48021504, cached=32332460032, shared=2261733376, slab=787664896) 2022-04-12 07:30:56.473044: epoch: 400 2022-04-12 07:32:46.599426: train loss : -0.7065 2022-04-12 07:32:59.669352: validation loss: -0.7002 2022-04-12 07:32:59.770522: Average global foreground Dice: [0.9117, 0.9749, 0.9831, 0.4256] 2022-04-12 07:32:59.835773: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:33:00.655686: lr: 0.002328 2022-04-12 07:33:00.685124: saving scheduled checkpoint file... 2022-04-12 07:33:00.854706: saving checkpoint... 2022-04-12 07:33:17.797977: done, saving took 17.07 seconds 2022-04-12 07:33:18.050209: done 2022-04-12 07:33:18.248028: saving checkpoint... 2022-04-12 07:33:35.945495: done, saving took 17.84 seconds 2022-04-12 07:33:36.091347: This epoch took 159.556485 s 2022-04-12 07:33:36.175864: svmem(total=99932160000, available=76137160704, percent=23.8, used=19713675264, free=47835426816, active=43436027904, inactive=6523604992, buffers=48320512, cached=32334737408, shared=2261749760, slab=786915328) 2022-04-12 07:33:36.252484: epoch: 401 2022-04-12 07:35:27.961187: train loss : -0.7045 2022-04-12 07:35:40.796966: validation loss: -0.6985 2022-04-12 07:35:40.833039: Average global foreground Dice: [0.8978, 0.9703, 0.9848, 0.4268] 2022-04-12 07:35:40.837190: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:35:41.717919: lr: 0.002307 2022-04-12 07:35:41.795727: saving scheduled checkpoint file... 2022-04-12 07:35:41.944761: saving checkpoint... 2022-04-12 07:35:59.472703: done, saving took 17.62 seconds 2022-04-12 07:35:59.621045: done 2022-04-12 07:35:59.735635: This epoch took 143.408048 s 2022-04-12 07:35:59.754089: svmem(total=99932160000, available=75900866560, percent=24.0, used=19950043136, free=47598940160, active=43675324416, inactive=6523359232, buffers=48611328, cached=32334565376, shared=2261569536, slab=789520384) 2022-04-12 07:35:59.864923: epoch: 402 2022-04-12 07:37:48.861035: train loss : -0.7126 2022-04-12 07:38:00.753039: validation loss: -0.6943 2022-04-12 07:38:00.834440: Average global foreground Dice: [0.9032, 0.9719, 0.9862, 0.4296] 2022-04-12 07:38:00.846876: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:38:01.605552: lr: 0.002286 2022-04-12 07:38:01.611955: saving scheduled checkpoint file... 2022-04-12 07:38:01.667003: saving checkpoint... 2022-04-12 07:38:22.644654: done, saving took 21.03 seconds 2022-04-12 07:38:22.726040: done 2022-04-12 07:38:22.800960: saving checkpoint... 2022-04-12 07:38:37.325566: done, saving took 14.59 seconds 2022-04-12 07:38:37.548022: This epoch took 157.560960 s 2022-04-12 07:38:37.622052: svmem(total=99932160000, available=76328239104, percent=23.6, used=19608162304, free=48023662592, active=43337424896, inactive=6439669760, buffers=48910336, cached=32251424768, shared=2176135168, slab=787763200) 2022-04-12 07:38:37.627741: epoch: 403 2022-04-12 07:40:25.746511: train loss : -0.6992 2022-04-12 07:40:37.913627: validation loss: -0.7073 2022-04-12 07:40:37.918950: Average global foreground Dice: [0.9003, 0.9704, 0.9844, 0.437] 2022-04-12 07:40:37.921288: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:40:39.013501: lr: 0.002264 2022-04-12 07:40:39.023465: saving scheduled checkpoint file... 2022-04-12 07:40:39.118035: saving checkpoint... 2022-04-12 07:40:48.100300: done, saving took 9.07 seconds 2022-04-12 07:40:48.183892: done 2022-04-12 07:40:48.280711: saving checkpoint... 2022-04-12 07:41:01.393395: done, saving took 13.19 seconds 2022-04-12 07:41:01.615955: This epoch took 143.937825 s 2022-04-12 07:41:01.624049: svmem(total=99932160000, available=76145664000, percent=23.8, used=19704791040, free=47838851072, active=43432103936, inactive=6527381504, buffers=49176576, cached=32339341312, shared=2262020096, slab=788258816) 2022-04-12 07:41:01.661772: epoch: 404 2022-04-12 07:42:50.164737: train loss : -0.7021 2022-04-12 07:43:01.743542: validation loss: -0.6992 2022-04-12 07:43:01.790885: Average global foreground Dice: [0.8957, 0.9724, 0.9835, 0.4241] 2022-04-12 07:43:01.842227: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:43:02.772104: lr: 0.002243 2022-04-12 07:43:02.823761: saving scheduled checkpoint file... 2022-04-12 07:43:02.962729: saving checkpoint... 2022-04-12 07:43:24.631143: done, saving took 21.75 seconds 2022-04-12 07:43:24.945195: done 2022-04-12 07:43:25.097221: This epoch took 143.397702 s 2022-04-12 07:43:25.212052: svmem(total=99932160000, available=76360400896, percent=23.6, used=19660431360, free=48053346304, active=43379744768, inactive=6358392832, buffers=49455104, cached=32168927232, shared=2091589632, slab=787214336) 2022-04-12 07:43:25.326033: epoch: 405 2022-04-12 07:45:14.557889: train loss : -0.7149 2022-04-12 07:45:26.557934: validation loss: -0.6807 2022-04-12 07:45:26.604563: Average global foreground Dice: [0.9002, 0.9692, 0.9822, 0.4319] 2022-04-12 07:45:26.650892: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:45:27.449561: lr: 0.002222 2022-04-12 07:45:27.539619: saving scheduled checkpoint file... 2022-04-12 07:45:27.730038: saving checkpoint... 2022-04-12 07:45:42.253910: done, saving took 14.66 seconds 2022-04-12 07:45:42.424247: done 2022-04-12 07:45:42.488175: This epoch took 136.987378 s 2022-04-12 07:45:42.507897: svmem(total=99932160000, available=76090368000, percent=23.9, used=19759435776, free=47781076992, active=43479474176, inactive=6531264512, buffers=49729536, cached=32341917696, shared=2262536192, slab=789483520) 2022-04-12 07:45:42.550931: epoch: 406 2022-04-12 07:47:31.915738: train loss : -0.7032 2022-04-12 07:47:43.864544: validation loss: -0.7039 2022-04-12 07:47:43.984792: Average global foreground Dice: [0.9091, 0.9718, 0.9854, 0.4193] 2022-04-12 07:47:44.047382: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:47:45.108008: lr: 0.002201 2022-04-12 07:47:45.187865: saving scheduled checkpoint file... 2022-04-12 07:47:45.382301: saving checkpoint... 2022-04-12 07:48:04.746757: done, saving took 19.48 seconds 2022-04-12 07:48:04.868809: done 2022-04-12 07:48:04.927646: This epoch took 142.371861 s 2022-04-12 07:48:04.958225: svmem(total=99932160000, available=76349243392, percent=23.6, used=19644502016, free=48138272768, active=43279773696, inactive=6380478464, buffers=49987584, cached=32099397632, shared=2118660096, slab=788889600) 2022-04-12 07:48:04.994737: epoch: 407 2022-04-12 07:49:55.603038: train loss : -0.7055 2022-04-12 07:50:07.796166: validation loss: -0.6907 2022-04-12 07:50:07.855884: Average global foreground Dice: [0.909, 0.9721, 0.9829, 0.4023] 2022-04-12 07:50:07.921591: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:50:09.098569: lr: 0.002179 2022-04-12 07:50:09.177247: saving scheduled checkpoint file... 2022-04-12 07:50:09.328829: saving checkpoint... 2022-04-12 07:50:21.221549: done, saving took 11.99 seconds 2022-04-12 07:50:21.402810: done 2022-04-12 07:50:21.417552: This epoch took 136.410704 s 2022-04-12 07:50:21.489937: svmem(total=99932160000, available=75871162368, percent=24.1, used=19979620352, free=47657938944, active=43609247744, inactive=6525214720, buffers=50241536, cached=32244359168, shared=2261426176, slab=789499904) 2022-04-12 07:50:21.544267: epoch: 408 2022-04-12 07:52:10.979413: train loss : -0.7016 2022-04-12 07:52:21.728666: validation loss: -0.6870 2022-04-12 07:52:21.795315: Average global foreground Dice: [0.8961, 0.9711, 0.9819, 0.4132] 2022-04-12 07:52:21.852471: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:52:23.044963: lr: 0.002158 2022-04-12 07:52:23.215706: saving scheduled checkpoint file... 2022-04-12 07:52:23.304343: saving checkpoint... 2022-04-12 07:52:42.282432: done, saving took 19.03 seconds 2022-04-12 07:52:42.357914: done 2022-04-12 07:52:42.390329: This epoch took 140.802504 s 2022-04-12 07:52:42.419375: svmem(total=99932160000, available=76048277504, percent=23.9, used=19788099584, free=47832735744, active=43419308032, inactive=6541139968, buffers=50507776, cached=32260816896, shared=2275823616, slab=790126592) 2022-04-12 07:52:42.481333: epoch: 409 2022-04-12 07:54:30.809453: train loss : -0.7094 2022-04-12 07:54:42.209373: validation loss: -0.7095 2022-04-12 07:54:42.264079: Average global foreground Dice: [0.9078, 0.9763, 0.9839, 0.4113] 2022-04-12 07:54:42.334697: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:54:43.271813: lr: 0.002137 2022-04-12 07:54:43.351284: saving scheduled checkpoint file... 2022-04-12 07:54:43.493868: saving checkpoint... 2022-04-12 07:55:00.703723: done, saving took 17.29 seconds 2022-04-12 07:55:00.880749: done 2022-04-12 07:55:00.935442: This epoch took 138.444918 s 2022-04-12 07:55:00.995848: svmem(total=99932160000, available=75839320064, percent=24.1, used=20011159552, free=47621517312, active=43640958976, inactive=6529134592, buffers=50778112, cached=32248705024, shared=2261725184, slab=789893120) 2022-04-12 07:55:01.074664: epoch: 410 2022-04-12 07:56:49.352129: train loss : -0.6972 2022-04-12 07:57:00.830008: validation loss: -0.7007 2022-04-12 07:57:00.860407: Average global foreground Dice: [0.9058, 0.9727, 0.9836, 0.4055] 2022-04-12 07:57:00.864535: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:57:01.865912: lr: 0.002115 2022-04-12 07:57:01.909056: saving scheduled checkpoint file... 2022-04-12 07:57:01.975823: saving checkpoint... 2022-04-12 07:57:13.186589: done, saving took 11.27 seconds 2022-04-12 07:57:13.378080: done 2022-04-12 07:57:13.486789: This epoch took 132.353921 s 2022-04-12 07:57:13.526454: svmem(total=99932160000, available=75795726336, percent=24.2, used=20054601728, free=47575867392, active=43692261376, inactive=6530945024, buffers=51040256, cached=32250650624, shared=2261811200, slab=787976192) 2022-04-12 07:57:13.577991: epoch: 411 2022-04-12 07:59:01.539442: train loss : -0.6993 2022-04-12 07:59:13.303147: validation loss: -0.6983 2022-04-12 07:59:13.399416: Average global foreground Dice: [0.9072, 0.9726, 0.9827, 0.4487] 2022-04-12 07:59:13.501353: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 07:59:14.712126: lr: 0.002094 2022-04-12 07:59:14.959241: saving scheduled checkpoint file... 2022-04-12 07:59:15.297496: saving checkpoint... 2022-04-12 07:59:33.020231: done, saving took 17.81 seconds 2022-04-12 07:59:33.156995: done 2022-04-12 07:59:33.177545: This epoch took 139.579366 s 2022-04-12 07:59:33.234164: svmem(total=99932160000, available=76066738176, percent=23.9, used=19783479296, free=47844478976, active=43414446080, inactive=6532939776, buffers=51310592, cached=32252891136, shared=2261843968, slab=790609920) 2022-04-12 07:59:33.240616: epoch: 412 2022-04-12 08:01:22.837315: train loss : -0.7027 2022-04-12 08:01:35.392804: validation loss: -0.6945 2022-04-12 08:01:35.433895: Average global foreground Dice: [0.8974, 0.9703, 0.9836, 0.4201] 2022-04-12 08:01:35.483516: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:01:36.261157: lr: 0.002072 2022-04-12 08:01:36.328265: saving scheduled checkpoint file... 2022-04-12 08:01:36.513788: saving checkpoint... 2022-04-12 08:01:54.308788: done, saving took 17.92 seconds 2022-04-12 08:01:54.647768: done 2022-04-12 08:01:54.717531: This epoch took 141.429155 s 2022-04-12 08:01:54.733867: svmem(total=99932160000, available=75873603584, percent=24.1, used=19976523776, free=47649275904, active=43609141248, inactive=6534627328, buffers=51568640, cached=32254791680, shared=2261766144, slab=790478848) 2022-04-12 08:01:54.778579: epoch: 413 2022-04-12 08:03:44.097785: train loss : -0.7038 2022-04-12 08:03:56.118860: validation loss: -0.6977 2022-04-12 08:03:56.235001: Average global foreground Dice: [0.8932, 0.9706, 0.9839, 0.4233] 2022-04-12 08:03:56.410695: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:03:57.701286: lr: 0.002051 2022-04-12 08:03:57.778422: saving scheduled checkpoint file... 2022-04-12 08:03:58.017667: saving checkpoint... 2022-04-12 08:04:15.514038: done, saving took 17.58 seconds 2022-04-12 08:04:15.703417: done 2022-04-12 08:04:15.851862: This epoch took 140.948155 s 2022-04-12 08:04:15.945495: svmem(total=99932160000, available=76093325312, percent=23.9, used=19742994432, free=47866585088, active=43378294784, inactive=6550646784, buffers=51818496, cached=32270761984, shared=2275635200, slab=790110208) 2022-04-12 08:04:16.037154: epoch: 414 2022-04-12 08:06:05.396752: train loss : -0.7110 2022-04-12 08:06:17.274884: validation loss: -0.6981 2022-04-12 08:06:17.341716: Average global foreground Dice: [0.8943, 0.9716, 0.9866, 0.4335] 2022-04-12 08:06:17.406021: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:06:18.285881: lr: 0.00203 2022-04-12 08:06:18.476740: saving scheduled checkpoint file... 2022-04-12 08:06:18.567157: saving checkpoint... 2022-04-12 08:06:36.691631: done, saving took 18.20 seconds 2022-04-12 08:06:36.889384: done 2022-04-12 08:06:36.969944: This epoch took 140.869012 s 2022-04-12 08:06:37.074882: svmem(total=99932160000, available=76051718144, percent=23.9, used=19788075008, free=47822639104, active=43420524544, inactive=6549184512, buffers=52125696, cached=32269320192, shared=2272165888, slab=790876160) 2022-04-12 08:06:37.224577: epoch: 415 2022-04-12 08:08:25.250075: train loss : -0.7169 2022-04-12 08:08:37.013261: validation loss: -0.7052 2022-04-12 08:08:37.062426: Average global foreground Dice: [0.8829, 0.9715, 0.9834, 0.4659] 2022-04-12 08:08:37.170464: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:08:38.506284: lr: 0.002008 2022-04-12 08:08:38.548543: saving scheduled checkpoint file... 2022-04-12 08:08:38.681300: saving checkpoint... 2022-04-12 08:08:57.361042: done, saving took 18.76 seconds 2022-04-12 08:08:57.504674: done 2022-04-12 08:08:57.525500: This epoch took 140.170347 s 2022-04-12 08:08:57.528703: svmem(total=99932160000, available=76143722496, percent=23.8, used=19706576896, free=47912452096, active=43338502144, inactive=6540632064, buffers=52375552, cached=32260755456, shared=2261680128, slab=791449600) 2022-04-12 08:08:57.542915: epoch: 416 2022-04-12 08:10:47.569997: train loss : -0.7187 2022-04-12 08:11:00.021554: validation loss: -0.7063 2022-04-12 08:11:00.119515: Average global foreground Dice: [0.8921, 0.9723, 0.9844, 0.453] 2022-04-12 08:11:00.401740: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:11:01.598190: lr: 0.001987 2022-04-12 08:11:01.739984: saving scheduled checkpoint file... 2022-04-12 08:11:01.899406: saving checkpoint... 2022-04-12 08:11:18.038908: done, saving took 16.24 seconds 2022-04-12 08:11:18.122488: done 2022-04-12 08:11:18.251956: saving checkpoint... 2022-04-12 08:11:32.508155: done, saving took 14.37 seconds 2022-04-12 08:11:32.665490: This epoch took 155.087125 s 2022-04-12 08:11:32.727834: svmem(total=99932160000, available=75996258304, percent=24.0, used=19853639680, free=47762489344, active=43486130176, inactive=6543015936, buffers=52678656, cached=32263352320, shared=2261929984, slab=791244800) 2022-04-12 08:11:32.731871: epoch: 417 2022-04-12 08:13:21.008599: train loss : -0.7110 2022-04-12 08:13:31.903664: validation loss: -0.7093 2022-04-12 08:13:31.975545: Average global foreground Dice: [0.9076, 0.9739, 0.9858, 0.4361] 2022-04-12 08:13:32.035501: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:13:33.113285: lr: 0.001965 2022-04-12 08:13:33.268503: saving scheduled checkpoint file... 2022-04-12 08:13:33.474883: saving checkpoint... 2022-04-12 08:13:53.436414: done, saving took 20.11 seconds 2022-04-12 08:13:53.546668: done 2022-04-12 08:13:53.746055: saving checkpoint... 2022-04-12 08:14:13.556807: done, saving took 19.95 seconds 2022-04-12 08:14:13.789342: This epoch took 161.000394 s 2022-04-12 08:14:13.899946: svmem(total=99932160000, available=75715911680, percent=24.2, used=20137783296, free=47479685120, active=43771404288, inactive=6541406208, buffers=52957184, cached=32261734400, shared=2258149376, slab=791220224) 2022-04-12 08:14:13.963510: epoch: 418 2022-04-12 08:16:03.901958: train loss : -0.7096 2022-04-12 08:16:16.772992: validation loss: -0.6913 2022-04-12 08:16:16.838055: Average global foreground Dice: [0.9072, 0.9705, 0.982, 0.4417] 2022-04-12 08:16:16.881444: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:16:17.757594: lr: 0.001943 2022-04-12 08:16:17.851071: saving scheduled checkpoint file... 2022-04-12 08:16:17.990751: saving checkpoint... 2022-04-12 08:16:29.823085: done, saving took 11.95 seconds 2022-04-12 08:16:30.032923: done 2022-04-12 08:16:30.206171: saving checkpoint... 2022-04-12 08:16:48.871703: done, saving took 18.78 seconds 2022-04-12 08:16:49.435592: This epoch took 155.418087 s 2022-04-12 08:16:49.493657: svmem(total=99932160000, available=75846578176, percent=24.1, used=20002729984, free=47607816192, active=43634253824, inactive=6548291584, buffers=53280768, cached=32268333056, shared=2262425600, slab=790982656) 2022-04-12 08:16:49.562802: epoch: 419 2022-04-12 08:18:40.250800: train loss : -0.7127 2022-04-12 08:18:52.889362: validation loss: -0.7150 2022-04-12 08:18:53.011352: Average global foreground Dice: [0.8988, 0.9724, 0.9843, 0.4419] 2022-04-12 08:18:53.124922: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:18:54.324144: lr: 0.001922 2022-04-12 08:18:54.386586: saving scheduled checkpoint file... 2022-04-12 08:18:54.514112: saving checkpoint... 2022-04-12 08:19:09.687089: done, saving took 15.26 seconds 2022-04-12 08:19:09.836769: done 2022-04-12 08:19:09.904907: saving checkpoint... 2022-04-12 08:19:23.770888: done, saving took 13.91 seconds 2022-04-12 08:19:23.919694: This epoch took 154.284875 s 2022-04-12 08:19:24.047047: svmem(total=99932160000, available=75985784832, percent=24.0, used=19860656128, free=47744507904, active=43491057664, inactive=6553378816, buffers=53583872, cached=32273412096, shared=2265305088, slab=790790144) 2022-04-12 08:19:24.196028: epoch: 420 2022-04-12 08:21:12.188523: train loss : -0.7203 2022-04-12 08:21:23.081841: validation loss: -0.7049 2022-04-12 08:21:23.146755: Average global foreground Dice: [0.8979, 0.9749, 0.9831, 0.454] 2022-04-12 08:21:23.219077: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:21:24.211665: lr: 0.0019 2022-04-12 08:21:24.315010: saving scheduled checkpoint file... 2022-04-12 08:21:24.455084: saving checkpoint... 2022-04-12 08:21:41.342369: done, saving took 16.96 seconds 2022-04-12 08:21:41.468067: done 2022-04-12 08:21:41.593933: saving checkpoint... 2022-04-12 08:21:58.971758: done, saving took 17.43 seconds 2022-04-12 08:21:59.156408: This epoch took 154.900376 s 2022-04-12 08:21:59.273514: svmem(total=99932160000, available=76109967360, percent=23.8, used=19862278144, free=47866281984, active=43490549760, inactive=6429741056, buffers=53891072, cached=32149708800, shared=2139545600, slab=790659072) 2022-04-12 08:21:59.333266: epoch: 421 2022-04-12 08:23:47.949031: train loss : -0.7035 2022-04-12 08:23:59.417181: validation loss: -0.7167 2022-04-12 08:23:59.504855: Average global foreground Dice: [0.9117, 0.9703, 0.9852, 0.4565] 2022-04-12 08:23:59.571941: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:24:00.540753: lr: 0.001879 2022-04-12 08:24:00.661542: saving scheduled checkpoint file... 2022-04-12 08:24:00.732588: saving checkpoint... 2022-04-12 08:24:15.948096: done, saving took 15.26 seconds 2022-04-12 08:24:16.157657: done 2022-04-12 08:24:16.319912: saving checkpoint... 2022-04-12 08:24:29.492969: done, saving took 13.22 seconds 2022-04-12 08:24:29.788046: This epoch took 150.406353 s 2022-04-12 08:24:29.901145: svmem(total=99932160000, available=76050042880, percent=23.9, used=19911864320, free=47803899904, active=43550490624, inactive=6442344448, buffers=54181888, cached=32162213888, shared=2149863424, slab=789966848) 2022-04-12 08:24:29.953315: epoch: 422 2022-04-12 08:26:19.169166: train loss : -0.6984 2022-04-12 08:26:31.227484: validation loss: -0.7049 2022-04-12 08:26:31.271537: Average global foreground Dice: [0.898, 0.9744, 0.9843, 0.4211] 2022-04-12 08:26:31.314261: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:26:32.366191: lr: 0.001857 2022-04-12 08:26:32.512518: saving scheduled checkpoint file... 2022-04-12 08:26:32.632126: saving checkpoint... 2022-04-12 08:26:43.509903: done, saving took 10.95 seconds 2022-04-12 08:26:43.957854: done 2022-04-12 08:26:44.034393: This epoch took 133.991483 s 2022-04-12 08:26:44.140515: svmem(total=99932160000, available=76089610240, percent=23.9, used=19760234496, free=47841284096, active=43391705088, inactive=6555619328, buffers=54456320, cached=32276185088, shared=2261954560, slab=790925312) 2022-04-12 08:26:44.202205: epoch: 423 2022-04-12 08:28:32.783446: train loss : -0.7115 2022-04-12 08:28:45.078617: validation loss: -0.7043 2022-04-12 08:28:45.130766: Average global foreground Dice: [0.9154, 0.9777, 0.9822, 0.4267] 2022-04-12 08:28:45.136786: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:28:46.080962: lr: 0.001835 2022-04-12 08:28:46.125815: saving scheduled checkpoint file... 2022-04-12 08:28:46.269103: saving checkpoint... 2022-04-12 08:29:30.682120: done, saving took 44.51 seconds 2022-04-12 08:29:30.955261: done 2022-04-12 08:29:31.002000: This epoch took 166.688947 s 2022-04-12 08:29:31.030621: svmem(total=99932160000, available=75766947840, percent=24.2, used=20086370304, free=47516024832, active=43717021696, inactive=6554120192, buffers=54784000, cached=32274980864, shared=2258374656, slab=792104960) 2022-04-12 08:29:31.074569: epoch: 424 2022-04-12 08:31:19.726488: train loss : -0.7254 2022-04-12 08:31:31.880136: validation loss: -0.7021 2022-04-12 08:31:31.956080: Average global foreground Dice: [0.9076, 0.9711, 0.982, 0.4397] 2022-04-12 08:31:32.027401: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:31:32.958666: lr: 0.001813 2022-04-12 08:31:33.143172: saving scheduled checkpoint file... 2022-04-12 08:31:33.427879: saving checkpoint... 2022-04-12 08:31:53.815382: done, saving took 20.49 seconds 2022-04-12 08:31:54.659089: done 2022-04-12 08:31:56.329945: This epoch took 145.201864 s 2022-04-12 08:31:56.592304: svmem(total=99932160000, available=76047298560, percent=23.9, used=19782582272, free=47793741824, active=43450347520, inactive=6551711744, buffers=55255040, cached=32300580864, shared=2281693184, slab=792485888) 2022-04-12 08:31:56.606244: epoch: 425 2022-04-12 08:33:46.307601: train loss : -0.7309 2022-04-12 08:33:57.974596: validation loss: -0.7138 2022-04-12 08:33:59.495518: Average global foreground Dice: [0.9111, 0.9768, 0.9838, 0.4213] 2022-04-12 08:33:59.647298: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:34:00.651086: lr: 0.001792 2022-04-12 08:34:00.716711: saving scheduled checkpoint file... 2022-04-12 08:34:00.886902: saving checkpoint... 2022-04-12 08:34:15.646734: done, saving took 14.87 seconds 2022-04-12 08:34:15.812212: done 2022-04-12 08:34:15.862979: This epoch took 139.158971 s 2022-04-12 08:34:15.910921: svmem(total=99932160000, available=76014874624, percent=23.9, used=19834982400, free=47759228928, active=43502903296, inactive=6533632000, buffers=55504896, cached=32282443776, shared=2261733376, slab=791588864) 2022-04-12 08:34:15.937056: epoch: 426 2022-04-12 08:36:06.157243: train loss : -0.7203 2022-04-12 08:36:18.177501: validation loss: -0.6954 2022-04-12 08:36:18.259268: Average global foreground Dice: [0.9034, 0.9737, 0.9808, 0.3964] 2022-04-12 08:36:18.420573: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:36:19.362928: lr: 0.00177 2022-04-12 08:36:19.557395: saving scheduled checkpoint file... 2022-04-12 08:36:19.654903: saving checkpoint... 2022-04-12 08:36:40.651346: done, saving took 21.09 seconds 2022-04-12 08:36:40.962465: done 2022-04-12 08:36:41.030005: This epoch took 145.040964 s 2022-04-12 08:36:41.170966: svmem(total=99932160000, available=76022382592, percent=23.9, used=19827494912, free=47764307968, active=43488772096, inactive=6535766016, buffers=55812096, cached=32284545024, shared=2261757952, slab=789700608) 2022-04-12 08:36:41.312619: epoch: 427 2022-04-12 08:38:29.641522: train loss : -0.7143 2022-04-12 08:38:41.160844: validation loss: -0.7038 2022-04-12 08:38:41.239653: Average global foreground Dice: [0.9013, 0.9769, 0.9844, 0.4448] 2022-04-12 08:38:41.285767: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:38:42.288964: lr: 0.001748 2022-04-12 08:38:42.353992: saving scheduled checkpoint file... 2022-04-12 08:38:42.477968: saving checkpoint... 2022-04-12 08:38:57.940877: done, saving took 15.58 seconds 2022-04-12 08:38:58.125233: done 2022-04-12 08:38:58.195861: This epoch took 136.821444 s 2022-04-12 08:38:58.252838: svmem(total=99932160000, available=75835330560, percent=24.1, used=20014313472, free=47574994944, active=43684122624, inactive=6537732096, buffers=56053760, cached=32286797824, shared=2261876736, slab=790876160) 2022-04-12 08:38:58.319747: epoch: 428 2022-04-12 08:40:47.812967: train loss : -0.7166 2022-04-12 08:40:59.989359: validation loss: -0.7130 2022-04-12 08:41:00.087732: Average global foreground Dice: [0.9234, 0.9724, 0.9836, 0.4246] 2022-04-12 08:41:00.192383: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:41:01.240754: lr: 0.001726 2022-04-12 08:41:01.303165: saving scheduled checkpoint file... 2022-04-12 08:41:01.435295: saving checkpoint... 2022-04-12 08:41:21.663485: done, saving took 20.34 seconds 2022-04-12 08:41:21.975082: done 2022-04-12 08:41:22.064828: This epoch took 143.689558 s 2022-04-12 08:41:22.134410: svmem(total=99932160000, available=75918323712, percent=24.0, used=19931369472, free=47655755776, active=43590828032, inactive=6539726848, buffers=56320000, cached=32288714752, shared=2261843968, slab=791777280) 2022-04-12 08:41:22.224849: epoch: 429 2022-04-12 08:43:11.626712: train loss : -0.7177 2022-04-12 08:43:23.941638: validation loss: -0.7186 2022-04-12 08:43:24.052505: Average global foreground Dice: [0.9147, 0.9766, 0.988, 0.4678] 2022-04-12 08:43:24.089799: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:43:24.993052: lr: 0.001704 2022-04-12 08:43:25.057147: saving scheduled checkpoint file... 2022-04-12 08:43:25.272879: saving checkpoint... 2022-04-12 08:43:40.605028: done, saving took 15.45 seconds 2022-04-12 08:43:40.895195: done 2022-04-12 08:43:41.082172: saving checkpoint... 2022-04-12 08:43:55.083899: done, saving took 14.13 seconds 2022-04-12 08:43:55.332250: This epoch took 153.037293 s 2022-04-12 08:43:55.445091: svmem(total=99932160000, available=76384870400, percent=23.6, used=19537367040, free=48119844864, active=43202154496, inactive=6469812224, buffers=56606720, cached=32218341376, shared=2189217792, slab=791003136) 2022-04-12 08:43:55.513815: epoch: 430 2022-04-12 08:45:44.847586: train loss : -0.7254 2022-04-12 08:45:57.809486: validation loss: -0.7135 2022-04-12 08:45:57.884137: Average global foreground Dice: [0.9013, 0.9732, 0.9865, 0.4407] 2022-04-12 08:45:57.905214: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:45:58.820852: lr: 0.001682 2022-04-12 08:45:58.887417: saving scheduled checkpoint file... 2022-04-12 08:45:59.043027: saving checkpoint... 2022-04-12 08:46:09.990841: done, saving took 11.10 seconds 2022-04-12 08:46:10.283806: done 2022-04-12 08:46:10.459602: saving checkpoint... 2022-04-12 08:46:27.316924: done, saving took 16.98 seconds 2022-04-12 08:46:27.524935: This epoch took 151.937584 s 2022-04-12 08:46:27.673818: svmem(total=99932160000, available=76207607808, percent=23.7, used=19631054848, free=47940222976, active=43291688960, inactive=6554836992, buffers=56926208, cached=32303955968, shared=2272739328, slab=792793088) 2022-04-12 08:46:27.748231: epoch: 431 2022-04-12 08:48:18.107899: train loss : -0.7177 2022-04-12 08:48:30.476682: validation loss: -0.7209 2022-04-12 08:48:30.646873: Average global foreground Dice: [0.8965, 0.9728, 0.9862, 0.4631] 2022-04-12 08:48:30.775852: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:48:31.951662: lr: 0.00166 2022-04-12 08:48:32.249263: saving scheduled checkpoint file... 2022-04-12 08:48:32.454965: saving checkpoint... 2022-04-12 08:48:44.638763: done, saving took 12.27 seconds 2022-04-12 08:48:44.803136: done 2022-04-12 08:48:44.923330: saving checkpoint... 2022-04-12 08:48:55.501505: done, saving took 10.63 seconds 2022-04-12 08:48:55.693581: This epoch took 147.862206 s 2022-04-12 08:48:55.711904: svmem(total=99932160000, available=75951517696, percent=24.0, used=19891929088, free=47681880064, active=43555876864, inactive=6552080384, buffers=57184256, cached=32301166592, shared=2267926528, slab=792137728) 2022-04-12 08:48:55.759850: epoch: 432 2022-04-12 08:50:44.551188: train loss : -0.7174 2022-04-12 08:50:56.527867: validation loss: -0.7175 2022-04-12 08:50:56.713940: Average global foreground Dice: [0.916, 0.9733, 0.9876, 0.4485] 2022-04-12 08:50:56.834764: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:50:57.829847: lr: 0.001638 2022-04-12 08:50:58.066966: saving scheduled checkpoint file... 2022-04-12 08:50:58.237976: saving checkpoint... 2022-04-12 08:51:13.934455: done, saving took 15.75 seconds 2022-04-12 08:51:14.134356: done 2022-04-12 08:51:14.232681: saving checkpoint... 2022-04-12 08:51:32.534098: done, saving took 18.35 seconds 2022-04-12 08:51:32.733730: This epoch took 156.957104 s 2022-04-12 08:51:32.780811: svmem(total=99932160000, available=75956027392, percent=24.0, used=19893399552, free=47683854336, active=43556540416, inactive=6548160512, buffers=57487360, cached=32297418752, shared=2261946368, slab=791797760) 2022-04-12 08:51:32.847398: epoch: 433 2022-04-12 08:53:21.477893: train loss : -0.7158 2022-04-12 08:53:33.582411: validation loss: -0.7022 2022-04-12 08:53:33.756739: Average global foreground Dice: [0.8868, 0.9749, 0.9814, 0.4121] 2022-04-12 08:53:33.800619: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:53:34.986463: lr: 0.001616 2022-04-12 08:53:35.098820: saving scheduled checkpoint file... 2022-04-12 08:53:35.333657: saving checkpoint... 2022-04-12 08:53:57.604448: done, saving took 22.34 seconds 2022-04-12 08:53:57.814090: done 2022-04-12 08:53:57.878335: This epoch took 144.980803 s 2022-04-12 08:53:57.899610: svmem(total=99932160000, available=76217516032, percent=23.7, used=19653976064, free=47943057408, active=43323772928, inactive=6528192512, buffers=57745408, cached=32277381120, shared=2239926272, slab=790765568) 2022-04-12 08:53:57.960280: epoch: 434 2022-04-12 08:55:44.382102: train loss : -0.7152 2022-04-12 08:55:56.351542: validation loss: -0.7097 2022-04-12 08:55:57.038378: Average global foreground Dice: [0.9061, 0.97, 0.9857, 0.4459] 2022-04-12 08:55:57.543156: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:55:58.488066: lr: 0.001594 2022-04-12 08:55:58.583663: saving scheduled checkpoint file... 2022-04-12 08:55:58.777493: saving checkpoint... 2022-04-12 08:56:17.163109: done, saving took 18.47 seconds 2022-04-12 08:56:17.342126: done 2022-04-12 08:56:17.409556: This epoch took 139.439993 s 2022-04-12 08:56:17.414336: svmem(total=99932160000, available=75855212544, percent=24.1, used=19993686016, free=47578587136, active=43662409728, inactive=6552473600, buffers=58032128, cached=32301854720, shared=2262401024, slab=790814720) 2022-04-12 08:56:17.455102: epoch: 435 2022-04-12 08:58:06.084938: train loss : -0.7268 2022-04-12 08:58:17.782247: validation loss: -0.7140 2022-04-12 08:58:17.821776: Average global foreground Dice: [0.928, 0.9743, 0.9867, 0.4367] 2022-04-12 08:58:17.894654: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 08:58:18.780194: lr: 0.001572 2022-04-12 08:58:18.833710: saving scheduled checkpoint file... 2022-04-12 08:58:18.966913: saving checkpoint... 2022-04-12 08:58:35.059527: done, saving took 16.21 seconds 2022-04-12 08:58:35.345971: done 2022-04-12 08:58:35.466489: This epoch took 137.965847 s 2022-04-12 08:58:35.590142: svmem(total=99932160000, available=75837239296, percent=24.1, used=20011778048, free=47558238208, active=43677573120, inactive=6554361856, buffers=58298368, cached=32303845376, shared=2262253568, slab=791859200) 2022-04-12 08:58:35.830398: epoch: 436 2022-04-12 09:00:23.354239: train loss : -0.7184 2022-04-12 09:00:34.975328: validation loss: -0.7144 2022-04-12 09:00:35.182282: Average global foreground Dice: [0.9056, 0.9761, 0.9847, 0.4459] 2022-04-12 09:00:35.307147: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:00:36.331454: lr: 0.00155 2022-04-12 09:00:36.472089: saving scheduled checkpoint file... 2022-04-12 09:00:36.630680: saving checkpoint... 2022-04-12 09:00:57.136927: done, saving took 20.62 seconds 2022-04-12 09:00:57.371157: done 2022-04-12 09:00:57.438480: This epoch took 141.507416 s 2022-04-12 09:00:57.496139: svmem(total=99932160000, available=75899953152, percent=24.0, used=19945320448, free=47618748416, active=43611848704, inactive=6559866880, buffers=58560512, cached=32309530624, shared=2265919488, slab=790704128) 2022-04-12 09:00:57.560860: epoch: 437 2022-04-12 09:02:48.116248: train loss : -0.7247 2022-04-12 09:03:00.448667: validation loss: -0.7269 2022-04-12 09:03:00.524745: Average global foreground Dice: [0.9015, 0.9746, 0.9884, 0.4564] 2022-04-12 09:03:00.590703: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:03:01.665258: lr: 0.001528 2022-04-12 09:03:01.846378: saving scheduled checkpoint file... 2022-04-12 09:03:02.019708: saving checkpoint... 2022-04-12 09:03:17.941401: done, saving took 16.00 seconds 2022-04-12 09:03:18.071722: done 2022-04-12 09:03:18.173649: saving checkpoint... 2022-04-12 09:03:29.118326: done, saving took 10.99 seconds 2022-04-12 09:03:29.778677: This epoch took 152.174695 s 2022-04-12 09:03:29.804365: svmem(total=99932160000, available=75979034624, percent=24.0, used=19856138240, free=47695265792, active=43517829120, inactive=6572335104, buffers=58843136, cached=32321912832, shared=2276020224, slab=790319104) 2022-04-12 09:03:29.856477: epoch: 438 2022-04-12 09:05:18.929478: train loss : -0.7261 2022-04-12 09:05:30.848821: validation loss: -0.7122 2022-04-12 09:05:30.908931: Average global foreground Dice: [0.9064, 0.9706, 0.9846, 0.4199] 2022-04-12 09:05:30.977944: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:05:33.309129: lr: 0.001506 2022-04-12 09:05:33.785899: saving scheduled checkpoint file... 2022-04-12 09:05:33.903449: saving checkpoint... 2022-04-12 09:05:49.020903: done, saving took 15.17 seconds 2022-04-12 09:05:49.198653: done 2022-04-12 09:05:49.253695: This epoch took 139.394086 s 2022-04-12 09:05:49.318812: svmem(total=99932160000, available=75686215680, percent=24.3, used=20163411968, free=47400337408, active=43828637696, inactive=6560223232, buffers=59121664, cached=32309288960, shared=2261647360, slab=791072768) 2022-04-12 09:05:49.358255: epoch: 439 2022-04-12 09:07:37.850579: train loss : -0.7176 2022-04-12 09:07:50.403580: validation loss: -0.7162 2022-04-12 09:07:50.468472: Average global foreground Dice: [0.9157, 0.9771, 0.9852, 0.4438] 2022-04-12 09:07:50.513173: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:07:51.183720: lr: 0.001483 2022-04-12 09:07:51.250564: saving scheduled checkpoint file... 2022-04-12 09:07:51.378764: saving checkpoint... 2022-04-12 09:08:06.949073: done, saving took 15.65 seconds 2022-04-12 09:08:07.190420: done 2022-04-12 09:08:07.231602: This epoch took 137.849848 s 2022-04-12 09:08:07.296552: svmem(total=99932160000, available=75924115456, percent=24.0, used=19911483392, free=47636123648, active=43581218816, inactive=6576078848, buffers=59371520, cached=32325181440, shared=2275635200, slab=792047616) 2022-04-12 09:08:07.339536: epoch: 440 2022-04-12 09:09:54.954034: train loss : -0.7196 2022-04-12 09:10:07.068194: validation loss: -0.7115 2022-04-12 09:10:07.284860: Average global foreground Dice: [0.8862, 0.9743, 0.9839, 0.4395] 2022-04-12 09:10:07.532044: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:10:08.706088: lr: 0.001461 2022-04-12 09:10:08.765537: saving scheduled checkpoint file... 2022-04-12 09:10:08.889176: saving checkpoint... 2022-04-12 09:10:24.242614: done, saving took 15.41 seconds 2022-04-12 09:10:24.436317: done 2022-04-12 09:10:24.492026: This epoch took 137.051038 s 2022-04-12 09:10:24.552158: svmem(total=99932160000, available=75856871424, percent=24.1, used=19992420352, free=47566761984, active=43659591680, inactive=6564048896, buffers=59621376, cached=32313356288, shared=2261827584, slab=792666112) 2022-04-12 09:10:24.617201: epoch: 441 2022-04-12 09:12:14.685973: train loss : -0.7159 2022-04-12 09:12:26.882353: validation loss: -0.7072 2022-04-12 09:12:26.967068: Average global foreground Dice: [0.9018, 0.9767, 0.9831, 0.4371] 2022-04-12 09:12:27.028002: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:12:28.117941: lr: 0.001439 2022-04-12 09:12:28.233485: saving scheduled checkpoint file... 2022-04-12 09:12:28.371161: saving checkpoint... 2022-04-12 09:12:43.971683: done, saving took 15.68 seconds 2022-04-12 09:12:44.292921: done 2022-04-12 09:12:44.517965: This epoch took 139.812774 s 2022-04-12 09:12:44.644960: svmem(total=99932160000, available=75756249088, percent=24.2, used=20093034496, free=47463870464, active=43757031424, inactive=6565593088, buffers=59883520, cached=32315371520, shared=2261860352, slab=793268224) 2022-04-12 09:12:44.768196: epoch: 442 2022-04-12 09:14:34.847015: train loss : -0.7190 2022-04-12 09:14:47.284689: validation loss: -0.7237 2022-04-12 09:14:47.333722: Average global foreground Dice: [0.9027, 0.9758, 0.9878, 0.4602] 2022-04-12 09:14:47.376082: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:14:48.597724: lr: 0.001416 2022-04-12 09:14:48.833356: saving scheduled checkpoint file... 2022-04-12 09:14:49.108690: saving checkpoint... 2022-04-12 09:15:07.963297: done, saving took 18.96 seconds 2022-04-12 09:15:08.446665: done 2022-04-12 09:15:08.553629: This epoch took 143.683353 s 2022-04-12 09:15:08.660593: svmem(total=99932160000, available=75829301248, percent=24.1, used=20019990528, free=47534706688, active=43686682624, inactive=6567387136, buffers=60166144, cached=32317296640, shared=2261741568, slab=791707648) 2022-04-12 09:15:08.726323: epoch: 443 2022-04-12 09:16:56.195689: train loss : -0.7134 2022-04-12 09:17:07.367232: validation loss: -0.7109 2022-04-12 09:17:07.446817: Average global foreground Dice: [0.915, 0.9705, 0.9818, 0.4097] 2022-04-12 09:17:07.498017: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:17:08.579978: lr: 0.001394 2022-04-12 09:17:08.596625: saving scheduled checkpoint file... 2022-04-12 09:17:08.732651: saving checkpoint... 2022-04-12 09:17:27.849025: done, saving took 19.20 seconds 2022-04-12 09:17:28.094588: done 2022-04-12 09:17:28.150775: This epoch took 139.374298 s 2022-04-12 09:17:28.220409: svmem(total=99932160000, available=75874914304, percent=24.1, used=19974303744, free=47577952256, active=43636965376, inactive=6569500672, buffers=60461056, cached=32319442944, shared=2261843968, slab=792371200) 2022-04-12 09:17:28.288272: epoch: 444 2022-04-12 09:19:17.904948: train loss : -0.7152 2022-04-12 09:19:29.234223: validation loss: -0.6934 2022-04-12 09:19:29.338559: Average global foreground Dice: [0.909, 0.975, 0.9832, 0.4176] 2022-04-12 09:19:29.461476: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:19:30.637044: lr: 0.001372 2022-04-12 09:19:30.786279: saving scheduled checkpoint file... 2022-04-12 09:19:30.968380: saving checkpoint... 2022-04-12 09:19:47.316781: done, saving took 16.41 seconds 2022-04-12 09:19:47.584543: done 2022-04-12 09:19:47.650159: This epoch took 139.297764 s 2022-04-12 09:19:47.701667: svmem(total=99932160000, available=75974049792, percent=24.0, used=19871293440, free=47674945536, active=43541196800, inactive=6575251456, buffers=60706816, cached=32325214208, shared=2265636864, slab=791003136) 2022-04-12 09:19:47.742908: epoch: 445 2022-04-12 09:21:36.647954: train loss : -0.7222 2022-04-12 09:21:49.371806: validation loss: -0.7026 2022-04-12 09:21:49.493620: Average global foreground Dice: [0.8911, 0.972, 0.9839, 0.4583] 2022-04-12 09:21:49.599908: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:21:50.383789: lr: 0.001349 2022-04-12 09:21:50.433750: saving scheduled checkpoint file... 2022-04-12 09:21:50.520822: saving checkpoint... 2022-04-12 09:22:04.681890: done, saving took 14.25 seconds 2022-04-12 09:22:04.893951: done 2022-04-12 09:22:04.965126: This epoch took 137.191978 s 2022-04-12 09:22:05.093257: svmem(total=99932160000, available=75982499840, percent=24.0, used=19866529792, free=47681069056, active=43535687680, inactive=6573858816, buffers=60968960, cached=32323592192, shared=2261921792, slab=792969216) 2022-04-12 09:22:05.295217: epoch: 446 2022-04-12 09:23:55.394680: train loss : -0.7285 2022-04-12 09:24:07.479546: validation loss: -0.7116 2022-04-12 09:24:07.537875: Average global foreground Dice: [0.9058, 0.9746, 0.9832, 0.454] 2022-04-12 09:24:07.575767: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:24:08.531539: lr: 0.001327 2022-04-12 09:24:08.578036: saving scheduled checkpoint file... 2022-04-12 09:24:08.732769: saving checkpoint... 2022-04-12 09:24:20.063866: done, saving took 11.45 seconds 2022-04-12 09:24:20.239664: done 2022-04-12 09:24:20.283470: This epoch took 134.957260 s 2022-04-12 09:24:20.338856: svmem(total=99932160000, available=75894161408, percent=24.1, used=19954982912, free=47590637568, active=43621466112, inactive=6575050752, buffers=61218816, cached=32325320704, shared=2261745664, slab=792199168) 2022-04-12 09:24:20.402031: epoch: 447 2022-04-12 09:26:10.024610: train loss : -0.7150 2022-04-12 09:26:22.225781: validation loss: -0.6965 2022-04-12 09:26:22.390811: Average global foreground Dice: [0.88, 0.9713, 0.9823, 0.4321] 2022-04-12 09:26:22.554254: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:26:24.049048: lr: 0.001304 2022-04-12 09:26:24.153833: saving scheduled checkpoint file... 2022-04-12 09:26:24.439388: saving checkpoint... 2022-04-12 09:26:38.125796: done, saving took 13.81 seconds 2022-04-12 09:26:38.369523: done 2022-04-12 09:26:38.468472: This epoch took 138.059901 s 2022-04-12 09:26:38.559996: svmem(total=99932160000, available=76019871744, percent=23.9, used=19829338112, free=47714172928, active=43495178240, inactive=6576783360, buffers=61530112, cached=32327118848, shared=2261569536, slab=794206208) 2022-04-12 09:26:38.637900: epoch: 448 2022-04-12 09:28:29.789920: train loss : -0.7093 2022-04-12 09:28:41.784683: validation loss: -0.7170 2022-04-12 09:28:41.871994: Average global foreground Dice: [0.9065, 0.9714, 0.9839, 0.4598] 2022-04-12 09:28:41.946548: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:28:42.918546: lr: 0.001282 2022-04-12 09:28:43.018838: saving scheduled checkpoint file... 2022-04-12 09:28:43.204382: saving checkpoint... 2022-04-12 09:28:55.954498: done, saving took 12.87 seconds 2022-04-12 09:28:56.259092: done 2022-04-12 09:28:56.436037: This epoch took 137.727097 s 2022-04-12 09:28:56.444509: svmem(total=99932160000, available=75806035968, percent=24.1, used=20033404928, free=47498088448, active=43705618432, inactive=6589833216, buffers=61775872, cached=32338890752, shared=2271322112, slab=792850432) 2022-04-12 09:28:56.529275: epoch: 449 2022-04-12 09:30:47.188398: train loss : -0.7138 2022-04-12 09:31:01.653784: validation loss: -0.7026 2022-04-12 09:31:01.706177: Average global foreground Dice: [0.9213, 0.9743, 0.9869, 0.4318] 2022-04-12 09:31:01.772963: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:31:02.712273: lr: 0.001259 2022-04-12 09:31:02.732512: saving scheduled checkpoint file... 2022-04-12 09:31:02.850225: saving checkpoint... 2022-04-12 09:31:14.648279: done, saving took 11.87 seconds 2022-04-12 09:31:14.875977: done 2022-04-12 09:31:14.984290: This epoch took 138.397202 s 2022-04-12 09:31:15.093554: svmem(total=99932160000, available=75898130432, percent=24.1, used=19940745216, free=47587897344, active=43614261248, inactive=6590889984, buffers=62025728, cached=32341491712, shared=2271846400, slab=791945216) 2022-04-12 09:31:15.233945: epoch: 450 2022-04-12 09:33:03.131469: train loss : -0.7161 2022-04-12 09:33:13.996048: validation loss: -0.7087 2022-04-12 09:33:14.101216: Average global foreground Dice: [0.9073, 0.9733, 0.983, 0.4191] 2022-04-12 09:33:14.199222: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:33:15.337137: lr: 0.001236 2022-04-12 09:33:15.382067: saving scheduled checkpoint file... 2022-04-12 09:33:15.486359: saving checkpoint... 2022-04-12 09:33:29.196835: done, saving took 13.78 seconds 2022-04-12 09:33:29.377005: done 2022-04-12 09:33:29.443643: This epoch took 134.140253 s 2022-04-12 09:33:29.484405: svmem(total=99932160000, available=75749105664, percent=24.2, used=20089860096, free=47436738560, active=43760181248, inactive=6593105920, buffers=62287872, cached=32343273472, shared=2271703040, slab=794660864) 2022-04-12 09:33:29.498569: epoch: 451 2022-04-12 09:35:18.375655: train loss : -0.7244 2022-04-12 09:35:30.170474: validation loss: -0.7138 2022-04-12 09:35:30.251146: Average global foreground Dice: [0.9164, 0.9734, 0.9863, 0.4095] 2022-04-12 09:35:30.337044: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:35:31.354971: lr: 0.001214 2022-04-12 09:35:31.471565: saving scheduled checkpoint file... 2022-04-12 09:35:31.581260: saving checkpoint... 2022-04-12 09:35:47.052774: done, saving took 15.52 seconds 2022-04-12 09:35:47.366035: done 2022-04-12 09:35:47.507415: This epoch took 137.962281 s 2022-04-12 09:35:47.520318: svmem(total=99932160000, available=75613216768, percent=24.3, used=20235427840, free=47298547712, active=43907768320, inactive=6584631296, buffers=62562304, cached=32335622144, shared=2261934080, slab=793010176) 2022-04-12 09:35:47.631957: epoch: 452 2022-04-12 09:37:36.538198: train loss : -0.7232 2022-04-12 09:37:49.022951: validation loss: -0.7012 2022-04-12 09:37:49.078996: Average global foreground Dice: [0.8907, 0.9748, 0.9817, 0.4378] 2022-04-12 09:37:49.109366: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:37:49.814968: lr: 0.001191 2022-04-12 09:37:49.863499: saving scheduled checkpoint file... 2022-04-12 09:37:49.949293: saving checkpoint... 2022-04-12 09:37:58.918580: done, saving took 9.04 seconds 2022-04-12 09:37:59.071980: done 2022-04-12 09:37:59.178501: This epoch took 131.486522 s 2022-04-12 09:37:59.294928: svmem(total=99932160000, available=75809914880, percent=24.1, used=20038873088, free=47493398528, active=43709292544, inactive=6586085376, buffers=62803968, cached=32337084416, shared=2261753856, slab=793403392) 2022-04-12 09:37:59.310997: epoch: 453 2022-04-12 09:39:48.459495: train loss : -0.7221 2022-04-12 09:39:59.958458: validation loss: -0.6976 2022-04-12 09:40:00.059260: Average global foreground Dice: [0.8747, 0.9745, 0.9825, 0.4437] 2022-04-12 09:40:00.165243: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:40:01.401823: lr: 0.001168 2022-04-12 09:40:01.459026: saving scheduled checkpoint file... 2022-04-12 09:40:01.559652: saving checkpoint... 2022-04-12 09:40:12.643247: done, saving took 11.18 seconds 2022-04-12 09:40:12.774224: done 2022-04-12 09:40:12.843904: This epoch took 133.488307 s 2022-04-12 09:40:12.904099: svmem(total=99932160000, available=75991433216, percent=24.0, used=19857137664, free=47672594432, active=43528298496, inactive=6588936192, buffers=63049728, cached=32339378176, shared=2261958656, slab=793944064) 2022-04-12 09:40:12.969831: epoch: 454 2022-04-12 09:42:02.669810: train loss : -0.7193 2022-04-12 09:42:15.389256: validation loss: -0.7083 2022-04-12 09:42:15.431178: Average global foreground Dice: [0.9207, 0.9746, 0.9851, 0.4495] 2022-04-12 09:42:15.504044: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:42:16.568036: lr: 0.001145 2022-04-12 09:42:16.589028: saving scheduled checkpoint file... 2022-04-12 09:42:16.748810: saving checkpoint... 2022-04-12 09:42:34.577718: done, saving took 17.93 seconds 2022-04-12 09:42:34.682379: done 2022-04-12 09:42:34.740435: This epoch took 141.736338 s 2022-04-12 09:42:34.751336: svmem(total=99932160000, available=75783200768, percent=24.2, used=20065345536, free=47462072320, active=43738009600, inactive=6590296064, buffers=63315968, cached=32341426176, shared=2261983232, slab=794054656) 2022-04-12 09:42:34.805346: epoch: 455 2022-04-12 09:44:25.373852: train loss : -0.7188 2022-04-12 09:44:36.437962: validation loss: -0.7213 2022-04-12 09:44:36.994911: Average global foreground Dice: [0.905, 0.9744, 0.9862, 0.4686] 2022-04-12 09:44:37.121148: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:44:38.096241: lr: 0.001122 2022-04-12 09:44:38.135981: saving scheduled checkpoint file... 2022-04-12 09:44:38.315216: saving checkpoint... 2022-04-12 09:44:46.480896: done, saving took 8.29 seconds 2022-04-12 09:44:46.619214: done 2022-04-12 09:44:46.643195: This epoch took 131.775235 s 2022-04-12 09:44:46.692946: svmem(total=99932160000, available=76070510592, percent=23.9, used=19765129216, free=47747334144, active=43436126208, inactive=6604976128, buffers=63553536, cached=32356143104, shared=2274836480, slab=793243648) 2022-04-12 09:44:46.747877: epoch: 456 2022-04-12 09:46:36.725579: train loss : -0.7297 2022-04-12 09:46:48.916904: validation loss: -0.7128 2022-04-12 09:46:49.054170: Average global foreground Dice: [0.9016, 0.9741, 0.9844, 0.4332] 2022-04-12 09:46:49.213277: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:46:50.273183: lr: 0.001099 2022-04-12 09:46:50.331841: saving scheduled checkpoint file... 2022-04-12 09:46:50.469252: saving checkpoint... 2022-04-12 09:47:10.915963: done, saving took 20.53 seconds 2022-04-12 09:47:11.120832: done 2022-04-12 09:47:11.244122: This epoch took 144.462559 s 2022-04-12 09:47:11.316315: svmem(total=99932160000, available=76121354240, percent=23.8, used=19727540224, free=47795863552, active=43401445376, inactive=6593880064, buffers=63844352, cached=32344911872, shared=2261344256, slab=792838144) 2022-04-12 09:47:11.385168: epoch: 457 2022-04-12 09:48:59.330557: train loss : -0.7321 2022-04-12 09:49:11.799668: validation loss: -0.7073 2022-04-12 09:49:11.863970: Average global foreground Dice: [0.9087, 0.9746, 0.9855, 0.4401] 2022-04-12 09:49:11.950947: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:49:12.861310: lr: 0.001076 2022-04-12 09:49:13.089695: saving scheduled checkpoint file... 2022-04-12 09:49:13.255484: saving checkpoint... 2022-04-12 09:49:25.263819: done, saving took 12.11 seconds 2022-04-12 09:49:25.402752: done 2022-04-12 09:49:25.440115: This epoch took 133.961656 s 2022-04-12 09:49:25.462965: svmem(total=99932160000, available=75915669504, percent=24.0, used=19932209152, free=47587975168, active=43610779648, inactive=6596538368, buffers=64106496, cached=32347869184, shared=2262515712, slab=791076864) 2022-04-12 09:49:25.487043: epoch: 458 2022-04-12 09:51:14.903824: train loss : -0.7286 2022-04-12 09:51:27.118204: validation loss: -0.7000 2022-04-12 09:51:27.134406: Average global foreground Dice: [0.8985, 0.9744, 0.984, 0.4096] 2022-04-12 09:51:27.190032: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:51:28.118809: lr: 0.001053 2022-04-12 09:51:28.192362: saving scheduled checkpoint file... 2022-04-12 09:51:28.368234: saving checkpoint... 2022-04-12 09:51:45.969388: done, saving took 17.71 seconds 2022-04-12 09:51:46.140685: done 2022-04-12 09:51:46.251279: This epoch took 140.734426 s 2022-04-12 09:51:46.298137: svmem(total=99932160000, available=76030459904, percent=23.9, used=19817992192, free=47700643840, active=43488976896, inactive=6597738496, buffers=64356352, cached=32349167616, shared=2261848064, slab=793128960) 2022-04-12 09:51:46.485532: epoch: 459 2022-04-12 09:53:37.318380: train loss : -0.7335 2022-04-12 09:53:49.842423: validation loss: -0.7107 2022-04-12 09:53:50.030429: Average global foreground Dice: [0.8972, 0.9741, 0.9836, 0.4361] 2022-04-12 09:53:50.147103: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:53:51.169796: lr: 0.00103 2022-04-12 09:53:51.231433: saving scheduled checkpoint file... 2022-04-12 09:53:51.350538: saving checkpoint... 2022-04-12 09:54:06.653203: done, saving took 15.42 seconds 2022-04-12 09:54:06.908142: done 2022-04-12 09:54:06.983125: This epoch took 140.424505 s 2022-04-12 09:54:07.047295: svmem(total=99932160000, available=75957977088, percent=24.0, used=19867693056, free=47625748480, active=43546804224, inactive=6620528640, buffers=64606208, cached=32374112256, shared=2284560384, slab=793042944) 2022-04-12 09:54:07.082422: epoch: 460 2022-04-12 09:55:56.552191: train loss : -0.7292 2022-04-12 09:56:08.256029: validation loss: -0.7064 2022-04-12 09:56:08.321870: Average global foreground Dice: [0.9204, 0.9753, 0.9837, 0.417] 2022-04-12 09:56:08.403236: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:56:09.585254: lr: 0.001007 2022-04-12 09:56:09.673002: saving scheduled checkpoint file... 2022-04-12 09:56:09.859554: saving checkpoint... 2022-04-12 09:57:09.128730: done, saving took 59.40 seconds 2022-04-12 09:57:09.352750: done 2022-04-12 09:57:09.389840: This epoch took 182.258896 s 2022-04-12 09:57:09.393403: svmem(total=99932160000, available=76019916800, percent=23.9, used=19828523008, free=47684939776, active=43494162432, inactive=6602637312, buffers=64987136, cached=32353710080, shared=2261848064, slab=795340800) 2022-04-12 09:57:09.460013: epoch: 461 2022-04-12 09:58:59.044973: train loss : -0.7311 2022-04-12 09:59:10.581056: validation loss: -0.7027 2022-04-12 09:59:10.658135: Average global foreground Dice: [0.9033, 0.9763, 0.9839, 0.4251] 2022-04-12 09:59:10.709627: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 09:59:12.035874: lr: 0.000983 2022-04-12 09:59:12.057307: saving scheduled checkpoint file... 2022-04-12 09:59:12.196615: saving checkpoint... 2022-04-12 09:59:23.976932: done, saving took 11.88 seconds 2022-04-12 09:59:24.156156: done 2022-04-12 09:59:24.186964: This epoch took 134.676955 s 2022-04-12 09:59:24.195415: svmem(total=99932160000, available=75712028672, percent=24.2, used=20140163072, free=47375024128, active=43812343808, inactive=6600306688, buffers=65232896, cached=32351739904, shared=2258096128, slab=794075136) 2022-04-12 09:59:24.229412: epoch: 462 2022-04-12 10:01:13.921248: train loss : -0.7232 2022-04-12 10:01:26.253117: validation loss: -0.6984 2022-04-12 10:01:26.363249: Average global foreground Dice: [0.8966, 0.974, 0.9841, 0.397] 2022-04-12 10:01:26.471825: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:01:27.562983: lr: 0.00096 2022-04-12 10:01:27.784473: saving scheduled checkpoint file... 2022-04-12 10:01:27.918030: saving checkpoint... 2022-04-12 10:01:42.607326: done, saving took 14.74 seconds 2022-04-12 10:01:42.848996: done 2022-04-12 10:01:42.970804: This epoch took 138.711996 s 2022-04-12 10:01:43.035626: svmem(total=99932160000, available=76007419904, percent=23.9, used=19827126272, free=47668092928, active=43501387776, inactive=6619762688, buffers=65499136, cached=32371441664, shared=2275639296, slab=793174016) 2022-04-12 10:01:43.141676: epoch: 463 2022-04-12 10:03:31.348979: train loss : -0.7204 2022-04-12 10:03:43.810635: validation loss: -0.7166 2022-04-12 10:03:43.870299: Average global foreground Dice: [0.9114, 0.9736, 0.9838, 0.4547] 2022-04-12 10:03:43.962274: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:03:44.978557: lr: 0.000937 2022-04-12 10:03:45.030962: saving scheduled checkpoint file... 2022-04-12 10:03:45.114607: saving checkpoint... 2022-04-12 10:03:58.656554: done, saving took 13.59 seconds 2022-04-12 10:03:58.901615: done 2022-04-12 10:03:58.956044: This epoch took 135.754566 s 2022-04-12 10:03:59.009556: svmem(total=99932160000, available=75859763200, percent=24.1, used=19981340672, free=47518318592, active=43662024704, inactive=6614593536, buffers=65744896, cached=32366755840, shared=2269130752, slab=792813568) 2022-04-12 10:03:59.061348: epoch: 464 2022-04-12 10:05:48.103860: train loss : -0.7304 2022-04-12 10:05:58.980424: validation loss: -0.7122 2022-04-12 10:05:59.059492: Average global foreground Dice: [0.8986, 0.9726, 0.986, 0.4402] 2022-04-12 10:05:59.096086: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:06:00.209364: lr: 0.000913 2022-04-12 10:06:00.253200: saving scheduled checkpoint file... 2022-04-12 10:06:00.378006: saving checkpoint... 2022-04-12 10:06:15.086632: done, saving took 14.78 seconds 2022-04-12 10:06:15.368074: done 2022-04-12 10:06:15.531288: This epoch took 136.378366 s 2022-04-12 10:06:15.641915: svmem(total=99932160000, available=76064829440, percent=23.9, used=19778985984, free=47721189376, active=43451416576, inactive=6614249472, buffers=66015232, cached=32365969408, shared=2266386432, slab=795086848) 2022-04-12 10:06:15.707519: epoch: 465 2022-04-12 10:08:03.745118: train loss : -0.7264 2022-04-12 10:08:15.209684: validation loss: -0.7046 2022-04-12 10:08:15.235366: Average global foreground Dice: [0.9088, 0.9763, 0.9849, 0.4181] 2022-04-12 10:08:15.302740: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:08:16.618310: lr: 0.00089 2022-04-12 10:08:16.675586: saving scheduled checkpoint file... 2022-04-12 10:08:16.825193: saving checkpoint... 2022-04-12 10:08:31.945497: done, saving took 15.21 seconds 2022-04-12 10:08:32.199018: done 2022-04-12 10:08:32.317618: This epoch took 136.559130 s 2022-04-12 10:08:32.379365: svmem(total=99932160000, available=75879915520, percent=24.1, used=19982524416, free=47534039040, active=43652931584, inactive=6597521408, buffers=66273280, cached=32349323264, shared=2247712768, slab=794288128) 2022-04-12 10:08:32.441606: epoch: 466 2022-04-12 10:10:21.500640: train loss : -0.7293 2022-04-12 10:10:32.676161: validation loss: -0.7178 2022-04-12 10:10:32.746160: Average global foreground Dice: [0.9089, 0.9726, 0.987, 0.4513] 2022-04-12 10:10:32.816256: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:10:33.718423: lr: 0.000866 2022-04-12 10:10:33.778710: saving scheduled checkpoint file... 2022-04-12 10:10:34.007262: saving checkpoint... 2022-04-12 10:10:47.497514: done, saving took 13.61 seconds 2022-04-12 10:10:47.562553: done 2022-04-12 10:10:47.573746: This epoch took 135.109710 s 2022-04-12 10:10:47.594054: svmem(total=99932160000, available=75772395520, percent=24.2, used=20070412288, free=47424438272, active=43745628160, inactive=6618873856, buffers=66527232, cached=32370782208, shared=2267299840, slab=795103232) 2022-04-12 10:10:47.620728: epoch: 467 2022-04-12 10:12:36.984428: train loss : -0.7273 2022-04-12 10:12:49.904049: validation loss: -0.7149 2022-04-12 10:12:50.084873: Average global foreground Dice: [0.9053, 0.976, 0.9861, 0.4275] 2022-04-12 10:12:50.133375: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:12:51.075684: lr: 0.000842 2022-04-12 10:12:51.158132: saving scheduled checkpoint file... 2022-04-12 10:12:51.276420: saving checkpoint... 2022-04-12 10:13:06.521004: done, saving took 15.32 seconds 2022-04-12 10:13:07.073392: done 2022-04-12 10:13:07.185206: This epoch took 139.531254 s 2022-04-12 10:13:07.301464: svmem(total=99932160000, available=75922104320, percent=24.0, used=19926163456, free=47571910656, active=43601879040, inactive=6615388160, buffers=66768896, cached=32367316992, shared=2261848064, slab=793423872) 2022-04-12 10:13:07.418223: epoch: 468 2022-04-12 10:14:55.551414: train loss : -0.7342 2022-04-12 10:15:07.196532: validation loss: -0.7164 2022-04-12 10:15:07.353468: Average global foreground Dice: [0.9096, 0.9766, 0.9853, 0.4282] 2022-04-12 10:15:07.398564: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:15:08.387355: lr: 0.000819 2022-04-12 10:15:08.500715: saving scheduled checkpoint file... 2022-04-12 10:15:08.659238: saving checkpoint... 2022-04-12 10:15:26.098568: done, saving took 17.49 seconds 2022-04-12 10:15:26.220686: done 2022-04-12 10:15:26.269710: This epoch took 138.786536 s 2022-04-12 10:15:26.306598: svmem(total=99932160000, available=75696017408, percent=24.3, used=20148195328, free=47343570944, active=43825057792, inactive=6621302784, buffers=67063808, cached=32373329920, shared=2265812992, slab=792530944) 2022-04-12 10:15:26.390392: epoch: 469 2022-04-12 10:17:14.333022: train loss : -0.7283 2022-04-12 10:17:26.293222: validation loss: -0.7131 2022-04-12 10:17:26.382615: Average global foreground Dice: [0.9131, 0.9753, 0.9833, 0.4385] 2022-04-12 10:17:26.438091: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:17:27.501236: lr: 0.000795 2022-04-12 10:17:27.546309: saving scheduled checkpoint file... 2022-04-12 10:17:27.679600: saving checkpoint... 2022-04-12 10:17:42.456923: done, saving took 14.85 seconds 2022-04-12 10:17:42.634217: done 2022-04-12 10:17:42.684001: This epoch took 136.291627 s 2022-04-12 10:17:42.824855: svmem(total=99932160000, available=76059541504, percent=23.9, used=19790561280, free=47704948736, active=43465707520, inactive=6617272320, buffers=67338240, cached=32369311744, shared=2259943424, slab=795738112) 2022-04-12 10:17:42.905948: epoch: 470 2022-04-12 10:19:31.621253: train loss : -0.7345 2022-04-12 10:19:43.610264: validation loss: -0.7097 2022-04-12 10:19:43.768312: Average global foreground Dice: [0.9099, 0.9778, 0.9852, 0.4488] 2022-04-12 10:19:43.888467: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:19:44.604093: lr: 0.000771 2022-04-12 10:19:44.717983: saving scheduled checkpoint file... 2022-04-12 10:19:44.826551: saving checkpoint... 2022-04-12 10:20:02.221099: done, saving took 17.45 seconds 2022-04-12 10:20:02.321107: done 2022-04-12 10:20:02.405175: This epoch took 139.438050 s 2022-04-12 10:20:02.409672: svmem(total=99932160000, available=75707015168, percent=24.2, used=20141314048, free=47350153216, active=43816525824, inactive=6620889088, buffers=67588096, cached=32373104640, shared=2261598208, slab=796798976) 2022-04-12 10:20:02.459807: epoch: 471 2022-04-12 10:21:50.557379: train loss : -0.7354 2022-04-12 10:22:02.211468: validation loss: -0.7130 2022-04-12 10:22:02.309422: Average global foreground Dice: [0.9088, 0.9749, 0.9873, 0.4407] 2022-04-12 10:22:02.373798: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:22:03.499635: lr: 0.000747 2022-04-12 10:22:03.610942: saving scheduled checkpoint file... 2022-04-12 10:22:03.725909: saving checkpoint... 2022-04-12 10:22:15.549989: done, saving took 11.88 seconds 2022-04-12 10:22:15.785582: done 2022-04-12 10:22:15.848687: This epoch took 133.356072 s 2022-04-12 10:22:15.903993: svmem(total=99932160000, available=75715776512, percent=24.2, used=20132642816, free=47393927168, active=43794092032, inactive=6603558912, buffers=67846144, cached=32337743872, shared=2261495808, slab=794791936) 2022-04-12 10:22:15.932716: epoch: 472 2022-04-12 10:24:03.883206: train loss : -0.7303 2022-04-12 10:24:15.332510: validation loss: -0.7175 2022-04-12 10:24:15.396894: Average global foreground Dice: [0.9105, 0.9772, 0.9858, 0.454] 2022-04-12 10:24:15.476344: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:24:16.616847: lr: 0.000723 2022-04-12 10:24:16.741403: saving scheduled checkpoint file... 2022-04-12 10:24:16.930396: saving checkpoint... 2022-04-12 10:24:34.019920: done, saving took 17.21 seconds 2022-04-12 10:24:34.274335: done 2022-04-12 10:24:34.346035: This epoch took 138.373331 s 2022-04-12 10:24:34.371688: svmem(total=99932160000, available=76162420736, percent=23.8, used=19679645696, free=47838388224, active=43350593536, inactive=6599909376, buffers=68104192, cached=32346021888, shared=2267709440, slab=794939392) 2022-04-12 10:24:34.466215: epoch: 473 2022-04-12 10:26:22.826964: train loss : -0.7199 2022-04-12 10:26:34.991534: validation loss: -0.7292 2022-04-12 10:26:35.038276: Average global foreground Dice: [0.9408, 0.976, 0.9872, 0.4164] 2022-04-12 10:26:35.096125: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:26:36.183786: lr: 0.000699 2022-04-12 10:26:36.347634: saving scheduled checkpoint file... 2022-04-12 10:26:36.494096: saving checkpoint... 2022-04-12 10:26:48.121784: done, saving took 11.71 seconds 2022-04-12 10:26:48.306926: done 2022-04-12 10:26:48.416407: saving checkpoint... 2022-04-12 10:27:04.436145: done, saving took 16.07 seconds 2022-04-12 10:27:04.658317: This epoch took 150.140742 s 2022-04-12 10:27:04.767594: svmem(total=99932160000, available=75909754880, percent=24.0, used=19919319040, free=47583268864, active=43590676480, inactive=6614618112, buffers=68419584, cached=32361152512, shared=2280787968, slab=794329088) 2022-04-12 10:27:04.830812: epoch: 474 2022-04-12 10:28:53.362084: train loss : -0.7212 2022-04-12 10:29:05.034806: validation loss: -0.7273 2022-04-12 10:29:05.242502: Average global foreground Dice: [0.9149, 0.9739, 0.9862, 0.4513] 2022-04-12 10:29:05.323119: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:29:06.202212: lr: 0.000675 2022-04-12 10:29:06.423207: saving scheduled checkpoint file... 2022-04-12 10:29:06.563015: saving checkpoint... 2022-04-12 10:29:21.523860: done, saving took 15.02 seconds 2022-04-12 10:29:21.680168: done 2022-04-12 10:29:21.798189: saving checkpoint... 2022-04-12 10:29:36.632397: done, saving took 14.88 seconds 2022-04-12 10:29:36.911813: This epoch took 152.031023 s 2022-04-12 10:29:36.976494: svmem(total=99932160000, available=75666362368, percent=24.3, used=20181798912, free=47337496576, active=43852283904, inactive=6598098944, buffers=68710400, cached=32344154112, shared=2261676032, slab=793718784) 2022-04-12 10:29:37.042647: epoch: 475 2022-04-12 10:31:25.915004: train loss : -0.7359 2022-04-12 10:31:37.255411: validation loss: -0.7167 2022-04-12 10:31:37.409821: Average global foreground Dice: [0.9129, 0.9741, 0.9848, 0.4214] 2022-04-12 10:31:37.579187: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:31:38.698156: lr: 0.00065 2022-04-12 10:31:38.797100: saving scheduled checkpoint file... 2022-04-12 10:31:38.986024: saving checkpoint... 2022-04-12 10:31:57.408284: done, saving took 18.47 seconds 2022-04-12 10:31:57.626906: done 2022-04-12 10:31:57.695095: This epoch took 140.493094 s 2022-04-12 10:31:57.755935: svmem(total=99932160000, available=75884515328, percent=24.1, used=19963375616, free=47553339392, active=43638898688, inactive=6599827456, buffers=68964352, cached=32346480640, shared=2261950464, slab=793788416) 2022-04-12 10:31:57.852424: epoch: 476 2022-04-12 10:33:45.818001: train loss : -0.7307 2022-04-12 10:33:57.206014: validation loss: -0.7111 2022-04-12 10:33:57.335142: Average global foreground Dice: [0.9072, 0.9753, 0.9851, 0.3982] 2022-04-12 10:33:57.361052: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:33:58.361533: lr: 0.000626 2022-04-12 10:33:58.407162: saving scheduled checkpoint file... 2022-04-12 10:33:58.562951: saving checkpoint... 2022-04-12 10:34:14.287210: done, saving took 15.82 seconds 2022-04-12 10:34:14.582330: done 2022-04-12 10:34:14.624081: This epoch took 136.696230 s 2022-04-12 10:34:14.685723: svmem(total=99932160000, available=75847270400, percent=24.1, used=19992707072, free=47513964544, active=43666616320, inactive=6609428480, buffers=69206016, cached=32356282368, shared=2269777920, slab=793686016) 2022-04-12 10:34:14.764657: epoch: 477 2022-04-12 10:36:03.474325: train loss : -0.7500 2022-04-12 10:36:15.051471: validation loss: -0.7329 2022-04-12 10:36:15.140783: Average global foreground Dice: [0.9209, 0.9799, 0.985, 0.4395] 2022-04-12 10:36:15.206306: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:36:16.226604: lr: 0.000601 2022-04-12 10:36:16.352742: saving scheduled checkpoint file... 2022-04-12 10:36:16.502886: saving checkpoint... 2022-04-12 10:36:33.060949: done, saving took 16.63 seconds 2022-04-12 10:36:33.157156: done 2022-04-12 10:36:33.177373: This epoch took 138.282618 s 2022-04-12 10:36:33.213595: svmem(total=99932160000, available=75763449856, percent=24.2, used=20083896320, free=47427784704, active=43751403520, inactive=6604410880, buffers=69496832, cached=32350982144, shared=2262257664, slab=794075136) 2022-04-12 10:36:33.445303: epoch: 478 2022-04-12 10:38:22.055734: train loss : -0.7433 2022-04-12 10:38:34.132002: validation loss: -0.7173 2022-04-12 10:38:34.271930: Average global foreground Dice: [0.9144, 0.9748, 0.9855, 0.425] 2022-04-12 10:38:34.315281: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:38:35.035644: lr: 0.000577 2022-04-12 10:38:35.039597: saving scheduled checkpoint file... 2022-04-12 10:38:35.156202: saving checkpoint... 2022-04-12 10:38:57.573182: done, saving took 22.51 seconds 2022-04-12 10:38:57.759957: done 2022-04-12 10:38:57.915369: This epoch took 143.899306 s 2022-04-12 10:38:58.025926: svmem(total=99932160000, available=75806367744, percent=24.1, used=20044967936, free=47468433408, active=43721355264, inactive=6601850880, buffers=69754880, cached=32349003776, shared=2258272256, slab=794116096) 2022-04-12 10:38:58.067098: epoch: 479 2022-04-12 10:40:48.240832: train loss : -0.7364 2022-04-12 10:41:00.609879: validation loss: -0.7262 2022-04-12 10:41:00.671142: Average global foreground Dice: [0.9177, 0.9775, 0.9861, 0.4327] 2022-04-12 10:41:00.758394: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:41:01.978309: lr: 0.000552 2022-04-12 10:41:02.102516: saving scheduled checkpoint file... 2022-04-12 10:41:02.296926: saving checkpoint... 2022-04-12 10:41:13.434425: done, saving took 11.21 seconds 2022-04-12 10:41:13.673511: done 2022-04-12 10:41:13.727156: This epoch took 135.524324 s 2022-04-12 10:41:13.793719: svmem(total=99932160000, available=76036730880, percent=23.9, used=19810873344, free=47696719872, active=43485446144, inactive=6607806464, buffers=70000640, cached=32354566144, shared=2261925888, slab=794226688) 2022-04-12 10:41:13.914758: epoch: 480 2022-04-12 10:43:03.349089: train loss : -0.7333 2022-04-12 10:43:15.474334: validation loss: -0.7050 2022-04-12 10:43:15.550043: Average global foreground Dice: [0.9094, 0.971, 0.9831, 0.411] 2022-04-12 10:43:15.597365: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:43:16.484948: lr: 0.000527 2022-04-12 10:43:16.533753: saving scheduled checkpoint file... 2022-04-12 10:43:16.645848: saving checkpoint... 2022-04-12 10:43:34.385201: done, saving took 17.82 seconds 2022-04-12 10:43:34.680252: done 2022-04-12 10:43:34.814973: This epoch took 140.838154 s 2022-04-12 10:43:34.877497: svmem(total=99932160000, available=75882778624, percent=24.1, used=19964518400, free=47540490240, active=43630858240, inactive=6609813504, buffers=70266880, cached=32356884480, shared=2262360064, slab=796745728) 2022-04-12 10:43:34.980468: epoch: 481 2022-04-12 10:45:23.395521: train loss : -0.7307 2022-04-12 10:45:35.782741: validation loss: -0.6965 2022-04-12 10:45:35.842164: Average global foreground Dice: [0.9113, 0.9724, 0.9841, 0.4174] 2022-04-12 10:45:35.855632: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:45:36.627784: lr: 0.000502 2022-04-12 10:45:36.657752: saving scheduled checkpoint file... 2022-04-12 10:45:36.783083: saving checkpoint... 2022-04-12 10:45:52.479511: done, saving took 15.79 seconds 2022-04-12 10:45:52.811617: done 2022-04-12 10:45:52.867383: This epoch took 137.810251 s 2022-04-12 10:45:52.903417: svmem(total=99932160000, available=75994640384, percent=24.0, used=19852947456, free=47650234368, active=43525152768, inactive=6611562496, buffers=70553600, cached=32358424576, shared=2261872640, slab=796778496) 2022-04-12 10:45:52.992704: epoch: 482 2022-04-12 10:47:41.850406: train loss : -0.7326 2022-04-12 10:47:53.039955: validation loss: -0.7232 2022-04-12 10:47:53.079314: Average global foreground Dice: [0.9146, 0.9778, 0.9836, 0.441] 2022-04-12 10:47:53.139496: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:47:54.299492: lr: 0.000477 2022-04-12 10:47:54.358561: saving scheduled checkpoint file... 2022-04-12 10:47:54.526279: saving checkpoint... 2022-04-12 10:48:48.061398: done, saving took 53.65 seconds 2022-04-12 10:48:54.594384: done 2022-04-12 10:48:54.916211: This epoch took 181.830252 s 2022-04-12 10:48:54.935877: svmem(total=99932160000, available=75792056320, percent=24.2, used=20079980544, free=47444787200, active=43748671488, inactive=6589784064, buffers=70877184, cached=32336515072, shared=2237509632, slab=797913088) 2022-04-12 10:48:55.211679: epoch: 483 2022-04-12 10:52:27.064398: train loss : -0.7352 2022-04-12 10:52:38.921549: validation loss: -0.7185 2022-04-12 10:52:39.129295: Average global foreground Dice: [0.9238, 0.9782, 0.9855, 0.4273] 2022-04-12 10:52:39.215162: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:52:40.868701: lr: 0.000451 2022-04-12 10:52:40.987984: saving scheduled checkpoint file... 2022-04-12 10:52:41.165072: saving checkpoint... 2022-04-12 10:53:34.288833: done, saving took 53.23 seconds 2022-04-12 10:53:34.837106: done 2022-04-12 10:53:34.962093: This epoch took 279.704067 s 2022-04-12 10:53:35.089096: svmem(total=99932160000, available=75734392832, percent=24.2, used=20113358848, free=47382773760, active=43782991872, inactive=6617300992, buffers=71397376, cached=32364630016, shared=2261762048, slab=796684288) 2022-04-12 10:53:35.156792: epoch: 484 2022-04-12 10:55:24.683448: train loss : -0.7389 2022-04-12 10:55:36.891610: validation loss: -0.7145 2022-04-12 10:55:36.948111: Average global foreground Dice: [0.9121, 0.9755, 0.9838, 0.4107] 2022-04-12 10:55:37.019764: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:55:37.974496: lr: 0.000426 2022-04-12 10:55:38.059890: saving scheduled checkpoint file... 2022-04-12 10:55:38.269281: saving checkpoint... 2022-04-12 10:55:52.072698: done, saving took 13.95 seconds 2022-04-12 10:55:52.282534: done 2022-04-12 10:55:52.297667: This epoch took 137.011203 s 2022-04-12 10:55:52.357333: svmem(total=99932160000, available=75965415424, percent=24.0, used=19882393600, free=47611621376, active=43563237376, inactive=6618836992, buffers=71684096, cached=32366460928, shared=2261532672, slab=794222592) 2022-04-12 10:55:52.408465: epoch: 485 2022-04-12 10:57:42.520194: train loss : -0.7402 2022-04-12 10:57:54.581080: validation loss: -0.7315 2022-04-12 10:57:54.746763: Average global foreground Dice: [0.9146, 0.9769, 0.9853, 0.4556] 2022-04-12 10:57:54.816900: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 10:57:55.785460: lr: 0.0004 2022-04-12 10:57:55.893896: saving scheduled checkpoint file... 2022-04-12 10:57:56.058970: saving checkpoint... 2022-04-12 10:58:07.204184: done, saving took 11.24 seconds 2022-04-12 10:58:07.379744: done 2022-04-12 10:58:07.398921: This epoch took 134.963354 s 2022-04-12 10:58:07.445925: svmem(total=99932160000, available=75859140608, percent=24.1, used=19988357120, free=47503167488, active=43665678336, inactive=6621069312, buffers=71925760, cached=32368709632, shared=2261815296, slab=794742784) 2022-04-12 10:58:07.454664: epoch: 486 2022-04-12 10:59:55.641972: train loss : -0.7419 2022-04-12 11:00:07.545637: validation loss: -0.7146 2022-04-12 11:00:07.579585: Average global foreground Dice: [0.9115, 0.9792, 0.9871, 0.4211] 2022-04-12 11:00:07.644707: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:00:08.568132: lr: 0.000375 2022-04-12 11:00:08.683993: saving scheduled checkpoint file... 2022-04-12 11:00:08.753741: saving checkpoint... 2022-04-12 11:00:26.131670: done, saving took 17.43 seconds 2022-04-12 11:00:26.974935: done 2022-04-12 11:00:27.058907: This epoch took 139.554219 s 2022-04-12 11:00:27.160805: svmem(total=99932160000, available=76078530560, percent=23.9, used=19762028544, free=47720247296, active=43441700864, inactive=6629994496, buffers=72204288, cached=32377679872, shared=2268786688, slab=794562560) 2022-04-12 11:00:27.260538: epoch: 487 2022-04-12 11:02:15.188205: train loss : -0.7417 2022-04-12 11:02:26.428232: validation loss: -0.7141 2022-04-12 11:02:26.581959: Average global foreground Dice: [0.914, 0.9777, 0.983, 0.4224] 2022-04-12 11:02:26.771866: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:02:28.042697: lr: 0.000348 2022-04-12 11:02:28.096394: saving scheduled checkpoint file... 2022-04-12 11:02:28.204109: saving checkpoint... 2022-04-12 11:02:38.390254: done, saving took 10.23 seconds 2022-04-12 11:02:38.663369: done 2022-04-12 11:02:38.793032: This epoch took 131.432704 s 2022-04-12 11:02:38.856022: svmem(total=99932160000, available=75891920896, percent=24.1, used=19955634176, free=47531470848, active=43630891008, inactive=6624919552, buffers=72437760, cached=32372617216, shared=2261762048, slab=797036544) 2022-04-12 11:02:38.868351: epoch: 488 2022-04-12 11:04:27.817524: train loss : -0.7350 2022-04-12 11:04:39.798079: validation loss: -0.7207 2022-04-12 11:04:39.850000: Average global foreground Dice: [0.9077, 0.9768, 0.9865, 0.4638] 2022-04-12 11:04:39.857057: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:04:40.729144: lr: 0.000322 2022-04-12 11:04:40.737594: saving scheduled checkpoint file... 2022-04-12 11:04:40.836410: saving checkpoint... 2022-04-12 11:04:56.053576: done, saving took 15.30 seconds 2022-04-12 11:04:56.167488: done 2022-04-12 11:04:56.214969: This epoch took 137.282062 s 2022-04-12 11:04:56.257176: svmem(total=99932160000, available=75885879296, percent=24.1, used=19961819136, free=47523381248, active=43637428224, inactive=6626521088, buffers=72679424, cached=32374280192, shared=2261590016, slab=795267072) 2022-04-12 11:04:56.315844: epoch: 489 2022-04-12 11:06:44.963383: train loss : -0.7387 2022-04-12 11:06:56.250008: validation loss: -0.7120 2022-04-12 11:06:56.453886: Average global foreground Dice: [0.9173, 0.9768, 0.9853, 0.4433] 2022-04-12 11:06:56.475766: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:06:57.587316: lr: 0.000296 2022-04-12 11:06:57.678770: saving scheduled checkpoint file... 2022-04-12 11:06:57.827534: saving checkpoint... 2022-04-12 11:07:11.326462: done, saving took 13.59 seconds 2022-04-12 11:07:11.503253: done 2022-04-12 11:07:11.566675: This epoch took 135.209692 s 2022-04-12 11:07:11.617198: svmem(total=99932160000, available=75849498624, percent=24.1, used=19998478336, free=47484764160, active=43677274112, inactive=6628057088, buffers=72957952, cached=32375959552, shared=2261241856, slab=796377088) 2022-04-12 11:07:11.678919: epoch: 490 2022-04-12 11:09:01.134983: train loss : -0.7371 2022-04-12 11:09:13.133630: validation loss: -0.7179 2022-04-12 11:09:13.142580: Average global foreground Dice: [0.9175, 0.9757, 0.9854, 0.437] 2022-04-12 11:09:13.182445: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:09:14.221014: lr: 0.000269 2022-04-12 11:09:14.290369: saving scheduled checkpoint file... 2022-04-12 11:09:14.421818: saving checkpoint... 2022-04-12 11:09:27.164427: done, saving took 12.86 seconds 2022-04-12 11:09:27.411121: done 2022-04-12 11:09:27.620694: saving checkpoint... 2022-04-12 11:09:42.281506: done, saving took 14.76 seconds 2022-04-12 11:09:42.600945: This epoch took 150.871243 s 2022-04-12 11:09:42.693071: svmem(total=99932160000, available=75708690432, percent=24.2, used=20129640448, free=47341666304, active=43808059392, inactive=6639538176, buffers=73232384, cached=32387620864, shared=2270760960, slab=796364800) 2022-04-12 11:09:42.728600: epoch: 491 2022-04-12 11:11:32.174294: train loss : -0.7435 2022-04-12 11:11:44.936863: validation loss: -0.7218 2022-04-12 11:11:45.011629: Average global foreground Dice: [0.9189, 0.975, 0.987, 0.423] 2022-04-12 11:11:45.147168: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:11:46.416648: lr: 0.000242 2022-04-12 11:11:46.603042: saving scheduled checkpoint file... 2022-04-12 11:11:46.745481: saving checkpoint... 2022-04-12 11:12:08.646208: done, saving took 21.96 seconds 2022-04-12 11:12:09.745825: done 2022-04-12 11:12:09.852379: This epoch took 147.070887 s 2022-04-12 11:12:09.967843: svmem(total=99932160000, available=75946696704, percent=24.0, used=19901009920, free=47577419776, active=43569754112, inactive=6632620032, buffers=73510912, cached=32380219392, shared=2261372928, slab=797319168) 2022-04-12 11:12:10.071542: epoch: 492 2022-04-12 11:13:57.376769: train loss : -0.7368 2022-04-12 11:14:09.794084: validation loss: -0.7074 2022-04-12 11:14:09.858174: Average global foreground Dice: [0.9027, 0.9724, 0.9868, 0.442] 2022-04-12 11:14:09.907185: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:14:10.853103: lr: 0.000215 2022-04-12 11:14:10.903217: saving scheduled checkpoint file... 2022-04-12 11:14:11.007520: saving checkpoint... 2022-04-12 11:14:20.107404: done, saving took 9.15 seconds 2022-04-12 11:14:20.203644: done 2022-04-12 11:14:20.237700: This epoch took 130.067636 s 2022-04-12 11:14:20.248378: svmem(total=99932160000, available=75937591296, percent=24.0, used=19904520192, free=47566200832, active=43574943744, inactive=6639693824, buffers=73736192, cached=32387702784, shared=2267017216, slab=797646848) 2022-04-12 11:14:20.253660: epoch: 493 2022-04-12 11:16:09.163416: train loss : -0.7422 2022-04-12 11:16:20.377297: validation loss: -0.7185 2022-04-12 11:16:20.443501: Average global foreground Dice: [0.9197, 0.9751, 0.9831, 0.4344] 2022-04-12 11:16:20.497571: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:16:21.450677: lr: 0.000187 2022-04-12 11:16:21.539180: saving scheduled checkpoint file... 2022-04-12 11:16:21.670580: saving checkpoint... 2022-04-12 11:16:35.190870: done, saving took 13.59 seconds 2022-04-12 11:16:35.369618: done 2022-04-12 11:16:35.468507: This epoch took 135.166664 s 2022-04-12 11:16:35.532279: svmem(total=99932160000, available=75929100288, percent=24.0, used=19908026368, free=47555530752, active=43584839680, inactive=6646575104, buffers=74027008, cached=32394575872, shared=2271965184, slab=797057024) 2022-04-12 11:16:35.584487: epoch: 494 2022-04-12 11:18:23.546472: train loss : -0.7438 2022-04-12 11:18:34.857328: validation loss: -0.7219 2022-04-12 11:18:34.978434: Average global foreground Dice: [0.9206, 0.9784, 0.9828, 0.4276] 2022-04-12 11:18:34.998838: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:18:35.909802: lr: 0.000158 2022-04-12 11:18:36.038920: saving scheduled checkpoint file... 2022-04-12 11:18:36.097627: saving checkpoint... 2022-04-12 11:19:14.480057: done, saving took 38.44 seconds 2022-04-12 11:19:14.771114: done 2022-04-12 11:19:14.884289: This epoch took 159.275969 s 2022-04-12 11:19:14.940473: svmem(total=99932160000, available=75771502592, percent=24.2, used=20075450368, free=47395045376, active=43750813696, inactive=6638739456, buffers=74330112, cached=32387334144, shared=2261934080, slab=796721152) 2022-04-12 11:19:14.997301: epoch: 495 2022-04-12 11:21:03.716167: train loss : -0.7321 2022-04-12 11:21:15.534091: validation loss: -0.7060 2022-04-12 11:21:15.764484: Average global foreground Dice: [0.911, 0.9776, 0.9861, 0.4455] 2022-04-12 11:21:15.899853: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:21:17.134649: lr: 0.00013 2022-04-12 11:21:17.239402: saving scheduled checkpoint file... 2022-04-12 11:21:17.414847: saving checkpoint... 2022-04-12 11:21:35.377669: done, saving took 18.02 seconds 2022-04-12 11:21:35.530983: done 2022-04-12 11:21:35.634978: saving checkpoint... 2022-04-12 11:22:00.419664: done, saving took 24.88 seconds 2022-04-12 11:22:12.927157: This epoch took 177.873593 s 2022-04-12 11:22:15.787505: svmem(total=99932160000, available=76024451072, percent=23.9, used=19836518400, free=47645310976, active=43501043712, inactive=6626111488, buffers=74723328, cached=32375607296, shared=2247966720, slab=808833024) 2022-04-12 11:22:17.450150: epoch: 496 2022-04-12 11:24:07.306147: train loss : -0.7317 2022-04-12 11:24:19.216552: validation loss: -0.7369 2022-04-12 11:24:19.331344: Average global foreground Dice: [0.9226, 0.9764, 0.9867, 0.4615] 2022-04-12 11:24:19.412209: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:24:20.480663: lr: 0.0001 2022-04-12 11:24:20.486059: saving scheduled checkpoint file... 2022-04-12 11:24:20.610521: saving checkpoint... 2022-04-12 11:27:11.912024: done, saving took 171.42 seconds 2022-04-12 11:27:12.475083: done 2022-04-12 11:27:16.348496: saving checkpoint... 2022-04-12 11:30:29.011006: done, saving took 192.77 seconds 2022-04-12 11:30:29.437093: This epoch took 490.888656 s 2022-04-12 11:30:29.603035: svmem(total=99932160000, available=76048662528, percent=23.9, used=19812339712, free=47662014464, active=43483779072, inactive=6632017920, buffers=75665408, cached=32382140416, shared=2247774208, slab=796905472) 2022-04-12 11:30:29.671784: epoch: 497 2022-04-12 11:32:18.569851: train loss : -0.7369 2022-04-12 11:32:30.621516: validation loss: -0.7025 2022-04-12 11:32:30.648131: Average global foreground Dice: [0.9058, 0.9734, 0.9854, 0.4143] 2022-04-12 11:32:30.655191: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:32:31.694760: lr: 6.9e-05 2022-04-12 11:32:31.709632: saving scheduled checkpoint file... 2022-04-12 11:32:31.832357: saving checkpoint... 2022-04-12 11:32:45.005280: done, saving took 13.24 seconds 2022-04-12 11:32:45.157073: done 2022-04-12 11:32:45.165280: This epoch took 135.352916 s 2022-04-12 11:32:45.187171: svmem(total=99932160000, available=74996391936, percent=25.0, used=20878106624, free=47571939328, active=43201978368, inactive=6939004928, buffers=75894784, cached=31406219264, shared=2233843712, slab=801591296) 2022-04-12 11:32:45.199540: epoch: 498 2022-04-12 11:34:33.541643: train loss : -0.7356 2022-04-12 11:34:54.188945: validation loss: -0.7241 2022-04-12 11:34:54.299271: Average global foreground Dice: [0.9288, 0.9779, 0.9875, 0.435] 2022-04-12 11:34:54.417848: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:34:55.344691: lr: 3.7e-05 2022-04-12 11:34:55.408286: saving scheduled checkpoint file... 2022-04-12 11:34:55.610139: saving checkpoint... 2022-04-12 11:35:11.366585: done, saving took 15.82 seconds 2022-04-12 11:35:11.783378: done 2022-04-12 11:35:11.872340: This epoch took 146.633716 s 2022-04-12 11:35:11.957624: svmem(total=99932160000, available=74304954368, percent=25.6, used=21569413120, free=46981783552, active=43825057792, inactive=6908915712, buffers=76189696, cached=31304773632, shared=2233843712, slab=802168832) 2022-04-12 11:35:12.062784: epoch: 499 2022-04-12 11:37:00.186463: train loss : -0.7356 2022-04-12 11:37:11.654036: validation loss: -0.7092 2022-04-12 11:37:11.910485: Average global foreground Dice: [0.9145, 0.9784, 0.9828, 0.4118] 2022-04-12 11:37:12.058902: (interpret this as an estimate for the Dice of the different classes. This is not exact.) 2022-04-12 11:37:13.159278: lr: 0.0 2022-04-12 11:37:13.390262: saving scheduled checkpoint file... 2022-04-12 11:37:13.589542: saving checkpoint... 2022-04-12 11:37:26.515817: done, saving took 12.97 seconds 2022-04-12 11:37:26.645198: done 2022-04-12 11:37:26.703269: This epoch took 134.560155 s 2022-04-12 11:37:26.760556: saving checkpoint... 2022-04-12 11:37:43.464576: done, saving took 16.75 seconds 2022-04-12 11:47:56.315863: finished prediction 2022-04-12 11:47:56.649734: evaluation of raw predictions