diff --git "a/Task500_Glacier_zonefronts/fold_4/training_log_2022_4_11_16_28_44.txt" "b/Task500_Glacier_zonefronts/fold_4/training_log_2022_4_11_16_28_44.txt" new file mode 100644--- /dev/null +++ "b/Task500_Glacier_zonefronts/fold_4/training_log_2022_4_11_16_28_44.txt" @@ -0,0 +1,7786 @@ +Starting... +2022-04-11 16:28:44.759183: Using splits from existing split file: /home/woody/iwi5/iwi5039h/nnUNet_data/nnUNet_preprocessed/Task500_Glacier_zonefronts/splits_final.pkl +2022-04-11 16:28:44.795875: The split file contains 5 splits. +2022-04-11 16:28:44.796594: Desired fold for training: 4 +2022-04-11 16:28:44.797462: This split has 448 training and 111 validation cases. +2022-04-11 16:28:45.543147: TRAINING KEYS: + odict_keys(['Crane_2002-11-18_ERS_20_3_195', 'Crane_2003-01-18_ERS_20_1_061', 'Crane_2006-11-27_ERS_20_1_195', 'Crane_2007-02-05_ERS_20_1_195', 'Crane_2007-06-09_ENVISAT_20_1_467', 'Crane_2007-07-14_ENVISAT_20_1_467', 'Crane_2007-09-22_ENVISAT_20_1_467', 'Crane_2007-09-24_PALSAR_17_1_137', 'Crane_2007-10-27_ENVISAT_20_2_467', 'Crane_2008-01-05_ENVISAT_20_2_467', 'Crane_2008-10-13_TSX_7_3_034', 'Crane_2008-10-24_TSX_7_2_034', 'Crane_2008-11-04_TSX_7_1_034', 'Crane_2008-11-23_PALSAR_17_2_135', 'Crane_2009-09-29_PALSAR_17_1_137', 'Crane_2009-11-02_TSX_7_3_034', 'Crane_2009-11-13_TSX_7_1_034', 'Crane_2010-05-29_ENVISAT_20_3_467', 'Crane_2010-07-03_ENVISAT_20_2_467', 'Crane_2010-10-14_PALSAR_17_2_135', 'Crane_2010-11-05_TSX_7_2_110', 'Crane_2010-11-16_TSX_7_2_110', 'Crane_2010-11-27_TSX_7_1_110', 'Crane_2010-11-29_PALSAR_17_1_135', 'Crane_2010-12-14_TSX_7_1_034', 'Crane_2010-12-19_TSX_7_1_110', 'Crane_2010-12-25_TSX_7_1_034', 'Crane_2010-12-30_TSX_7_1_110', 'Crane_2011-06-24_TSX_7_1_110', 'Crane_2011-08-13_TSX_7_1_034', 'Crane_2011-09-04_TSX_7_1_034', 'Crane_2011-10-18_TSX_7_1_034', 'Crane_2011-12-01_TSX_7_1_034', 'Crane_2011-12-12_TSX_7_2_034', 'Crane_2012-05-14_TSX_7_2_034', 'Crane_2012-06-27_TSX_7_2_034', 'Crane_2012-07-03_TSX_7_1_125', 'Crane_2012-07-25_TSX_7_2_125', 'Crane_2012-07-30_TSX_7_2_034', 'Crane_2012-08-10_TSX_7_3_034', 'Crane_2012-08-21_TSX_7_3_034', 'Crane_2012-09-01_TSX_7_1_034', 'Crane_2012-10-04_TSX_7_3_034', 'Crane_2012-11-06_TSX_7_2_034', 'Crane_2013-02-24_TSX_7_2_034', 'Crane_2013-03-07_TSX_7_2_034', 'Crane_2013-03-29_TSX_7_2_034', 'Crane_2013-06-30_TSX_7_2_110', 'Crane_2013-07-17_TSX_7_3_034', 'Crane_2013-08-19_TSX_7_4_034', 'Crane_2013-11-04_TSX_7_4_034', 'Crane_2013-12-07_TSX_7_5_034', 'Crane_2013-12-29_TSX_7_4_034', 'Crane_2014-01-20_TSX_7_2_034', 'Crane_2014-02-11_TSX_7_3_034', 'DBE_1995-11-16_ERS_20_1_152', 'DBE_1996-01-25_ERS_20_1_152', 'DBE_1997-01-29_ERS_20_2_424', 'DBE_1998-02-18_ERS_20_1_424', 'DBE_2000-09-10_RSAT_20_3_282', 'DBE_2000-10-13_RSAT_20_4_063', 'DBE_2002-02-27_ERS_20_1_424', 'DBE_2003-11-19_ERS_20_3_424', 'DBE_2004-03-03_ENVISAT_20_3_424', 'DBE_2004-03-03_ERS_20_3_424', 'DBE_2004-04-07_ENVISAT_20_3_424', 'DBE_2004-05-12_ENVISAT_20_3_424', 'DBE_2004-06-16_ENVISAT_20_3_424', 'DBE_2004-07-21_ENVISAT_20_3_424', 'DBE_2004-09-13_RSAT_12_3_282', 'DBE_2004-09-29_ENVISAT_20_3_424', 'DBE_2004-10-07_RSAT_12_4_282', 'DBE_2004-10-16_RSAT_12_3_063', 'DBE_2004-11-03_ENVISAT_20_4_424', 'DBE_2004-11-09_RSAT_12_3_063', 'DBE_2004-11-24_RSAT_12_3_282', 'DBE_2004-12-08_ENVISAT_20_5_424', 'DBE_2004-12-08_ERS_20_5_424', 'DBE_2004-12-11_RSAT_12_4_182', 'DBE_2004-12-18_RSAT_12_3_282', 'DBE_2005-01-12_ERS_20_6_424', 'DBE_2005-02-16_ENVISAT_20_1_424', 'DBE_2005-04-27_ENVISAT_20_2_424', 'DBE_2005-07-06_ENVISAT_20_3_424', 'DBE_2005-08-10_ENVISAT_20_2_424', 'DBE_2006-01-06_RSAT_12_4_282', 'DBE_2006-02-23_RSAT_12_3_282', 'DBE_2006-03-08_ENVISAT_20_1_424', 'DBE_2006-03-19_RSAT_12_4_282', 'DBE_2006-04-12_ENVISAT_20_1_424', 'DBE_2006-04-12_RSAT_12_1_282', 'DBE_2006-05-06_RSAT_12_3_282', 'DBE_2006-06-21_ENVISAT_20_2_424', 'DBE_2006-06-21_PALSAR_17_2_137', 'DBE_2006-07-26_ENVISAT_20_3_424', 'DBE_2006-08-06_PALSAR_17_2_137', 'DBE_2006-08-30_ENVISAT_20_2_424', 'DBE_2006-09-03_RSAT_20_5_282', 'DBE_2006-10-04_ENVISAT_20_3_424', 'DBE_2006-11-06_PALSAR_17_3_137', 'DBE_2007-01-01_RSAT_12_5_282', 'DBE_2007-01-25_RSAT_12_5_282', 'DBE_2007-02-18_RSAT_12_4_282', 'DBE_2007-03-14_RSAT_12_3_282', 'DBE_2007-05-01_RSAT_12_4_282', 'DBE_2007-08-15_ENVISAT_20_5_424', 'DBE_2007-09-19_ENVISAT_20_6_424', 'DBE_2007-11-11_PALSAR_17_4_129', 'DBE_2007-12-03_RSAT_12_4_282', 'DBE_2007-12-27_PALSAR_17_5_129', 'DBE_2007-12-27_RSAT_12_5_282', 'DBE_2008-01-20_RSAT_12_4_282', 'DBE_2008-03-30_TSX_7_3_049', 'DBE_2008-10-14_TSX_7_3_049', 'DBE_2008-10-25_TSX_7_2_049', 'DBE_2008-11-13_PALSAR_17_5_129', 'DBE_2008-12-29_PALSAR_17_5_129', 'DBE_2009-02-13_PALSAR_17_5_129', 'DBE_2009-10-23_TSX_7_2_049', 'DBE_2009-11-03_TSX_7_2_049', 'DBE_2009-12-03_PALSAR_17_5_132', 'DBE_2010-01-18_PALSAR_17_5_132', 'DBE_2010-01-22_ERS_20_6_152', 'DBE_2010-04-21_ENVISAT_20_6_424', 'DBE_2010-10-10_TSX_7_2_049', 'DBE_2010-10-16_PALSAR_17_4_127', 'DBE_2010-10-21_PALSAR_17_4_132', 'DBE_2010-11-12_TSX_7_4_049', 'DBE_2010-11-19_PALSAR_17_4_129', 'DBE_2010-11-23_TSX_7_5_049', 'DBE_2010-12-01_PALSAR_17_4_127', 'DBE_2010-12-06_PALSAR_17_3_132', 'DBE_2011-01-04_PALSAR_17_4_129', 'DBE_2011-01-17_TSX_7_4_049', 'DBE_2011-01-28_TSX_7_2_049', 'DBE_2011-07-05_TSX_7_1_110', 'DBE_2011-07-27_TSX_7_1_110', 'DBE_2011-10-23_TSX_7_1_110', 'DBE_2011-11-14_TSX_7_1_110', 'DBE_2011-12-06_TSX_7_5_110', 'DBE_2011-12-28_TSX_7_2_110', 'DBE_2012-03-14_TSX_7_1_110', 'DBE_2012-05-08_TSX_7_1_110', 'DBE_2012-06-10_TSX_7_1_110', 'DBE_2012-06-21_TSX_7_1_110', 'DBE_2012-08-04_TSX_7_1_110', 'DBE_2012-08-15_TSX_7_1_110', 'DBE_2012-10-31_TSX_7_1_110', 'DBE_2013-01-16_TSX_7_5_110', 'DBE_2013-03-12_TSX_7_1_110', 'DBE_2013-03-23_TSX_7_1_110', 'DBE_2013-04-03_TSX_7_1_110', 'DBE_2013-05-28_TSX_7_1_110', 'DBE_2013-06-19_TSX_7_1_110', 'DBE_2013-07-11_TSX_7_1_110', 'DBE_2013-09-04_TSX_7_1_110', 'DBE_2013-11-09_TSX_7_1_110', 'DBE_2013-12-01_TSX_7_2_110', 'DBE_2013-12-12_TSX_7_3_110', 'DBE_2013-12-23_TSX_7_2_110', 'DBE_2014-02-05_TSX_7_1_110', 'DBE_2014-03-21_TSX_7_2_110', 'JAC_2009-04-16_TSX_6_1_005', 'JAC_2009-04-27_TSX_6_1_005', 'JAC_2009-05-08_TSX_6_1_005', 'JAC_2009-05-19_TSX_6_1_005', 'JAC_2009-05-30_TSX_6_1_005', 'JAC_2009-06-10_TSX_6_1_005', 'JAC_2009-07-02_TSX_6_1_005', 'JAC_2009-07-13_TSX_6_1_005', 'JAC_2009-07-24_TSX_6_1_005', 'JAC_2009-08-15_TSX_6_1_005', 'JAC_2009-09-06_TSX_6_1_005', 'JAC_2009-09-17_TSX_6_1_005', 'JAC_2009-09-28_TSX_6_1_005', 'JAC_2009-10-20_TSX_6_1_005', 'JAC_2009-10-31_TSX_6_1_005', 'JAC_2009-11-11_TSX_6_1_005', 'JAC_2009-11-22_TSX_6_1_005', 'JAC_2009-12-03_TSX_6_1_005', 'JAC_2009-12-25_TSX_6_1_005', 'JAC_2010-02-07_TSX_6_1_005', 'JAC_2010-02-18_TSX_6_1_005', 'JAC_2010-03-12_TSX_6_1_005', 'JAC_2010-04-25_TSX_6_1_005', 'JAC_2010-05-06_TSX_6_1_005', 'JAC_2010-06-30_TSX_6_1_005', 'JAC_2010-07-11_TSX_6_1_005', 'JAC_2010-07-22_TSX_6_1_005', 'JAC_2010-09-04_TSX_6_1_005', 'JAC_2010-09-15_TSX_6_1_005', 'JAC_2010-10-07_TSX_6_1_005', 'JAC_2010-10-18_TSX_6_1_005', 'JAC_2010-11-09_TSX_6_1_005', 'JAC_2010-11-20_TSX_6_1_005', 'JAC_2010-12-01_TSX_6_1_005', 'JAC_2010-12-23_TSX_6_1_005', 'JAC_2011-01-25_TSX_6_1_005', 'JAC_2011-02-16_TSX_6_1_005', 'JAC_2011-03-21_TSX_6_1_005', 'JAC_2011-04-12_TSX_6_1_005', 'JAC_2011-04-23_TSX_6_1_005', 'JAC_2011-05-15_TSX_6_1_005', 'JAC_2011-05-26_TSX_6_1_005', 'JAC_2011-06-17_TSX_6_1_005', 'JAC_2011-07-20_TSX_6_1_005', 'JAC_2011-09-03_TSX_6_1_027', 'JAC_2011-10-05_TSX_6_1_005', 'JAC_2011-11-08_TSX_6_1_027', 'JAC_2011-11-18_TSX_6_1_005', 'JAC_2011-12-10_TSX_6_1_005', 'JAC_2012-01-13_TSX_6_1_027', 'JAC_2012-01-23_TSX_6_1_005', 'JAC_2012-02-14_TSX_6_1_005', 'JAC_2012-02-25_TSX_6_1_005', 'JAC_2012-03-07_TSX_6_1_005', 'JAC_2012-03-29_TSX_6_1_005', 'JAC_2012-04-20_TSX_6_1_005', 'JAC_2012-05-24_TSX_6_1_027', 'JAC_2012-06-03_TSX_6_1_005', 'JAC_2012-08-09_TSX_6_1_027', 'JAC_2012-08-20_TSX_6_1_027', 'JAC_2012-09-11_TSX_6_1_027', 'JAC_2012-09-21_TSX_6_1_005', 'JAC_2012-10-03_TSX_6_1_027', 'JAC_2012-11-04_TSX_6_1_005', 'JAC_2012-12-19_TSX_6_1_027', 'JAC_2013-01-09_TSX_6_1_005', 'JAC_2013-01-20_TSX_6_1_005', 'JAC_2013-02-11_TSX_6_1_005', 'JAC_2013-03-17_TSX_6_1_027', 'JAC_2013-04-07_TSX_6_1_005', 'JAC_2013-05-10_TSX_6_1_005', 'JAC_2013-05-22_TSX_6_1_027', 'JAC_2013-06-13_TSX_6_1_027', 'JAC_2013-06-24_TSX_6_1_027', 'JAC_2013-07-27_TSX_6_1_027', 'JAC_2013-08-17_TSX_6_1_005', 'JAC_2013-08-28_TSX_6_1_005', 'JAC_2013-09-08_TSX_6_1_005', 'JAC_2013-09-20_TSX_6_1_027', 'JAC_2013-10-12_TSX_6_1_027', 'JAC_2013-11-14_TSX_6_1_027', 'JAC_2013-11-25_TSX_6_1_027', 'JAC_2013-12-06_TSX_6_1_027', 'JAC_2013-12-17_TSX_6_1_027', 'JAC_2014-01-29_TSX_6_1_005', 'JAC_2014-03-14_TSX_6_1_005', 'JAC_2014-03-25_TSX_6_1_005', 'JAC_2014-04-16_TSX_6_1_005', 'JAC_2014-04-27_TSX_6_1_005', 'JAC_2014-06-10_TSX_6_1_005', 'JAC_2014-06-21_TSX_6_1_005', 'JAC_2014-07-02_TSX_6_1_005', 'JAC_2014-07-03_TSX_6_1_027', 'JAC_2014-07-14_TSX_6_1_027', 'JAC_2014-07-25_TSX_6_1_027', 'JAC_2014-08-04_TSX_6_1_005', 'JAC_2014-09-18_TSX_6_1_027', 'JAC_2014-09-28_TSX_6_1_005', 'JAC_2014-10-09_TSX_6_1_005', 'JAC_2014-10-20_TSX_6_1_005', 'JAC_2014-11-11_TSX_6_1_005', 'JAC_2014-12-03_TSX_6_1_005', 'JAC_2014-12-14_TSX_6_1_005', 'JAC_2014-12-25_TSX_6_1_005', 'JAC_2015-01-05_TSX_6_1_005', 'JAC_2015-01-27_TSX_6_1_005', 'JAC_2015-02-07_TSX_6_1_005', 'JAC_2015-02-19_TSX_6_1_027', 'JAC_2015-03-12_TSX_6_1_005', 'JAC_2015-03-23_TSX_6_1_005', 'JAC_2015-04-14_TSX_6_1_005', 'JAC_2015-05-06_TSX_6_1_005', 'JAC_2015-05-17_TSX_6_1_005', 'JAC_2015-06-08_TSX_6_1_005', 'JAC_2015-06-19_TSX_6_1_005', 'JAC_2015-07-11_TSX_6_1_005', 'JAC_2015-07-22_TSX_6_1_005', 'JAC_2015-08-02_TSX_6_1_005', 'JAC_2015-08-13_TSX_6_1_005', 'JAC_2015-09-26_TSX_6_1_005', 'JAC_2015-10-18_TSX_6_1_005', 'JAC_2015-10-29_TSX_6_1_005', 'JAC_2015-11-20_TSX_6_1_005', 'JAC_2015-12-12_TSX_6_1_005', 'JAC_2015-12-23_TSX_6_1_005', 'Jorum_2003-01-27_ERS_20_1_195', 'Jorum_2006-11-27_ERS_20_3_195', 'Jorum_2007-02-05_ERS_20_1_195', 'Jorum_2007-03-31_ENVISAT_20_2_467', 'Jorum_2007-05-05_ENVISAT_20_2_467', 'Jorum_2007-06-09_ENVISAT_20_2_467', 'Jorum_2007-07-14_ENVISAT_20_2_467', 'Jorum_2007-09-22_ENVISAT_20_3_467', 'Jorum_2008-11-23_PALSAR_17_2_135', 'Jorum_2009-12-25_PALSAR_17_1_136', 'Jorum_2010-05-29_ENVISAT_20_2_467', 'Jorum_2010-07-03_ENVISAT_20_2_467', 'Jorum_2010-09-15_PALSAR_17_2_138', 'Jorum_2010-10-14_PALSAR_17_2_135', 'Jorum_2010-11-27_TSX_7_2_110', 'Jorum_2010-12-08_TSX_7_2_110', 'Jorum_2010-12-14_TSX_7_1_034', 'Jorum_2010-12-19_TSX_7_1_110', 'Jorum_2010-12-25_TSX_7_1_034', 'Jorum_2011-06-02_TSX_7_3_110', 'Jorum_2011-07-16_TSX_7_3_110', 'Jorum_2011-08-13_TSX_7_2_034', 'Jorum_2011-09-04_TSX_7_4_034', 'Jorum_2011-10-18_TSX_7_3_034', 'Jorum_2011-12-01_TSX_7_1_034', 'Jorum_2011-12-12_TSX_7_3_034', 'Jorum_2012-05-14_TSX_7_2_034', 'Jorum_2012-06-16_TSX_7_2_034', 'Jorum_2012-07-03_TSX_7_2_125', 'Jorum_2012-07-25_TSX_7_2_125', 'Jorum_2012-07-30_TSX_7_2_034', 'Jorum_2012-08-10_TSX_7_2_034', 'Jorum_2012-08-21_TSX_7_2_034', 'Jorum_2012-09-01_TSX_7_2_034', 'Jorum_2012-11-06_TSX_7_2_034', 'Jorum_2013-02-24_TSX_7_3_034', 'Jorum_2013-03-07_TSX_7_2_034', 'Jorum_2013-03-29_TSX_7_2_034', 'Jorum_2013-07-17_TSX_7_1_034', 'Jorum_2013-07-22_TSX_7_1_110', 'Jorum_2013-08-02_TSX_7_2_110', 'Jorum_2013-11-04_TSX_7_2_034', 'Jorum_2013-11-15_TSX_7_2_034', 'Jorum_2013-11-26_TSX_7_5_034', 'Jorum_2013-12-07_TSX_7_5_034', 'Jorum_2013-12-18_TSX_7_5_034', 'Jorum_2014-01-20_TSX_7_2_034', 'Jorum_2014-02-11_TSX_7_2_034', 'Jorum_2016-07-03_S1_20_2_009', 'Jorum_2016-09-01_S1_20_2_009', 'Jorum_2017-06-16_S1_20_2_009', 'Jorum_2017-09-08_S1_20_2_009', 'Jorum_2018-01-06_S1_20_3_009', 'Jorum_2018-03-07_S1_20_3_009', 'Jorum_2018-06-11_S1_20_2_009', 'Jorum_2018-09-03_S1_20_3_009', 'Jorum_2019-01-01_S1_20_3_009', 'Jorum_2019-09-22_S1_20_3_009', 'Jorum_2020-01-08_S1_20_3_009', 'Jorum_2020-03-08_S1_20_3_009', 'Jorum_2020-06-12_S1_20_3_009', 'SI_1995-11-13_ERS_20_3_109', 'SI_1995-11-15_ERS_20_3_118', 'SI_1996-01-22_ERS_20_3_109', 'SI_1996-01-23_ERS_20_5_109', 'SI_1996-01-24_ERS_20_4_118', 'SI_1996-02-12_ERS_20_2_390', 'SI_1996-03-18_ERS_20_2_390', 'SI_1997-03-03_ERS_20_4_390', 'SI_1999-11-07_ERS_20_2_381', 'SI_2000-02-20_ERS_20_3_381', 'SI_2000-09-19_RSAT_20_5_063', 'SI_2000-10-04_RSAT_20_5_282', 'SI_2000-10-13_RSAT_20_5_063', 'SI_2002-02-25_ERS_20_3_390', 'SI_2002-11-12_ERS_20_4_109', 'SI_2002-12-21_ERS_20_3_161', 'SI_2003-01-25_ERS_20_3_161', 'SI_2003-12-05_ENVISAT_20_4_152', 'SI_2004-02-13_ENVISAT_20_4_152', 'SI_2004-07-02_ENVISAT_20_4_152', 'SI_2004-08-06_ENVISAT_20_4_152', 'SI_2004-09-13_RSAT_12_5_282', 'SI_2004-10-15_ERS_20_5_152', 'SI_2004-11-19_ERS_20_4_152', 'SI_2004-11-24_RSAT_12_4_282', 'SI_2004-12-03_RSAT_12_5_063', 'SI_2004-12-18_RSAT_12_4_282', 'SI_2004-12-24_ENVISAT_20_5_152', 'SI_2004-12-24_ERS_20_5_152', 'SI_2005-01-28_ERS_20_5_152', 'SI_2005-02-28_RSAT_12_5_282', 'SI_2005-03-04_ENVISAT_20_3_152', 'SI_2005-04-08_ENVISAT_20_2_152', 'SI_2005-04-17_RSAT_12_3_282', 'SI_2005-05-11_RSAT_12_2_282', 'SI_2005-08-26_ENVISAT_20_2_152', 'SI_2006-01-29_ENVISAT_20_3_381', 'SI_2006-02-23_RSAT_12_1_282', 'SI_2006-03-19_RSAT_12_2_282', 'SI_2006-04-12_RSAT_12_1_282', 'SI_2006-05-06_RSAT_12_3_282', 'SI_2006-05-18_PALSAR_17_2_135', 'SI_2006-07-03_PALSAR_17_3_135', 'SI_2006-07-23_ENVISAT_20_4_381', 'SI_2006-08-10_RSAT_20_3_282', 'SI_2006-08-27_ENVISAT_20_3_381', 'SI_2006-09-03_RSAT_12_2_282', 'SI_2006-09-03_RSAT_20_2_282', 'SI_2006-10-18_ERS_20_4_118', 'SI_2006-11-21_ERS_20_3_109', 'SI_2007-01-01_RSAT_12_3_282', 'SI_2007-01-14_ENVISAT_20_3_381', 'SI_2007-01-25_RSAT_12_1_282', 'SI_2007-03-14_RSAT_12_2_282', 'SI_2007-03-25_ENVISAT_20_3_381', 'SI_2007-05-01_RSAT_12_2_282', 'SI_2007-06-03_ENVISAT_20_3_381', 'SI_2007-07-08_ENVISAT_20_3_381', 'SI_2007-09-16_ENVISAT_20_3_381', 'SI_2007-10-21_ENVISAT_20_3_381', 'SI_2007-12-03_RSAT_12_4_282', 'SI_2008-05-18_ENVISAT_20_3_381', 'SI_2008-07-27_ENVISAT_20_4_381', 'SI_2009-02-22_ENVISAT_20_2_381', 'SI_2009-03-29_ENVISAT_20_4_381', 'SI_2009-08-16_ENVISAT_20_3_381', 'SI_2010-01-13_PALSAR_17_2_127', 'SI_2010-01-22_ERS_20_3_152', 'SI_2010-02-26_ERS_20_2_152', 'SI_2010-04-02_ERS_20_2_152', 'SI_2010-10-16_PALSAR_17_3_127', 'SI_2011-03-03_PALSAR_17_2_127', 'SI_2011-06-14_TSX_7_1_125', 'SI_2011-07-06_TSX_7_1_125', 'SI_2011-07-28_TSX_7_1_125', 'SI_2011-08-19_TSX_7_1_125', 'SI_2011-09-10_TSX_7_1_125', 'SI_2011-09-21_TSX_7_1_125', 'SI_2011-11-26_TSX_7_2_125', 'SI_2011-12-29_TSX_7_1_125', 'SI_2012-01-09_TSX_7_1_125', 'SI_2012-03-26_TSX_7_1_125', 'SI_2012-04-28_TSX_7_1_125', 'SI_2012-05-20_TSX_7_1_125', 'SI_2012-07-03_TSX_7_1_125', 'SI_2012-08-05_TSX_7_1_125', 'SI_2012-08-16_TSX_7_1_125', 'SI_2013-07-12_TSX_7_1_125', 'SI_2013-09-16_TSX_7_1_125', 'SI_2013-10-19_TSX_7_1_125', 'SI_2013-10-28_TSX_7_1_104', 'SI_2013-11-08_TSX_7_2_104', 'SI_2013-11-21_TSX_7_1_125', 'SI_2013-12-02_TSX_7_1_125', 'SI_2013-12-13_TSX_7_2_125', 'SI_2014-01-04_TSX_7_2_125', 'SI_2014-03-22_TSX_7_1_125', 'SI_2014-04-02_TSX_7_1_125', 'SI_2014-12-11_TSX_7_4_125', 'SI_2014-12-22_TSX_7_2_125']) +2022-04-11 16:28:45.544271: VALIDATION KEYS: + odict_keys(['Crane_2002-11-09_ERS_20_2_061', 'Crane_2007-03-31_ENVISAT_20_3_467', 'Crane_2007-05-05_ENVISAT_20_1_467', 'Crane_2007-11-21_PALSAR_17_2_135', 'Crane_2010-09-15_PALSAR_17_4_138', 'Crane_2010-12-08_TSX_7_1_110', 'Crane_2011-06-02_TSX_7_2_110', 'Crane_2011-07-16_TSX_7_1_110', 'Crane_2012-06-16_TSX_7_2_034', 'Crane_2013-07-22_TSX_7_3_110', 'Crane_2013-08-02_TSX_7_3_110', 'Crane_2013-11-15_TSX_7_3_034', 'Crane_2013-11-26_TSX_7_4_034', 'Crane_2013-12-18_TSX_7_4_034', 'DBE_1997-03-05_ERS_20_1_424', 'DBE_1999-11-10_ERS_20_4_424', 'DBE_2000-10-04_RSAT_20_4_282', 'DBE_2000-10-06_RSAT_12_3_306', 'DBE_2002-11-16_ERS_20_3_161', 'DBE_2002-12-21_ERS_20_4_161', 'DBE_2003-01-08_ERS_20_4_424', 'DBE_2003-02-12_ERS_20_3_424', 'DBE_2004-08-25_ENVISAT_20_3_424', 'DBE_2005-03-23_ENVISAT_20_1_424', 'DBE_2005-05-11_RSAT_12_3_282', 'DBE_2005-06-01_ENVISAT_20_2_424', 'DBE_2006-05-18_PALSAR_17_2_135', 'DBE_2006-07-03_PALSAR_17_3_135', 'DBE_2006-08-10_RSAT_20_4_282', 'DBE_2006-11-08_ENVISAT_20_3_424', 'DBE_2008-09-28_PALSAR_17_5_129', 'DBE_2008-11-05_TSX_7_2_049', 'DBE_2010-03-17_ENVISAT_20_6_424', 'DBE_2011-01-16_PALSAR_17_4_127', 'DBE_2011-09-09_TSX_7_1_110', 'DBE_2011-09-20_TSX_7_1_110', 'DBE_2012-04-05_TSX_7_1_110', 'DBE_2012-11-11_TSX_7_1_110', 'DBE_2013-10-07_TSX_7_1_110', 'DBE_2014-01-03_TSX_7_5_110', 'JAC_2009-06-21_TSX_6_1_005', 'JAC_2009-08-04_TSX_6_1_005', 'JAC_2009-08-26_TSX_6_1_005', 'JAC_2009-10-09_TSX_6_1_005', 'JAC_2009-12-14_TSX_6_1_005', 'JAC_2010-01-05_TSX_6_1_005', 'JAC_2010-01-16_TSX_6_1_005', 'JAC_2010-03-23_TSX_6_1_005', 'JAC_2010-04-14_TSX_6_1_005', 'JAC_2010-05-17_TSX_6_1_005', 'JAC_2010-06-19_TSX_6_1_005', 'JAC_2011-01-03_TSX_6_1_005', 'JAC_2011-03-10_TSX_6_1_005', 'JAC_2011-08-23_TSX_6_1_027', 'JAC_2011-09-14_TSX_6_1_027', 'JAC_2011-10-16_TSX_6_1_005', 'JAC_2011-12-21_TSX_6_1_005', 'JAC_2012-06-14_TSX_6_1_005', 'JAC_2012-07-07_TSX_6_1_027', 'JAC_2012-07-18_TSX_6_1_027', 'JAC_2012-10-14_TSX_6_1_027', 'JAC_2012-11-15_TSX_6_1_005', 'JAC_2012-12-07_TSX_6_1_005', 'JAC_2013-02-22_TSX_6_1_005', 'JAC_2013-03-06_TSX_6_1_027', 'JAC_2013-04-18_TSX_6_1_005', 'JAC_2013-07-05_TSX_6_1_027', 'JAC_2013-10-23_TSX_6_1_027', 'JAC_2014-03-03_TSX_6_1_005', 'JAC_2014-04-05_TSX_6_1_005', 'JAC_2015-01-16_TSX_6_1_005', 'JAC_2015-04-25_TSX_6_1_005', 'JAC_2015-09-04_TSX_6_1_005', 'JAC_2015-11-09_TSX_6_1_005', 'Jorum_2007-10-27_ENVISAT_20_3_467', 'Jorum_2007-11-21_PALSAR_17_2_135', 'Jorum_2008-01-05_ENVISAT_20_2_467', 'Jorum_2010-03-20_ENVISAT_20_2_467', 'Jorum_2010-11-05_TSX_7_1_110', 'Jorum_2010-11-16_TSX_7_2_110', 'Jorum_2010-11-29_PALSAR_17_1_135', 'Jorum_2010-12-30_TSX_7_1_110', 'Jorum_2011-06-24_TSX_7_2_110', 'Jorum_2012-06-27_TSX_7_2_034', 'Jorum_2012-10-04_TSX_7_2_034', 'Jorum_2013-06-30_TSX_7_2_110', 'Jorum_2013-08-19_TSX_7_2_034', 'Jorum_2013-12-29_TSX_7_4_034', 'Jorum_2016-12-30_S1_20_2_009', 'Jorum_2019-03-02_S1_20_3_009', 'SI_1995-11-14_ERS_20_4_118', 'SI_1995-11-16_ERS_20_4_152', 'SI_1996-02-29_ERS_20_2_152', 'SI_1997-01-27_ERS_20_4_390', 'SI_2000-09-10_RSAT_20_5_282', 'SI_2002-01-21_ERS_20_2_390', 'SI_2004-11-09_RSAT_12_5_063', 'SI_2005-07-22_ENVISAT_20_2_152', 'SI_2006-01-06_RSAT_12_2_282', 'SI_2006-03-05_ENVISAT_20_2_381', 'SI_2007-12-27_RSAT_12_2_282', 'SI_2008-01-20_RSAT_12_2_282', 'SI_2008-04-13_ENVISAT_20_2_381', 'SI_2008-08-31_ENVISAT_20_4_381', 'SI_2009-07-12_ENVISAT_20_3_381', 'SI_2010-12-01_PALSAR_17_3_127', 'SI_2011-01-16_PALSAR_17_2_127', 'SI_2012-03-04_TSX_7_1_125', 'SI_2012-03-15_TSX_7_1_125', 'SI_2012-06-22_TSX_7_1_125', 'SI_2013-08-14_TSX_7_1_125']) +2022-04-11 16:29:02.291670: lr: 0.01 +2022-04-11 16:30:09.979400: Unable to plot network architecture: +2022-04-11 16:30:10.033950: No module named 'hiddenlayer' +2022-04-11 16:30:10.051191: +printing the network instead: + +2022-04-11 16:30:10.107288: Generic_UNet( + (conv_blocks_localization): ModuleList( + (0): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + (1): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + (2): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + (3): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + (4): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(512, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + (5): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(256, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + (6): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(128, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + (7): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(64, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + ) + (conv_blocks_context): ModuleList( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(1, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + (1): ConvDropoutNormNonlin( + (conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(32, 64, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) + (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + (1): ConvDropoutNormNonlin( + (conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (2): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(64, 128, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) + (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + (1): ConvDropoutNormNonlin( + (conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (3): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(128, 256, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) + (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + (1): ConvDropoutNormNonlin( + (conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (4): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(256, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + (1): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (5): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + (1): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (6): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + (1): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (7): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + (1): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (8): Sequential( + (0): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + (1): StackedConvLayers( + (blocks): Sequential( + (0): ConvDropoutNormNonlin( + (conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1]) + (instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False) + (lrelu): LeakyReLU(negative_slope=0.01, inplace=True) + ) + ) + ) + ) + ) + (td): ModuleList() + (tu): ModuleList( + (0): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False) + (1): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False) + (2): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False) + (3): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False) + (4): ConvTranspose2d(480, 256, kernel_size=[2, 2], stride=[2, 2], bias=False) + (5): ConvTranspose2d(256, 128, kernel_size=[2, 2], stride=[2, 2], bias=False) + (6): ConvTranspose2d(128, 64, kernel_size=[2, 2], stride=[2, 2], bias=False) + (7): ConvTranspose2d(64, 32, kernel_size=[2, 2], stride=[2, 2], bias=False) + ) + (seg_outputs): ModuleList( + (0): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) + (1): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) + (2): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) + (3): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) + (4): Conv2d(256, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) + (5): Conv2d(128, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) + (6): Conv2d(64, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) + (7): Conv2d(32, 5, kernel_size=(1, 1), stride=(1, 1), bias=False) + ) +) +2022-04-11 16:30:10.156767: + +2022-04-11 16:30:10.222662: svmem(total=99932160000, available=80544206848, percent=19.4, used=15621857280, free=71972573184, active=16320491520, inactive=9945157632, buffers=11112448, cached=12326617088, shared=1974157312, slab=690577408) +2022-04-11 16:30:10.278398: +epoch: 0 +2022-04-11 16:32:28.915750: train loss : 0.8115 +2022-04-11 16:32:41.373700: validation loss: 0.6441 +2022-04-11 16:32:41.388361: Average global foreground Dice: [0.0607, 0.5993, 0.4555, 0.0] +2022-04-11 16:32:41.390602: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:32:42.284588: lr: 0.009982 +2022-04-11 16:32:42.286186: saving scheduled checkpoint file... +2022-04-11 16:32:42.424067: saving checkpoint... +2022-04-11 16:32:46.713814: done, saving took 4.43 seconds +2022-04-11 16:32:46.735545: done +2022-04-11 16:32:46.739042: This epoch took 156.418662 s + +2022-04-11 16:32:46.743874: svmem(total=99932160000, available=78567493632, percent=21.4, used=17373577216, free=56948731904, active=27280195584, inactive=13964165120, buffers=11235328, cached=25598615552, shared=2182742016, slab=728305664) +2022-04-11 16:32:46.747407: +epoch: 1 +2022-04-11 16:34:37.582255: train loss : 0.5875 +2022-04-11 16:34:48.833301: validation loss: 0.5976 +2022-04-11 16:34:48.952447: Average global foreground Dice: [0.1244, 0.6572, 0.1783, 0.0] +2022-04-11 16:34:49.094463: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:34:50.043136: lr: 0.009964 +2022-04-11 16:34:50.194371: saving scheduled checkpoint file... +2022-04-11 16:34:50.449136: saving checkpoint... +2022-04-11 16:34:55.111908: done, saving took 4.78 seconds +2022-04-11 16:34:55.165554: done +2022-04-11 16:34:55.177876: This epoch took 128.424792 s + +2022-04-11 16:34:55.186192: svmem(total=99932160000, available=78756438016, percent=21.2, used=17183318016, free=52701818880, active=33206583296, inactive=12275585024, buffers=11280384, cached=30035742720, shared=2181476352, slab=740175872) +2022-04-11 16:34:55.189191: +epoch: 2 +2022-04-11 16:36:44.371337: train loss : 0.5283 +2022-04-11 16:36:57.150279: validation loss: 0.3810 +2022-04-11 16:36:57.260215: Average global foreground Dice: [0.3234, 0.7021, 0.5834, 0.0] +2022-04-11 16:36:57.376856: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:36:58.082944: lr: 0.009946 +2022-04-11 16:36:58.083847: saving scheduled checkpoint file... +2022-04-11 16:36:58.211029: saving checkpoint... +2022-04-11 16:37:05.413625: done, saving took 7.33 seconds +2022-04-11 16:37:05.435006: done +2022-04-11 16:37:05.580209: saving checkpoint... +2022-04-11 16:37:09.841580: done, saving took 4.41 seconds +2022-04-11 16:37:09.863396: This epoch took 134.670996 s + +2022-04-11 16:37:09.864407: svmem(total=99932160000, available=78608859136, percent=21.3, used=17329532928, free=50605826048, active=36703449088, inactive=10872483840, buffers=11366400, cached=31985434624, shared=2181021696, slab=743731200) +2022-04-11 16:37:09.865075: +epoch: 3 +2022-04-11 16:39:00.287536: train loss : 0.5004 +2022-04-11 16:39:12.848682: validation loss: 0.4419 +2022-04-11 16:39:12.973960: Average global foreground Dice: [0.2264, 0.7004, 0.5675, 0.0] +2022-04-11 16:39:13.107016: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:39:14.121175: lr: 0.009928 +2022-04-11 16:39:14.255270: saving scheduled checkpoint file... +2022-04-11 16:39:14.531676: saving checkpoint... +2022-04-11 16:39:20.056033: done, saving took 5.65 seconds +2022-04-11 16:39:20.069483: done +2022-04-11 16:39:20.117644: saving checkpoint... +2022-04-11 16:39:24.513986: done, saving took 4.44 seconds +2022-04-11 16:39:24.528041: This epoch took 134.662278 s + +2022-04-11 16:39:24.529269: svmem(total=99932160000, available=78344933376, percent=21.6, used=17605734400, free=49544204288, active=38821265408, inactive=9815982080, buffers=11419648, cached=32770801664, shared=2167689216, slab=745771008) +2022-04-11 16:39:24.530084: +epoch: 4 +2022-04-11 16:41:12.344302: train loss : 0.4405 +2022-04-11 16:41:23.862708: validation loss: 0.3602 +2022-04-11 16:41:23.990284: Average global foreground Dice: [0.3554, 0.6582, 0.6359, 0.0] +2022-04-11 16:41:24.141759: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:41:25.089687: lr: 0.00991 +2022-04-11 16:41:25.196268: saving scheduled checkpoint file... +2022-04-11 16:41:25.430735: saving checkpoint... +2022-04-11 16:41:33.334412: done, saving took 8.07 seconds +2022-04-11 16:41:33.349384: done +2022-04-11 16:41:33.402950: saving checkpoint... +2022-04-11 16:41:37.769484: done, saving took 4.42 seconds +2022-04-11 16:41:37.785155: This epoch took 133.254270 s + +2022-04-11 16:41:37.786171: svmem(total=99932160000, available=78135463936, percent=21.8, used=17784016896, free=48951287808, active=40223027200, inactive=9002512384, buffers=11481088, cached=33185374208, shared=2198409216, slab=747208704) +2022-04-11 16:41:37.786987: +epoch: 5 +2022-04-11 16:43:26.792997: train loss : 0.3341 +2022-04-11 16:43:39.152438: validation loss: 0.2785 +2022-04-11 16:43:39.299618: Average global foreground Dice: [0.29, 0.7247, 0.6979, 0.0] +2022-04-11 16:43:39.459362: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:43:40.413740: lr: 0.009892 +2022-04-11 16:43:40.416284: saving scheduled checkpoint file... +2022-04-11 16:43:40.560840: saving checkpoint... +2022-04-11 16:43:48.422770: done, saving took 8.00 seconds +2022-04-11 16:43:48.447145: done +2022-04-11 16:43:48.606709: saving checkpoint... +2022-04-11 16:43:53.013040: done, saving took 4.56 seconds +2022-04-11 16:43:53.036348: This epoch took 135.248552 s + +2022-04-11 16:43:53.037477: svmem(total=99932160000, available=78112219136, percent=21.8, used=17823932416, free=48761917440, active=40877928448, inactive=8537735168, buffers=11538432, cached=33334771712, shared=2181443584, slab=748011520) +2022-04-11 16:43:53.038170: +epoch: 6 +2022-04-11 16:45:42.733771: train loss : 0.3457 +2022-04-11 16:45:55.309705: validation loss: 0.2533 +2022-04-11 16:45:55.440421: Average global foreground Dice: [0.4208, 0.7319, 0.6997, 0.0] +2022-04-11 16:45:55.597025: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:45:56.509526: lr: 0.009874 +2022-04-11 16:45:56.510361: saving scheduled checkpoint file... +2022-04-11 16:45:56.637624: saving checkpoint... +2022-04-11 16:46:04.059707: done, saving took 7.55 seconds +2022-04-11 16:46:04.081889: done +2022-04-11 16:46:04.235004: saving checkpoint... +2022-04-11 16:46:08.593563: done, saving took 4.51 seconds +2022-04-11 16:46:08.616977: This epoch took 135.578122 s + +2022-04-11 16:46:08.617938: svmem(total=99932160000, available=77606866944, percent=22.3, used=18322710528, free=48232378368, active=42677178368, inactive=7268659200, buffers=11571200, cached=33365499904, shared=2187202560, slab=749002752) +2022-04-11 16:46:08.618534: +epoch: 7 +2022-04-11 16:47:58.059472: train loss : 0.2641 +2022-04-11 16:48:10.585145: validation loss: 0.1710 +2022-04-11 16:48:10.596377: Average global foreground Dice: [0.4832, 0.7542, 0.7233, 0.0] +2022-04-11 16:48:10.597505: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:48:11.430062: lr: 0.009856 +2022-04-11 16:48:11.563856: saving scheduled checkpoint file... +2022-04-11 16:48:11.834223: saving checkpoint... +2022-04-11 16:48:18.092528: done, saving took 6.41 seconds +2022-04-11 16:48:18.113484: done +2022-04-11 16:48:18.261880: saving checkpoint... +2022-04-11 16:48:22.645096: done, saving took 4.53 seconds +2022-04-11 16:48:22.670233: This epoch took 134.051130 s + +2022-04-11 16:48:22.671389: svmem(total=99932160000, available=77573230592, percent=22.4, used=18353766400, free=48139079680, active=42902818816, inactive=7135174656, buffers=11636736, cached=33427677184, shared=2190483456, slab=749498368) +2022-04-11 16:48:22.672153: +epoch: 8 +2022-04-11 16:50:12.636234: train loss : 0.2573 +2022-04-11 16:50:24.438226: validation loss: 0.2944 +2022-04-11 16:50:24.452034: Average global foreground Dice: [0.2945, 0.728, 0.6839, 0.0] +2022-04-11 16:50:24.452769: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:50:25.012655: lr: 0.009838 +2022-04-11 16:50:25.013525: saving scheduled checkpoint file... +2022-04-11 16:50:25.172898: saving checkpoint... +2022-04-11 16:50:32.347454: done, saving took 7.33 seconds +2022-04-11 16:50:32.370704: done +2022-04-11 16:50:32.517018: saving checkpoint... +2022-04-11 16:50:36.939329: done, saving took 4.57 seconds +2022-04-11 16:50:36.962424: This epoch took 134.289636 s + +2022-04-11 16:50:36.963475: svmem(total=99932160000, available=77717762048, percent=22.2, used=18217975808, free=48266420224, active=42797162496, inactive=7112613888, buffers=11706368, cached=33436057600, shared=2181468160, slab=749285376) +2022-04-11 16:50:36.964165: +epoch: 9 +2022-04-11 16:52:27.212385: train loss : 0.2517 +2022-04-11 16:52:39.079212: validation loss: 0.1618 +2022-04-11 16:52:39.204315: Average global foreground Dice: [0.4578, 0.7427, 0.7547, 0.0] +2022-04-11 16:52:39.367909: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:52:40.169844: lr: 0.00982 +2022-04-11 16:52:40.170775: saving scheduled checkpoint file... +2022-04-11 16:52:40.299435: saving checkpoint... +2022-04-11 16:52:48.084236: done, saving took 7.91 seconds +2022-04-11 16:52:48.110812: done +2022-04-11 16:52:48.276182: saving checkpoint... +2022-04-11 16:52:52.674378: done, saving took 4.56 seconds +2022-04-11 16:52:52.701744: This epoch took 135.736696 s + +2022-04-11 16:52:52.702897: svmem(total=99932160000, available=77477154816, percent=22.5, used=18458394624, free=48132030464, active=43010334720, inactive=7035281408, buffers=11735040, cached=33329999872, shared=2181562368, slab=749498368) +2022-04-11 16:52:52.703520: +epoch: 10 +2022-04-11 16:54:40.970308: train loss : 0.2012 +2022-04-11 16:55:07.156918: validation loss: 0.1434 +2022-04-11 16:55:07.418470: Average global foreground Dice: [0.3743, 0.7585, 0.7968, 0.0] +2022-04-11 16:55:07.469459: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:55:08.014517: lr: 0.009802 +2022-04-11 16:55:08.066677: saving scheduled checkpoint file... +2022-04-11 16:55:08.240106: saving checkpoint... +2022-04-11 16:55:12.772668: done, saving took 4.66 seconds +2022-04-11 16:55:12.878101: done +2022-04-11 16:55:13.054289: saving checkpoint... +2022-04-11 16:55:18.561852: done, saving took 5.64 seconds +2022-04-11 16:55:18.863960: This epoch took 146.159729 s + +2022-04-11 16:55:19.025679: svmem(total=99932160000, available=77538910208, percent=22.4, used=18592481280, free=71675240448, active=18899484672, inactive=7641378816, buffers=8839168, cached=9655599104, shared=2010226688, slab=714543104) +2022-04-11 16:55:19.184778: +epoch: 11 +2022-04-11 16:57:30.221735: train loss : 0.1692 +2022-04-11 16:57:43.650400: validation loss: 0.1125 +2022-04-11 16:57:43.658293: Average global foreground Dice: [0.3568, 0.7963, 0.7808, 0.0] +2022-04-11 16:57:43.662486: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:57:44.562484: lr: 0.009784 +2022-04-11 16:57:44.574534: saving scheduled checkpoint file... +2022-04-11 16:57:44.709427: saving checkpoint... +2022-04-11 16:57:49.157838: done, saving took 4.57 seconds +2022-04-11 16:57:49.234792: done +2022-04-11 16:57:49.326731: saving checkpoint... +2022-04-11 16:57:57.346549: done, saving took 8.09 seconds +2022-04-11 16:57:57.370789: This epoch took 158.019200 s + +2022-04-11 16:57:57.375751: svmem(total=99932160000, available=77198372864, percent=22.7, used=18721611776, free=53646180352, active=27696807936, inactive=16766312448, buffers=12410880, cached=27551956992, shared=2167574528, slab=814518272) +2022-04-11 16:57:57.379411: +epoch: 12 +2022-04-11 16:59:45.681371: train loss : 0.1786 +2022-04-11 16:59:56.425632: validation loss: 0.1365 +2022-04-11 16:59:56.431280: Average global foreground Dice: [0.5038, 0.8042, 0.8136, 0.0] +2022-04-11 16:59:56.435093: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 16:59:57.274267: lr: 0.009766 +2022-04-11 16:59:57.276253: saving scheduled checkpoint file... +2022-04-11 16:59:57.402656: saving checkpoint... +2022-04-11 17:00:01.887023: done, saving took 4.61 seconds +2022-04-11 17:00:01.930631: done +2022-04-11 17:00:02.031088: saving checkpoint... +2022-04-11 17:00:09.444780: done, saving took 7.50 seconds +2022-04-11 17:00:09.790030: This epoch took 132.404510 s + +2022-04-11 17:00:09.899356: svmem(total=99932160000, available=77312000000, percent=22.6, used=18622570496, free=50906038272, active=34165235712, inactive=13060288512, buffers=12754944, cached=30390796288, shared=2167316480, slab=795762688) +2022-04-11 17:00:10.005942: +epoch: 13 +2022-04-11 17:03:20.897981: train loss : 0.1151 +2022-04-11 17:03:36.648901: validation loss: 0.1617 +2022-04-11 17:03:36.657867: Average global foreground Dice: [0.44, 0.7795, 0.7189, 0.0] +2022-04-11 17:03:36.660785: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:03:37.842270: lr: 0.009748 +2022-04-11 17:03:37.866740: saving scheduled checkpoint file... +2022-04-11 17:03:38.045642: saving checkpoint... +2022-04-11 17:03:45.618663: done, saving took 7.72 seconds +2022-04-11 17:03:45.662736: done +2022-04-11 17:03:45.767462: saving checkpoint... +2022-04-11 17:03:53.632073: done, saving took 7.96 seconds +2022-04-11 17:03:53.650704: This epoch took 223.607900 s + +2022-04-11 17:03:53.651968: svmem(total=99932160000, available=76974911488, percent=23.0, used=18852356096, free=44811669504, active=26422562816, inactive=26744233984, buffers=11255808, cached=36256878592, shared=2191364096, slab=950841344) +2022-04-11 17:03:53.652825: +epoch: 14 +2022-04-11 17:05:42.429314: train loss : 0.0766 +2022-04-11 17:05:54.311445: validation loss: -0.0079 +2022-04-11 17:05:54.320906: Average global foreground Dice: [0.5097, 0.8339, 0.8606, 0.0] +2022-04-11 17:05:54.329045: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:05:55.196023: lr: 0.00973 +2022-04-11 17:05:55.197113: saving scheduled checkpoint file... +2022-04-11 17:05:55.331633: saving checkpoint... +2022-04-11 17:06:03.020419: done, saving took 7.82 seconds +2022-04-11 17:06:03.042399: done +2022-04-11 17:06:03.170992: saving checkpoint... +2022-04-11 17:06:11.080729: done, saving took 8.04 seconds +2022-04-11 17:06:11.100446: This epoch took 137.446488 s + +2022-04-11 17:06:11.101577: svmem(total=99932160000, available=77051371520, percent=22.9, used=18729250816, free=35688136704, active=33384366080, inactive=28781580288, buffers=15073280, cached=45499699200, shared=2177839104, slab=1073881088) +2022-04-11 17:06:11.102414: +epoch: 15 +2022-04-11 17:08:00.219816: train loss : 0.0430 +2022-04-11 17:08:12.688771: validation loss: 0.0492 +2022-04-11 17:08:12.693157: Average global foreground Dice: [0.4924, 0.7882, 0.8145, 0.0] +2022-04-11 17:08:12.694204: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:08:13.582440: lr: 0.009712 +2022-04-11 17:08:13.583848: saving scheduled checkpoint file... +2022-04-11 17:08:13.702173: saving checkpoint... +2022-04-11 17:08:21.411238: done, saving took 7.83 seconds +2022-04-11 17:08:21.433668: done +2022-04-11 17:08:21.563979: saving checkpoint... +2022-04-11 17:08:29.470772: done, saving took 8.03 seconds +2022-04-11 17:08:29.497628: This epoch took 138.393256 s + +2022-04-11 17:08:29.500047: svmem(total=99932160000, available=76975149056, percent=23.0, used=18742767616, free=29717004288, active=37400494080, inactive=30605094912, buffers=20119552, cached=51452268544, shared=2177810432, slab=1202061312) +2022-04-11 17:08:29.502023: +epoch: 16 +2022-04-11 17:10:18.349904: train loss : 0.0398 +2022-04-11 17:10:30.395445: validation loss: 0.0073 +2022-04-11 17:10:30.397113: Average global foreground Dice: [0.5119, 0.842, 0.8268, 0.0] +2022-04-11 17:10:30.398093: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:10:31.176621: lr: 0.009693 +2022-04-11 17:10:31.179099: saving scheduled checkpoint file... +2022-04-11 17:10:31.299704: saving checkpoint... +2022-04-11 17:10:37.545132: done, saving took 6.36 seconds +2022-04-11 17:10:37.886368: done +2022-04-11 17:10:38.137178: saving checkpoint... +2022-04-11 17:10:44.689179: done, saving took 6.68 seconds +2022-04-11 17:10:44.984501: This epoch took 135.479997 s + +2022-04-11 17:10:45.110140: svmem(total=99932160000, available=76560306176, percent=23.4, used=19128131584, free=31390613504, active=39911608320, inactive=26314571776, buffers=31354880, cached=49382060032, shared=2167574528, slab=1302450176) +2022-04-11 17:10:45.235015: +epoch: 17 +2022-04-11 17:12:33.619274: train loss : 0.0142 +2022-04-11 17:12:45.890574: validation loss: 0.0426 +2022-04-11 17:12:45.904065: Average global foreground Dice: [0.3771, 0.8153, 0.8652, 0.0] +2022-04-11 17:12:45.904964: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:12:46.894276: lr: 0.009675 +2022-04-11 17:12:46.895159: saving scheduled checkpoint file... +2022-04-11 17:12:46.983260: saving checkpoint... +2022-04-11 17:12:51.522449: done, saving took 4.63 seconds +2022-04-11 17:12:51.889750: done +2022-04-11 17:12:52.077348: saving checkpoint... +2022-04-11 17:12:58.831448: done, saving took 6.81 seconds +2022-04-11 17:12:59.162761: This epoch took 133.803637 s + +2022-04-11 17:12:59.305727: svmem(total=99932160000, available=76903047168, percent=23.0, used=18927783936, free=40051564544, active=40991166464, inactive=16870293504, buffers=31866880, cached=40920944640, shared=2167574528, slab=1009303552) +2022-04-11 17:12:59.439311: +epoch: 18 +2022-04-11 17:14:48.371114: train loss : 0.0391 +2022-04-11 17:15:00.329636: validation loss: -0.0417 +2022-04-11 17:15:00.344035: Average global foreground Dice: [0.5515, 0.8389, 0.8656, 0.0] +2022-04-11 17:15:00.344957: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:15:01.350607: lr: 0.009657 +2022-04-11 17:15:01.351494: saving scheduled checkpoint file... +2022-04-11 17:15:01.449877: saving checkpoint... +2022-04-11 17:15:09.091078: done, saving took 7.74 seconds +2022-04-11 17:15:09.106421: done +2022-04-11 17:15:09.159956: saving checkpoint... +2022-04-11 17:15:17.049817: done, saving took 7.94 seconds +2022-04-11 17:15:17.063027: This epoch took 137.498344 s + +2022-04-11 17:15:17.064293: svmem(total=99932160000, available=76803342336, percent=23.1, used=19005366272, free=39800795136, active=41860083712, inactive=16266137600, buffers=31948800, cached=41094049792, shared=2189791232, slab=997359616) +2022-04-11 17:15:17.064951: +epoch: 19 +2022-04-11 17:17:04.013731: train loss : 0.0449 +2022-04-11 17:17:16.328982: validation loss: -0.0878 +2022-04-11 17:17:16.337509: Average global foreground Dice: [0.5477, 0.8502, 0.8595, 0.0] +2022-04-11 17:17:16.340266: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:17:17.224820: lr: 0.009639 +2022-04-11 17:17:17.227479: saving scheduled checkpoint file... +2022-04-11 17:17:17.301040: saving checkpoint... +2022-04-11 17:17:23.532808: done, saving took 6.30 seconds +2022-04-11 17:17:23.837632: done +2022-04-11 17:17:24.018805: saving checkpoint... +2022-04-11 17:17:30.709950: done, saving took 6.75 seconds +2022-04-11 17:17:31.058849: This epoch took 133.993258 s + +2022-04-11 17:17:31.327152: svmem(total=99932160000, available=76925427712, percent=23.0, used=18905182208, free=39837163520, active=41953742848, inactive=16122646528, buffers=32038912, cached=41157775360, shared=2167660544, slab=1011482624) +2022-04-11 17:17:31.460803: +epoch: 20 +2022-04-11 17:20:37.712162: train loss : 0.1069 +2022-04-11 17:20:50.877975: validation loss: -0.0571 +2022-04-11 17:20:50.909437: Average global foreground Dice: [0.5138, 0.838, 0.8429, 0.0] +2022-04-11 17:20:50.962348: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:20:51.516384: lr: 0.009621 +2022-04-11 17:20:51.549595: saving scheduled checkpoint file... +2022-04-11 17:20:51.672852: saving checkpoint... +2022-04-11 17:20:56.938884: done, saving took 5.37 seconds +2022-04-11 17:20:57.245236: done +2022-04-11 17:20:57.452471: saving checkpoint... +2022-04-11 17:21:04.022400: done, saving took 6.63 seconds +2022-04-11 17:21:04.342449: This epoch took 212.748103 s + +2022-04-11 17:21:04.477158: svmem(total=99932160000, available=76925026304, percent=23.0, used=18976354304, free=45684269056, active=25461776384, inactive=26820673536, buffers=11464704, cached=35260071936, shared=2118541312, slab=958713856) +2022-04-11 17:21:04.618300: +epoch: 21 +2022-04-11 17:22:54.340552: train loss : -0.0394 +2022-04-11 17:23:05.840497: validation loss: -0.0982 +2022-04-11 17:23:05.847731: Average global foreground Dice: [0.5968, 0.8415, 0.8611, 0.0] +2022-04-11 17:23:05.849826: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:23:06.660325: lr: 0.009603 +2022-04-11 17:23:06.661140: saving scheduled checkpoint file... +2022-04-11 17:23:06.787699: saving checkpoint... +2022-04-11 17:23:11.252267: done, saving took 4.59 seconds +2022-04-11 17:23:11.321626: done +2022-04-11 17:23:11.435068: saving checkpoint... +2022-04-11 17:23:19.015632: done, saving took 7.67 seconds +2022-04-11 17:23:19.041738: This epoch took 134.280128 s + +2022-04-11 17:23:19.047930: svmem(total=99932160000, available=76920422400, percent=23.0, used=18870738944, free=35570814976, active=33028243456, inactive=29259096064, buffers=15462400, cached=45475143680, shared=2167455744, slab=1070592000) +2022-04-11 17:23:19.052634: +epoch: 22 +2022-04-11 17:25:07.806077: train loss : -0.0660 +2022-04-11 17:25:19.460015: validation loss: -0.0699 +2022-04-11 17:25:19.463646: Average global foreground Dice: [0.5763, 0.8469, 0.8175, 0.0] +2022-04-11 17:25:19.467323: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:25:20.375448: lr: 0.009585 +2022-04-11 17:25:20.378095: saving scheduled checkpoint file... +2022-04-11 17:25:20.509869: saving checkpoint... +2022-04-11 17:25:24.894874: done, saving took 4.51 seconds +2022-04-11 17:25:24.909635: done +2022-04-11 17:25:25.010650: saving checkpoint... +2022-04-11 17:25:32.811768: done, saving took 7.90 seconds +2022-04-11 17:25:32.831193: This epoch took 133.774125 s + +2022-04-11 17:25:32.832351: svmem(total=99932160000, available=76790964224, percent=23.2, used=18934149120, free=29379252224, active=37126307840, inactive=31212384256, buffers=21147648, cached=51597611008, shared=2167488512, slab=1208356864) +2022-04-11 17:25:32.833154: +epoch: 23 +2022-04-11 17:27:23.914539: train loss : -0.0500 +2022-04-11 17:27:36.427160: validation loss: -0.1038 +2022-04-11 17:27:36.429213: Average global foreground Dice: [0.5563, 0.8556, 0.8731, 0.0] +2022-04-11 17:27:36.430447: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:27:37.400556: lr: 0.009567 +2022-04-11 17:27:37.401460: saving scheduled checkpoint file... +2022-04-11 17:27:37.495400: saving checkpoint... +2022-04-11 17:27:42.133246: done, saving took 4.73 seconds +2022-04-11 17:27:42.482877: done +2022-04-11 17:27:42.685255: saving checkpoint... +2022-04-11 17:27:49.395685: done, saving took 6.77 seconds +2022-04-11 17:27:49.736177: This epoch took 136.901684 s + +2022-04-11 17:27:49.861795: svmem(total=99932160000, available=76818329600, percent=23.1, used=18883186688, free=28783587328, active=39311056896, inactive=29546876928, buffers=31436800, cached=52233949184, shared=2167488512, slab=1276477440) +2022-04-11 17:27:49.986198: +epoch: 24 +2022-04-11 17:29:38.339847: train loss : -0.0655 +2022-04-11 17:29:49.619399: validation loss: -0.1961 +2022-04-11 17:29:49.620867: Average global foreground Dice: [0.6121, 0.8659, 0.9168, 0.0] +2022-04-11 17:29:49.621605: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:29:50.510173: lr: 0.009549 +2022-04-11 17:29:50.511212: saving scheduled checkpoint file... +2022-04-11 17:29:50.601478: saving checkpoint... +2022-04-11 17:29:55.034173: done, saving took 4.52 seconds +2022-04-11 17:29:55.058815: done +2022-04-11 17:29:55.141455: saving checkpoint... +2022-04-11 17:30:02.922730: done, saving took 7.86 seconds +2022-04-11 17:30:02.942122: This epoch took 132.824024 s + +2022-04-11 17:30:02.943227: svmem(total=99932160000, available=73014235136, percent=26.9, used=22651850752, free=27298619392, active=44235628544, inactive=25996345344, buffers=32251904, cached=49949437952, shared=2195550208, slab=1288753152) +2022-04-11 17:30:02.943956: +epoch: 25 +2022-04-11 17:31:50.093130: train loss : -0.0882 +2022-04-11 17:32:01.729632: validation loss: -0.1491 +2022-04-11 17:32:01.744573: Average global foreground Dice: [0.5182, 0.8658, 0.8653, 0.0] +2022-04-11 17:32:01.745654: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:32:02.440406: lr: 0.009531 +2022-04-11 17:32:02.441448: saving scheduled checkpoint file... +2022-04-11 17:32:02.531621: saving checkpoint... +2022-04-11 17:32:06.969281: done, saving took 4.53 seconds +2022-04-11 17:32:06.987296: done +2022-04-11 17:32:07.076667: saving checkpoint... +2022-04-11 17:32:12.214795: done, saving took 5.23 seconds +2022-04-11 17:32:12.557163: This epoch took 129.612638 s + +2022-04-11 17:32:12.691202: svmem(total=99932160000, available=72530407424, percent=27.4, used=23151616000, free=26872475648, active=45725458432, inactive=24925233152, buffers=32325632, cached=49875742720, shared=2182426624, slab=1295212544) +2022-04-11 17:32:12.817348: +epoch: 26 +2022-04-11 17:33:59.986779: train loss : -0.0178 +2022-04-11 17:34:12.006717: validation loss: 0.0533 +2022-04-11 17:34:12.010263: Average global foreground Dice: [0.5471, 0.7958, 0.8192, 0.0] +2022-04-11 17:34:12.011205: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:34:12.893502: lr: 0.009513 +2022-04-11 17:34:12.894790: saving scheduled checkpoint file... +2022-04-11 17:34:13.016027: saving checkpoint... +2022-04-11 17:34:17.392090: done, saving took 4.50 seconds +2022-04-11 17:34:17.412055: done +2022-04-11 17:34:17.516880: saving checkpoint... +2022-04-11 17:34:21.868986: done, saving took 4.46 seconds +2022-04-11 17:34:21.998057: This epoch took 129.037292 s + +2022-04-11 17:34:21.999130: svmem(total=99932160000, available=72684748800, percent=27.3, used=22998585344, free=26957537280, active=46043766784, inactive=24534081536, buffers=32395264, cached=49943642112, shared=2182082560, slab=1280929792) +2022-04-11 17:34:21.999711: +epoch: 27 +2022-04-11 17:36:11.465950: train loss : -0.0514 +2022-04-11 17:36:23.814878: validation loss: -0.2015 +2022-04-11 17:36:23.818096: Average global foreground Dice: [0.6451, 0.8874, 0.9151, 0.0] +2022-04-11 17:36:23.820679: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:36:24.542908: lr: 0.009495 +2022-04-11 17:36:24.544606: saving scheduled checkpoint file... +2022-04-11 17:36:24.688169: saving checkpoint... +2022-04-11 17:36:29.042838: done, saving took 4.50 seconds +2022-04-11 17:36:29.059755: done +2022-04-11 17:36:29.179539: saving checkpoint... +2022-04-11 17:36:33.638077: done, saving took 4.58 seconds +2022-04-11 17:36:33.658824: This epoch took 131.658283 s + +2022-04-11 17:36:33.660095: svmem(total=99932160000, available=72515043328, percent=27.4, used=23166910464, free=26966061056, active=46554058752, inactive=24018169856, buffers=32505856, cached=49766682624, shared=2182426624, slab=1284911104) +2022-04-11 17:36:33.660762: +epoch: 28 +2022-04-11 17:38:21.595961: train loss : -0.0978 +2022-04-11 17:38:33.205333: validation loss: -0.1464 +2022-04-11 17:38:33.213428: Average global foreground Dice: [0.5809, 0.8564, 0.8993, 0.0] +2022-04-11 17:38:33.214438: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:38:34.113566: lr: 0.009476 +2022-04-11 17:38:34.114851: saving scheduled checkpoint file... +2022-04-11 17:38:34.263383: saving checkpoint... +2022-04-11 17:38:38.640817: done, saving took 4.52 seconds +2022-04-11 17:38:38.658620: done +2022-04-11 17:38:38.770763: saving checkpoint... +2022-04-11 17:38:43.110934: done, saving took 4.45 seconds +2022-04-11 17:38:43.135348: This epoch took 129.473689 s + +2022-04-11 17:38:43.138138: svmem(total=99932160000, available=72704303104, percent=27.2, used=23029587968, free=26844372992, active=46665547776, inactive=24032210944, buffers=32555008, cached=50025644032, shared=2131742720, slab=1280823296) +2022-04-11 17:38:43.140432: +epoch: 29 +2022-04-11 17:40:33.221166: train loss : -0.0946 +2022-04-11 17:40:45.276181: validation loss: -0.2023 +2022-04-11 17:40:45.278610: Average global foreground Dice: [0.6213, 0.8617, 0.9198, 0.0] +2022-04-11 17:40:45.279571: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:40:46.027590: lr: 0.009458 +2022-04-11 17:40:46.028836: saving scheduled checkpoint file... +2022-04-11 17:40:46.114105: saving checkpoint... +2022-04-11 17:40:50.398499: done, saving took 4.37 seconds +2022-04-11 17:40:50.410820: done +2022-04-11 17:40:50.458993: saving checkpoint... +2022-04-11 17:40:54.725308: done, saving took 4.31 seconds +2022-04-11 17:40:54.738329: This epoch took 131.595364 s + +2022-04-11 17:40:54.739611: svmem(total=99932160000, available=72628678656, percent=27.3, used=23077838848, free=26747674624, active=46831075328, inactive=23960162304, buffers=32636928, cached=50074009600, shared=2158473216, slab=1281122304) +2022-04-11 17:40:54.740404: +epoch: 30 +2022-04-11 17:42:40.893281: train loss : -0.1602 +2022-04-11 17:42:53.824648: validation loss: -0.2550 +2022-04-11 17:42:53.826371: Average global foreground Dice: [0.6428, 0.888, 0.9398, 0.0001] +2022-04-11 17:42:53.827226: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:42:54.698192: lr: 0.00944 +2022-04-11 17:42:54.699355: saving scheduled checkpoint file... +2022-04-11 17:42:54.772191: saving checkpoint... +2022-04-11 17:42:59.034606: done, saving took 4.33 seconds +2022-04-11 17:42:59.048501: done +2022-04-11 17:42:59.096295: saving checkpoint... +2022-04-11 17:43:03.506198: done, saving took 4.46 seconds +2022-04-11 17:43:03.526624: This epoch took 128.785355 s + +2022-04-11 17:43:03.527960: svmem(total=99932160000, available=72715599872, percent=27.2, used=23049965568, free=26811551744, active=46909194240, inactive=23813693440, buffers=32702464, cached=50037940224, shared=2099339264, slab=1280204800) +2022-04-11 17:43:03.528824: +epoch: 31 +2022-04-11 17:44:50.207533: train loss : -0.0854 +2022-04-11 17:45:01.952050: validation loss: -0.1942 +2022-04-11 17:45:01.954078: Average global foreground Dice: [0.6543, 0.8792, 0.9101, 0.0] +2022-04-11 17:45:01.955254: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:45:02.764568: lr: 0.009422 +2022-04-11 17:45:02.765598: saving scheduled checkpoint file... +2022-04-11 17:45:02.851956: saving checkpoint... +2022-04-11 17:45:07.390444: done, saving took 4.62 seconds +2022-04-11 17:45:07.399186: done +2022-04-11 17:45:07.447770: saving checkpoint... +2022-04-11 17:45:11.825083: done, saving took 4.42 seconds +2022-04-11 17:45:11.838171: This epoch took 128.308633 s + +2022-04-11 17:45:11.839430: svmem(total=99932160000, available=72498749440, percent=27.5, used=23168925696, free=26575867904, active=47133085696, inactive=23829377024, buffers=32796672, cached=50154569728, shared=2197233664, slab=1280749568) +2022-04-11 17:45:11.840116: +epoch: 32 +2022-04-11 17:46:58.763399: train loss : -0.1626 +2022-04-11 17:47:10.816393: validation loss: -0.1866 +2022-04-11 17:47:10.818597: Average global foreground Dice: [0.4695, 0.8659, 0.9112, 0.0287] +2022-04-11 17:47:10.820432: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:47:11.722924: lr: 0.009404 +2022-04-11 17:47:11.724090: saving scheduled checkpoint file... +2022-04-11 17:47:11.807429: saving checkpoint... +2022-04-11 17:47:16.236042: done, saving took 4.51 seconds +2022-04-11 17:47:16.254839: done +2022-04-11 17:47:16.318847: saving checkpoint... +2022-04-11 17:47:20.689697: done, saving took 4.43 seconds +2022-04-11 17:47:20.707149: This epoch took 128.866422 s + +2022-04-11 17:47:20.708260: svmem(total=99932160000, available=72638271488, percent=27.3, used=23054983168, free=26701324288, active=47069278208, inactive=23762493440, buffers=32870400, cached=50142982144, shared=2171539456, slab=1281363968) +2022-04-11 17:47:20.709155: +epoch: 33 +2022-04-11 17:49:07.981102: train loss : -0.1355 +2022-04-11 17:49:19.088469: validation loss: -0.1452 +2022-04-11 17:49:19.096927: Average global foreground Dice: [0.5958, 0.8597, 0.8495, 0.0324] +2022-04-11 17:49:19.099699: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:49:19.845646: lr: 0.009386 +2022-04-11 17:49:19.846409: saving scheduled checkpoint file... +2022-04-11 17:49:19.946099: saving checkpoint... +2022-04-11 17:49:24.378577: done, saving took 4.53 seconds +2022-04-11 17:49:24.394604: done +2022-04-11 17:49:24.496571: saving checkpoint... +2022-04-11 17:49:28.948981: done, saving took 4.55 seconds +2022-04-11 17:49:28.967541: This epoch took 128.257744 s + +2022-04-11 17:49:28.968732: svmem(total=99932160000, available=72589037568, percent=27.4, used=23073366016, free=26648145920, active=47121653760, inactive=23759949824, buffers=32952320, cached=50177695744, shared=2202599424, slab=1281396736) +2022-04-11 17:49:28.969642: +epoch: 34 +2022-04-11 17:51:18.594008: train loss : -0.1189 +2022-04-11 17:51:30.069335: validation loss: -0.1774 +2022-04-11 17:51:30.071650: Average global foreground Dice: [0.6762, 0.8731, 0.8907, 0.0723] +2022-04-11 17:51:30.072833: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:51:30.974719: lr: 0.009368 +2022-04-11 17:51:30.975876: saving scheduled checkpoint file... +2022-04-11 17:51:31.105781: saving checkpoint... +2022-04-11 17:51:35.523103: done, saving took 4.55 seconds +2022-04-11 17:51:35.542399: done +2022-04-11 17:51:35.646850: saving checkpoint... +2022-04-11 17:51:40.156778: done, saving took 4.61 seconds +2022-04-11 17:51:40.177367: This epoch took 131.206611 s + +2022-04-11 17:51:40.178771: svmem(total=99932160000, available=72466202624, percent=27.5, used=23202557952, free=26514063360, active=47298195456, inactive=23726415872, buffers=33017856, cached=50182520832, shared=2196074496, slab=1280749568) +2022-04-11 17:51:40.179779: +epoch: 35 +2022-04-11 17:53:28.961627: train loss : -0.1739 +2022-04-11 17:53:41.781502: validation loss: -0.2483 +2022-04-11 17:53:41.796599: Average global foreground Dice: [0.4814, 0.8862, 0.931, 0.1353] +2022-04-11 17:53:41.797762: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:53:42.598184: lr: 0.00935 +2022-04-11 17:53:42.599269: saving scheduled checkpoint file... +2022-04-11 17:53:42.692631: saving checkpoint... +2022-04-11 17:53:47.006041: done, saving took 4.41 seconds +2022-04-11 17:53:47.029983: done +2022-04-11 17:53:47.077221: saving checkpoint... +2022-04-11 17:53:51.392221: done, saving took 4.36 seconds +2022-04-11 17:53:51.405175: This epoch took 131.224713 s + +2022-04-11 17:53:51.406653: svmem(total=99932160000, available=71683604480, percent=28.3, used=23992000512, free=26342543360, active=48129720320, inactive=23076958208, buffers=33116160, cached=49564499968, shared=2196004864, slab=1268240384) +2022-04-11 17:53:51.407427: +epoch: 36 +2022-04-11 17:55:38.607474: train loss : -0.1667 +2022-04-11 17:55:50.390426: validation loss: -0.2156 +2022-04-11 17:55:50.400633: Average global foreground Dice: [0.5716, 0.889, 0.8867, 0.1566] +2022-04-11 17:55:50.402859: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:55:51.275792: lr: 0.009331 +2022-04-11 17:55:51.276935: saving scheduled checkpoint file... +2022-04-11 17:55:51.334177: saving checkpoint... +2022-04-11 17:55:55.699186: done, saving took 4.42 seconds +2022-04-11 17:55:55.711171: done +2022-04-11 17:55:55.758731: saving checkpoint... +2022-04-11 17:56:00.049188: done, saving took 4.34 seconds +2022-04-11 17:56:00.068410: This epoch took 128.660265 s + +2022-04-11 17:56:00.070723: svmem(total=99932160000, available=71717457920, percent=28.2, used=23957704704, free=26374295552, active=48111980544, inactive=23059308544, buffers=33230848, cached=49566928896, shared=2196328448, slab=1268219904) +2022-04-11 17:56:00.071522: +epoch: 37 +2022-04-11 17:57:48.259306: train loss : -0.1756 +2022-04-11 17:57:59.673591: validation loss: -0.2530 +2022-04-11 17:57:59.675459: Average global foreground Dice: [0.6838, 0.882, 0.9022, 0.152] +2022-04-11 17:57:59.676420: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 17:58:00.556088: lr: 0.009313 +2022-04-11 17:58:00.557207: saving scheduled checkpoint file... +2022-04-11 17:58:00.641125: saving checkpoint... +2022-04-11 17:58:04.918866: done, saving took 4.36 seconds +2022-04-11 17:58:06.432168: done +2022-04-11 17:58:06.479966: saving checkpoint... +2022-04-11 17:58:10.815622: done, saving took 4.38 seconds +2022-04-11 17:58:10.830016: This epoch took 130.757651 s + +2022-04-11 17:58:10.831250: svmem(total=99932160000, available=71628136448, percent=28.3, used=24047157248, free=26281361408, active=48226160640, inactive=23040598016, buffers=33296384, cached=49570344960, shared=2196242432, slab=1266597888) +2022-04-11 17:58:10.831935: +epoch: 38 +2022-04-11 17:59:58.090957: train loss : -0.2465 +2022-04-11 18:00:10.021915: validation loss: -0.2913 +2022-04-11 18:00:10.023896: Average global foreground Dice: [0.6582, 0.9002, 0.9285, 0.1388] +2022-04-11 18:00:10.024770: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:00:10.682244: lr: 0.009295 +2022-04-11 18:00:10.683542: saving scheduled checkpoint file... +2022-04-11 18:00:10.770340: saving checkpoint... +2022-04-11 18:00:15.201299: done, saving took 4.52 seconds +2022-04-11 18:00:15.225258: done +2022-04-11 18:00:15.276376: saving checkpoint... +2022-04-11 18:00:19.784731: done, saving took 4.56 seconds +2022-04-11 18:00:19.798435: This epoch took 128.965402 s + +2022-04-11 18:00:19.799945: svmem(total=99932160000, available=71662424064, percent=28.3, used=24012623872, free=26314448896, active=48201822208, inactive=23032979456, buffers=33382400, cached=49571704832, shared=2196516864, slab=1268215808) +2022-04-11 18:00:19.800673: +epoch: 39 +2022-04-11 18:02:06.059399: train loss : -0.2509 +2022-04-11 18:02:17.195197: validation loss: -0.2172 +2022-04-11 18:02:17.199909: Average global foreground Dice: [0.658, 0.8674, 0.8788, 0.1857] +2022-04-11 18:02:17.200920: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:02:18.099625: lr: 0.009277 +2022-04-11 18:02:18.100718: saving scheduled checkpoint file... +2022-04-11 18:02:18.204732: saving checkpoint... +2022-04-11 18:02:23.045764: done, saving took 4.94 seconds +2022-04-11 18:02:23.068339: done +2022-04-11 18:02:23.132725: saving checkpoint... +2022-04-11 18:02:27.538586: done, saving took 4.47 seconds +2022-04-11 18:02:27.568020: This epoch took 127.766590 s + +2022-04-11 18:02:27.569363: svmem(total=99932160000, available=71528693760, percent=28.4, used=24146714624, free=26180079616, active=48342061056, inactive=23025238016, buffers=33456128, cached=49571909632, shared=2196234240, slab=1267740672) +2022-04-11 18:02:27.570184: +epoch: 40 +2022-04-11 18:04:14.703655: train loss : -0.2314 +2022-04-11 18:04:27.221083: validation loss: -0.2872 +2022-04-11 18:04:27.235953: Average global foreground Dice: [0.657, 0.8859, 0.9227, 0.1831] +2022-04-11 18:04:27.237153: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:04:28.056935: lr: 0.009259 +2022-04-11 18:04:28.058538: saving scheduled checkpoint file... +2022-04-11 18:04:28.172277: saving checkpoint... +2022-04-11 18:04:32.636987: done, saving took 4.58 seconds +2022-04-11 18:04:32.663728: done +2022-04-11 18:04:32.745012: saving checkpoint... +2022-04-11 18:04:37.262603: done, saving took 4.60 seconds +2022-04-11 18:04:37.280566: This epoch took 129.709617 s + +2022-04-11 18:04:37.282191: svmem(total=99932160000, available=71672274944, percent=28.3, used=24003051520, free=26322038784, active=48201543680, inactive=23023456256, buffers=33521664, cached=49573548032, shared=2196242432, slab=1268715520) +2022-04-11 18:04:37.283252: +epoch: 41 +2022-04-11 18:06:25.047718: train loss : -0.2537 +2022-04-11 18:06:37.353164: validation loss: -0.3425 +2022-04-11 18:06:37.369532: Average global foreground Dice: [0.6938, 0.9172, 0.9274, 0.2092] +2022-04-11 18:06:37.370620: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:06:38.103533: lr: 0.009241 +2022-04-11 18:06:38.104604: saving scheduled checkpoint file... +2022-04-11 18:06:38.202850: saving checkpoint... +2022-04-11 18:06:42.806414: done, saving took 4.70 seconds +2022-04-11 18:06:42.820515: done +2022-04-11 18:06:42.870794: saving checkpoint... +2022-04-11 18:06:47.424655: done, saving took 4.60 seconds +2022-04-11 18:06:47.433012: This epoch took 130.148904 s + +2022-04-11 18:06:47.434160: svmem(total=99932160000, available=71845617664, percent=28.1, used=23816331264, free=26494177280, active=48022134784, inactive=23033192448, buffers=33628160, cached=49588023296, shared=2209701888, slab=1266647040) +2022-04-11 18:06:47.435188: +epoch: 42 +2022-04-11 18:08:35.805219: train loss : -0.3023 +2022-04-11 18:08:47.599961: validation loss: -0.3499 +2022-04-11 18:08:47.607869: Average global foreground Dice: [0.6879, 0.9062, 0.9389, 0.2151] +2022-04-11 18:08:47.608906: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:08:48.329960: lr: 0.009223 +2022-04-11 18:08:48.331387: saving scheduled checkpoint file... +2022-04-11 18:08:48.405823: saving checkpoint... +2022-04-11 18:08:52.953016: done, saving took 4.62 seconds +2022-04-11 18:08:52.972173: done +2022-04-11 18:08:53.050466: saving checkpoint... +2022-04-11 18:08:57.495852: done, saving took 4.52 seconds +2022-04-11 18:08:57.530961: This epoch took 130.095050 s + +2022-04-11 18:08:57.532340: svmem(total=99932160000, available=71816028160, percent=28.1, used=23860207616, free=26463170560, active=48064442368, inactive=23018893312, buffers=33685504, cached=49575096320, shared=2195361792, slab=1267789824) +2022-04-11 18:08:57.533106: +epoch: 43 +2022-04-11 18:10:46.160861: train loss : -0.2702 +2022-04-11 18:10:57.969609: validation loss: -0.2082 +2022-04-11 18:10:57.977633: Average global foreground Dice: [0.6654, 0.8481, 0.9105, 0.1982] +2022-04-11 18:10:57.978516: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:10:58.652878: lr: 0.009204 +2022-04-11 18:10:58.653985: saving scheduled checkpoint file... +2022-04-11 18:10:58.746648: saving checkpoint... +2022-04-11 18:11:03.261231: done, saving took 4.61 seconds +2022-04-11 18:11:03.274947: done +2022-04-11 18:11:03.368240: saving checkpoint... +2022-04-11 18:11:08.266320: done, saving took 4.99 seconds +2022-04-11 18:11:08.297967: This epoch took 130.763880 s + +2022-04-11 18:11:08.303693: svmem(total=99932160000, available=71869943808, percent=28.1, used=23805878272, free=26516189184, active=48024776704, inactive=23013216256, buffers=33751040, cached=49576341504, shared=2195681280, slab=1268711424) +2022-04-11 18:11:08.308928: +epoch: 44 +2022-04-11 18:12:55.369489: train loss : -0.2349 +2022-04-11 18:13:06.845438: validation loss: -0.2912 +2022-04-11 18:13:06.855357: Average global foreground Dice: [0.6846, 0.8943, 0.9277, 0.204] +2022-04-11 18:13:06.856243: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:13:07.711790: lr: 0.009186 +2022-04-11 18:13:07.712953: saving scheduled checkpoint file... +2022-04-11 18:13:07.846677: saving checkpoint... +2022-04-11 18:13:12.279533: done, saving took 4.57 seconds +2022-04-11 18:13:12.321957: done +2022-04-11 18:13:12.430094: saving checkpoint... +2022-04-11 18:13:16.924383: done, saving took 4.60 seconds +2022-04-11 18:13:16.944057: This epoch took 128.629416 s + +2022-04-11 18:13:16.945210: svmem(total=99932160000, available=71967395840, percent=28.0, used=23708123136, free=26611437568, active=47929511936, inactive=23009800192, buffers=33816576, cached=49578782720, shared=2195968000, slab=1266991104) +2022-04-11 18:13:16.946081: +epoch: 45 +2022-04-11 18:15:05.418838: train loss : -0.2683 +2022-04-11 18:15:16.993828: validation loss: -0.2822 +2022-04-11 18:15:17.006854: Average global foreground Dice: [0.6742, 0.887, 0.9125, 0.2255] +2022-04-11 18:15:17.008621: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:15:17.803252: lr: 0.009168 +2022-04-11 18:15:17.804230: saving scheduled checkpoint file... +2022-04-11 18:15:17.944124: saving checkpoint... +2022-04-11 18:15:22.448521: done, saving took 4.64 seconds +2022-04-11 18:15:22.470047: done +2022-04-11 18:15:22.591187: saving checkpoint... +2022-04-11 18:15:27.037232: done, saving took 4.57 seconds +2022-04-11 18:15:27.059900: This epoch took 130.113118 s + +2022-04-11 18:15:27.061117: svmem(total=99932160000, available=71769849856, percent=28.2, used=23910928384, free=26411966464, active=48141303808, inactive=22998003712, buffers=33931264, cached=49575333888, shared=2190790656, slab=1265270784) +2022-04-11 18:15:27.061800: +epoch: 46 +2022-04-11 18:17:15.704180: train loss : -0.2276 +2022-04-11 18:17:27.847235: validation loss: -0.2460 +2022-04-11 18:17:27.849338: Average global foreground Dice: [0.5908, 0.8965, 0.8909, 0.2117] +2022-04-11 18:17:27.850488: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:17:28.470910: lr: 0.00915 +2022-04-11 18:17:28.471856: saving scheduled checkpoint file... +2022-04-11 18:17:28.629243: saving checkpoint... +2022-04-11 18:17:33.094712: done, saving took 4.62 seconds +2022-04-11 18:17:33.118334: done +2022-04-11 18:17:33.253474: saving checkpoint... +2022-04-11 18:17:37.738729: done, saving took 4.62 seconds +2022-04-11 18:17:37.762346: This epoch took 130.699828 s + +2022-04-11 18:17:37.763723: svmem(total=99932160000, available=71608434688, percent=28.3, used=24073736192, free=26279845888, active=48306630656, inactive=22958751744, buffers=36200448, cached=49542377472, shared=2189103104, slab=1267236864) +2022-04-11 18:17:37.764469: +epoch: 47 +2022-04-11 18:19:27.182035: train loss : -0.2779 +2022-04-11 18:19:39.669902: validation loss: -0.3089 +2022-04-11 18:19:39.679551: Average global foreground Dice: [0.6047, 0.8984, 0.9226, 0.2415] +2022-04-11 18:19:39.680734: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:19:40.634269: lr: 0.009132 +2022-04-11 18:19:40.635638: saving scheduled checkpoint file... +2022-04-11 18:19:40.726617: saving checkpoint... +2022-04-11 18:19:45.164888: done, saving took 4.53 seconds +2022-04-11 18:19:45.178617: done +2022-04-11 18:19:45.226301: saving checkpoint... +2022-04-11 18:19:49.695523: done, saving took 4.52 seconds +2022-04-11 18:19:49.704718: This epoch took 131.939290 s + +2022-04-11 18:19:49.706110: svmem(total=99932160000, available=71670407168, percent=28.3, used=23991009280, free=26340552704, active=48224555008, inactive=22980042752, buffers=36278272, cached=49564319744, shared=2209726464, slab=1269080064) +2022-04-11 18:19:49.706919: +epoch: 48 +2022-04-11 18:21:37.064239: train loss : -0.2881 +2022-04-11 18:21:48.938533: validation loss: -0.3023 +2022-04-11 18:21:48.940470: Average global foreground Dice: [0.6597, 0.9076, 0.9067, 0.2341] +2022-04-11 18:21:48.941611: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:21:49.755537: lr: 0.009114 +2022-04-11 18:21:49.756776: saving scheduled checkpoint file... +2022-04-11 18:21:49.827629: saving checkpoint... +2022-04-11 18:21:54.251754: done, saving took 4.49 seconds +2022-04-11 18:21:54.264327: done +2022-04-11 18:21:54.314246: saving checkpoint... +2022-04-11 18:21:58.852721: done, saving took 4.59 seconds +2022-04-11 18:21:58.866265: This epoch took 129.158326 s + +2022-04-11 18:21:58.867603: svmem(total=99932160000, available=71412047872, percent=28.5, used=24262901760, free=26080571392, active=48503476224, inactive=22964740096, buffers=36368384, cached=49552318464, shared=2196180992, slab=1267544064) +2022-04-11 18:21:58.868602: +epoch: 49 +2022-04-11 18:23:46.001378: train loss : -0.2621 +2022-04-11 18:23:56.842020: validation loss: -0.2880 +2022-04-11 18:23:56.844160: Average global foreground Dice: [0.6701, 0.8991, 0.9128, 0.2344] +2022-04-11 18:23:56.845083: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:23:57.561248: lr: 0.009095 +2022-04-11 18:23:57.562429: saving scheduled checkpoint file... +2022-04-11 18:23:57.629569: saving checkpoint... +2022-04-11 18:24:02.159654: done, saving took 4.60 seconds +2022-04-11 18:24:02.182249: done +2022-04-11 18:24:02.259319: saving checkpoint... +2022-04-11 18:24:06.829621: done, saving took 4.65 seconds +2022-04-11 18:24:06.847664: This epoch took 127.978388 s + +2022-04-11 18:24:06.849019: svmem(total=99932160000, available=71646982144, percent=28.3, used=24014819328, free=26314625024, active=48251424768, inactive=22977568768, buffers=36433920, cached=49566281728, shared=2209316864, slab=1269067776) +2022-04-11 18:24:06.849840: +epoch: 50 +2022-04-11 18:25:54.688497: train loss : -0.3154 +2022-04-11 18:26:06.182189: validation loss: -0.3657 +2022-04-11 18:26:06.192132: Average global foreground Dice: [0.6919, 0.8934, 0.9294, 0.2583] +2022-04-11 18:26:06.193136: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:26:07.069997: lr: 0.009077 +2022-04-11 18:26:07.071355: saving scheduled checkpoint file... +2022-04-11 18:26:07.172444: saving checkpoint... +2022-04-11 18:26:11.774269: done, saving took 4.70 seconds +2022-04-11 18:26:11.793447: done +2022-04-11 18:26:11.894217: saving checkpoint... +2022-04-11 18:26:16.449231: done, saving took 4.65 seconds +2022-04-11 18:26:16.467571: This epoch took 129.616878 s + +2022-04-11 18:26:16.468773: svmem(total=99932160000, available=71520280576, percent=28.4, used=24140906496, free=26193039360, active=48377069568, inactive=22973128704, buffers=36552704, cached=49561661440, shared=2210033664, slab=1269325824) +2022-04-11 18:26:16.469502: +epoch: 51 +2022-04-11 18:28:05.240090: train loss : -0.2947 +2022-04-11 18:28:17.222424: validation loss: -0.3041 +2022-04-11 18:28:17.224815: Average global foreground Dice: [0.6788, 0.8881, 0.921, 0.2634] +2022-04-11 18:28:17.231312: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:28:18.109228: lr: 0.009059 +2022-04-11 18:28:18.110629: saving scheduled checkpoint file... +2022-04-11 18:28:18.246113: saving checkpoint... +2022-04-11 18:28:23.486511: done, saving took 5.37 seconds +2022-04-11 18:28:23.508159: done +2022-04-11 18:28:23.618336: saving checkpoint... +2022-04-11 18:28:28.279020: done, saving took 4.77 seconds +2022-04-11 18:28:28.299248: This epoch took 131.829089 s + +2022-04-11 18:28:28.300552: svmem(total=99932160000, available=71511785472, percent=28.4, used=24150175744, free=26184527872, active=48388386816, inactive=22972461056, buffers=36626432, cached=49560829952, shared=2209259520, slab=1268604928) +2022-04-11 18:28:28.301557: +epoch: 52 +2022-04-11 18:30:16.967725: train loss : -0.3031 +2022-04-11 18:30:29.849122: validation loss: -0.3679 +2022-04-11 18:30:29.861084: Average global foreground Dice: [0.7137, 0.9014, 0.9416, 0.2497] +2022-04-11 18:30:29.862144: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:30:30.454842: lr: 0.009041 +2022-04-11 18:30:30.455911: saving scheduled checkpoint file... +2022-04-11 18:30:30.576693: saving checkpoint... +2022-04-11 18:30:35.098886: done, saving took 4.64 seconds +2022-04-11 18:30:35.115855: done +2022-04-11 18:30:35.223767: saving checkpoint... +2022-04-11 18:30:39.723451: done, saving took 4.61 seconds +2022-04-11 18:30:39.757392: This epoch took 131.454903 s + +2022-04-11 18:30:39.759086: svmem(total=99932160000, available=71660638208, percent=28.3, used=24014323712, free=26334027776, active=48251936768, inactive=22958690304, buffers=36712448, cached=49547096064, shared=2196131840, slab=1268867072) +2022-04-11 18:30:39.760441: +epoch: 53 +2022-04-11 18:32:28.624094: train loss : -0.3768 +2022-04-11 18:32:40.718122: validation loss: -0.4194 +2022-04-11 18:32:40.720535: Average global foreground Dice: [0.709, 0.9241, 0.9492, 0.2638] +2022-04-11 18:32:40.721632: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:32:41.668246: lr: 0.009023 +2022-04-11 18:32:41.669288: saving scheduled checkpoint file... +2022-04-11 18:32:41.759511: saving checkpoint... +2022-04-11 18:32:46.421063: done, saving took 4.75 seconds +2022-04-11 18:32:46.433383: done +2022-04-11 18:32:46.481598: saving checkpoint... +2022-04-11 18:32:50.923074: done, saving took 4.49 seconds +2022-04-11 18:32:50.935340: This epoch took 131.174044 s + +2022-04-11 18:32:50.936290: svmem(total=99932160000, available=71682412544, percent=28.3, used=23992377344, free=26355888128, active=48238084096, inactive=22958497792, buffers=36777984, cached=49547116544, shared=2196234240, slab=1268899840) +2022-04-11 18:32:50.937012: +epoch: 54 +2022-04-11 18:34:37.667846: train loss : -0.3251 +2022-04-11 18:34:49.726325: validation loss: -0.3586 +2022-04-11 18:34:49.737571: Average global foreground Dice: [0.6454, 0.8996, 0.937, 0.2872] +2022-04-11 18:34:49.739474: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:34:50.631107: lr: 0.009004 +2022-04-11 18:34:50.633163: saving scheduled checkpoint file... +2022-04-11 18:34:50.706120: saving checkpoint... +2022-04-11 18:34:55.058240: done, saving took 4.42 seconds +2022-04-11 18:34:55.069563: done +2022-04-11 18:34:55.120415: saving checkpoint... +2022-04-11 18:34:59.403939: done, saving took 4.33 seconds +2022-04-11 18:34:59.411613: This epoch took 128.474006 s + +2022-04-11 18:34:59.412882: svmem(total=99932160000, available=71726559232, percent=28.2, used=23948042240, free=26470100992, active=48229625856, inactive=22842605568, buffers=38047744, cached=49475969024, shared=2197147648, slab=1267843072) +2022-04-11 18:34:59.413836: +epoch: 55 +2022-04-11 18:36:46.872693: train loss : -0.3692 +2022-04-11 18:36:58.302178: validation loss: -0.4244 +2022-04-11 18:36:58.326955: Average global foreground Dice: [0.7445, 0.9206, 0.9335, 0.2747] +2022-04-11 18:36:58.329629: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:36:59.171144: lr: 0.008986 +2022-04-11 18:36:59.173079: saving scheduled checkpoint file... +2022-04-11 18:36:59.242760: saving checkpoint... +2022-04-11 18:37:03.711619: done, saving took 4.54 seconds +2022-04-11 18:37:03.725688: done +2022-04-11 18:37:03.773688: saving checkpoint... +2022-04-11 18:37:09.032716: done, saving took 5.31 seconds +2022-04-11 18:37:09.046173: This epoch took 129.631531 s + +2022-04-11 18:37:09.047697: svmem(total=99932160000, available=71820447744, percent=28.1, used=23854874624, free=26563035136, active=48137535488, inactive=22842576896, buffers=38146048, cached=49476104192, shared=2196283392, slab=1269370880) +2022-04-11 18:37:09.048469: +epoch: 56 +2022-04-11 18:38:56.540147: train loss : -0.3970 +2022-04-11 18:39:09.018270: validation loss: -0.4092 +2022-04-11 18:39:09.026973: Average global foreground Dice: [0.7419, 0.9191, 0.9458, 0.2878] +2022-04-11 18:39:09.028124: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:39:09.901635: lr: 0.008968 +2022-04-11 18:39:09.902968: saving scheduled checkpoint file... +2022-04-11 18:39:09.987071: saving checkpoint... +2022-04-11 18:39:14.469324: done, saving took 4.57 seconds +2022-04-11 18:39:14.483902: done +2022-04-11 18:39:14.533688: saving checkpoint... +2022-04-11 18:39:19.199581: done, saving took 4.71 seconds +2022-04-11 18:39:19.223735: This epoch took 130.174454 s + +2022-04-11 18:39:19.224780: svmem(total=99932160000, available=71753383936, percent=28.2, used=23896240128, free=26422767616, active=48167915520, inactive=22951837696, buffers=38244352, cached=49574907904, shared=2220503040, slab=1272446976) +2022-04-11 18:39:19.225772: +epoch: 57 +2022-04-11 18:41:07.317251: train loss : -0.3703 +2022-04-11 18:41:17.838792: validation loss: -0.3191 +2022-04-11 18:41:17.840514: Average global foreground Dice: [0.7134, 0.8992, 0.9281, 0.2444] +2022-04-11 18:41:17.841522: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:41:18.671868: lr: 0.00895 +2022-04-11 18:41:18.674716: saving scheduled checkpoint file... +2022-04-11 18:41:18.778330: saving checkpoint... +2022-04-11 18:41:23.254517: done, saving took 4.58 seconds +2022-04-11 18:41:23.273777: done +2022-04-11 18:41:23.350428: saving checkpoint... +2022-04-11 18:41:27.679279: done, saving took 4.40 seconds +2022-04-11 18:41:27.734111: This epoch took 128.507520 s + +2022-04-11 18:41:27.735766: svmem(total=99932160000, available=71853785088, percent=28.1, used=23820189696, free=26524856320, active=48011423744, inactive=23005171712, buffers=38326272, cached=49548787712, shared=2196156416, slab=1272606720) +2022-04-11 18:41:27.736654: +epoch: 58 +2022-04-11 18:43:14.813223: train loss : -0.3428 +2022-04-11 18:43:26.728497: validation loss: -0.3630 +2022-04-11 18:43:26.731688: Average global foreground Dice: [0.6763, 0.9149, 0.9393, 0.2745] +2022-04-11 18:43:26.735094: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:43:27.553338: lr: 0.008931 +2022-04-11 18:43:27.557782: saving scheduled checkpoint file... +2022-04-11 18:43:27.687447: saving checkpoint... +2022-04-11 18:43:32.182281: done, saving took 4.62 seconds +2022-04-11 18:43:32.205749: done +2022-04-11 18:43:32.303085: saving checkpoint... +2022-04-11 18:43:36.744379: done, saving took 4.54 seconds +2022-04-11 18:43:36.776512: This epoch took 129.039052 s + +2022-04-11 18:43:36.784048: svmem(total=99932160000, available=71674396672, percent=28.3, used=23999864832, free=26344976384, active=48423510016, inactive=22769029120, buffers=38383616, cached=49548935168, shared=2196017152, slab=1269686272) +2022-04-11 18:43:36.794336: +epoch: 59 +2022-04-11 18:45:24.233442: train loss : -0.3347 +2022-04-11 18:45:35.894710: validation loss: -0.3808 +2022-04-11 18:45:35.905703: Average global foreground Dice: [0.6936, 0.9112, 0.9347, 0.2865] +2022-04-11 18:45:35.907654: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:45:36.858814: lr: 0.008913 +2022-04-11 18:45:36.860265: saving scheduled checkpoint file... +2022-04-11 18:45:36.965877: saving checkpoint... +2022-04-11 18:45:41.733386: done, saving took 4.87 seconds +2022-04-11 18:45:41.770356: done +2022-04-11 18:45:41.817978: saving checkpoint... +2022-04-11 18:45:46.242729: done, saving took 4.47 seconds +2022-04-11 18:45:46.255179: This epoch took 129.453653 s + +2022-04-11 18:45:46.256604: svmem(total=99932160000, available=71798345728, percent=28.2, used=23862317056, free=26468126720, active=48403472384, inactive=22671319040, buffers=38465536, cached=49563250688, shared=2209521664, slab=1270640640) +2022-04-11 18:45:46.257446: +epoch: 60 +2022-04-11 18:47:34.721798: train loss : -0.3701 +2022-04-11 18:47:45.742934: validation loss: -0.4405 +2022-04-11 18:47:45.745199: Average global foreground Dice: [0.7501, 0.9254, 0.9527, 0.2804] +2022-04-11 18:47:45.746419: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:47:46.636870: lr: 0.008895 +2022-04-11 18:47:46.638619: saving scheduled checkpoint file... +2022-04-11 18:47:46.724052: saving checkpoint... +2022-04-11 18:47:51.413126: done, saving took 4.77 seconds +2022-04-11 18:47:51.421919: done +2022-04-11 18:47:51.470228: saving checkpoint... +2022-04-11 18:47:56.181871: done, saving took 4.76 seconds +2022-04-11 18:47:56.197866: This epoch took 129.939450 s + +2022-04-11 18:47:56.199324: svmem(total=99932160000, available=71693828096, percent=28.3, used=23969693696, free=26363559936, active=48517279744, inactive=22662692864, buffers=38531072, cached=49560375296, shared=2206650368, slab=1270464512) +2022-04-11 18:47:56.200166: +epoch: 61 +2022-04-11 18:49:43.759281: train loss : -0.3975 +2022-04-11 18:49:55.426508: validation loss: -0.3942 +2022-04-11 18:49:55.439315: Average global foreground Dice: [0.7163, 0.919, 0.9324, 0.2803] +2022-04-11 18:49:55.440440: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:49:56.386636: lr: 0.008877 +2022-04-11 18:49:56.387607: saving scheduled checkpoint file... +2022-04-11 18:49:56.495818: saving checkpoint... +2022-04-11 18:50:01.254612: done, saving took 4.87 seconds +2022-04-11 18:50:01.297468: done +2022-04-11 18:50:01.375372: saving checkpoint... +2022-04-11 18:50:05.977811: done, saving took 4.68 seconds +2022-04-11 18:50:05.995223: This epoch took 129.794418 s + +2022-04-11 18:50:05.996326: svmem(total=99932160000, available=71570751488, percent=28.4, used=24089505792, free=26240876544, active=48636485632, inactive=22664658944, buffers=38608896, cached=49563168768, shared=2209931264, slab=1270452224) +2022-04-11 18:50:05.996881: +epoch: 62 +2022-04-11 18:51:54.028112: train loss : -0.4031 +2022-04-11 18:52:05.188961: validation loss: -0.4136 +2022-04-11 18:52:05.195951: Average global foreground Dice: [0.7337, 0.9083, 0.9516, 0.2878] +2022-04-11 18:52:05.196945: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:52:06.005736: lr: 0.008859 +2022-04-11 18:52:06.007090: saving scheduled checkpoint file... +2022-04-11 18:52:06.141356: saving checkpoint... +2022-04-11 18:52:10.634848: done, saving took 4.63 seconds +2022-04-11 18:52:10.654228: done +2022-04-11 18:52:10.751653: saving checkpoint... +2022-04-11 18:52:15.252165: done, saving took 4.60 seconds +2022-04-11 18:52:15.285738: This epoch took 129.287948 s + +2022-04-11 18:52:15.287256: svmem(total=99932160000, available=71710216192, percent=28.2, used=23967973376, free=26422775808, active=48511619072, inactive=22604644352, buffers=38674432, cached=49502736384, shared=2192363520, slab=1269415936) +2022-04-11 18:52:15.287986: +epoch: 63 +2022-04-11 18:54:03.332053: train loss : -0.3975 +2022-04-11 18:54:14.545367: validation loss: -0.4228 +2022-04-11 18:54:14.558565: Average global foreground Dice: [0.7212, 0.9081, 0.9485, 0.3151] +2022-04-11 18:54:14.559683: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:54:15.566501: lr: 0.00884 +2022-04-11 18:54:15.567366: saving scheduled checkpoint file... +2022-04-11 18:54:15.695309: saving checkpoint... +2022-04-11 18:54:20.329912: done, saving took 4.76 seconds +2022-04-11 18:54:20.359336: done +2022-04-11 18:54:20.483702: saving checkpoint... +2022-04-11 18:54:25.088330: done, saving took 4.73 seconds +2022-04-11 18:54:25.108469: This epoch took 129.819561 s + +2022-04-11 18:54:25.109867: svmem(total=99932160000, available=71738171392, percent=28.2, used=23939317760, free=26449747968, active=48484470784, inactive=22606426112, buffers=38735872, cached=49504358400, shared=2193182720, slab=1269141504) +2022-04-11 18:54:25.110504: +epoch: 64 +2022-04-11 18:56:15.570628: train loss : -0.3774 +2022-04-11 18:56:27.840337: validation loss: -0.3773 +2022-04-11 18:56:27.851278: Average global foreground Dice: [0.7141, 0.9182, 0.9263, 0.2496] +2022-04-11 18:56:27.852256: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:56:28.722017: lr: 0.008822 +2022-04-11 18:56:28.723807: saving scheduled checkpoint file... +2022-04-11 18:56:28.869378: saving checkpoint... +2022-04-11 18:56:33.356452: done, saving took 4.63 seconds +2022-04-11 18:56:33.378145: done +2022-04-11 18:56:33.502068: saving checkpoint... +2022-04-11 18:56:38.199621: done, saving took 4.82 seconds +2022-04-11 18:56:38.221232: This epoch took 133.110068 s + +2022-04-11 18:56:38.222543: svmem(total=99932160000, available=71580631040, percent=28.4, used=24082960384, free=26291171328, active=48633544704, inactive=22620708864, buffers=38871040, cached=49519157248, shared=2206822400, slab=1268670464) +2022-04-11 18:56:38.223726: +epoch: 65 +2022-04-11 18:58:27.712118: train loss : -0.3917 +2022-04-11 18:58:40.018239: validation loss: -0.4325 +2022-04-11 18:58:40.020084: Average global foreground Dice: [0.7747, 0.9231, 0.9574, 0.2816] +2022-04-11 18:58:40.021408: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 18:58:40.935200: lr: 0.008804 +2022-04-11 18:58:40.936211: saving scheduled checkpoint file... +2022-04-11 18:58:41.027947: saving checkpoint... +2022-04-11 18:58:45.390052: done, saving took 4.45 seconds +2022-04-11 18:58:45.432248: done +2022-04-11 18:58:45.484301: saving checkpoint... +2022-04-11 18:58:49.830793: done, saving took 4.40 seconds +2022-04-11 18:58:49.843248: This epoch took 131.618716 s + +2022-04-11 18:58:49.844605: svmem(total=99932160000, available=71663607808, percent=28.3, used=24004153344, free=26373419008, active=48555257856, inactive=22617260032, buffers=38936576, cached=49515651072, shared=2202632192, slab=1269895168) +2022-04-11 18:58:49.845569: +epoch: 66 +2022-04-11 19:00:37.259517: train loss : -0.3242 +2022-04-11 19:00:49.588156: validation loss: -0.3894 +2022-04-11 19:00:49.610891: Average global foreground Dice: [0.7135, 0.9096, 0.9415, 0.2917] +2022-04-11 19:00:49.612144: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:00:50.231796: lr: 0.008785 +2022-04-11 19:00:50.232986: saving scheduled checkpoint file... +2022-04-11 19:00:50.297171: saving checkpoint... +2022-04-11 19:00:54.772312: done, saving took 4.54 seconds +2022-04-11 19:00:54.783021: done +2022-04-11 19:00:54.830988: saving checkpoint... +2022-04-11 19:00:59.352774: done, saving took 4.57 seconds +2022-04-11 19:00:59.364459: This epoch took 129.517904 s + +2022-04-11 19:00:59.365817: svmem(total=99932160000, available=71413137408, percent=28.5, used=24260730880, free=26122067968, active=48810696704, inactive=22611030016, buffers=39010304, cached=49510350848, shared=2196492288, slab=1270104064) +2022-04-11 19:00:59.366770: +epoch: 67 +2022-04-11 19:02:46.873090: train loss : -0.3738 +2022-04-11 19:02:58.110735: validation loss: -0.4657 +2022-04-11 19:02:58.112841: Average global foreground Dice: [0.7307, 0.9383, 0.9626, 0.302] +2022-04-11 19:02:58.117852: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:02:58.837849: lr: 0.008767 +2022-04-11 19:02:58.838950: saving scheduled checkpoint file... +2022-04-11 19:02:58.901926: saving checkpoint... +2022-04-11 19:03:03.441553: done, saving took 4.60 seconds +2022-04-11 19:03:03.462142: done +2022-04-11 19:03:03.519165: saving checkpoint... +2022-04-11 19:03:08.184125: done, saving took 4.71 seconds +2022-04-11 19:03:08.206252: This epoch took 128.838502 s + +2022-04-11 19:03:08.207552: svmem(total=99932160000, available=71398780928, percent=28.6, used=24269832192, free=26093256704, active=48819556352, inactive=22625640448, buffers=39104512, cached=49529966592, shared=2198601728, slab=1276366848) +2022-04-11 19:03:08.208521: +epoch: 68 +2022-04-11 19:04:54.897284: train loss : -0.3967 +2022-04-11 19:05:06.870562: validation loss: -0.3821 +2022-04-11 19:05:06.876331: Average global foreground Dice: [0.7369, 0.9196, 0.9429, 0.3008] +2022-04-11 19:05:06.881974: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:05:07.798418: lr: 0.008749 +2022-04-11 19:05:07.799808: saving scheduled checkpoint file... +2022-04-11 19:05:07.881236: saving checkpoint... +2022-04-11 19:05:12.368467: done, saving took 4.57 seconds +2022-04-11 19:05:12.402638: done +2022-04-11 19:05:12.465402: saving checkpoint... +2022-04-11 19:05:16.938370: done, saving took 4.53 seconds +2022-04-11 19:05:16.955183: This epoch took 128.745821 s + +2022-04-11 19:05:16.956802: svmem(total=99932160000, available=71597088768, percent=28.4, used=24074186752, free=26293022720, active=47968587776, inactive=23273504768, buffers=39186432, cached=49525764096, shared=2196123648, slab=1275830272) +2022-04-11 19:05:16.957529: +epoch: 69 +2022-04-11 19:07:04.665476: train loss : -0.3635 +2022-04-11 19:07:16.373395: validation loss: -0.3527 +2022-04-11 19:07:16.375566: Average global foreground Dice: [0.7455, 0.907, 0.9259, 0.2971] +2022-04-11 19:07:16.376827: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:07:17.383925: lr: 0.008731 +2022-04-11 19:07:17.422357: saving scheduled checkpoint file... +2022-04-11 19:07:17.522975: saving checkpoint... +2022-04-11 19:07:21.955659: done, saving took 4.53 seconds +2022-04-11 19:07:21.988881: done +2022-04-11 19:07:22.070941: saving checkpoint... +2022-04-11 19:07:26.610742: done, saving took 4.62 seconds +2022-04-11 19:07:26.651790: This epoch took 129.693489 s + +2022-04-11 19:07:26.653172: svmem(total=99932160000, available=71607312384, percent=28.3, used=24063770624, free=26302193664, active=48237252608, inactive=22996811776, buffers=39276544, cached=49526919168, shared=2196242432, slab=1275887616) +2022-04-11 19:07:26.654317: +epoch: 70 +2022-04-11 19:09:14.616590: train loss : -0.4213 +2022-04-11 19:09:26.486490: validation loss: -0.4579 +2022-04-11 19:09:26.488686: Average global foreground Dice: [0.6324, 0.9278, 0.9648, 0.3321] +2022-04-11 19:09:26.489962: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:09:27.151810: lr: 0.008712 +2022-04-11 19:09:27.153429: saving scheduled checkpoint file... +2022-04-11 19:09:27.275683: saving checkpoint... +2022-04-11 19:09:31.954727: done, saving took 4.80 seconds +2022-04-11 19:09:31.978604: done +2022-04-11 19:09:32.077263: saving checkpoint... +2022-04-11 19:09:36.780628: done, saving took 4.80 seconds +2022-04-11 19:09:36.800784: This epoch took 130.145452 s + +2022-04-11 19:09:36.801938: svmem(total=99932160000, available=71708024832, percent=28.2, used=23965155328, free=26402111488, active=48235008000, inactive=22900576256, buffers=39354368, cached=49525538816, shared=2194235392, slab=1274970112) +2022-04-11 19:09:36.802657: +epoch: 71 +2022-04-11 19:11:25.280174: train loss : -0.3866 +2022-04-11 19:11:37.569556: validation loss: -0.4725 +2022-04-11 19:11:37.575956: Average global foreground Dice: [0.7801, 0.933, 0.952, 0.3374] +2022-04-11 19:11:37.577073: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:11:38.554029: lr: 0.008694 +2022-04-11 19:11:38.556143: saving scheduled checkpoint file... +2022-04-11 19:11:38.664203: saving checkpoint... +2022-04-11 19:11:43.329226: done, saving took 4.77 seconds +2022-04-11 19:11:43.338286: done +2022-04-11 19:11:43.385927: saving checkpoint... +2022-04-11 19:11:47.975751: done, saving took 4.64 seconds +2022-04-11 19:11:47.991270: This epoch took 131.187947 s + +2022-04-11 19:11:47.993764: svmem(total=99932160000, available=71450378240, percent=28.5, used=24220749824, free=26143649792, active=48530534400, inactive=22869098496, buffers=39424000, cached=49528336384, shared=2196168704, slab=1273880576) +2022-04-11 19:11:47.995868: +epoch: 72 +2022-04-11 19:13:37.358001: train loss : -0.4066 +2022-04-11 19:13:49.722756: validation loss: -0.4664 +2022-04-11 19:13:49.724530: Average global foreground Dice: [0.7728, 0.9246, 0.9616, 0.3047] +2022-04-11 19:13:49.725712: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:13:50.460244: lr: 0.008676 +2022-04-11 19:13:50.461146: saving scheduled checkpoint file... +2022-04-11 19:13:50.551116: saving checkpoint... +2022-04-11 19:13:54.991128: done, saving took 4.53 seconds +2022-04-11 19:13:55.008350: done +2022-04-11 19:13:55.084999: saving checkpoint... +2022-04-11 19:13:59.538089: done, saving took 4.53 seconds +2022-04-11 19:13:59.590923: This epoch took 131.592515 s + +2022-04-11 19:13:59.592330: svmem(total=99932160000, available=71525838848, percent=28.4, used=24145092608, free=26218074112, active=48458756096, inactive=22864785408, buffers=39489536, cached=49529503744, shared=2196430848, slab=1276456960) +2022-04-11 19:13:59.593503: +epoch: 73 +2022-04-11 19:15:48.509385: train loss : -0.3745 +2022-04-11 19:16:00.545888: validation loss: -0.3755 +2022-04-11 19:16:00.552369: Average global foreground Dice: [0.7343, 0.9104, 0.9403, 0.2881] +2022-04-11 19:16:00.553371: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:16:01.442603: lr: 0.008658 +2022-04-11 19:16:01.443604: saving scheduled checkpoint file... +2022-04-11 19:16:01.538140: saving checkpoint... +2022-04-11 19:16:05.990880: done, saving took 4.55 seconds +2022-04-11 19:16:06.009314: done +2022-04-11 19:16:06.091265: saving checkpoint... +2022-04-11 19:16:10.573809: done, saving took 4.56 seconds +2022-04-11 19:16:10.596658: This epoch took 131.002057 s + +2022-04-11 19:16:10.598020: svmem(total=99932160000, available=71704649728, percent=28.2, used=23959838720, free=26395992064, active=48286740480, inactive=22854500352, buffers=39591936, cached=49536737280, shared=2202935296, slab=1276108800) +2022-04-11 19:16:10.599321: +epoch: 74 +2022-04-11 19:17:59.759021: train loss : -0.4096 +2022-04-11 19:18:12.075463: validation loss: -0.4448 +2022-04-11 19:18:12.077405: Average global foreground Dice: [0.7225, 0.9282, 0.9448, 0.3142] +2022-04-11 19:18:12.078672: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:18:12.869212: lr: 0.008639 +2022-04-11 19:18:12.870769: saving scheduled checkpoint file... +2022-04-11 19:18:13.001551: saving checkpoint... +2022-04-11 19:18:17.427347: done, saving took 4.56 seconds +2022-04-11 19:18:17.446983: done +2022-04-11 19:18:17.542412: saving checkpoint... +2022-04-11 19:18:22.004999: done, saving took 4.56 seconds +2022-04-11 19:18:22.025470: This epoch took 131.425343 s + +2022-04-11 19:18:22.026949: svmem(total=99932160000, available=71768338432, percent=28.2, used=23898910720, free=26458910720, active=48222732288, inactive=22851002368, buffers=39665664, cached=49534672896, shared=2200207360, slab=1276297216) +2022-04-11 19:18:22.027636: +epoch: 75 +2022-04-11 19:20:08.332307: train loss : -0.3743 +2022-04-11 19:20:19.422991: validation loss: -0.4216 +2022-04-11 19:20:19.425634: Average global foreground Dice: [0.7549, 0.9166, 0.9373, 0.3209] +2022-04-11 19:20:19.426721: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:20:20.203928: lr: 0.008621 +2022-04-11 19:20:20.205698: saving scheduled checkpoint file... +2022-04-11 19:20:20.329213: saving checkpoint... +2022-04-11 19:20:24.874214: done, saving took 4.67 seconds +2022-04-11 19:20:24.896098: done +2022-04-11 19:20:25.015186: saving checkpoint... +2022-04-11 19:20:29.542266: done, saving took 4.64 seconds +2022-04-11 19:20:29.594911: This epoch took 127.566438 s + +2022-04-11 19:20:29.596197: svmem(total=99932160000, available=71796019200, percent=28.2, used=23874588672, free=26485571584, active=48203329536, inactive=22846525440, buffers=39755776, cached=49532243968, shared=2196652032, slab=1275703296) +2022-04-11 19:20:29.597275: +epoch: 76 +2022-04-11 19:22:19.008609: train loss : -0.3528 +2022-04-11 19:22:31.720677: validation loss: -0.4158 +2022-04-11 19:22:31.726728: Average global foreground Dice: [0.7236, 0.9098, 0.9466, 0.3375] +2022-04-11 19:22:31.727680: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:22:32.321308: lr: 0.008603 +2022-04-11 19:22:32.322792: saving scheduled checkpoint file... +2022-04-11 19:22:32.451273: saving checkpoint... +2022-04-11 19:22:36.973559: done, saving took 4.65 seconds +2022-04-11 19:22:36.996704: done +2022-04-11 19:22:37.123287: saving checkpoint... +2022-04-11 19:22:41.586112: done, saving took 4.59 seconds +2022-04-11 19:22:41.608291: This epoch took 132.009753 s + +2022-04-11 19:22:41.609439: svmem(total=99932160000, available=71718756352, percent=28.2, used=23952408576, free=26406576128, active=48282681344, inactive=22847135744, buffers=39833600, cached=49533341696, shared=2196185088, slab=1277026304) +2022-04-11 19:22:41.610415: +epoch: 77 +2022-04-11 19:24:29.301479: train loss : -0.3870 +2022-04-11 19:24:41.360894: validation loss: -0.4520 +2022-04-11 19:24:41.379554: Average global foreground Dice: [0.765, 0.9227, 0.9495, 0.3166] +2022-04-11 19:24:41.380760: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:24:42.355647: lr: 0.008584 +2022-04-11 19:24:42.356690: saving scheduled checkpoint file... +2022-04-11 19:24:42.446141: saving checkpoint... +2022-04-11 19:24:47.039620: done, saving took 4.68 seconds +2022-04-11 19:24:47.053351: done +2022-04-11 19:24:47.102750: saving checkpoint... +2022-04-11 19:24:51.443958: done, saving took 4.39 seconds +2022-04-11 19:24:51.457770: This epoch took 129.846250 s + +2022-04-11 19:24:51.459010: svmem(total=99932160000, available=71525326848, percent=28.4, used=24139100160, free=26212397056, active=48480055296, inactive=22848315392, buffers=39907328, cached=49540755456, shared=2202853376, slab=1274957824) +2022-04-11 19:24:51.459783: +epoch: 78 +2022-04-11 19:26:40.120519: train loss : -0.4274 +2022-04-11 19:26:52.290891: validation loss: -0.4902 +2022-04-11 19:26:52.296248: Average global foreground Dice: [0.7765, 0.9305, 0.953, 0.3349] +2022-04-11 19:26:52.297661: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:26:53.129289: lr: 0.008566 +2022-04-11 19:26:53.130759: saving scheduled checkpoint file... +2022-04-11 19:26:53.208538: saving checkpoint... +2022-04-11 19:26:57.787100: done, saving took 4.66 seconds +2022-04-11 19:26:57.802129: done +2022-04-11 19:26:57.849536: saving checkpoint... +2022-04-11 19:27:02.358984: done, saving took 4.56 seconds +2022-04-11 19:27:02.383101: This epoch took 130.922239 s + +2022-04-11 19:27:02.384650: svmem(total=99932160000, available=71597539328, percent=28.4, used=24072024064, free=26277388288, active=48442671104, inactive=22815555584, buffers=40034304, cached=49542713344, shared=2196414464, slab=1279004672) +2022-04-11 19:27:02.385552: +epoch: 79 +2022-04-11 19:28:49.061388: train loss : -0.4574 +2022-04-11 19:29:00.778975: validation loss: -0.5091 +2022-04-11 19:29:00.791915: Average global foreground Dice: [0.7742, 0.9363, 0.963, 0.3251] +2022-04-11 19:29:00.792967: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:29:01.686133: lr: 0.008548 +2022-04-11 19:29:01.687498: saving scheduled checkpoint file... +2022-04-11 19:29:01.753598: saving checkpoint... +2022-04-11 19:29:06.214257: done, saving took 4.53 seconds +2022-04-11 19:29:06.232352: done +2022-04-11 19:29:06.299061: saving checkpoint... +2022-04-11 19:29:10.724781: done, saving took 4.49 seconds +2022-04-11 19:29:10.741008: This epoch took 128.354532 s + +2022-04-11 19:29:10.742436: svmem(total=99932160000, available=71533592576, percent=28.4, used=24136253440, free=26212630528, active=48509165568, inactive=22816210944, buffers=40112128, cached=49543163904, shared=2196156416, slab=1278066688) +2022-04-11 19:29:10.743698: +epoch: 80 +2022-04-11 19:30:57.089462: train loss : -0.4802 +2022-04-11 19:31:08.205732: validation loss: -0.4996 +2022-04-11 19:31:08.211947: Average global foreground Dice: [0.7725, 0.9349, 0.9602, 0.3297] +2022-04-11 19:31:08.212958: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:31:09.273666: lr: 0.008529 +2022-04-11 19:31:09.274963: saving scheduled checkpoint file... +2022-04-11 19:31:09.383308: saving checkpoint... +2022-04-11 19:31:13.877820: done, saving took 4.60 seconds +2022-04-11 19:31:13.898395: done +2022-04-11 19:31:13.981096: saving checkpoint... +2022-04-11 19:31:18.459979: done, saving took 4.56 seconds +2022-04-11 19:31:18.478554: This epoch took 127.733956 s + +2022-04-11 19:31:18.479614: svmem(total=99932160000, available=71603027968, percent=28.3, used=24056287232, free=26281426944, active=48424480768, inactive=22827335680, buffers=40185856, cached=49554259968, shared=2206642176, slab=1277890560) +2022-04-11 19:31:18.480368: +epoch: 81 +2022-04-11 19:33:04.383478: train loss : -0.4581 +2022-04-11 19:33:15.166869: validation loss: -0.4579 +2022-04-11 19:33:15.168599: Average global foreground Dice: [0.7517, 0.9299, 0.964, 0.3421] +2022-04-11 19:33:15.169380: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:33:15.879930: lr: 0.008511 +2022-04-11 19:33:15.881199: saving scheduled checkpoint file... +2022-04-11 19:33:16.002822: saving checkpoint... +2022-04-11 19:33:20.570220: done, saving took 4.69 seconds +2022-04-11 19:33:20.615238: done +2022-04-11 19:33:20.709787: saving checkpoint... +2022-04-11 19:33:26.325140: done, saving took 5.71 seconds +2022-04-11 19:33:26.370126: This epoch took 127.888835 s + +2022-04-11 19:33:26.371533: svmem(total=99932160000, available=71672238080, percent=28.3, used=23985459200, free=26349531136, active=48355979264, inactive=22827405312, buffers=40263680, cached=49556905984, shared=2208301056, slab=1277964288) +2022-04-11 19:33:26.372684: +epoch: 82 +2022-04-11 19:35:12.556623: train loss : -0.4702 +2022-04-11 19:35:25.275555: validation loss: -0.4037 +2022-04-11 19:35:25.279437: Average global foreground Dice: [0.7445, 0.923, 0.9395, 0.2989] +2022-04-11 19:35:25.280670: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:35:25.918546: lr: 0.008493 +2022-04-11 19:35:25.919775: saving scheduled checkpoint file... +2022-04-11 19:35:26.056378: saving checkpoint... +2022-04-11 19:35:30.469879: done, saving took 4.55 seconds +2022-04-11 19:35:30.517189: done +2022-04-11 19:35:30.518667: This epoch took 124.145215 s + +2022-04-11 19:35:30.520242: svmem(total=99932160000, available=71582441472, percent=28.4, used=24087609344, free=26258608128, active=48461287424, inactive=22816858112, buffers=40366080, cached=49545576448, shared=2195800064, slab=1277624320) +2022-04-11 19:35:30.521516: +epoch: 83 +2022-04-11 19:37:20.107598: train loss : -0.4083 +2022-04-11 19:37:32.836789: validation loss: -0.4785 +2022-04-11 19:37:32.839076: Average global foreground Dice: [0.7513, 0.9216, 0.9617, 0.3311] +2022-04-11 19:37:32.840673: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:37:33.572269: lr: 0.008474 +2022-04-11 19:37:33.573698: saving scheduled checkpoint file... +2022-04-11 19:37:33.670155: saving checkpoint... +2022-04-11 19:37:38.090527: done, saving took 4.52 seconds +2022-04-11 19:37:38.108334: done +2022-04-11 19:37:38.156561: saving checkpoint... +2022-04-11 19:37:42.561908: done, saving took 4.45 seconds +2022-04-11 19:37:42.600926: This epoch took 132.078533 s + +2022-04-11 19:37:42.602480: svmem(total=99932160000, available=71563407360, percent=28.4, used=24106115072, free=26238881792, active=48478117888, inactive=22818480128, buffers=40431616, cached=49546731520, shared=2196299776, slab=1277743104) +2022-04-11 19:37:42.603313: +epoch: 84 +2022-04-11 19:39:31.213066: train loss : -0.4054 +2022-04-11 19:39:44.377310: validation loss: -0.4174 +2022-04-11 19:39:44.379654: Average global foreground Dice: [0.7353, 0.9202, 0.9453, 0.3307] +2022-04-11 19:39:44.380704: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:39:45.131722: lr: 0.008456 +2022-04-11 19:39:45.133522: saving scheduled checkpoint file... +2022-04-11 19:39:45.214099: saving checkpoint... +2022-04-11 19:39:49.611459: done, saving took 4.48 seconds +2022-04-11 19:39:49.662814: done +2022-04-11 19:39:49.731624: saving checkpoint... +2022-04-11 19:39:54.188886: done, saving took 4.52 seconds +2022-04-11 19:39:54.208674: This epoch took 131.604588 s + +2022-04-11 19:39:54.210290: svmem(total=99932160000, available=71727423488, percent=28.2, used=23942197248, free=26402181120, active=48314273792, inactive=22818578432, buffers=40497152, cached=49547284480, shared=2196242432, slab=1277800448) +2022-04-11 19:39:54.211156: +epoch: 85 +2022-04-11 19:41:43.171819: train loss : -0.3202 +2022-04-11 19:41:54.214986: validation loss: -0.3822 +2022-04-11 19:41:54.227345: Average global foreground Dice: [0.7157, 0.903, 0.9414, 0.2878] +2022-04-11 19:41:54.228753: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:41:55.056335: lr: 0.008438 +2022-04-11 19:41:55.057794: saving scheduled checkpoint file... +2022-04-11 19:41:55.160272: saving checkpoint... +2022-04-11 19:42:00.174040: done, saving took 5.12 seconds +2022-04-11 19:42:00.192563: done +2022-04-11 19:42:00.193530: This epoch took 125.981419 s + +2022-04-11 19:42:00.194437: svmem(total=99932160000, available=71703367680, percent=28.2, used=23958626304, free=26377265152, active=48331395072, inactive=22826803200, buffers=40579072, cached=49555689472, shared=2203787264, slab=1277280256) +2022-04-11 19:42:00.195645: +epoch: 86 +2022-04-11 19:43:50.511739: train loss : -0.3843 +2022-04-11 19:44:03.696019: validation loss: -0.4464 +2022-04-11 19:44:03.698052: Average global foreground Dice: [0.7342, 0.923, 0.9373, 0.3293] +2022-04-11 19:44:03.699140: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:44:04.568508: lr: 0.008419 +2022-04-11 19:44:04.569714: saving scheduled checkpoint file... +2022-04-11 19:44:04.683113: saving checkpoint... +2022-04-11 19:44:09.100654: done, saving took 4.53 seconds +2022-04-11 19:44:09.146212: done +2022-04-11 19:44:09.147393: This epoch took 128.950845 s + +2022-04-11 19:44:09.148203: svmem(total=99932160000, available=71742115840, percent=28.2, used=23927234560, free=26415427584, active=48303431680, inactive=22819885056, buffers=40644608, cached=49548853248, shared=2196492288, slab=1277181952) +2022-04-11 19:44:09.148846: +epoch: 87 +2022-04-11 19:45:58.210088: train loss : -0.4604 +2022-04-11 19:46:10.200972: validation loss: -0.5159 +2022-04-11 19:46:10.210716: Average global foreground Dice: [0.7432, 0.9396, 0.9705, 0.3311] +2022-04-11 19:46:10.211967: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:46:11.098555: lr: 0.008401 +2022-04-11 19:46:11.100128: saving scheduled checkpoint file... +2022-04-11 19:46:11.209246: saving checkpoint... +2022-04-11 19:46:15.640654: done, saving took 4.54 seconds +2022-04-11 19:46:15.672036: done +2022-04-11 19:46:15.673503: This epoch took 126.523921 s + +2022-04-11 19:46:15.674755: svmem(total=99932160000, available=71847346176, percent=28.1, used=23822303232, free=26520076288, active=48200187904, inactive=22820470784, buffers=40726528, cached=49549053952, shared=2196176896, slab=1276792832) +2022-04-11 19:46:15.675805: +epoch: 88 +2022-04-11 19:48:05.877325: train loss : -0.4681 +2022-04-11 19:48:17.835390: validation loss: -0.4393 +2022-04-11 19:48:17.845375: Average global foreground Dice: [0.7679, 0.9147, 0.9436, 0.3273] +2022-04-11 19:48:17.846165: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:48:18.713967: lr: 0.008383 +2022-04-11 19:48:18.715193: saving scheduled checkpoint file... +2022-04-11 19:48:18.838598: saving checkpoint... +2022-04-11 19:48:23.356436: done, saving took 4.64 seconds +2022-04-11 19:48:23.377454: done +2022-04-11 19:48:23.495977: saving checkpoint... +2022-04-11 19:48:28.086476: done, saving took 4.71 seconds +2022-04-11 19:48:28.196707: This epoch took 132.519892 s + +2022-04-11 19:48:28.209306: svmem(total=99932160000, available=71534116864, percent=28.4, used=24126214144, free=26206158848, active=48501075968, inactive=22830202880, buffers=40816640, cached=49558970368, shared=2205548544, slab=1276620800) +2022-04-11 19:48:28.214314: +epoch: 89 +2022-04-11 19:50:17.410523: train loss : -0.4364 +2022-04-11 19:50:30.621792: validation loss: -0.4538 +2022-04-11 19:50:30.636366: Average global foreground Dice: [0.7579, 0.9219, 0.9571, 0.3302] +2022-04-11 19:50:30.637580: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:50:31.442158: lr: 0.008364 +2022-04-11 19:50:31.443489: saving scheduled checkpoint file... +2022-04-11 19:50:31.542506: saving checkpoint... +2022-04-11 19:50:35.850993: done, saving took 4.41 seconds +2022-04-11 19:50:35.865740: done +2022-04-11 19:50:35.913440: saving checkpoint... +2022-04-11 19:50:40.304385: done, saving took 4.44 seconds +2022-04-11 19:50:40.319217: This epoch took 132.089951 s + +2022-04-11 19:50:40.320199: svmem(total=99932160000, available=71655743488, percent=28.3, used=24013365248, free=26328092672, active=48414425088, inactive=22795624448, buffers=40923136, cached=49549778944, shared=2196328448, slab=1277825024) +2022-04-11 19:50:40.321064: +epoch: 90 +2022-04-11 19:52:27.681289: train loss : -0.4596 +2022-04-11 19:52:39.525730: validation loss: -0.4427 +2022-04-11 19:52:39.527856: Average global foreground Dice: [0.7647, 0.9155, 0.9394, 0.3302] +2022-04-11 19:52:39.528646: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:52:40.360944: lr: 0.008346 +2022-04-11 19:52:40.362247: saving scheduled checkpoint file... +2022-04-11 19:52:40.448262: saving checkpoint... +2022-04-11 19:52:45.003880: done, saving took 4.64 seconds +2022-04-11 19:52:45.012632: done +2022-04-11 19:52:45.060823: saving checkpoint... +2022-04-11 19:52:49.374249: done, saving took 4.36 seconds +2022-04-11 19:52:49.425280: This epoch took 129.103592 s + +2022-04-11 19:52:49.426875: svmem(total=99932160000, available=71516622848, percent=28.4, used=24152801280, free=26188926976, active=48551112704, inactive=22795325440, buffers=40984576, cached=49549447168, shared=2195996672, slab=1276821504) +2022-04-11 19:52:49.427972: +epoch: 91 +2022-04-11 19:54:37.660686: train loss : -0.4438 +2022-04-11 19:54:48.975024: validation loss: -0.5248 +2022-04-11 19:54:48.984420: Average global foreground Dice: [0.7692, 0.9361, 0.9645, 0.3625] +2022-04-11 19:54:48.985302: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:54:49.883906: lr: 0.008328 +2022-04-11 19:54:49.885413: saving scheduled checkpoint file... +2022-04-11 19:54:49.957854: saving checkpoint... +2022-04-11 19:54:54.623448: done, saving took 4.74 seconds +2022-04-11 19:54:54.667132: done +2022-04-11 19:54:54.713525: saving checkpoint... +2022-04-11 19:54:59.231657: done, saving took 4.56 seconds +2022-04-11 19:54:59.244180: This epoch took 129.815142 s + +2022-04-11 19:54:59.245710: svmem(total=99932160000, available=71617658880, percent=28.3, used=24041103360, free=26289606656, active=48442191872, inactive=22805905408, buffers=41062400, cached=49560387584, shared=2206728192, slab=1278525440) +2022-04-11 19:54:59.246720: +epoch: 92 +2022-04-11 19:56:47.296640: train loss : -0.4102 +2022-04-11 19:56:59.331266: validation loss: -0.4337 +2022-04-11 19:56:59.351643: Average global foreground Dice: [0.7542, 0.9242, 0.942, 0.3019] +2022-04-11 19:56:59.352829: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:57:00.148041: lr: 0.008309 +2022-04-11 19:57:00.149364: saving scheduled checkpoint file... +2022-04-11 19:57:00.216193: saving checkpoint... +2022-04-11 19:57:04.664807: done, saving took 4.51 seconds +2022-04-11 19:57:04.712203: done +2022-04-11 19:57:04.713516: This epoch took 125.466143 s + +2022-04-11 19:57:04.715050: svmem(total=99932160000, available=71610875904, percent=28.3, used=24044548096, free=26282696704, active=48444690432, inactive=22808231936, buffers=41168896, cached=49563746304, shared=2210033664, slab=1277997056) +2022-04-11 19:57:04.716006: +epoch: 93 +2022-04-11 19:58:52.500543: train loss : -0.3431 +2022-04-11 19:59:03.799776: validation loss: -0.3682 +2022-04-11 19:59:03.801943: Average global foreground Dice: [0.7117, 0.8777, 0.9341, 0.3204] +2022-04-11 19:59:03.803176: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 19:59:04.677224: lr: 0.008291 +2022-04-11 19:59:04.678337: saving scheduled checkpoint file... +2022-04-11 19:59:04.779145: saving checkpoint... +2022-04-11 19:59:09.177545: done, saving took 4.50 seconds +2022-04-11 19:59:09.239446: done +2022-04-11 19:59:09.240758: This epoch took 124.523548 s + +2022-04-11 19:59:09.241841: svmem(total=99932160000, available=71688409088, percent=28.3, used=23980638208, free=26359779328, active=48379084800, inactive=22794612736, buffers=41226240, cached=49550516224, shared=2196328448, slab=1278676992) +2022-04-11 19:59:09.242532: +epoch: 94 +2022-04-11 20:00:56.094079: train loss : -0.4063 +2022-04-11 20:01:07.798992: validation loss: -0.4532 +2022-04-11 20:01:07.801080: Average global foreground Dice: [0.7428, 0.9248, 0.9548, 0.3276] +2022-04-11 20:01:07.802108: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:01:08.684552: lr: 0.008272 +2022-04-11 20:01:08.685799: saving scheduled checkpoint file... +2022-04-11 20:01:08.814292: saving checkpoint... +2022-04-11 20:01:13.355457: done, saving took 4.67 seconds +2022-04-11 20:01:13.376362: done +2022-04-11 20:01:13.377206: This epoch took 124.133699 s + +2022-04-11 20:01:13.378939: svmem(total=99932160000, available=71627755520, percent=28.3, used=24041164800, free=26298974208, active=48439140352, inactive=22794510336, buffers=41295872, cached=49550725120, shared=2196328448, slab=1281007616) +2022-04-11 20:01:13.379882: +epoch: 95 +2022-04-11 20:03:01.443920: train loss : -0.4558 +2022-04-11 20:03:13.284811: validation loss: -0.4307 +2022-04-11 20:03:13.287550: Average global foreground Dice: [0.8242, 0.9163, 0.9326, 0.3388] +2022-04-11 20:03:13.291231: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:03:14.105758: lr: 0.008254 +2022-04-11 20:03:14.107241: saving scheduled checkpoint file... +2022-04-11 20:03:14.194159: saving checkpoint... +2022-04-11 20:03:19.161468: done, saving took 5.05 seconds +2022-04-11 20:03:19.175343: done +2022-04-11 20:03:19.176344: This epoch took 125.795720 s + +2022-04-11 20:03:19.177421: svmem(total=99932160000, available=71717683200, percent=28.2, used=23951409152, free=26395000832, active=48352690176, inactive=22788714496, buffers=41373696, cached=49544376320, shared=2196439040, slab=1278529536) +2022-04-11 20:03:19.178094: +epoch: 96 +2022-04-11 20:05:08.125480: train loss : -0.4521 +2022-04-11 20:05:19.516427: validation loss: -0.4908 +2022-04-11 20:05:19.519865: Average global foreground Dice: [0.748, 0.9369, 0.9506, 0.3505] +2022-04-11 20:05:19.522933: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:05:20.396015: lr: 0.008236 +2022-04-11 20:05:20.397592: saving scheduled checkpoint file... +2022-04-11 20:05:20.490832: saving checkpoint... +2022-04-11 20:05:24.995656: done, saving took 4.60 seconds +2022-04-11 20:05:25.012825: done +2022-04-11 20:05:25.077903: saving checkpoint... +2022-04-11 20:05:29.562397: done, saving took 4.55 seconds +2022-04-11 20:05:29.601523: This epoch took 130.422769 s + +2022-04-11 20:05:29.603278: svmem(total=99932160000, available=71429181440, percent=28.5, used=24240611328, free=26105638912, active=48641748992, inactive=22788390912, buffers=41480192, cached=49544429568, shared=2195746816, slab=1278095360) +2022-04-11 20:05:29.604114: +epoch: 97 +2022-04-11 20:07:17.273744: train loss : -0.4645 +2022-04-11 20:07:29.562621: validation loss: -0.4961 +2022-04-11 20:07:29.564618: Average global foreground Dice: [0.7961, 0.9384, 0.9608, 0.3306] +2022-04-11 20:07:29.565914: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:07:30.450982: lr: 0.008217 +2022-04-11 20:07:30.452187: saving scheduled checkpoint file... +2022-04-11 20:07:30.574170: saving checkpoint... +2022-04-11 20:07:34.972037: done, saving took 4.52 seconds +2022-04-11 20:07:34.990670: done +2022-04-11 20:07:35.069842: saving checkpoint... +2022-04-11 20:07:39.565091: done, saving took 4.57 seconds +2022-04-11 20:07:39.583057: This epoch took 129.977882 s + +2022-04-11 20:07:39.584332: svmem(total=99932160000, available=71770509312, percent=28.2, used=23912284160, free=26446163968, active=48315981824, inactive=22775914496, buffers=41553920, cached=49532157952, shared=2182701056, slab=1278447616) +2022-04-11 20:07:39.585241: +epoch: 98 +2022-04-11 20:09:29.817050: train loss : -0.4764 +2022-04-11 20:09:43.396559: validation loss: -0.5226 +2022-04-11 20:09:43.398961: Average global foreground Dice: [0.7636, 0.9437, 0.9528, 0.3432] +2022-04-11 20:09:43.400099: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:09:44.098642: lr: 0.008199 +2022-04-11 20:09:44.099946: saving scheduled checkpoint file... +2022-04-11 20:09:44.212514: saving checkpoint... +2022-04-11 20:09:48.623689: done, saving took 4.52 seconds +2022-04-11 20:09:48.639612: done +2022-04-11 20:09:48.734945: saving checkpoint... +2022-04-11 20:09:53.203893: done, saving took 4.56 seconds +2022-04-11 20:09:53.223815: This epoch took 133.637886 s + +2022-04-11 20:09:53.225058: svmem(total=99932160000, available=71707500544, percent=28.2, used=23937384448, free=26382172160, active=48339468288, inactive=22814633984, buffers=41627648, cached=49570975744, shared=2220527616, slab=1278545920) +2022-04-11 20:09:53.226327: +epoch: 99 +2022-04-11 20:11:44.333015: train loss : -0.3427 +2022-04-11 20:11:56.305505: validation loss: -0.3411 +2022-04-11 20:11:56.313252: Average global foreground Dice: [0.7488, 0.9, 0.9047, 0.3173] +2022-04-11 20:11:56.314309: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:11:56.936810: lr: 0.008181 +2022-04-11 20:11:56.937819: saving scheduled checkpoint file... +2022-04-11 20:11:57.067589: saving checkpoint... +2022-04-11 20:12:01.943211: done, saving took 5.00 seconds +2022-04-11 20:12:02.002761: done +2022-04-11 20:12:02.003729: This epoch took 128.776212 s + +2022-04-11 20:12:02.004902: svmem(total=99932160000, available=71639007232, percent=28.3, used=24028905472, free=26336571392, active=48451371008, inactive=22746206208, buffers=41721856, cached=49524961280, shared=2196709376, slab=1280249856) +2022-04-11 20:12:02.005872: +epoch: 100 +2022-04-11 20:13:51.168970: train loss : -0.2836 +2022-04-11 20:14:04.276722: validation loss: -0.3857 +2022-04-11 20:14:04.285666: Average global foreground Dice: [0.6953, 0.9211, 0.9435, 0.328] +2022-04-11 20:14:04.286666: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:14:05.101148: lr: 0.008162 +2022-04-11 20:14:05.102355: saving scheduled checkpoint file... +2022-04-11 20:14:05.229079: saving checkpoint... +2022-04-11 20:14:09.671656: done, saving took 4.57 seconds +2022-04-11 20:14:09.694640: done +2022-04-11 20:14:09.695612: This epoch took 127.689042 s + +2022-04-11 20:14:09.697409: svmem(total=99932160000, available=71708499968, percent=28.2, used=23949361152, free=26405134336, active=48376119296, inactive=22755561472, buffers=41783296, cached=49535881216, shared=2206728192, slab=1279012864) +2022-04-11 20:14:09.698337: +epoch: 101 +2022-04-11 20:15:59.067595: train loss : -0.4204 +2022-04-11 20:16:11.101132: validation loss: -0.4421 +2022-04-11 20:16:11.108765: Average global foreground Dice: [0.769, 0.9182, 0.9511, 0.3349] +2022-04-11 20:16:11.110035: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:16:12.114200: lr: 0.008144 +2022-04-11 20:16:12.115568: saving scheduled checkpoint file... +2022-04-11 20:16:12.207164: saving checkpoint... +2022-04-11 20:16:16.726911: done, saving took 4.61 seconds +2022-04-11 20:16:16.736675: done +2022-04-11 20:16:16.738105: This epoch took 127.038663 s + +2022-04-11 20:16:16.741214: svmem(total=99932160000, available=71455514624, percent=28.5, used=24213213184, free=26151526400, active=48641552384, inactive=22745202688, buffers=41869312, cached=49525551104, shared=2195816448, slab=1280503808) +2022-04-11 20:16:16.742545: +epoch: 102 +2022-04-11 20:18:03.778753: train loss : -0.4509 +2022-04-11 20:18:15.695846: validation loss: -0.4445 +2022-04-11 20:18:15.709476: Average global foreground Dice: [0.7634, 0.9121, 0.9492, 0.3278] +2022-04-11 20:18:15.716839: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:18:16.740419: lr: 0.008125 +2022-04-11 20:18:16.743628: saving scheduled checkpoint file... +2022-04-11 20:18:16.831194: saving checkpoint... +2022-04-11 20:18:21.399173: done, saving took 4.65 seconds +2022-04-11 20:18:21.420949: done +2022-04-11 20:18:21.424227: This epoch took 124.679314 s + +2022-04-11 20:18:21.427010: svmem(total=99932160000, available=71436951552, percent=28.5, used=24229994496, free=26132086784, active=48655818752, inactive=22747639808, buffers=41955328, cached=49528123392, shared=2197577728, slab=1279803392) +2022-04-11 20:18:21.429293: +epoch: 103 +2022-04-11 20:20:09.503761: train loss : -0.4628 +2022-04-11 20:20:20.736617: validation loss: -0.4900 +2022-04-11 20:20:20.740211: Average global foreground Dice: [0.7805, 0.9264, 0.957, 0.3415] +2022-04-11 20:20:20.742145: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:20:21.643398: lr: 0.008107 +2022-04-11 20:20:21.645384: saving scheduled checkpoint file... +2022-04-11 20:20:21.722448: saving checkpoint... +2022-04-11 20:20:26.291178: done, saving took 4.64 seconds +2022-04-11 20:20:26.332982: done +2022-04-11 20:20:26.336414: This epoch took 124.904565 s + +2022-04-11 20:20:26.339477: svmem(total=99932160000, available=71699697664, percent=28.3, used=23972491264, free=26393919488, active=48394973184, inactive=22743203840, buffers=42041344, cached=49523707904, shared=2192277504, slab=1280540672) +2022-04-11 20:20:26.341500: +epoch: 104 +2022-04-11 20:22:13.708849: train loss : -0.4834 +2022-04-11 20:22:25.165470: validation loss: -0.5211 +2022-04-11 20:22:25.183551: Average global foreground Dice: [0.8128, 0.9316, 0.9665, 0.3468] +2022-04-11 20:22:25.184925: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:22:26.017034: lr: 0.008088 +2022-04-11 20:22:26.018180: saving scheduled checkpoint file... +2022-04-11 20:22:26.105288: saving checkpoint... +2022-04-11 20:22:31.050795: done, saving took 5.03 seconds +2022-04-11 20:22:31.114577: done +2022-04-11 20:22:31.197971: saving checkpoint... +2022-04-11 20:22:35.734997: done, saving took 4.62 seconds +2022-04-11 20:22:35.805544: This epoch took 129.461495 s + +2022-04-11 20:22:35.807298: svmem(total=99932160000, available=71818985472, percent=28.1, used=23849070592, free=26512338944, active=48276901888, inactive=22747852800, buffers=42139648, cached=49528610816, shared=2196328448, slab=1279913984) +2022-04-11 20:22:35.808584: +epoch: 105 +2022-04-11 20:24:23.598526: train loss : -0.4405 +2022-04-11 20:24:34.385007: validation loss: -0.4637 +2022-04-11 20:24:34.397634: Average global foreground Dice: [0.7799, 0.9185, 0.9457, 0.331] +2022-04-11 20:24:34.400700: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:24:35.588029: lr: 0.00807 +2022-04-11 20:24:36.022978: saving scheduled checkpoint file... +2022-04-11 20:24:36.308958: saving checkpoint... +2022-04-11 20:24:43.164092: done, saving took 6.96 seconds +2022-04-11 20:24:43.221241: done +2022-04-11 20:24:43.319618: saving checkpoint... +2022-04-11 20:24:47.914749: done, saving took 4.69 seconds +2022-04-11 20:24:47.935236: This epoch took 132.125664 s + +2022-04-11 20:24:47.936634: svmem(total=99932160000, available=71737339904, percent=28.2, used=23920795648, free=26575073280, active=48349949952, inactive=22614302720, buffers=42221568, cached=49394069504, shared=2207825920, slab=1276907520) +2022-04-11 20:24:47.937424: +epoch: 106 +2022-04-11 20:26:36.465475: train loss : -0.4733 +2022-04-11 20:26:48.518471: validation loss: -0.4855 +2022-04-11 20:26:48.540033: Average global foreground Dice: [0.7782, 0.936, 0.9389, 0.3301] +2022-04-11 20:26:48.541495: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:26:49.434900: lr: 0.008052 +2022-04-11 20:26:49.436562: saving scheduled checkpoint file... +2022-04-11 20:26:49.571988: saving checkpoint... +2022-04-11 20:26:53.455501: done, saving took 4.02 seconds +2022-04-11 20:26:53.509523: done +2022-04-11 20:26:53.619824: saving checkpoint... +2022-04-11 20:26:58.037401: done, saving took 4.53 seconds +2022-04-11 20:26:58.102072: This epoch took 130.163920 s + +2022-04-11 20:26:58.103650: svmem(total=99932160000, available=71630635008, percent=28.3, used=24039325696, free=26467573760, active=48462790656, inactive=22603915264, buffers=42315776, cached=49382944768, shared=2196131840, slab=1278287872) +2022-04-11 20:26:58.113725: +epoch: 107 +2022-04-11 20:28:48.082075: train loss : -0.4678 +2022-04-11 20:29:00.151264: validation loss: -0.5431 +2022-04-11 20:29:00.153405: Average global foreground Dice: [0.8083, 0.9379, 0.966, 0.3837] +2022-04-11 20:29:00.154734: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:29:01.144052: lr: 0.008033 +2022-04-11 20:29:01.145212: saving scheduled checkpoint file... +2022-04-11 20:29:01.219059: saving checkpoint... +2022-04-11 20:29:05.740968: done, saving took 4.59 seconds +2022-04-11 20:29:05.793245: done +2022-04-11 20:29:05.844022: saving checkpoint... +2022-04-11 20:29:10.385345: done, saving took 4.59 seconds +2022-04-11 20:29:10.438917: This epoch took 132.321242 s + +2022-04-11 20:29:10.440449: svmem(total=99932160000, available=71508541440, percent=28.4, used=24151064576, free=26338197504, active=48581664768, inactive=22615011328, buffers=43102208, cached=49399795712, shared=2206371840, slab=1277415424) +2022-04-11 20:29:10.441638: +epoch: 108 +2022-04-11 20:30:59.434181: train loss : -0.5052 +2022-04-11 20:31:11.830385: validation loss: -0.5090 +2022-04-11 20:31:11.832367: Average global foreground Dice: [0.7813, 0.9363, 0.9669, 0.332] +2022-04-11 20:31:11.833839: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:31:12.732276: lr: 0.008015 +2022-04-11 20:31:12.733949: saving scheduled checkpoint file... +2022-04-11 20:31:12.829277: saving checkpoint... +2022-04-11 20:31:17.297561: done, saving took 4.56 seconds +2022-04-11 20:31:17.314816: done +2022-04-11 20:31:17.382270: saving checkpoint... +2022-04-11 20:31:22.132257: done, saving took 4.82 seconds +2022-04-11 20:31:22.149631: This epoch took 131.707211 s + +2022-04-11 20:31:22.150950: svmem(total=99932160000, available=71586701312, percent=28.4, used=24083103744, free=26384613376, active=48516894720, inactive=22633537536, buffers=44007424, cached=49420435456, shared=2196242432, slab=1277968384) +2022-04-11 20:31:22.151989: +epoch: 109 +2022-04-11 20:33:11.018990: train loss : -0.5130 +2022-04-11 20:33:22.608850: validation loss: -0.5369 +2022-04-11 20:33:22.611518: Average global foreground Dice: [0.7783, 0.9392, 0.97, 0.3754] +2022-04-11 20:33:22.613386: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:33:23.566383: lr: 0.007996 +2022-04-11 20:33:23.568236: saving scheduled checkpoint file... +2022-04-11 20:33:23.673861: saving checkpoint... +2022-04-11 20:33:28.235552: done, saving took 4.67 seconds +2022-04-11 20:33:28.253519: done +2022-04-11 20:33:28.334209: saving checkpoint... +2022-04-11 20:33:32.853342: done, saving took 4.60 seconds +2022-04-11 20:33:32.911354: This epoch took 130.758650 s + +2022-04-11 20:33:32.912913: svmem(total=99932160000, available=71662063616, percent=28.3, used=24005353472, free=26459230208, active=48446902272, inactive=22636572672, buffers=44101632, cached=49423474688, shared=2198609920, slab=1277313024) +2022-04-11 20:33:32.914163: +epoch: 110 +2022-04-11 20:35:22.399561: train loss : -0.5009 +2022-04-11 20:35:34.434807: validation loss: -0.4880 +2022-04-11 20:35:34.438943: Average global foreground Dice: [0.7935, 0.9327, 0.9544, 0.3655] +2022-04-11 20:35:34.439850: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:35:35.120995: lr: 0.007978 +2022-04-11 20:35:35.122247: saving scheduled checkpoint file... +2022-04-11 20:35:35.235175: saving checkpoint... +2022-04-11 20:35:39.928493: done, saving took 4.81 seconds +2022-04-11 20:35:40.010977: done +2022-04-11 20:35:40.112713: saving checkpoint... +2022-04-11 20:35:44.632060: done, saving took 4.62 seconds +2022-04-11 20:35:44.652263: This epoch took 131.736685 s + +2022-04-11 20:35:44.653429: svmem(total=99932160000, available=71678062592, percent=28.3, used=23977635840, free=26356469760, active=48427839488, inactive=22747127808, buffers=44228608, cached=49553825792, shared=2208026624, slab=1282789376) +2022-04-11 20:35:44.654497: +epoch: 111 +2022-04-11 20:37:33.437816: train loss : -0.4903 +2022-04-11 20:37:45.056941: validation loss: -0.4996 +2022-04-11 20:37:45.070031: Average global foreground Dice: [0.819, 0.9379, 0.9573, 0.3614] +2022-04-11 20:37:45.070970: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:37:45.971579: lr: 0.007959 +2022-04-11 20:37:45.972953: saving scheduled checkpoint file... +2022-04-11 20:37:46.108601: saving checkpoint... +2022-04-11 20:37:50.551648: done, saving took 4.58 seconds +2022-04-11 20:37:50.572083: done +2022-04-11 20:37:50.676980: saving checkpoint... +2022-04-11 20:37:55.124000: done, saving took 4.55 seconds +2022-04-11 20:37:55.145448: This epoch took 130.490215 s + +2022-04-11 20:37:55.146985: svmem(total=99932160000, available=71498215424, percent=28.5, used=24168759296, free=26175741952, active=48618868736, inactive=22738489344, buffers=44290048, cached=49543368704, shared=2196652032, slab=1281175552) +2022-04-11 20:37:55.148083: +epoch: 112 +2022-04-11 20:39:45.113776: train loss : -0.5007 +2022-04-11 20:39:57.947649: validation loss: -0.5156 +2022-04-11 20:39:57.955647: Average global foreground Dice: [0.7601, 0.9386, 0.976, 0.3356] +2022-04-11 20:39:57.957496: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:39:58.666395: lr: 0.007941 +2022-04-11 20:39:58.668204: saving scheduled checkpoint file... +2022-04-11 20:39:58.801155: saving checkpoint... +2022-04-11 20:40:03.275004: done, saving took 4.61 seconds +2022-04-11 20:40:03.320543: done +2022-04-11 20:40:03.416317: saving checkpoint... +2022-04-11 20:40:07.952632: done, saving took 4.63 seconds +2022-04-11 20:40:07.975125: This epoch took 132.826371 s + +2022-04-11 20:40:07.976776: svmem(total=99932160000, available=71485255680, percent=28.5, used=24195993600, free=26161725440, active=48640716800, inactive=22724534272, buffers=44380160, cached=49530060800, shared=2182279168, slab=1281806336) +2022-04-11 20:40:07.977942: +epoch: 113 +2022-04-11 20:41:57.463986: train loss : -0.4919 +2022-04-11 20:42:09.555755: validation loss: -0.4855 +2022-04-11 20:42:09.567356: Average global foreground Dice: [0.7727, 0.9334, 0.946, 0.3231] +2022-04-11 20:42:09.568524: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:42:10.549752: lr: 0.007922 +2022-04-11 20:42:10.552033: saving scheduled checkpoint file... +2022-04-11 20:42:10.643505: saving checkpoint... +2022-04-11 20:42:15.138790: done, saving took 4.59 seconds +2022-04-11 20:42:15.191174: done +2022-04-11 20:42:15.192763: This epoch took 127.214070 s + +2022-04-11 20:42:15.194462: svmem(total=99932160000, available=71592177664, percent=28.4, used=24070557696, free=26267873280, active=48523902976, inactive=22743822336, buffers=44445696, cached=49549283328, shared=2200821760, slab=1282953216) +2022-04-11 20:42:15.195761: +epoch: 114 +2022-04-11 20:44:03.738406: train loss : -0.4724 +2022-04-11 20:44:15.895952: validation loss: -0.5037 +2022-04-11 20:44:15.898422: Average global foreground Dice: [0.7894, 0.9344, 0.965, 0.3444] +2022-04-11 20:44:15.899907: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:44:16.576769: lr: 0.007904 +2022-04-11 20:44:16.578342: saving scheduled checkpoint file... +2022-04-11 20:44:16.650731: saving checkpoint... +2022-04-11 20:44:21.099458: done, saving took 4.52 seconds +2022-04-11 20:44:21.112441: done +2022-04-11 20:44:21.159327: saving checkpoint... +2022-04-11 20:44:25.598624: done, saving took 4.49 seconds +2022-04-11 20:44:25.645007: This epoch took 130.448179 s + +2022-04-11 20:44:25.646888: svmem(total=99932160000, available=71419883520, percent=28.5, used=24247738368, free=26094710784, active=48692203520, inactive=22739976192, buffers=44511232, cached=49545199616, shared=2196062208, slab=1283727360) +2022-04-11 20:44:25.647975: +epoch: 115 +2022-04-11 20:46:14.799149: train loss : -0.3722 +2022-04-11 20:46:26.783284: validation loss: -0.4308 +2022-04-11 20:46:26.785627: Average global foreground Dice: [0.7674, 0.904, 0.935, 0.335] +2022-04-11 20:46:26.786598: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:46:27.704578: lr: 0.007885 +2022-04-11 20:46:27.705961: saving scheduled checkpoint file... +2022-04-11 20:46:27.767013: saving checkpoint... +2022-04-11 20:46:32.141296: done, saving took 4.43 seconds +2022-04-11 20:46:32.156654: done +2022-04-11 20:46:32.158067: This epoch took 126.509213 s + +2022-04-11 20:46:32.159442: svmem(total=99932160000, available=71426781184, percent=28.5, used=24238784512, free=26101047296, active=48689754112, inactive=22742573056, buffers=44617728, cached=49547710464, shared=2198224896, slab=1281937408) +2022-04-11 20:46:32.160553: +epoch: 116 +2022-04-11 20:48:18.292104: train loss : -0.4090 +2022-04-11 20:48:30.182600: validation loss: -0.4556 +2022-04-11 20:48:30.184661: Average global foreground Dice: [0.7157, 0.9188, 0.9584, 0.3454] +2022-04-11 20:48:30.185823: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:48:31.018576: lr: 0.007867 +2022-04-11 20:48:31.019856: saving scheduled checkpoint file... +2022-04-11 20:48:31.084492: saving checkpoint... +2022-04-11 20:48:35.782312: done, saving took 4.76 seconds +2022-04-11 20:48:35.821203: done +2022-04-11 20:48:35.822765: This epoch took 123.661412 s + +2022-04-11 20:48:35.824450: svmem(total=99932160000, available=71346282496, percent=28.6, used=24320839680, free=26019717120, active=48771362816, inactive=22741704704, buffers=44679168, cached=49546924032, shared=2196660224, slab=1282502656) +2022-04-11 20:48:35.825265: +epoch: 117 +2022-04-11 20:50:24.304200: train loss : -0.4381 +2022-04-11 20:50:36.297696: validation loss: -0.5169 +2022-04-11 20:50:36.308900: Average global foreground Dice: [0.789, 0.9339, 0.964, 0.345] +2022-04-11 20:50:36.310455: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:50:37.004154: lr: 0.007848 +2022-04-11 20:50:37.005509: saving scheduled checkpoint file... +2022-04-11 20:50:37.093701: saving checkpoint... +2022-04-11 20:50:41.601203: done, saving took 4.59 seconds +2022-04-11 20:50:41.627970: done +2022-04-11 20:50:41.630848: This epoch took 125.804806 s + +2022-04-11 20:50:41.631909: svmem(total=99932160000, available=71524962304, percent=28.4, used=24142512128, free=26197557248, active=48589508608, inactive=22742069248, buffers=44752896, cached=49547337728, shared=2196234240, slab=1285083136) +2022-04-11 20:50:41.632761: +epoch: 118 +2022-04-11 20:52:30.370024: train loss : -0.4659 +2022-04-11 20:52:42.022299: validation loss: -0.4414 +2022-04-11 20:52:42.035672: Average global foreground Dice: [0.79, 0.9155, 0.9334, 0.3449] +2022-04-11 20:52:42.036703: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:52:42.899782: lr: 0.00783 +2022-04-11 20:52:42.901308: saving scheduled checkpoint file... +2022-04-11 20:52:43.022070: saving checkpoint... +2022-04-11 20:52:47.578856: done, saving took 4.68 seconds +2022-04-11 20:52:47.593957: done +2022-04-11 20:52:47.595707: This epoch took 125.961823 s + +2022-04-11 20:52:47.597520: svmem(total=99932160000, available=71456710656, percent=28.5, used=24210665472, free=26128424960, active=48660451328, inactive=22742700032, buffers=44814336, cached=49548255232, shared=2196156416, slab=1282957312) +2022-04-11 20:52:47.598539: +epoch: 119 +2022-04-11 20:54:36.484715: train loss : -0.4015 +2022-04-11 20:54:47.992453: validation loss: -0.4368 +2022-04-11 20:54:48.000375: Average global foreground Dice: [0.6933, 0.9194, 0.9408, 0.334] +2022-04-11 20:54:48.001694: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:54:49.017499: lr: 0.007811 +2022-04-11 20:54:49.018784: saving scheduled checkpoint file... +2022-04-11 20:54:49.124067: saving checkpoint... +2022-04-11 20:54:53.767345: done, saving took 4.75 seconds +2022-04-11 20:54:53.782513: done +2022-04-11 20:54:53.783253: This epoch took 126.183996 s + +2022-04-11 20:54:53.784100: svmem(total=99932160000, available=71335907328, percent=28.6, used=24311652352, free=26006814720, active=48761458688, inactive=22763483136, buffers=44916736, cached=49568776192, shared=2216144896, slab=1282416640) +2022-04-11 20:54:53.785199: +epoch: 120 +2022-04-11 20:56:41.287821: train loss : -0.4751 +2022-04-11 20:56:53.132255: validation loss: -0.5146 +2022-04-11 20:56:53.142299: Average global foreground Dice: [0.8041, 0.9433, 0.9581, 0.3817] +2022-04-11 20:56:53.148962: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:56:54.066800: lr: 0.007793 +2022-04-11 20:56:54.068898: saving scheduled checkpoint file... +2022-04-11 20:56:54.149039: saving checkpoint... +2022-04-11 20:56:58.561109: done, saving took 4.49 seconds +2022-04-11 20:56:58.629482: done +2022-04-11 20:56:58.631171: This epoch took 124.844769 s + +2022-04-11 20:56:58.632513: svmem(total=99932160000, available=71571595264, percent=28.4, used=24095842304, free=26241433600, active=48543232000, inactive=22744211456, buffers=44998656, cached=49549885440, shared=2196070400, slab=1282215936) +2022-04-11 20:56:58.633459: +epoch: 121 +2022-04-11 20:58:46.992417: train loss : -0.4874 +2022-04-11 20:58:59.076386: validation loss: -0.4784 +2022-04-11 20:58:59.078800: Average global foreground Dice: [0.7899, 0.9207, 0.9533, 0.3522] +2022-04-11 20:58:59.080055: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 20:59:00.274241: lr: 0.007774 +2022-04-11 20:59:00.398766: saving scheduled checkpoint file... +2022-04-11 20:59:00.689221: saving checkpoint... +2022-04-11 20:59:07.232520: done, saving took 6.65 seconds +2022-04-11 20:59:07.255284: done +2022-04-11 20:59:07.256330: This epoch took 128.621698 s + +2022-04-11 20:59:07.257886: svmem(total=99932160000, available=71616233472, percent=28.3, used=24038633472, free=26285420544, active=48531308544, inactive=22709989376, buffers=45088768, cached=49563017216, shared=2206736384, slab=1286574080) +2022-04-11 20:59:07.259023: +epoch: 122 +2022-04-11 21:00:55.841857: train loss : -0.4657 +2022-04-11 21:01:07.031559: validation loss: -0.4349 +2022-04-11 21:01:07.034293: Average global foreground Dice: [0.7743, 0.9136, 0.931, 0.339] +2022-04-11 21:01:07.035560: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:01:08.243424: lr: 0.007756 +2022-04-11 21:01:08.400146: saving scheduled checkpoint file... +2022-04-11 21:01:08.637685: saving checkpoint... +2022-04-11 21:01:14.838482: done, saving took 6.31 seconds +2022-04-11 21:01:14.862389: done +2022-04-11 21:01:14.863669: This epoch took 127.603828 s + +2022-04-11 21:01:14.865123: svmem(total=99932160000, available=71499472896, percent=28.5, used=24164933632, free=26167861248, active=48658038784, inactive=22701047808, buffers=45178880, cached=49554186240, shared=2197184512, slab=1285566464) +2022-04-11 21:01:14.866178: +epoch: 123 +2022-04-11 21:03:02.223758: train loss : -0.4590 +2022-04-11 21:03:14.656101: validation loss: -0.4717 +2022-04-11 21:03:14.792511: Average global foreground Dice: [0.7864, 0.9233, 0.9255, 0.3449] +2022-04-11 21:03:14.979487: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:03:15.948938: lr: 0.007737 +2022-04-11 21:03:16.091031: saving scheduled checkpoint file... +2022-04-11 21:03:16.333203: saving checkpoint... +2022-04-11 21:03:21.909596: done, saving took 5.69 seconds +2022-04-11 21:03:21.927078: done +2022-04-11 21:03:21.927954: This epoch took 127.060937 s + +2022-04-11 21:03:21.929106: svmem(total=99932160000, available=71531642880, percent=28.4, used=24133656576, free=26199011328, active=48619040768, inactive=22701182976, buffers=45244416, cached=49554247680, shared=2196328448, slab=1286119424) +2022-04-11 21:03:21.929971: +epoch: 124 +2022-04-11 21:05:10.647982: train loss : -0.4662 +2022-04-11 21:05:22.877161: validation loss: -0.4767 +2022-04-11 21:05:23.055545: Average global foreground Dice: [0.6783, 0.9369, 0.9524, 0.3644] +2022-04-11 21:05:23.188075: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:05:24.254387: lr: 0.007719 +2022-04-11 21:05:24.447563: saving scheduled checkpoint file... +2022-04-11 21:05:24.695569: saving checkpoint... +2022-04-11 21:05:29.907998: done, saving took 5.34 seconds +2022-04-11 21:05:29.925440: done +2022-04-11 21:05:29.926769: This epoch took 127.996088 s + +2022-04-11 21:05:29.927753: svmem(total=99932160000, available=71394762752, percent=28.6, used=24270561280, free=26061279232, active=48766218240, inactive=22701862912, buffers=45359104, cached=49554960384, shared=2196267008, slab=1286799360) +2022-04-11 21:05:29.928446: +epoch: 125 +2022-04-11 21:07:19.600377: train loss : -0.4409 +2022-04-11 21:07:31.133442: validation loss: -0.5393 +2022-04-11 21:07:31.292279: Average global foreground Dice: [0.7561, 0.9377, 0.9713, 0.4039] +2022-04-11 21:07:31.465396: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:07:32.565995: lr: 0.0077 +2022-04-11 21:07:32.699268: saving scheduled checkpoint file... +2022-04-11 21:07:32.912778: saving checkpoint... +2022-04-11 21:07:38.394992: done, saving took 5.57 seconds +2022-04-11 21:07:38.409825: done +2022-04-11 21:07:38.410987: This epoch took 128.481290 s + +2022-04-11 21:07:38.411848: svmem(total=99932160000, available=71313358848, percent=28.6, used=24352309248, free=25978974208, active=48847581184, inactive=22702817280, buffers=45428736, cached=49555447808, shared=2195881984, slab=1286610944) +2022-04-11 21:07:38.412556: +epoch: 126 +2022-04-11 21:09:27.365198: train loss : -0.4815 +2022-04-11 21:09:40.319370: validation loss: -0.5372 +2022-04-11 21:09:40.420922: Average global foreground Dice: [0.769, 0.9362, 0.9714, 0.3772] +2022-04-11 21:09:40.423044: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:09:41.079147: lr: 0.007682 +2022-04-11 21:09:41.082593: saving scheduled checkpoint file... +2022-04-11 21:09:41.172300: saving checkpoint... +2022-04-11 21:09:45.806231: done, saving took 4.72 seconds +2022-04-11 21:09:45.820878: done +2022-04-11 21:09:45.821676: This epoch took 127.408471 s + +2022-04-11 21:09:45.823067: svmem(total=99932160000, available=71511150592, percent=28.4, used=24154288128, free=26176126976, active=48648736768, inactive=22702964736, buffers=45477888, cached=49556267008, shared=2196111360, slab=1286778880) +2022-04-11 21:09:45.823812: +epoch: 127 +2022-04-11 21:11:32.750351: train loss : -0.4850 +2022-04-11 21:11:43.894622: validation loss: -0.5552 +2022-04-11 21:11:44.053045: Average global foreground Dice: [0.7942, 0.9452, 0.9703, 0.3618] +2022-04-11 21:11:44.195133: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:11:45.252296: lr: 0.007663 +2022-04-11 21:11:45.395090: saving scheduled checkpoint file... +2022-04-11 21:11:45.586365: saving checkpoint... +2022-04-11 21:11:50.246766: done, saving took 4.73 seconds +2022-04-11 21:11:50.276145: done +2022-04-11 21:11:50.324639: saving checkpoint... +2022-04-11 21:11:54.820816: done, saving took 4.54 seconds +2022-04-11 21:11:54.839593: This epoch took 129.015077 s + +2022-04-11 21:11:54.842461: svmem(total=99932160000, available=71378886656, percent=28.6, used=24286212096, free=26043035648, active=48778006528, inactive=22704115712, buffers=45547520, cached=49557364736, shared=2196455424, slab=1286361088) +2022-04-11 21:11:54.845645: +epoch: 128 +2022-04-11 21:13:42.259369: train loss : -0.4964 +2022-04-11 21:13:53.818973: validation loss: -0.4855 +2022-04-11 21:13:53.824656: Average global foreground Dice: [0.7711, 0.9386, 0.9568, 0.3144] +2022-04-11 21:13:53.827243: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:13:54.695991: lr: 0.007645 +2022-04-11 21:13:54.697974: saving scheduled checkpoint file... +2022-04-11 21:13:54.794477: saving checkpoint... +2022-04-11 21:13:59.432817: done, saving took 4.73 seconds +2022-04-11 21:13:59.446023: done +2022-04-11 21:13:59.447429: This epoch took 124.599167 s + +2022-04-11 21:13:59.448343: svmem(total=99932160000, available=71511035904, percent=28.4, used=24153718784, free=26175647744, active=48859136000, inactive=22492766208, buffers=45625344, cached=49557168128, shared=2196738048, slab=1286066176) +2022-04-11 21:13:59.449072: +epoch: 129 +2022-04-11 21:15:47.245525: train loss : -0.5096 +2022-04-11 21:15:59.993968: validation loss: -0.4480 +2022-04-11 21:15:59.996670: Average global foreground Dice: [0.7964, 0.9227, 0.9262, 0.3383] +2022-04-11 21:15:59.998151: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:16:00.862623: lr: 0.007626 +2022-04-11 21:16:00.864132: saving scheduled checkpoint file... +2022-04-11 21:16:00.945404: saving checkpoint... +2022-04-11 21:16:05.575185: done, saving took 4.71 seconds +2022-04-11 21:16:05.621312: done +2022-04-11 21:16:05.622481: This epoch took 126.172658 s + +2022-04-11 21:16:05.623847: svmem(total=99932160000, available=71464947712, percent=28.5, used=24200675328, free=26129338368, active=48919998464, inactive=22476918784, buffers=45723648, cached=49556422656, shared=2195763200, slab=1286262784) +2022-04-11 21:16:05.625136: +epoch: 130 +2022-04-11 21:17:54.145343: train loss : -0.5023 +2022-04-11 21:18:06.232425: validation loss: -0.5198 +2022-04-11 21:18:06.234556: Average global foreground Dice: [0.8007, 0.9319, 0.9557, 0.3777] +2022-04-11 21:18:06.235259: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:18:07.019012: lr: 0.007608 +2022-04-11 21:18:07.020521: saving scheduled checkpoint file... +2022-04-11 21:18:07.151726: saving checkpoint... +2022-04-11 21:18:11.733579: done, saving took 4.71 seconds +2022-04-11 21:18:11.821954: done +2022-04-11 21:18:11.919008: saving checkpoint... +2022-04-11 21:18:16.778406: done, saving took 4.96 seconds +2022-04-11 21:18:16.811508: This epoch took 131.185179 s + +2022-04-11 21:18:16.812951: svmem(total=99932160000, available=71431675904, percent=28.5, used=24233230336, free=26098274304, active=49039794176, inactive=22389186560, buffers=45805568, cached=49554849792, shared=2196471808, slab=1286180864) +2022-04-11 21:18:16.813864: +epoch: 131 +2022-04-11 21:20:05.181855: train loss : -0.4969 +2022-04-11 21:20:16.542033: validation loss: -0.5234 +2022-04-11 21:20:16.550102: Average global foreground Dice: [0.7882, 0.9441, 0.9612, 0.3551] +2022-04-11 21:20:16.551448: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:20:17.577571: lr: 0.007589 +2022-04-11 21:20:17.579404: saving scheduled checkpoint file... +2022-04-11 21:20:17.687671: saving checkpoint... +2022-04-11 21:20:22.580390: done, saving took 5.00 seconds +2022-04-11 21:20:22.608492: done +2022-04-11 21:20:22.658909: saving checkpoint... +2022-04-11 21:20:27.479501: done, saving took 4.87 seconds +2022-04-11 21:20:27.529412: This epoch took 130.714831 s + +2022-04-11 21:20:27.530373: svmem(total=99932160000, available=71391977472, percent=28.6, used=24265945088, free=26057924608, active=49128218624, inactive=22342909952, buffers=45907968, cached=49562382336, shared=2203504640, slab=1287905280) +2022-04-11 21:20:27.531064: +epoch: 132 +2022-04-11 21:22:17.879557: train loss : -0.5239 +2022-04-11 21:22:29.466589: validation loss: -0.5611 +2022-04-11 21:22:29.511882: Average global foreground Dice: [0.7801, 0.9464, 0.9747, 0.3665] +2022-04-11 21:22:29.513153: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:22:30.370632: lr: 0.007571 +2022-04-11 21:22:30.372011: saving scheduled checkpoint file... +2022-04-11 21:22:30.463858: saving checkpoint... +2022-04-11 21:22:34.957999: done, saving took 4.58 seconds +2022-04-11 21:22:34.978854: done +2022-04-11 21:22:35.050335: saving checkpoint... +2022-04-11 21:22:39.639724: done, saving took 4.66 seconds +2022-04-11 21:22:39.656218: This epoch took 132.124532 s + +2022-04-11 21:22:39.657759: svmem(total=99932160000, available=71632003072, percent=28.3, used=24027828224, free=26388783104, active=48982261760, inactive=22160855040, buffers=46010368, cached=49469538304, shared=2201214976, slab=1288331264) +2022-04-11 21:22:39.658879: +epoch: 133 +2022-04-11 21:24:29.342956: train loss : -0.4791 +2022-04-11 21:24:41.892823: validation loss: -0.4695 +2022-04-11 21:24:41.896582: Average global foreground Dice: [0.773, 0.9346, 0.9356, 0.3427] +2022-04-11 21:24:41.898515: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:24:42.634391: lr: 0.007552 +2022-04-11 21:24:42.637274: saving scheduled checkpoint file... +2022-04-11 21:24:42.732162: saving checkpoint... +2022-04-11 21:24:47.140540: done, saving took 4.50 seconds +2022-04-11 21:24:47.161583: done +2022-04-11 21:24:47.164625: This epoch took 127.504980 s + +2022-04-11 21:24:47.167431: svmem(total=99932160000, available=71798239232, percent=28.2, used=23866449920, free=26554032128, active=48829014016, inactive=22145368064, buffers=46084096, cached=49465593856, shared=2196320256, slab=1287692288) +2022-04-11 21:24:47.169605: +epoch: 134 +2022-04-11 21:26:35.064553: train loss : -0.4984 +2022-04-11 21:26:46.364128: validation loss: -0.5079 +2022-04-11 21:26:46.370568: Average global foreground Dice: [0.8039, 0.9388, 0.9533, 0.3654] +2022-04-11 21:26:46.372458: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:26:47.277702: lr: 0.007533 +2022-04-11 21:26:47.280540: saving scheduled checkpoint file... +2022-04-11 21:26:47.413722: saving checkpoint... +2022-04-11 21:26:51.898875: done, saving took 4.62 seconds +2022-04-11 21:26:51.919075: done +2022-04-11 21:26:52.015148: saving checkpoint... +2022-04-11 21:26:56.758585: done, saving took 4.84 seconds +2022-04-11 21:26:56.818812: This epoch took 129.646679 s + +2022-04-11 21:26:56.820627: svmem(total=99932160000, available=71752376320, percent=28.2, used=23912292352, free=26507386880, active=48873947136, inactive=22145609728, buffers=46178304, cached=49466302464, shared=2196369408, slab=1288388608) +2022-04-11 21:26:56.821478: +epoch: 135 +2022-04-11 21:28:44.237480: train loss : -0.4996 +2022-04-11 21:28:56.398791: validation loss: -0.5151 +2022-04-11 21:28:56.403921: Average global foreground Dice: [0.8109, 0.9338, 0.945, 0.3872] +2022-04-11 21:28:56.405203: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:28:57.374783: lr: 0.007515 +2022-04-11 21:28:57.376008: saving scheduled checkpoint file... +2022-04-11 21:28:57.517851: saving checkpoint... +2022-04-11 21:29:01.979076: done, saving took 4.60 seconds +2022-04-11 21:29:02.000587: done +2022-04-11 21:29:02.116566: saving checkpoint... +2022-04-11 21:29:06.635186: done, saving took 4.63 seconds +2022-04-11 21:29:06.665105: This epoch took 129.842874 s + +2022-04-11 21:29:06.666623: svmem(total=99932160000, available=71698534400, percent=28.3, used=23955628032, free=26452705280, active=48914694144, inactive=22156804096, buffers=46235648, cached=49477591040, shared=2206838784, slab=1287839744) +2022-04-11 21:29:06.667439: +epoch: 136 +2022-04-11 21:30:55.501871: train loss : -0.4511 +2022-04-11 21:31:07.527912: validation loss: -0.4620 +2022-04-11 21:31:07.530916: Average global foreground Dice: [0.7468, 0.9225, 0.9412, 0.3591] +2022-04-11 21:31:07.532095: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:31:08.065747: lr: 0.007496 +2022-04-11 21:31:08.066991: saving scheduled checkpoint file... +2022-04-11 21:31:08.187782: saving checkpoint... +2022-04-11 21:31:12.663491: done, saving took 4.60 seconds +2022-04-11 21:31:12.681321: done +2022-04-11 21:31:12.682347: This epoch took 126.014193 s + +2022-04-11 21:31:12.683458: svmem(total=99932160000, available=71610683392, percent=28.3, used=24054013952, free=26364092416, active=49015984128, inactive=22146691072, buffers=46309376, cached=49467744256, shared=2196099072, slab=1289084928) +2022-04-11 21:31:12.684082: +epoch: 137 +2022-04-11 21:33:01.434668: train loss : -0.4769 +2022-04-11 21:33:14.166875: validation loss: -0.4970 +2022-04-11 21:33:14.169148: Average global foreground Dice: [0.7678, 0.9344, 0.9559, 0.3673] +2022-04-11 21:33:14.169969: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:33:14.888841: lr: 0.007478 +2022-04-11 21:33:14.889993: saving scheduled checkpoint file... +2022-04-11 21:33:14.980451: saving checkpoint... +2022-04-11 21:33:19.437632: done, saving took 4.55 seconds +2022-04-11 21:33:19.489821: done +2022-04-11 21:33:19.491001: This epoch took 126.806283 s + +2022-04-11 21:33:19.492095: svmem(total=99932160000, available=71452770304, percent=28.5, used=24204853248, free=26205102080, active=49171648512, inactive=22154883072, buffers=46374912, cached=49475829760, shared=2203267072, slab=1288269824) +2022-04-11 21:33:19.492751: +epoch: 138 +2022-04-11 21:35:07.325093: train loss : -0.4766 +2022-04-11 21:35:19.049758: validation loss: -0.5516 +2022-04-11 21:35:19.059617: Average global foreground Dice: [0.8228, 0.9397, 0.9717, 0.3794] +2022-04-11 21:35:19.060596: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:35:20.044229: lr: 0.007459 +2022-04-11 21:35:20.047770: saving scheduled checkpoint file... +2022-04-11 21:35:20.122822: saving checkpoint... +2022-04-11 21:35:24.874203: done, saving took 4.82 seconds +2022-04-11 21:35:24.889620: done +2022-04-11 21:35:24.938423: saving checkpoint... +2022-04-11 21:35:29.554469: done, saving took 4.66 seconds +2022-04-11 21:35:29.625782: This epoch took 130.132370 s + +2022-04-11 21:35:29.627531: svmem(total=99932160000, available=71293382656, percent=28.7, used=24378368000, free=26045104128, active=49340309504, inactive=22141288448, buffers=46497792, cached=49462190080, shared=2189148160, slab=1289285632) +2022-04-11 21:35:29.628834: +epoch: 139 +2022-04-11 21:37:17.708292: train loss : -0.5380 +2022-04-11 21:37:29.443242: validation loss: -0.5223 +2022-04-11 21:37:29.451026: Average global foreground Dice: [0.798, 0.939, 0.9544, 0.3741] +2022-04-11 21:37:29.454705: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:37:30.431288: lr: 0.00744 +2022-04-11 21:37:30.432956: saving scheduled checkpoint file... +2022-04-11 21:37:30.517812: saving checkpoint... +2022-04-11 21:37:35.334579: done, saving took 4.90 seconds +2022-04-11 21:37:35.411736: done +2022-04-11 21:37:35.482349: saving checkpoint... +2022-04-11 21:37:40.700676: done, saving took 5.29 seconds +2022-04-11 21:37:40.784846: This epoch took 131.154781 s + +2022-04-11 21:37:40.786827: svmem(total=99932160000, available=71580549120, percent=28.4, used=24061419520, free=26352570368, active=48774414336, inactive=22396588032, buffers=46546944, cached=49471623168, shared=2218930176, slab=1287618560) +2022-04-11 21:37:40.788083: +epoch: 140 +2022-04-11 21:39:29.352212: train loss : -0.5298 +2022-04-11 21:39:40.345420: validation loss: -0.5450 +2022-04-11 21:39:40.360572: Average global foreground Dice: [0.782, 0.9362, 0.9639, 0.3925] +2022-04-11 21:39:40.361684: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:39:41.264640: lr: 0.007422 +2022-04-11 21:39:41.266166: saving scheduled checkpoint file... +2022-04-11 21:39:41.343868: saving checkpoint... +2022-04-11 21:39:45.788997: done, saving took 4.52 seconds +2022-04-11 21:39:45.807258: done +2022-04-11 21:39:45.891550: saving checkpoint... +2022-04-11 21:39:50.421890: done, saving took 4.61 seconds +2022-04-11 21:39:50.442554: This epoch took 129.653209 s + +2022-04-11 21:39:50.444310: svmem(total=99932160000, available=71546351616, percent=28.4, used=24117596160, free=26317778944, active=49084755968, inactive=22121816064, buffers=46620672, cached=49450164224, shared=2196910080, slab=1287835648) +2022-04-11 21:39:50.445528: +epoch: 141 +2022-04-11 21:41:37.377149: train loss : -0.5379 +2022-04-11 21:41:48.748932: validation loss: -0.5753 +2022-04-11 21:41:48.751079: Average global foreground Dice: [0.8197, 0.9414, 0.9718, 0.3745] +2022-04-11 21:41:48.752216: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:41:49.575795: lr: 0.007403 +2022-04-11 21:41:49.577451: saving scheduled checkpoint file... +2022-04-11 21:41:49.708031: saving checkpoint... +2022-04-11 21:41:54.328340: done, saving took 4.75 seconds +2022-04-11 21:41:54.349447: done +2022-04-11 21:41:54.445624: saving checkpoint... +2022-04-11 21:41:59.131706: done, saving took 4.78 seconds +2022-04-11 21:41:59.152629: This epoch took 128.706229 s + +2022-04-11 21:41:59.154182: svmem(total=99932160000, available=71558578176, percent=28.4, used=24091774976, free=26329096192, active=49098534912, inactive=22098235392, buffers=46694400, cached=49464594432, shared=2210435072, slab=1288597504) +2022-04-11 21:41:59.154983: +epoch: 142 +2022-04-11 21:43:46.992702: train loss : -0.5534 +2022-04-11 21:43:58.517536: validation loss: -0.4904 +2022-04-11 21:43:58.526312: Average global foreground Dice: [0.835, 0.9269, 0.9496, 0.3737] +2022-04-11 21:43:58.528214: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:43:59.333520: lr: 0.007385 +2022-04-11 21:43:59.334999: saving scheduled checkpoint file... +2022-04-11 21:43:59.483252: saving checkpoint... +2022-04-11 21:44:04.070083: done, saving took 4.73 seconds +2022-04-11 21:44:04.109663: done +2022-04-11 21:44:04.223714: saving checkpoint... +2022-04-11 21:44:08.893916: done, saving took 4.78 seconds +2022-04-11 21:44:08.951298: This epoch took 129.795585 s + +2022-04-11 21:44:08.952381: svmem(total=99932160000, available=71383539712, percent=28.6, used=24280653824, free=26153267200, active=49283887104, inactive=22083854336, buffers=46759936, cached=49451479040, shared=2196713472, slab=1287409664) +2022-04-11 21:44:08.953196: +epoch: 143 +2022-04-11 21:45:59.132953: train loss : -0.5448 +2022-04-11 21:46:11.441608: validation loss: -0.5441 +2022-04-11 21:46:11.448062: Average global foreground Dice: [0.7957, 0.9406, 0.965, 0.378] +2022-04-11 21:46:11.449155: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:46:12.156277: lr: 0.007366 +2022-04-11 21:46:12.159342: saving scheduled checkpoint file... +2022-04-11 21:46:12.253706: saving checkpoint... +2022-04-11 21:46:16.895257: done, saving took 4.73 seconds +2022-04-11 21:46:16.913381: done +2022-04-11 21:46:16.963897: saving checkpoint... +2022-04-11 21:46:21.609983: done, saving took 4.69 seconds +2022-04-11 21:46:21.624206: This epoch took 132.670223 s + +2022-04-11 21:46:21.625503: svmem(total=99932160000, available=71477866496, percent=28.5, used=24182775808, free=26226278400, active=49234345984, inactive=22064758784, buffers=46878720, cached=49476227072, shared=2199371776, slab=1290129408) +2022-04-11 21:46:21.626193: +epoch: 144 +2022-04-11 21:48:09.982239: train loss : -0.5573 +2022-04-11 21:48:22.888192: validation loss: -0.5095 +2022-04-11 21:48:22.893513: Average global foreground Dice: [0.8274, 0.9332, 0.9469, 0.3754] +2022-04-11 21:48:22.895336: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:48:23.754428: lr: 0.007347 +2022-04-11 21:48:23.756155: saving scheduled checkpoint file... +2022-04-11 21:48:23.835511: saving checkpoint... +2022-04-11 21:48:28.355756: done, saving took 4.60 seconds +2022-04-11 21:48:28.368728: done +2022-04-11 21:48:28.417004: saving checkpoint... +2022-04-11 21:48:32.830277: done, saving took 4.46 seconds +2022-04-11 21:48:32.853035: This epoch took 131.226186 s + +2022-04-11 21:48:32.854249: svmem(total=99932160000, available=71532339200, percent=28.4, used=24123666432, free=26279936000, active=49189666816, inactive=22054354944, buffers=46960640, cached=49481596928, shared=2203881472, slab=1290604544) +2022-04-11 21:48:32.854935: +epoch: 145 +2022-04-11 21:50:21.443520: train loss : -0.5240 +2022-04-11 21:50:33.344244: validation loss: -0.5648 +2022-04-11 21:50:33.353851: Average global foreground Dice: [0.8032, 0.9434, 0.9739, 0.3825] +2022-04-11 21:50:33.355305: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:50:34.152976: lr: 0.007329 +2022-04-11 21:50:34.154241: saving scheduled checkpoint file... +2022-04-11 21:50:34.236171: saving checkpoint... +2022-04-11 21:50:38.791460: done, saving took 4.64 seconds +2022-04-11 21:50:38.823127: done +2022-04-11 21:50:38.874311: saving checkpoint... +2022-04-11 21:50:43.633116: done, saving took 4.81 seconds +2022-04-11 21:50:43.690636: This epoch took 130.834980 s + +2022-04-11 21:50:43.692180: svmem(total=99932160000, available=71470481408, percent=28.5, used=24182800384, free=26217152512, active=49250566144, inactive=22059040768, buffers=47034368, cached=49485172736, shared=2206707712, slab=1289977856) +2022-04-11 21:50:43.692963: +epoch: 146 +2022-04-11 21:52:31.567680: train loss : -0.4945 +2022-04-11 21:52:42.928053: validation loss: -0.4901 +2022-04-11 21:52:42.929937: Average global foreground Dice: [0.7816, 0.9328, 0.9594, 0.3471] +2022-04-11 21:52:42.930702: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:52:43.761930: lr: 0.00731 +2022-04-11 21:52:43.763922: saving scheduled checkpoint file... +2022-04-11 21:52:43.858194: saving checkpoint... +2022-04-11 21:52:48.539933: done, saving took 4.77 seconds +2022-04-11 21:52:48.562629: done +2022-04-11 21:52:48.563637: This epoch took 124.869324 s + +2022-04-11 21:52:48.564652: svmem(total=99932160000, available=71546609664, percent=28.4, used=24117067776, free=26292432896, active=49184722944, inactive=22049366016, buffers=47112192, cached=49475547136, shared=2196258816, slab=1288503296) +2022-04-11 21:52:48.566282: +epoch: 147 +2022-04-11 21:54:37.298382: train loss : -0.4170 +2022-04-11 21:54:49.116320: validation loss: -0.4737 +2022-04-11 21:54:49.127340: Average global foreground Dice: [0.8044, 0.9307, 0.9466, 0.3544] +2022-04-11 21:54:49.128776: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:54:49.959997: lr: 0.007291 +2022-04-11 21:54:49.962004: saving scheduled checkpoint file... +2022-04-11 21:54:50.076908: saving checkpoint... +2022-04-11 21:54:54.499018: done, saving took 4.54 seconds +2022-04-11 21:54:54.516545: done +2022-04-11 21:54:54.517787: This epoch took 125.950165 s + +2022-04-11 21:54:54.519185: svmem(total=99932160000, available=71687561216, percent=28.3, used=23989891072, free=26432524288, active=49058275328, inactive=22036303872, buffers=47202304, cached=49462542336, shared=2182561792, slab=1289605120) +2022-04-11 21:54:54.520137: +epoch: 148 +2022-04-11 21:56:43.446766: train loss : -0.3695 +2022-04-11 21:56:55.625146: validation loss: -0.4985 +2022-04-11 21:56:55.628534: Average global foreground Dice: [0.7434, 0.9253, 0.9633, 0.3431] +2022-04-11 21:56:55.629416: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:56:56.430835: lr: 0.007273 +2022-04-11 21:56:56.432121: saving scheduled checkpoint file... +2022-04-11 21:56:56.543640: saving checkpoint... +2022-04-11 21:57:01.154055: done, saving took 4.72 seconds +2022-04-11 21:57:01.175370: done +2022-04-11 21:57:01.176642: This epoch took 126.655672 s + +2022-04-11 21:57:01.177426: svmem(total=99932160000, available=71488114688, percent=28.5, used=24175484928, free=26232193024, active=49243906048, inactive=22051311616, buffers=47284224, cached=49477197824, shared=2196234240, slab=1291120640) +2022-04-11 21:57:01.177957: +epoch: 149 +2022-04-11 21:58:50.670374: train loss : -0.5013 +2022-04-11 21:59:02.701236: validation loss: -0.4815 +2022-04-11 21:59:02.708304: Average global foreground Dice: [0.7991, 0.9406, 0.9404, 0.3504] +2022-04-11 21:59:02.710111: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 21:59:03.720349: lr: 0.007254 +2022-04-11 21:59:03.721566: saving scheduled checkpoint file... +2022-04-11 21:59:03.829562: saving checkpoint... +2022-04-11 21:59:09.131998: done, saving took 5.41 seconds +2022-04-11 21:59:09.234113: done +2022-04-11 21:59:09.243155: This epoch took 128.064640 s + +2022-04-11 21:59:09.254190: svmem(total=99932160000, available=71483928576, percent=28.5, used=24179523584, free=26227224576, active=49241735168, inactive=22051565568, buffers=47349760, cached=49478062080, shared=2196320256, slab=1290809344) +2022-04-11 21:59:09.257431: +epoch: 150 +2022-04-11 22:00:56.247394: train loss : -0.4638 +2022-04-11 22:01:08.089420: validation loss: -0.5045 +2022-04-11 22:01:08.091786: Average global foreground Dice: [0.7727, 0.9351, 0.9489, 0.3626] +2022-04-11 22:01:08.092624: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:01:08.668325: lr: 0.007236 +2022-04-11 22:01:08.669523: saving scheduled checkpoint file... +2022-04-11 22:01:08.735621: saving checkpoint... +2022-04-11 22:01:13.446151: done, saving took 4.78 seconds +2022-04-11 22:01:13.461756: done +2022-04-11 22:01:13.463206: This epoch took 124.196459 s + +2022-04-11 22:01:13.464312: svmem(total=99932160000, available=71486291968, percent=28.5, used=24176807936, free=26228670464, active=49243336704, inactive=22052777984, buffers=47427584, cached=49479254016, shared=2196738048, slab=1290407936) +2022-04-11 22:01:13.465172: +epoch: 151 +2022-04-11 22:03:02.284807: train loss : -0.5039 +2022-04-11 22:03:14.181590: validation loss: -0.5410 +2022-04-11 22:03:14.184661: Average global foreground Dice: [0.7715, 0.9399, 0.9671, 0.3783] +2022-04-11 22:03:14.185691: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:03:14.896372: lr: 0.007217 +2022-04-11 22:03:14.897779: saving scheduled checkpoint file... +2022-04-11 22:03:14.959122: saving checkpoint... +2022-04-11 22:03:19.185854: done, saving took 4.29 seconds +2022-04-11 22:03:19.212459: done +2022-04-11 22:03:19.213859: This epoch took 125.747754 s + +2022-04-11 22:03:19.215131: svmem(total=99932160000, available=71572492288, percent=28.4, used=24087433216, free=26314035200, active=49150308352, inactive=22056960000, buffers=47489024, cached=49483202560, shared=2199814144, slab=1291001856) +2022-04-11 22:03:19.215961: +epoch: 152 +2022-04-11 22:05:06.616492: train loss : -0.5085 +2022-04-11 22:05:18.055840: validation loss: -0.5363 +2022-04-11 22:05:18.058069: Average global foreground Dice: [0.7802, 0.9429, 0.9677, 0.3672] +2022-04-11 22:05:18.059295: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:05:19.005196: lr: 0.007198 +2022-04-11 22:05:19.007013: saving scheduled checkpoint file... +2022-04-11 22:05:19.074647: saving checkpoint... +2022-04-11 22:05:23.689963: done, saving took 4.68 seconds +2022-04-11 22:05:23.705325: done +2022-04-11 22:05:23.706351: This epoch took 124.489238 s + +2022-04-11 22:05:23.707678: svmem(total=99932160000, available=71476051968, percent=28.5, used=24187203584, free=26216820736, active=49247436800, inactive=22053785600, buffers=47558656, cached=49480577024, shared=2196422656, slab=1290838016) +2022-04-11 22:05:23.708699: +epoch: 153 +2022-04-11 22:07:11.383161: train loss : -0.5323 +2022-04-11 22:07:23.632058: validation loss: -0.5594 +2022-04-11 22:07:23.642469: Average global foreground Dice: [0.8144, 0.9507, 0.9664, 0.354] +2022-04-11 22:07:23.643521: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:07:24.210372: lr: 0.00718 +2022-04-11 22:07:24.211588: saving scheduled checkpoint file... +2022-04-11 22:07:24.315306: saving checkpoint... +2022-04-11 22:07:29.078520: done, saving took 4.87 seconds +2022-04-11 22:07:29.122433: done +2022-04-11 22:07:29.123613: This epoch took 125.414114 s + +2022-04-11 22:07:29.125195: svmem(total=99932160000, available=71425978368, percent=28.5, used=24233402368, free=26165907456, active=49296551936, inactive=22058528768, buffers=47656960, cached=49485193216, shared=2200354816, slab=1290653696) +2022-04-11 22:07:29.126170: +epoch: 154 +2022-04-11 22:09:16.727462: train loss : -0.5340 +2022-04-11 22:09:27.247009: validation loss: -0.5501 +2022-04-11 22:09:27.264090: Average global foreground Dice: [0.8199, 0.9398, 0.9666, 0.3803] +2022-04-11 22:09:27.266822: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:09:28.154733: lr: 0.007161 +2022-04-11 22:09:28.156135: saving scheduled checkpoint file... +2022-04-11 22:09:28.278435: saving checkpoint... +2022-04-11 22:09:34.446382: done, saving took 6.29 seconds +2022-04-11 22:09:34.461044: done +2022-04-11 22:09:34.462482: This epoch took 125.335587 s + +2022-04-11 22:09:34.463723: svmem(total=99932160000, available=71604498432, percent=28.3, used=24058531840, free=26346090496, active=49139032064, inactive=22037544960, buffers=47751168, cached=49479786496, shared=2195587072, slab=1292447744) +2022-04-11 22:09:34.464449: +epoch: 155 +2022-04-11 22:11:22.599652: train loss : -0.5354 +2022-04-11 22:11:33.902853: validation loss: -0.5925 +2022-04-11 22:11:33.905080: Average global foreground Dice: [0.7993, 0.9575, 0.9736, 0.4016] +2022-04-11 22:11:33.905893: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:11:34.897671: lr: 0.007142 +2022-04-11 22:11:34.898782: saving scheduled checkpoint file... +2022-04-11 22:11:34.991482: saving checkpoint... +2022-04-11 22:11:39.565735: done, saving took 4.67 seconds +2022-04-11 22:11:39.627152: done +2022-04-11 22:11:39.675477: saving checkpoint... +2022-04-11 22:11:44.414682: done, saving took 4.79 seconds +2022-04-11 22:11:44.429848: This epoch took 129.964844 s + +2022-04-11 22:11:44.431425: svmem(total=99932160000, available=71487897600, percent=28.5, used=24183697408, free=26228555776, active=49267720192, inactive=22029393920, buffers=47824896, cached=49472081920, shared=2186977280, slab=1293406208) +2022-04-11 22:11:44.432554: +epoch: 156 +2022-04-11 22:13:32.364223: train loss : -0.5324 +2022-04-11 22:13:44.192539: validation loss: -0.5368 +2022-04-11 22:13:44.200830: Average global foreground Dice: [0.8212, 0.9493, 0.9561, 0.3856] +2022-04-11 22:13:44.201999: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:13:45.081499: lr: 0.007123 +2022-04-11 22:13:45.083142: saving scheduled checkpoint file... +2022-04-11 22:13:45.136341: saving checkpoint... +2022-04-11 22:13:49.692194: done, saving took 4.61 seconds +2022-04-11 22:13:49.708695: done +2022-04-11 22:13:49.776094: saving checkpoint... +2022-04-11 22:13:54.461599: done, saving took 4.75 seconds +2022-04-11 22:13:54.519687: This epoch took 130.086460 s + +2022-04-11 22:13:54.521115: svmem(total=99932160000, available=71360843776, percent=28.6, used=24287416320, free=26100862976, active=49365409792, inactive=22053834752, buffers=47898624, cached=49495982080, shared=2210361344, slab=1292783616) +2022-04-11 22:13:54.522061: +epoch: 157 +2022-04-11 22:15:43.044291: train loss : -0.5427 +2022-04-11 22:15:54.881791: validation loss: -0.5038 +2022-04-11 22:15:54.883901: Average global foreground Dice: [0.835, 0.936, 0.9594, 0.3553] +2022-04-11 22:15:54.885333: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:15:55.730006: lr: 0.007105 +2022-04-11 22:15:55.731007: saving scheduled checkpoint file... +2022-04-11 22:15:55.824601: saving checkpoint... +2022-04-11 22:16:00.670074: done, saving took 4.94 seconds +2022-04-11 22:16:00.691229: done +2022-04-11 22:16:00.773984: saving checkpoint... +2022-04-11 22:16:05.813862: done, saving took 5.12 seconds +2022-04-11 22:16:05.877878: This epoch took 131.355107 s + +2022-04-11 22:16:05.879752: svmem(total=99932160000, available=71459540992, percent=28.5, used=24202690560, free=26198429696, active=49280282624, inactive=22040162304, buffers=47996928, cached=49483042816, shared=2196267008, slab=1293004800) +2022-04-11 22:16:05.880906: +epoch: 158 +2022-04-11 22:17:54.245060: train loss : -0.5326 +2022-04-11 22:18:05.947041: validation loss: -0.5299 +2022-04-11 22:18:05.949292: Average global foreground Dice: [0.8117, 0.9385, 0.9549, 0.3833] +2022-04-11 22:18:05.949960: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:18:06.917882: lr: 0.007086 +2022-04-11 22:18:06.920587: saving scheduled checkpoint file... +2022-04-11 22:18:07.035402: saving checkpoint... +2022-04-11 22:18:11.596361: done, saving took 4.67 seconds +2022-04-11 22:18:11.611919: done +2022-04-11 22:18:11.715151: saving checkpoint... +2022-04-11 22:18:16.272111: done, saving took 4.66 seconds +2022-04-11 22:18:16.292368: This epoch took 130.410745 s + +2022-04-11 22:18:16.293583: svmem(total=99932160000, available=71547817984, percent=28.4, used=24107720704, free=26285969408, active=49182924800, inactive=22047981568, buffers=48082944, cached=49490386944, shared=2202943488, slab=1293647872) +2022-04-11 22:18:16.294391: +epoch: 159 +2022-04-11 22:20:04.721882: train loss : -0.4805 +2022-04-11 22:20:17.069196: validation loss: -0.5337 +2022-04-11 22:20:17.083225: Average global foreground Dice: [0.8043, 0.9474, 0.9633, 0.3467] +2022-04-11 22:20:17.085293: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:20:17.734856: lr: 0.007067 +2022-04-11 22:20:17.736343: saving scheduled checkpoint file... +2022-04-11 22:20:17.856798: saving checkpoint... +2022-04-11 22:20:22.681116: done, saving took 4.94 seconds +2022-04-11 22:20:22.728019: done +2022-04-11 22:20:22.729195: This epoch took 126.433914 s + +2022-04-11 22:20:22.730326: svmem(total=99932160000, available=71520071680, percent=28.4, used=24142569472, free=26257244160, active=49222901760, inactive=22041456640, buffers=48168960, cached=49484177408, shared=2195898368, slab=1293004800) +2022-04-11 22:20:22.731502: +epoch: 160 +2022-04-11 22:22:10.646039: train loss : -0.5389 +2022-04-11 22:22:23.393131: validation loss: -0.5768 +2022-04-11 22:22:23.404967: Average global foreground Dice: [0.8205, 0.9432, 0.9686, 0.3902] +2022-04-11 22:22:23.406182: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:22:24.170622: lr: 0.007049 +2022-04-11 22:22:24.172213: saving scheduled checkpoint file... +2022-04-11 22:22:24.291790: saving checkpoint... +2022-04-11 22:22:28.679869: done, saving took 4.51 seconds +2022-04-11 22:22:28.702863: done +2022-04-11 22:22:28.814642: saving checkpoint... +2022-04-11 22:22:33.432156: done, saving took 4.73 seconds +2022-04-11 22:22:33.455698: This epoch took 130.723079 s + +2022-04-11 22:22:33.457533: svmem(total=99932160000, available=71408599040, percent=28.5, used=24267935744, free=26144907264, active=49350348800, inactive=22028054528, buffers=48250880, cached=49471066112, shared=2181869568, slab=1293549568) +2022-04-11 22:22:33.458967: +epoch: 161 +2022-04-11 22:24:21.531599: train loss : -0.5509 +2022-04-11 22:24:33.669461: validation loss: -0.5426 +2022-04-11 22:24:33.675509: Average global foreground Dice: [0.7854, 0.9486, 0.9672, 0.3885] +2022-04-11 22:24:33.676891: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:24:34.421399: lr: 0.00703 +2022-04-11 22:24:34.422810: saving scheduled checkpoint file... +2022-04-11 22:24:34.512901: saving checkpoint... +2022-04-11 22:24:39.001664: done, saving took 4.58 seconds +2022-04-11 22:24:39.011257: done +2022-04-11 22:24:39.058682: saving checkpoint... +2022-04-11 22:24:43.398909: done, saving took 4.39 seconds +2022-04-11 22:24:43.447389: This epoch took 129.987434 s + +2022-04-11 22:24:43.449064: svmem(total=99932160000, available=71517655040, percent=28.4, used=24149893120, free=26253168640, active=49228300288, inactive=22037778432, buffers=48324608, cached=49480773632, shared=2190876672, slab=1291415552) +2022-04-11 22:24:43.449823: +epoch: 162 +2022-04-11 22:26:30.137000: train loss : -0.5588 +2022-04-11 22:26:42.392670: validation loss: -0.5865 +2022-04-11 22:26:42.405084: Average global foreground Dice: [0.8251, 0.9543, 0.9702, 0.352] +2022-04-11 22:26:42.405836: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:26:44.889885: lr: 0.007011 +2022-04-11 22:26:44.891367: saving scheduled checkpoint file... +2022-04-11 22:26:44.973976: saving checkpoint... +2022-04-11 22:26:49.772020: done, saving took 4.88 seconds +2022-04-11 22:26:49.795617: done +2022-04-11 22:26:49.846404: saving checkpoint... +2022-04-11 22:26:54.292678: done, saving took 4.50 seconds +2022-04-11 22:26:54.310977: This epoch took 130.860427 s + +2022-04-11 22:26:54.312441: svmem(total=99932160000, available=71473782784, percent=28.5, used=24179609600, free=26208284672, active=49260539904, inactive=22052622336, buffers=48422912, cached=49495842816, shared=2204921856, slab=1292865536) +2022-04-11 22:26:54.313321: +epoch: 163 +2022-04-11 22:28:41.743163: train loss : -0.5600 +2022-04-11 22:28:53.560413: validation loss: -0.5107 +2022-04-11 22:28:54.717676: Average global foreground Dice: [0.7808, 0.9331, 0.9597, 0.3547] +2022-04-11 22:28:54.720291: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:28:55.215724: lr: 0.006992 +2022-04-11 22:28:55.217463: saving scheduled checkpoint file... +2022-04-11 22:28:55.292576: saving checkpoint... +2022-04-11 22:28:59.795919: done, saving took 4.58 seconds +2022-04-11 22:28:59.842029: done +2022-04-11 22:28:59.843063: This epoch took 125.528865 s + +2022-04-11 22:28:59.844453: svmem(total=99932160000, available=71577190400, percent=28.4, used=24085348352, free=26310897664, active=49168646144, inactive=22044323840, buffers=48488448, cached=49487425536, shared=2195816448, slab=1291046912) +2022-04-11 22:28:59.845295: +epoch: 164 +2022-04-11 22:30:47.838415: train loss : -0.5511 +2022-04-11 22:30:59.375947: validation loss: -0.5901 +2022-04-11 22:30:59.378343: Average global foreground Dice: [0.8121, 0.9599, 0.9788, 0.3597] +2022-04-11 22:30:59.379210: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:31:00.265370: lr: 0.006974 +2022-04-11 22:31:00.266588: saving scheduled checkpoint file... +2022-04-11 22:31:00.358915: saving checkpoint... +2022-04-11 22:31:04.918629: done, saving took 4.65 seconds +2022-04-11 22:31:04.996887: done +2022-04-11 22:31:04.998261: This epoch took 125.152150 s + +2022-04-11 22:31:04.999243: svmem(total=99932160000, available=71636475904, percent=28.3, used=24024203264, free=26361761792, active=49110994944, inactive=22042898432, buffers=48562176, cached=49497632768, shared=2196353024, slab=1294524416) +2022-04-11 22:31:05.000046: +epoch: 165 +2022-04-11 22:32:54.098679: train loss : -0.5557 +2022-04-11 22:33:05.396886: validation loss: -0.5169 +2022-04-11 22:33:05.404131: Average global foreground Dice: [0.8409, 0.9386, 0.9573, 0.4112] +2022-04-11 22:33:05.405648: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:33:06.454403: lr: 0.006955 +2022-04-11 22:33:06.456149: saving scheduled checkpoint file... +2022-04-11 22:33:06.568038: saving checkpoint... +2022-04-11 22:33:11.353041: done, saving took 4.90 seconds +2022-04-11 22:33:11.390335: done +2022-04-11 22:33:11.474169: saving checkpoint... +2022-04-11 22:33:15.913969: done, saving took 4.52 seconds +2022-04-11 22:33:15.935711: This epoch took 130.934176 s + +2022-04-11 22:33:15.937452: svmem(total=99932160000, available=71760736256, percent=28.2, used=23899701248, free=26485211136, active=48992022528, inactive=22042902528, buffers=48640000, cached=49498607616, shared=2196541440, slab=1294602240) +2022-04-11 22:33:15.938258: +epoch: 166 +2022-04-11 22:35:03.273185: train loss : -0.5586 +2022-04-11 22:35:14.670492: validation loss: -0.5826 +2022-04-11 22:35:14.679039: Average global foreground Dice: [0.8253, 0.9477, 0.9709, 0.3789] +2022-04-11 22:35:14.680141: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:35:15.541729: lr: 0.006936 +2022-04-11 22:35:15.543121: saving scheduled checkpoint file... +2022-04-11 22:35:15.672404: saving checkpoint... +2022-04-11 22:35:20.385637: done, saving took 4.84 seconds +2022-04-11 22:35:20.400325: done +2022-04-11 22:35:20.511061: saving checkpoint... +2022-04-11 22:35:25.139162: done, saving took 4.74 seconds +2022-04-11 22:35:25.158116: This epoch took 129.218765 s + +2022-04-11 22:35:25.159314: svmem(total=99932160000, available=71544164352, percent=28.4, used=24116760576, free=26267828224, active=49207074816, inactive=22043582464, buffers=48734208, cached=49498836992, shared=2196070400, slab=1294823424) +2022-04-11 22:35:25.160031: +epoch: 167 +2022-04-11 22:37:14.324181: train loss : -0.5613 +2022-04-11 22:37:27.016914: validation loss: -0.5001 +2022-04-11 22:37:27.018603: Average global foreground Dice: [0.8372, 0.9324, 0.9499, 0.3799] +2022-04-11 22:37:27.019488: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:37:27.717587: lr: 0.006918 +2022-04-11 22:37:27.718764: saving scheduled checkpoint file... +2022-04-11 22:37:27.823906: saving checkpoint... +2022-04-11 22:37:32.379378: done, saving took 4.66 seconds +2022-04-11 22:37:32.426479: done +2022-04-11 22:37:32.473981: saving checkpoint... +2022-04-11 22:37:36.959343: done, saving took 4.53 seconds +2022-04-11 22:37:37.031157: This epoch took 131.869966 s + +2022-04-11 22:37:37.032732: svmem(total=99932160000, available=71475154944, percent=28.5, used=24185516032, free=26197999616, active=49276960768, inactive=22044549120, buffers=48816128, cached=49499828224, shared=2196328448, slab=1296510976) +2022-04-11 22:37:37.033877: +epoch: 168 +2022-04-11 22:39:23.929616: train loss : -0.5808 +2022-04-11 22:39:36.525734: validation loss: -0.5695 +2022-04-11 22:39:36.529675: Average global foreground Dice: [0.8288, 0.9528, 0.9663, 0.3809] +2022-04-11 22:39:36.531431: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:39:37.284938: lr: 0.006899 +2022-04-11 22:39:37.286362: saving scheduled checkpoint file... +2022-04-11 22:39:37.364299: saving checkpoint... +2022-04-11 22:39:41.911347: done, saving took 4.62 seconds +2022-04-11 22:39:41.991359: done +2022-04-11 22:39:42.039463: saving checkpoint... +2022-04-11 22:39:46.489192: done, saving took 4.50 seconds +2022-04-11 22:39:46.541231: This epoch took 129.506415 s + +2022-04-11 22:39:46.543055: svmem(total=99932160000, available=71263731712, percent=28.7, used=24376737792, free=25985769472, active=49468289024, inactive=22065541120, buffers=48885760, cached=49520766976, shared=2216476672, slab=1296003072) +2022-04-11 22:39:46.543803: +epoch: 169 +2022-04-11 22:41:34.745998: train loss : -0.5846 +2022-04-11 22:41:46.564064: validation loss: -0.6092 +2022-04-11 22:41:46.575600: Average global foreground Dice: [0.8478, 0.9542, 0.9748, 0.3841] +2022-04-11 22:41:46.579317: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:41:47.575771: lr: 0.00688 +2022-04-11 22:41:47.577228: saving scheduled checkpoint file... +2022-04-11 22:41:47.672447: saving checkpoint... +2022-04-11 22:41:52.080484: done, saving took 4.50 seconds +2022-04-11 22:41:52.094471: done +2022-04-11 22:41:52.142574: saving checkpoint... +2022-04-11 22:41:56.469161: done, saving took 4.37 seconds +2022-04-11 22:41:56.482830: This epoch took 129.937825 s + +2022-04-11 22:41:56.484191: svmem(total=99932160000, available=71326146560, percent=28.6, used=24334536704, free=26047238144, active=49422893056, inactive=22046101504, buffers=48959488, cached=49501425664, shared=2196267008, slab=1296257024) +2022-04-11 22:41:56.485232: +epoch: 170 +2022-04-11 22:43:43.265433: train loss : -0.5128 +2022-04-11 22:43:55.209113: validation loss: -0.4956 +2022-04-11 22:43:55.212096: Average global foreground Dice: [0.7719, 0.9286, 0.9627, 0.3404] +2022-04-11 22:43:55.213068: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:43:56.058661: lr: 0.006861 +2022-04-11 22:43:56.060117: saving scheduled checkpoint file... +2022-04-11 22:43:56.146342: saving checkpoint... +2022-04-11 22:44:00.734224: done, saving took 4.67 seconds +2022-04-11 22:44:00.751183: done +2022-04-11 22:44:00.752926: This epoch took 124.267087 s + +2022-04-11 22:44:00.754458: svmem(total=99932160000, available=71477596160, percent=28.5, used=24179478528, free=26197901312, active=49270370304, inactive=22050398208, buffers=49012736, cached=49505767424, shared=2199851008, slab=1295626240) +2022-04-11 22:44:00.755417: +epoch: 171 +2022-04-11 22:45:48.615556: train loss : -0.4987 +2022-04-11 22:46:01.015941: validation loss: -0.5474 +2022-04-11 22:46:01.017885: Average global foreground Dice: [0.8251, 0.9441, 0.9626, 0.371] +2022-04-11 22:46:01.018678: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:46:01.789436: lr: 0.006842 +2022-04-11 22:46:01.791308: saving scheduled checkpoint file... +2022-04-11 22:46:01.883980: saving checkpoint... +2022-04-11 22:46:06.566787: done, saving took 4.77 seconds +2022-04-11 22:46:06.653044: done +2022-04-11 22:46:06.654449: This epoch took 125.898206 s + +2022-04-11 22:46:06.655962: svmem(total=99932160000, available=71684493312, percent=28.3, used=23975833600, free=26403905536, active=49062703104, inactive=22047465472, buffers=49123328, cached=49503297536, shared=2196140032, slab=1295605760) +2022-04-11 22:46:06.657290: +epoch: 172 +2022-04-11 22:47:53.952786: train loss : -0.5350 +2022-04-11 22:48:05.125464: validation loss: -0.5329 +2022-04-11 22:48:05.150975: Average global foreground Dice: [0.7968, 0.9421, 0.9577, 0.3693] +2022-04-11 22:48:05.166604: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:48:06.059360: lr: 0.006824 +2022-04-11 22:48:06.065943: saving scheduled checkpoint file... +2022-04-11 22:48:06.192811: saving checkpoint... +2022-04-11 22:48:11.226500: done, saving took 5.15 seconds +2022-04-11 22:48:11.243415: done +2022-04-11 22:48:11.244487: This epoch took 124.586456 s + +2022-04-11 22:48:11.245398: svmem(total=99932160000, available=71583674368, percent=28.4, used=24077021184, free=26302328832, active=49166376960, inactive=22048157696, buffers=49188864, cached=49503621120, shared=2196156416, slab=1296744448) +2022-04-11 22:48:11.246069: +epoch: 173 +2022-04-11 22:49:58.389414: train loss : -0.5399 +2022-04-11 22:50:09.472773: validation loss: -0.5618 +2022-04-11 22:50:09.481906: Average global foreground Dice: [0.804, 0.9493, 0.9705, 0.3967] +2022-04-11 22:50:09.487547: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:50:10.503922: lr: 0.006805 +2022-04-11 22:50:10.506658: saving scheduled checkpoint file... +2022-04-11 22:50:10.604156: saving checkpoint... +2022-04-11 22:50:15.418806: done, saving took 4.91 seconds +2022-04-11 22:50:15.434026: done +2022-04-11 22:50:15.435549: This epoch took 124.188745 s + +2022-04-11 22:50:15.436941: svmem(total=99932160000, available=71217901568, percent=28.7, used=24432537600, free=25935781888, active=49525637120, inactive=22058995712, buffers=49262592, cached=49514577920, shared=2206371840, slab=1296748544) +2022-04-11 22:50:15.437745: +epoch: 174 +2022-04-11 22:52:03.837888: train loss : -0.5650 +2022-04-11 22:52:15.376706: validation loss: -0.5753 +2022-04-11 22:52:15.379277: Average global foreground Dice: [0.8417, 0.9414, 0.9681, 0.3658] +2022-04-11 22:52:15.381001: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:52:16.264966: lr: 0.006786 +2022-04-11 22:52:16.268036: saving scheduled checkpoint file... +2022-04-11 22:52:16.359346: saving checkpoint... +2022-04-11 22:52:20.968029: done, saving took 4.70 seconds +2022-04-11 22:52:20.982827: done +2022-04-11 22:52:20.984223: This epoch took 125.545615 s + +2022-04-11 22:52:20.985645: svmem(total=99932160000, available=71486734336, percent=28.5, used=24167079936, free=26203627520, active=49257295872, inactive=22049611776, buffers=49324032, cached=49512128512, shared=2203017216, slab=1296158720) +2022-04-11 22:52:20.986391: +epoch: 175 +2022-04-11 22:54:09.160716: train loss : -0.5734 +2022-04-11 22:54:22.273135: validation loss: -0.5919 +2022-04-11 22:54:22.287285: Average global foreground Dice: [0.8345, 0.9516, 0.9686, 0.3955] +2022-04-11 22:54:22.288493: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:54:23.237987: lr: 0.006767 +2022-04-11 22:54:23.239378: saving scheduled checkpoint file... +2022-04-11 22:54:23.333362: saving checkpoint... +2022-04-11 22:54:27.987783: done, saving took 4.75 seconds +2022-04-11 22:54:28.068556: done +2022-04-11 22:54:28.117091: saving checkpoint... +2022-04-11 22:54:32.669557: done, saving took 4.60 seconds +2022-04-11 22:54:32.704580: This epoch took 131.717445 s + +2022-04-11 22:54:32.707935: svmem(total=99932160000, available=71453237248, percent=28.5, used=24204865536, free=26145816576, active=49298317312, inactive=22065754112, buffers=49434624, cached=49532043264, shared=2196480000, slab=1300787200) +2022-04-11 22:54:32.709093: +epoch: 176 +2022-04-11 22:56:21.346977: train loss : -0.5702 +2022-04-11 22:56:33.160700: validation loss: -0.5981 +2022-04-11 22:56:33.182496: Average global foreground Dice: [0.8423, 0.9525, 0.9719, 0.3914] +2022-04-11 22:56:33.194192: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:56:34.278240: lr: 0.006749 +2022-04-11 22:56:34.285117: saving scheduled checkpoint file... +2022-04-11 22:56:34.462197: saving checkpoint... +2022-04-11 22:56:39.136272: done, saving took 4.78 seconds +2022-04-11 22:56:39.154648: done +2022-04-11 22:56:39.228916: saving checkpoint... +2022-04-11 22:56:43.755065: done, saving took 4.60 seconds +2022-04-11 22:56:43.774060: This epoch took 131.063787 s + +2022-04-11 22:56:43.775592: svmem(total=99932160000, available=71386308608, percent=28.6, used=24261566464, free=26078191616, active=49363910656, inactive=22076444672, buffers=49541120, cached=49542860800, shared=2206617600, slab=1299451904) +2022-04-11 22:56:43.776786: +epoch: 177 +2022-04-11 22:58:32.427156: train loss : -0.5572 +2022-04-11 22:58:44.409282: validation loss: -0.5859 +2022-04-11 22:58:44.415544: Average global foreground Dice: [0.8345, 0.9446, 0.9701, 0.4033] +2022-04-11 22:58:44.416714: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 22:58:45.058124: lr: 0.00673 +2022-04-11 22:58:45.059709: saving scheduled checkpoint file... +2022-04-11 22:58:45.179050: saving checkpoint... +2022-04-11 22:58:49.970481: done, saving took 4.91 seconds +2022-04-11 22:58:50.011170: done +2022-04-11 22:58:50.099830: saving checkpoint... +2022-04-11 22:58:54.607472: done, saving took 4.59 seconds +2022-04-11 22:58:54.627522: This epoch took 130.850003 s + +2022-04-11 22:58:54.629139: svmem(total=99932160000, available=71470792704, percent=28.5, used=24187535360, free=26161750016, active=49285021696, inactive=22066999296, buffers=49590272, cached=49533284352, shared=2196156416, slab=1298882560) +2022-04-11 22:58:54.629841: +epoch: 178 +2022-04-11 23:00:44.365696: train loss : -0.5533 +2022-04-11 23:00:56.871047: validation loss: -0.5597 +2022-04-11 23:00:56.874169: Average global foreground Dice: [0.8097, 0.9396, 0.965, 0.4] +2022-04-11 23:00:56.875519: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:00:57.587459: lr: 0.006711 +2022-04-11 23:00:57.588559: saving scheduled checkpoint file... +2022-04-11 23:00:57.706627: saving checkpoint... +2022-04-11 23:01:02.145762: done, saving took 4.56 seconds +2022-04-11 23:01:02.162070: done +2022-04-11 23:01:02.267627: saving checkpoint... +2022-04-11 23:01:06.788880: done, saving took 4.62 seconds +2022-04-11 23:01:06.838059: This epoch took 132.207275 s + +2022-04-11 23:01:06.839859: svmem(total=99932160000, available=71299432448, percent=28.7, used=24350605312, free=25989730304, active=49448894464, inactive=22075908096, buffers=49664000, cached=49542160384, shared=2204508160, slab=1299030016) +2022-04-11 23:01:06.840502: +epoch: 179 +2022-04-11 23:02:56.186361: train loss : -0.5335 +2022-04-11 23:03:08.912003: validation loss: -0.5735 +2022-04-11 23:03:08.914180: Average global foreground Dice: [0.8274, 0.9484, 0.9696, 0.3688] +2022-04-11 23:03:08.915430: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:03:09.638509: lr: 0.006692 +2022-04-11 23:03:09.639969: saving scheduled checkpoint file... +2022-04-11 23:03:09.733422: saving checkpoint... +2022-04-11 23:03:14.197672: done, saving took 4.56 seconds +2022-04-11 23:03:14.206959: done +2022-04-11 23:03:14.254390: saving checkpoint... +2022-04-11 23:03:18.558988: done, saving took 4.35 seconds +2022-04-11 23:03:18.572589: This epoch took 131.730950 s + +2022-04-11 23:03:18.573881: svmem(total=99932160000, available=71616446464, percent=28.3, used=24041775104, free=26306007040, active=49136373760, inactive=22068748288, buffers=49725440, cached=49534652416, shared=2196156416, slab=1299398656) +2022-04-11 23:03:18.574596: +epoch: 180 +2022-04-11 23:05:04.680432: train loss : -0.5331 +2022-04-11 23:05:16.682658: validation loss: -0.5145 +2022-04-11 23:05:16.684590: Average global foreground Dice: [0.7912, 0.9355, 0.9548, 0.3806] +2022-04-11 23:05:16.685532: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:05:17.496210: lr: 0.006673 +2022-04-11 23:05:17.497885: saving scheduled checkpoint file... +2022-04-11 23:05:17.587594: saving checkpoint... +2022-04-11 23:05:22.107980: done, saving took 4.61 seconds +2022-04-11 23:05:22.122172: done +2022-04-11 23:05:22.123151: This epoch took 123.547544 s + +2022-04-11 23:05:22.124383: svmem(total=99932160000, available=71430070272, percent=28.5, used=24228237312, free=26118578176, active=49325137920, inactive=22069612544, buffers=49823744, cached=49535520768, shared=2196144128, slab=1300795392) +2022-04-11 23:05:22.125183: +epoch: 181 +2022-04-11 23:07:10.710960: train loss : -0.5553 +2022-04-11 23:07:22.709574: validation loss: -0.5343 +2022-04-11 23:07:22.717910: Average global foreground Dice: [0.8261, 0.9412, 0.9657, 0.3781] +2022-04-11 23:07:22.719416: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:07:23.579829: lr: 0.006654 +2022-04-11 23:07:23.581485: saving scheduled checkpoint file... +2022-04-11 23:07:23.663046: saving checkpoint... +2022-04-11 23:07:28.249532: done, saving took 4.67 seconds +2022-04-11 23:07:28.263391: done +2022-04-11 23:07:28.264708: This epoch took 126.138770 s + +2022-04-11 23:07:28.265849: svmem(total=99932160000, available=71400783872, percent=28.6, used=24247644160, free=26088468480, active=49345818624, inactive=22079823872, buffers=49917952, cached=49546129408, shared=2206068736, slab=1300504576) +2022-04-11 23:07:28.266938: +epoch: 182 +2022-04-11 23:09:16.063560: train loss : -0.5369 +2022-04-11 23:09:27.330341: validation loss: -0.5948 +2022-04-11 23:09:27.333084: Average global foreground Dice: [0.799, 0.9508, 0.9688, 0.414] +2022-04-11 23:09:27.334038: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:09:28.325241: lr: 0.006636 +2022-04-11 23:09:28.326953: saving scheduled checkpoint file... +2022-04-11 23:09:28.379209: saving checkpoint... +2022-04-11 23:09:33.451003: done, saving took 5.12 seconds +2022-04-11 23:09:33.460568: done +2022-04-11 23:09:33.461817: This epoch took 125.194053 s + +2022-04-11 23:09:33.463050: svmem(total=99932160000, available=71415930880, percent=28.5, used=24241999872, free=26105643008, active=49251794944, inactive=22159294464, buffers=49983488, cached=49534533632, shared=2196414464, slab=1301651456) +2022-04-11 23:09:33.463820: +epoch: 183 +2022-04-11 23:11:20.317242: train loss : -0.5731 +2022-04-11 23:11:32.473270: validation loss: -0.5894 +2022-04-11 23:11:32.622195: Average global foreground Dice: [0.8288, 0.9419, 0.9687, 0.39] +2022-04-11 23:11:32.746158: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:11:33.782958: lr: 0.006617 +2022-04-11 23:11:33.933732: saving scheduled checkpoint file... +2022-04-11 23:11:34.166981: saving checkpoint... +2022-04-11 23:11:41.019409: done, saving took 6.94 seconds +2022-04-11 23:11:41.053034: done +2022-04-11 23:11:41.132134: saving checkpoint... +2022-04-11 23:11:46.129270: done, saving took 5.07 seconds +2022-04-11 23:11:46.150523: This epoch took 132.685991 s + +2022-04-11 23:11:46.153804: svmem(total=99932160000, available=71384174592, percent=28.6, used=24271126528, free=26072973312, active=49345908736, inactive=22090379264, buffers=50044928, cached=49538015232, shared=2199130112, slab=1299980288) +2022-04-11 23:11:46.160134: +epoch: 184 +2022-04-11 23:13:33.239923: train loss : -0.5240 +2022-04-11 23:13:44.725003: validation loss: -0.5586 +2022-04-11 23:13:44.858975: Average global foreground Dice: [0.8105, 0.9419, 0.9683, 0.4023] +2022-04-11 23:13:45.009405: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:13:46.039281: lr: 0.006598 +2022-04-11 23:13:46.172121: saving scheduled checkpoint file... +2022-04-11 23:13:46.466386: saving checkpoint... +2022-04-11 23:13:51.684107: done, saving took 5.32 seconds +2022-04-11 23:13:51.725893: done +2022-04-11 23:13:51.823221: saving checkpoint... +2022-04-11 23:13:58.762051: done, saving took 7.04 seconds +2022-04-11 23:13:58.793725: This epoch took 132.631373 s + +2022-04-11 23:13:58.795268: svmem(total=99932160000, available=71602188288, percent=28.3, used=24055660544, free=26290200576, active=49159622656, inactive=22062317568, buffers=50122752, cached=49536176128, shared=2196553728, slab=1300684800) +2022-04-11 23:13:58.796468: +epoch: 185 +2022-04-11 23:15:47.447680: train loss : -0.5809 +2022-04-11 23:15:59.303447: validation loss: -0.5997 +2022-04-11 23:15:59.431902: Average global foreground Dice: [0.8099, 0.9517, 0.9767, 0.4054] +2022-04-11 23:15:59.564937: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:16:00.688308: lr: 0.006579 +2022-04-11 23:16:00.689370: saving scheduled checkpoint file... +2022-04-11 23:16:00.774248: saving checkpoint... +2022-04-11 23:16:05.453616: done, saving took 4.76 seconds +2022-04-11 23:16:05.466331: done +2022-04-11 23:16:05.516213: saving checkpoint... +2022-04-11 23:16:10.173428: done, saving took 4.71 seconds +2022-04-11 23:16:10.205193: This epoch took 131.407563 s + +2022-04-11 23:16:10.206820: svmem(total=99932160000, available=71617134592, percent=28.3, used=24041185280, free=26304430080, active=49146257408, inactive=22062542848, buffers=50208768, cached=49536335872, shared=2196045824, slab=1301204992) +2022-04-11 23:16:10.207616: +epoch: 186 +2022-04-11 23:17:57.522152: train loss : -0.5893 +2022-04-11 23:18:10.262193: validation loss: -0.6061 +2022-04-11 23:18:10.395092: Average global foreground Dice: [0.835, 0.9603, 0.9649, 0.387] +2022-04-11 23:18:10.537500: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:18:11.424383: lr: 0.00656 +2022-04-11 23:18:11.548013: saving scheduled checkpoint file... +2022-04-11 23:18:11.775493: saving checkpoint... +2022-04-11 23:18:16.767938: done, saving took 5.08 seconds +2022-04-11 23:18:16.829305: done +2022-04-11 23:18:16.898891: saving checkpoint... +2022-04-11 23:18:21.976142: done, saving took 5.15 seconds +2022-04-11 23:18:21.993008: This epoch took 131.784677 s + +2022-04-11 23:18:21.994824: svmem(total=99932160000, available=71565545472, percent=28.4, used=24090996736, free=26301050880, active=49218326528, inactive=21987512320, buffers=50315264, cached=49489797120, shared=2196328448, slab=1303912448) +2022-04-11 23:18:21.995668: +epoch: 187 +2022-04-11 23:20:09.971362: train loss : -0.5877 +2022-04-11 23:20:20.993549: validation loss: -0.6049 +2022-04-11 23:20:20.998884: Average global foreground Dice: [0.8066, 0.9525, 0.9744, 0.4154] +2022-04-11 23:20:20.999868: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:20:21.910565: lr: 0.006541 +2022-04-11 23:20:21.911645: saving scheduled checkpoint file... +2022-04-11 23:20:21.990230: saving checkpoint... +2022-04-11 23:20:26.535928: done, saving took 4.62 seconds +2022-04-11 23:20:26.554010: done +2022-04-11 23:20:26.634919: saving checkpoint... +2022-04-11 23:20:31.148525: done, saving took 4.59 seconds +2022-04-11 23:20:31.167766: This epoch took 129.171355 s + +2022-04-11 23:20:31.169199: svmem(total=99932160000, available=71360188416, percent=28.6, used=24296751104, free=26094792704, active=49426313216, inactive=21988106240, buffers=50413568, cached=49490202624, shared=2196008960, slab=1305518080) +2022-04-11 23:20:31.170302: +epoch: 188 +2022-04-11 23:22:18.679235: train loss : -0.5655 +2022-04-11 23:22:30.444684: validation loss: -0.5563 +2022-04-11 23:22:30.453272: Average global foreground Dice: [0.8428, 0.9461, 0.9567, 0.375] +2022-04-11 23:22:30.455189: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:22:31.338480: lr: 0.006522 +2022-04-11 23:22:31.340168: saving scheduled checkpoint file... +2022-04-11 23:22:31.461833: saving checkpoint... +2022-04-11 23:22:36.006741: done, saving took 4.67 seconds +2022-04-11 23:22:36.029773: done +2022-04-11 23:22:36.031230: This epoch took 124.860220 s + +2022-04-11 23:22:36.032559: svmem(total=99932160000, available=71672049664, percent=28.3, used=23997104128, free=26405949440, active=49126240256, inactive=21976428544, buffers=50499584, cached=49478606848, shared=2183766016, slab=1305333760) +2022-04-11 23:22:36.033466: +epoch: 189 +2022-04-11 23:24:23.408360: train loss : -0.5179 +2022-04-11 23:24:35.352465: validation loss: -0.5740 +2022-04-11 23:24:35.360722: Average global foreground Dice: [0.8275, 0.955, 0.972, 0.3988] +2022-04-11 23:24:35.362035: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:24:36.220353: lr: 0.006504 +2022-04-11 23:24:36.221553: saving scheduled checkpoint file... +2022-04-11 23:24:36.348354: saving checkpoint... +2022-04-11 23:24:40.968540: done, saving took 4.75 seconds +2022-04-11 23:24:40.989367: done +2022-04-11 23:24:41.105750: saving checkpoint... +2022-04-11 23:24:45.626165: done, saving took 4.64 seconds +2022-04-11 23:24:45.645227: This epoch took 129.610926 s + +2022-04-11 23:24:45.646229: svmem(total=99932160000, available=71452729344, percent=28.5, used=24203399168, free=26185842688, active=49338093568, inactive=21990076416, buffers=50573312, cached=49492344832, shared=2196750336, slab=1304563712) +2022-04-11 23:24:45.646952: +epoch: 190 +2022-04-11 23:26:34.354793: train loss : -0.5745 +2022-04-11 23:26:47.010321: validation loss: -0.5137 +2022-04-11 23:26:47.012785: Average global foreground Dice: [0.835, 0.9324, 0.9521, 0.4044] +2022-04-11 23:26:47.014086: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:26:47.900843: lr: 0.006485 +2022-04-11 23:26:47.902826: saving scheduled checkpoint file... +2022-04-11 23:26:48.040207: saving checkpoint... +2022-04-11 23:26:52.530125: done, saving took 4.63 seconds +2022-04-11 23:26:52.612051: done +2022-04-11 23:26:52.613540: This epoch took 126.965908 s + +2022-04-11 23:26:52.615488: svmem(total=99932160000, available=71491031040, percent=28.5, used=24165445632, free=26223239168, active=49297145856, inactive=21990498304, buffers=50679808, cached=49492795392, shared=2196406272, slab=1302134784) +2022-04-11 23:26:52.616394: +epoch: 191 +2022-04-11 23:28:42.607740: train loss : -0.5621 +2022-04-11 23:28:54.540487: validation loss: -0.5712 +2022-04-11 23:28:54.542571: Average global foreground Dice: [0.8371, 0.9461, 0.9602, 0.3995] +2022-04-11 23:28:54.543723: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:28:55.628628: lr: 0.006466 +2022-04-11 23:28:55.630133: saving scheduled checkpoint file... +2022-04-11 23:28:55.737190: saving checkpoint... +2022-04-11 23:29:00.511427: done, saving took 4.88 seconds +2022-04-11 23:29:00.521415: done +2022-04-11 23:29:00.568783: saving checkpoint... +2022-04-11 23:29:05.347332: done, saving took 4.82 seconds +2022-04-11 23:29:05.399186: This epoch took 132.781862 s + +2022-04-11 23:29:05.400309: svmem(total=99932160000, available=71425536000, percent=28.5, used=24231239680, free=26156814336, active=49360662528, inactive=21991141376, buffers=50733056, cached=49493372928, shared=2196156416, slab=1302298624) +2022-04-11 23:29:05.400979: +epoch: 192 +2022-04-11 23:30:53.397749: train loss : -0.5803 +2022-04-11 23:31:05.461732: validation loss: -0.5993 +2022-04-11 23:31:05.471546: Average global foreground Dice: [0.8379, 0.9512, 0.9741, 0.3838] +2022-04-11 23:31:05.473071: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:31:06.343820: lr: 0.006447 +2022-04-11 23:31:06.347563: saving scheduled checkpoint file... +2022-04-11 23:31:06.410458: saving checkpoint... +2022-04-11 23:31:11.038465: done, saving took 4.69 seconds +2022-04-11 23:31:11.093558: done +2022-04-11 23:31:11.143784: saving checkpoint... +2022-04-11 23:31:15.718664: done, saving took 4.62 seconds +2022-04-11 23:31:15.733414: This epoch took 130.331771 s + +2022-04-11 23:31:15.735534: svmem(total=99932160000, available=71302684672, percent=28.6, used=24354738176, free=26033262592, active=49488031744, inactive=21990887424, buffers=50794496, cached=49493364736, shared=2195353600, slab=1304580096) +2022-04-11 23:31:15.736509: +epoch: 193 +2022-04-11 23:33:03.553565: train loss : -0.5864 +2022-04-11 23:33:15.867962: validation loss: -0.6278 +2022-04-11 23:33:15.873138: Average global foreground Dice: [0.8099, 0.957, 0.9796, 0.4225] +2022-04-11 23:33:15.876055: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:33:16.854462: lr: 0.006428 +2022-04-11 23:33:16.857311: saving scheduled checkpoint file... +2022-04-11 23:33:16.951118: saving checkpoint... +2022-04-11 23:33:21.595854: done, saving took 4.74 seconds +2022-04-11 23:33:21.613832: done +2022-04-11 23:33:21.680461: saving checkpoint... +2022-04-11 23:33:26.308623: done, saving took 4.69 seconds +2022-04-11 23:33:26.326058: This epoch took 130.588326 s + +2022-04-11 23:33:26.327306: svmem(total=99932160000, available=71395573760, percent=28.6, used=24264097792, free=26125320192, active=49397587968, inactive=21985931264, buffers=50864128, cached=49491877888, shared=2193133568, slab=1303977984) +2022-04-11 23:33:26.328699: +epoch: 194 +2022-04-11 23:35:14.394179: train loss : -0.5962 +2022-04-11 23:35:26.330861: validation loss: -0.6098 +2022-04-11 23:35:26.344039: Average global foreground Dice: [0.8397, 0.9547, 0.974, 0.3969] +2022-04-11 23:35:26.344828: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:35:27.203722: lr: 0.006409 +2022-04-11 23:35:27.206539: saving scheduled checkpoint file... +2022-04-11 23:35:27.306502: saving checkpoint... +2022-04-11 23:35:31.854153: done, saving took 4.65 seconds +2022-04-11 23:35:31.876047: done +2022-04-11 23:35:31.962887: saving checkpoint... +2022-04-11 23:35:36.630217: done, saving took 4.75 seconds +2022-04-11 23:35:36.644333: This epoch took 130.314746 s + +2022-04-11 23:35:36.645400: svmem(total=99932160000, available=71476989952, percent=28.5, used=24179556352, free=26206019584, active=49315749888, inactive=21993148416, buffers=50974720, cached=49495609344, shared=2196242432, slab=1302548480) +2022-04-11 23:35:36.646785: +epoch: 195 +2022-04-11 23:37:24.793297: train loss : -0.5853 +2022-04-11 23:37:35.927051: validation loss: -0.5837 +2022-04-11 23:37:35.931068: Average global foreground Dice: [0.8597, 0.9525, 0.9683, 0.3604] +2022-04-11 23:37:35.932365: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:37:36.886479: lr: 0.00639 +2022-04-11 23:37:36.887992: saving scheduled checkpoint file... +2022-04-11 23:37:37.000347: saving checkpoint... +2022-04-11 23:37:42.018674: done, saving took 5.13 seconds +2022-04-11 23:37:42.035482: done +2022-04-11 23:37:42.131332: saving checkpoint... +2022-04-11 23:37:46.934710: done, saving took 4.90 seconds +2022-04-11 23:37:46.954535: This epoch took 130.306891 s + +2022-04-11 23:37:46.955993: svmem(total=99932160000, available=71536443392, percent=28.4, used=24119685120, free=26264608768, active=49250578432, inactive=21994508288, buffers=51044352, cached=49496821760, shared=2196738048, slab=1303003136) +2022-04-11 23:37:46.956944: +epoch: 196 +2022-04-11 23:39:34.744710: train loss : -0.5865 +2022-04-11 23:39:47.464912: validation loss: -0.6048 +2022-04-11 23:39:47.485094: Average global foreground Dice: [0.8316, 0.9466, 0.9756, 0.4003] +2022-04-11 23:39:47.488680: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:39:48.211929: lr: 0.006371 +2022-04-11 23:39:48.215013: saving scheduled checkpoint file... +2022-04-11 23:39:48.332015: saving checkpoint... +2022-04-11 23:39:52.967052: done, saving took 4.75 seconds +2022-04-11 23:39:53.020503: done +2022-04-11 23:39:53.134386: saving checkpoint... +2022-04-11 23:39:58.090014: done, saving took 5.07 seconds +2022-04-11 23:39:58.114100: This epoch took 131.156258 s + +2022-04-11 23:39:58.115774: svmem(total=99932160000, available=71515975680, percent=28.4, used=24126586880, free=26243112960, active=49254596608, inactive=22008696832, buffers=51118080, cached=49511342080, shared=2210144256, slab=1304567808) +2022-04-11 23:39:58.116670: +epoch: 197 +2022-04-11 23:41:47.134023: train loss : -0.5882 +2022-04-11 23:42:00.100377: validation loss: -0.6308 +2022-04-11 23:42:00.106302: Average global foreground Dice: [0.8483, 0.9583, 0.9827, 0.4043] +2022-04-11 23:42:00.108296: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:42:01.024544: lr: 0.006352 +2022-04-11 23:42:01.027568: saving scheduled checkpoint file... +2022-04-11 23:42:01.134300: saving checkpoint... +2022-04-11 23:42:05.682439: done, saving took 4.65 seconds +2022-04-11 23:42:05.752127: done +2022-04-11 23:42:05.800273: saving checkpoint... +2022-04-11 23:42:10.880363: done, saving took 5.13 seconds +2022-04-11 23:42:10.911070: This epoch took 132.793293 s + +2022-04-11 23:42:10.912697: svmem(total=99932160000, available=71523139584, percent=28.4, used=24132132864, free=26251984896, active=49249402880, inactive=22008442880, buffers=51216384, cached=49496825856, shared=2196164608, slab=1306624000) +2022-04-11 23:42:10.913893: +epoch: 198 +2022-04-11 23:43:59.298209: train loss : -0.5378 +2022-04-11 23:44:10.834767: validation loss: -0.5742 +2022-04-11 23:44:10.852264: Average global foreground Dice: [0.8364, 0.9534, 0.9638, 0.3771] +2022-04-11 23:44:10.861173: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:44:11.767539: lr: 0.006333 +2022-04-11 23:44:11.788019: saving scheduled checkpoint file... +2022-04-11 23:44:11.932151: saving checkpoint... +2022-04-11 23:44:17.473968: done, saving took 5.62 seconds +2022-04-11 23:44:17.558064: done +2022-04-11 23:44:17.596846: This epoch took 126.682093 s + +2022-04-11 23:44:17.628775: svmem(total=99932160000, available=71354470400, percent=28.6, used=24300953600, free=26139996160, active=49412120576, inactive=21956816896, buffers=51286016, cached=49439924224, shared=2196328448, slab=1305202688) +2022-04-11 23:44:17.630958: +epoch: 199 +2022-04-11 23:46:05.629098: train loss : -0.5086 +2022-04-11 23:46:16.787832: validation loss: -0.5713 +2022-04-11 23:46:17.109598: Average global foreground Dice: [0.8357, 0.9497, 0.9653, 0.3665] +2022-04-11 23:46:17.118106: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:46:18.019361: lr: 0.006314 +2022-04-11 23:46:18.031739: saving scheduled checkpoint file... +2022-04-11 23:46:18.162178: saving checkpoint... +2022-04-11 23:46:22.794752: done, saving took 4.74 seconds +2022-04-11 23:46:22.813125: done +2022-04-11 23:46:22.814285: This epoch took 125.164019 s + +2022-04-11 23:46:22.815340: svmem(total=99932160000, available=71439892480, percent=28.5, used=24205262848, free=26224619520, active=49311612928, inactive=21967867904, buffers=51392512, cached=49450885120, shared=2206621696, slab=1306742784) +2022-04-11 23:46:22.816146: +epoch: 200 +2022-04-11 23:48:09.862580: train loss : -0.5806 +2022-04-11 23:48:20.511304: validation loss: -0.6110 +2022-04-11 23:48:20.513665: Average global foreground Dice: [0.8236, 0.9578, 0.9721, 0.3878] +2022-04-11 23:48:20.514709: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:48:21.351625: lr: 0.006296 +2022-04-11 23:48:21.353605: saving scheduled checkpoint file... +2022-04-11 23:48:21.469589: saving checkpoint... +2022-04-11 23:48:26.381353: done, saving took 5.03 seconds +2022-04-11 23:48:26.405978: done +2022-04-11 23:48:26.407331: This epoch took 123.590006 s + +2022-04-11 23:48:26.408596: svmem(total=99932160000, available=71561113600, percent=28.4, used=24083914752, free=26344947712, active=49192845312, inactive=21968793600, buffers=51445760, cached=49451851776, shared=2206752768, slab=1306169344) +2022-04-11 23:48:26.409849: +epoch: 201 +2022-04-11 23:50:13.030603: train loss : -0.5747 +2022-04-11 23:50:24.840876: validation loss: -0.5382 +2022-04-11 23:50:24.843427: Average global foreground Dice: [0.819, 0.9312, 0.9424, 0.3793] +2022-04-11 23:50:24.844187: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:50:25.674432: lr: 0.006277 +2022-04-11 23:50:25.676033: saving scheduled checkpoint file... +2022-04-11 23:50:25.824491: saving checkpoint... +2022-04-11 23:50:30.845467: done, saving took 5.17 seconds +2022-04-11 23:50:30.895125: done +2022-04-11 23:50:30.896451: This epoch took 124.485533 s + +2022-04-11 23:50:30.897400: svmem(total=99932160000, available=71534944256, percent=28.4, used=24120455168, free=26317955072, active=49228570624, inactive=21959024640, buffers=51535872, cached=49442213888, shared=2196275200, slab=1307688960) +2022-04-11 23:50:30.898659: +epoch: 202 +2022-04-11 23:52:18.657782: train loss : -0.5414 +2022-04-11 23:52:30.489487: validation loss: -0.5541 +2022-04-11 23:52:30.499911: Average global foreground Dice: [0.8164, 0.9478, 0.964, 0.3843] +2022-04-11 23:52:30.513200: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:52:31.348176: lr: 0.006258 +2022-04-11 23:52:31.356697: saving scheduled checkpoint file... +2022-04-11 23:52:31.510931: saving checkpoint... +2022-04-11 23:52:37.640519: done, saving took 6.28 seconds +2022-04-11 23:52:37.664289: done +2022-04-11 23:52:37.665515: This epoch took 126.766025 s + +2022-04-11 23:52:37.666469: svmem(total=99932160000, available=71462273024, percent=28.5, used=24197103616, free=26244423680, active=49307836416, inactive=21956407296, buffers=51609600, cached=49439023104, shared=2192355328, slab=1306062848) +2022-04-11 23:52:37.667228: +epoch: 203 +2022-04-11 23:54:27.090654: train loss : -0.5507 +2022-04-11 23:54:39.583055: validation loss: -0.5982 +2022-04-11 23:54:39.585445: Average global foreground Dice: [0.8103, 0.9592, 0.9701, 0.4116] +2022-04-11 23:54:39.586519: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:54:40.385814: lr: 0.006239 +2022-04-11 23:54:40.386962: saving scheduled checkpoint file... +2022-04-11 23:54:40.459661: saving checkpoint... +2022-04-11 23:54:45.120512: done, saving took 4.73 seconds +2022-04-11 23:54:45.129655: done +2022-04-11 23:54:45.130805: This epoch took 127.462651 s + +2022-04-11 23:54:45.132182: svmem(total=99932160000, available=71425114112, percent=28.5, used=24219156480, free=26206380032, active=49331552256, inactive=21971673088, buffers=51683328, cached=49454940160, shared=2207387648, slab=1306648576) +2022-04-11 23:54:45.133031: +epoch: 204 +2022-04-11 23:56:32.007399: train loss : -0.5669 +2022-04-11 23:56:44.031082: validation loss: -0.6265 +2022-04-11 23:56:44.033066: Average global foreground Dice: [0.8712, 0.9543, 0.982, 0.4108] +2022-04-11 23:56:44.034487: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:56:44.791920: lr: 0.00622 +2022-04-11 23:56:44.793689: saving scheduled checkpoint file... +2022-04-11 23:56:44.861160: saving checkpoint... +2022-04-11 23:56:49.502521: done, saving took 4.71 seconds +2022-04-11 23:56:49.516214: done +2022-04-11 23:56:49.518018: This epoch took 124.384183 s + +2022-04-11 23:56:49.519060: svmem(total=99932160000, available=71414870016, percent=28.5, used=24243834880, free=26195574784, active=49357721600, inactive=21957881856, buffers=51761152, cached=49440989184, shared=2193047552, slab=1306288128) +2022-04-11 23:56:49.519818: +epoch: 205 +2022-04-11 23:58:36.620265: train loss : -0.5955 +2022-04-11 23:58:47.606535: validation loss: -0.5714 +2022-04-11 23:58:47.618255: Average global foreground Dice: [0.8621, 0.9442, 0.9649, 0.3978] +2022-04-11 23:58:47.619189: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-11 23:58:48.485480: lr: 0.006201 +2022-04-11 23:58:48.487084: saving scheduled checkpoint file... +2022-04-11 23:58:48.574629: saving checkpoint... +2022-04-11 23:58:53.389167: done, saving took 4.90 seconds +2022-04-11 23:58:53.410174: done +2022-04-11 23:58:53.480341: saving checkpoint... +2022-04-11 23:58:57.998338: done, saving took 4.59 seconds +2022-04-11 23:58:58.079254: This epoch took 128.558208 s + +2022-04-11 23:58:58.082012: svmem(total=99932160000, available=71420395520, percent=28.5, used=24229580800, free=26200236032, active=49343229952, inactive=21961977856, buffers=51834880, cached=49450508288, shared=2201796608, slab=1305567232) +2022-04-11 23:58:58.084353: +epoch: 206 +2022-04-12 00:00:45.735904: train loss : -0.5979 +2022-04-12 00:00:57.005054: validation loss: -0.6280 +2022-04-12 00:00:57.011581: Average global foreground Dice: [0.8357, 0.9595, 0.9805, 0.4032] +2022-04-12 00:00:57.012833: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:00:57.907465: lr: 0.006182 +2022-04-12 00:00:57.908627: saving scheduled checkpoint file... +2022-04-12 00:00:58.020409: saving checkpoint... +2022-04-12 00:01:02.588186: done, saving took 4.68 seconds +2022-04-12 00:01:02.607674: done +2022-04-12 00:01:02.695033: saving checkpoint... +2022-04-12 00:01:07.518188: done, saving took 4.91 seconds +2022-04-12 00:01:07.537517: This epoch took 129.449636 s + +2022-04-12 00:01:07.538796: svmem(total=99932160000, available=71465873408, percent=28.5, used=24183562240, free=26227953664, active=49308106752, inactive=21962780672, buffers=60456960, cached=49460187136, shared=2196140032, slab=1318481920) +2022-04-12 00:01:07.539659: +epoch: 207 +2022-04-12 00:02:54.472651: train loss : -0.6196 +2022-04-12 00:03:06.221611: validation loss: -0.6362 +2022-04-12 00:03:06.223506: Average global foreground Dice: [0.8508, 0.9569, 0.9783, 0.4128] +2022-04-12 00:03:06.224765: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:03:06.898914: lr: 0.006163 +2022-04-12 00:03:06.900432: saving scheduled checkpoint file... +2022-04-12 00:03:07.024260: saving checkpoint... +2022-04-12 00:03:11.642844: done, saving took 4.74 seconds +2022-04-12 00:03:11.709569: done +2022-04-12 00:03:11.809884: saving checkpoint... +2022-04-12 00:03:16.647170: done, saving took 4.94 seconds +2022-04-12 00:03:16.667785: This epoch took 129.127228 s + +2022-04-12 00:03:16.669449: svmem(total=99932160000, available=71417344000, percent=28.5, used=24232693760, free=26178621440, active=49354010624, inactive=21962964992, buffers=60522496, cached=49460322304, shared=2195566592, slab=1317728256) +2022-04-12 00:03:16.670160: +epoch: 208 +2022-04-12 00:05:04.429231: train loss : -0.6007 +2022-04-12 00:05:16.188056: validation loss: -0.6351 +2022-04-12 00:05:16.190060: Average global foreground Dice: [0.8727, 0.9537, 0.9791, 0.387] +2022-04-12 00:05:16.191848: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:05:17.064525: lr: 0.006144 +2022-04-12 00:05:17.066436: saving scheduled checkpoint file... +2022-04-12 00:05:17.194910: saving checkpoint... +2022-04-12 00:05:21.857152: done, saving took 4.79 seconds +2022-04-12 00:05:21.942260: done +2022-04-12 00:05:22.069657: saving checkpoint... +2022-04-12 00:05:26.608222: done, saving took 4.66 seconds +2022-04-12 00:05:26.631529: This epoch took 129.960660 s + +2022-04-12 00:05:26.633096: svmem(total=99932160000, available=71570411520, percent=28.4, used=24064892928, free=26299695104, active=49141805056, inactive=22047764480, buffers=60665856, cached=49506906112, shared=2209796096, slab=1320177664) +2022-04-12 00:05:26.634315: +epoch: 209 +2022-04-12 00:07:15.113090: train loss : -0.5832 +2022-04-12 00:07:27.251100: validation loss: -0.5934 +2022-04-12 00:07:27.267414: Average global foreground Dice: [0.8365, 0.9457, 0.9645, 0.411] +2022-04-12 00:07:27.268635: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:07:28.248036: lr: 0.006125 +2022-04-12 00:07:28.254198: saving scheduled checkpoint file... +2022-04-12 00:07:28.345556: saving checkpoint... +2022-04-12 00:07:33.054482: done, saving took 4.80 seconds +2022-04-12 00:07:33.069831: done +2022-04-12 00:07:33.117795: saving checkpoint... +2022-04-12 00:07:37.780838: done, saving took 4.71 seconds +2022-04-12 00:07:37.794541: This epoch took 131.159268 s + +2022-04-12 00:07:37.796319: svmem(total=99932160000, available=71536353280, percent=28.4, used=24112713728, free=26265632768, active=49197850624, inactive=22034268160, buffers=60739584, cached=49493073920, shared=2195976192, slab=1318150144) +2022-04-12 00:07:37.797323: +epoch: 210 +2022-04-12 00:09:25.383321: train loss : -0.5950 +2022-04-12 00:09:38.065244: validation loss: -0.6057 +2022-04-12 00:09:38.069113: Average global foreground Dice: [0.8202, 0.9616, 0.9732, 0.3995] +2022-04-12 00:09:38.070421: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:09:38.952087: lr: 0.006106 +2022-04-12 00:09:38.953690: saving scheduled checkpoint file... +2022-04-12 00:09:39.044158: saving checkpoint... +2022-04-12 00:09:43.586214: done, saving took 4.63 seconds +2022-04-12 00:09:43.656530: done +2022-04-12 00:09:43.658233: This epoch took 125.860022 s + +2022-04-12 00:09:43.659923: svmem(total=99932160000, available=71396569088, percent=28.6, used=24245243904, free=26125762560, active=49331929088, inactive=22041968640, buffers=60792832, cached=49500360704, shared=2203267072, slab=1318801408) +2022-04-12 00:09:43.660787: +epoch: 211 +2022-04-12 00:11:30.955980: train loss : -0.5959 +2022-04-12 00:11:42.311220: validation loss: -0.6319 +2022-04-12 00:11:42.313241: Average global foreground Dice: [0.8598, 0.959, 0.9724, 0.4092] +2022-04-12 00:11:42.318265: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:11:43.085178: lr: 0.006087 +2022-04-12 00:11:43.086388: saving scheduled checkpoint file... +2022-04-12 00:11:43.171085: saving checkpoint... +2022-04-12 00:11:49.503368: done, saving took 6.42 seconds +2022-04-12 00:11:49.603575: done +2022-04-12 00:11:49.730881: saving checkpoint... +2022-04-12 00:11:54.676903: done, saving took 5.03 seconds +2022-04-12 00:11:54.713742: This epoch took 131.051565 s + +2022-04-12 00:11:54.715235: svmem(total=99932160000, available=71464382464, percent=28.5, used=24170999808, free=26192920576, active=49252974592, inactive=22048985088, buffers=60870656, cached=49507368960, shared=2209624064, slab=1319170048) +2022-04-12 00:11:54.716424: +epoch: 212 +2022-04-12 00:13:44.326228: train loss : -0.6028 +2022-04-12 00:13:57.320766: validation loss: -0.6038 +2022-04-12 00:13:57.483260: Average global foreground Dice: [0.8306, 0.958, 0.9739, 0.4104] +2022-04-12 00:13:57.496523: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:13:58.262756: lr: 0.006068 +2022-04-12 00:13:58.348691: saving scheduled checkpoint file... +2022-04-12 00:13:58.463010: saving checkpoint... +2022-04-12 00:14:03.135041: done, saving took 4.78 seconds +2022-04-12 00:14:03.223209: done +2022-04-12 00:14:03.319412: saving checkpoint... +2022-04-12 00:14:08.429917: done, saving took 5.21 seconds +2022-04-12 00:14:08.451325: This epoch took 133.734209 s + +2022-04-12 00:14:08.452936: svmem(total=99932160000, available=71490228224, percent=28.5, used=24144416768, free=26256117760, active=49166049280, inactive=22068797440, buffers=60936192, cached=49470689280, shared=2210377728, slab=1318936576) +2022-04-12 00:14:08.454224: +epoch: 213 +2022-04-12 00:15:56.529848: train loss : -0.5444 +2022-04-12 00:16:07.979949: validation loss: -0.4693 +2022-04-12 00:16:07.982846: Average global foreground Dice: [0.7827, 0.9268, 0.919, 0.3547] +2022-04-12 00:16:07.983861: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:16:08.904313: lr: 0.006049 +2022-04-12 00:16:08.906245: saving scheduled checkpoint file... +2022-04-12 00:16:09.029900: saving checkpoint... +2022-04-12 00:16:13.795613: done, saving took 4.89 seconds +2022-04-12 00:16:13.825151: done +2022-04-12 00:16:13.826420: This epoch took 125.370900 s + +2022-04-12 00:16:13.827955: svmem(total=99932160000, available=71584550912, percent=28.4, used=24073764864, free=26350383104, active=49127964672, inactive=22017134592, buffers=61030400, cached=49446981632, shared=2186788864, slab=1318891520) +2022-04-12 00:16:13.828735: +epoch: 214 +2022-04-12 00:18:04.080656: train loss : -0.5281 +2022-04-12 00:18:15.984379: validation loss: -0.5928 +2022-04-12 00:18:16.004026: Average global foreground Dice: [0.8079, 0.9511, 0.9717, 0.3746] +2022-04-12 00:18:16.005589: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:18:16.907345: lr: 0.00603 +2022-04-12 00:18:16.910406: saving scheduled checkpoint file... +2022-04-12 00:18:17.052106: saving checkpoint... +2022-04-12 00:18:21.619760: done, saving took 4.71 seconds +2022-04-12 00:18:21.654485: done +2022-04-12 00:18:21.656113: This epoch took 127.826612 s + +2022-04-12 00:18:21.657547: svmem(total=99932160000, available=71491629056, percent=28.5, used=24147361792, free=26257031168, active=49213194240, inactive=22022955008, buffers=61132800, cached=49466634240, shared=2206121984, slab=1319882752) +2022-04-12 00:18:21.658454: +epoch: 215 +2022-04-12 00:20:12.205744: train loss : -0.5675 +2022-04-12 00:20:24.564936: validation loss: -0.5785 +2022-04-12 00:20:24.584795: Average global foreground Dice: [0.8108, 0.9526, 0.9725, 0.379] +2022-04-12 00:20:24.586641: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:20:25.599825: lr: 0.006011 +2022-04-12 00:20:25.601318: saving scheduled checkpoint file... +2022-04-12 00:20:25.707131: saving checkpoint... +2022-04-12 00:20:30.129404: done, saving took 4.53 seconds +2022-04-12 00:20:30.146453: done +2022-04-12 00:20:30.147695: This epoch took 128.488336 s + +2022-04-12 00:20:30.149030: svmem(total=99932160000, available=71357583360, percent=28.6, used=24291115008, free=26122371072, active=49369042944, inactive=22007472128, buffers=61214720, cached=49457459200, shared=2196328448, slab=1319157760) +2022-04-12 00:20:30.149634: +epoch: 216 +2022-04-12 00:22:18.319218: train loss : -0.6023 +2022-04-12 00:22:30.499745: validation loss: -0.5947 +2022-04-12 00:22:30.504538: Average global foreground Dice: [0.8537, 0.9512, 0.9695, 0.4124] +2022-04-12 00:22:30.507405: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:22:31.759948: lr: 0.005991 +2022-04-12 00:22:31.766421: saving scheduled checkpoint file... +2022-04-12 00:22:31.842268: saving checkpoint... +2022-04-12 00:22:36.328315: done, saving took 4.56 seconds +2022-04-12 00:22:36.346324: done +2022-04-12 00:22:36.347740: This epoch took 126.197478 s + +2022-04-12 00:22:36.348780: svmem(total=99932160000, available=71264509952, percent=28.7, used=24377520128, free=26028929024, active=49453035520, inactive=22014459904, buffers=61296640, cached=49464414208, shared=2202943488, slab=1319538688) +2022-04-12 00:22:36.349451: +epoch: 217 +2022-04-12 00:24:23.787814: train loss : -0.6218 +2022-04-12 00:24:35.978881: validation loss: -0.5966 +2022-04-12 00:24:35.980995: Average global foreground Dice: [0.8399, 0.9553, 0.9681, 0.3886] +2022-04-12 00:24:35.981954: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:24:36.785167: lr: 0.005972 +2022-04-12 00:24:36.788046: saving scheduled checkpoint file... +2022-04-12 00:24:36.854931: saving checkpoint... +2022-04-12 00:24:41.524848: done, saving took 4.73 seconds +2022-04-12 00:24:41.537453: done +2022-04-12 00:24:41.538486: This epoch took 125.188352 s + +2022-04-12 00:24:41.539523: svmem(total=99932160000, available=71514763264, percent=28.4, used=24134299648, free=26280153088, active=49209942016, inactive=22005039104, buffers=61358080, cached=49456349184, shared=2196217856, slab=1318768640) +2022-04-12 00:24:41.540681: +epoch: 218 +2022-04-12 00:26:29.091560: train loss : -0.6110 +2022-04-12 00:26:41.394124: validation loss: -0.6556 +2022-04-12 00:26:41.396540: Average global foreground Dice: [0.859, 0.9571, 0.9831, 0.4255] +2022-04-12 00:26:41.397329: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:26:42.029889: lr: 0.005953 +2022-04-12 00:26:42.031795: saving scheduled checkpoint file... +2022-04-12 00:26:42.123599: saving checkpoint... +2022-04-12 00:26:46.916268: done, saving took 4.88 seconds +2022-04-12 00:26:46.931739: done +2022-04-12 00:26:46.933148: This epoch took 125.391729 s + +2022-04-12 00:26:46.934226: svmem(total=99932160000, available=71490965504, percent=28.5, used=24141393920, free=26255708160, active=49218490368, inactive=22022156288, buffers=61444096, cached=49473613824, shared=2212847616, slab=1319317504) +2022-04-12 00:26:46.935064: +epoch: 219 +2022-04-12 00:28:34.060567: train loss : -0.6091 +2022-04-12 00:28:45.803354: validation loss: -0.6246 +2022-04-12 00:28:45.813953: Average global foreground Dice: [0.8499, 0.9549, 0.9711, 0.4412] +2022-04-12 00:28:45.815419: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:28:46.791361: lr: 0.005934 +2022-04-12 00:28:46.792899: saving scheduled checkpoint file... +2022-04-12 00:28:46.903021: saving checkpoint... +2022-04-12 00:28:51.535359: done, saving took 4.74 seconds +2022-04-12 00:28:51.551090: done +2022-04-12 00:28:51.552491: This epoch took 124.616092 s + +2022-04-12 00:28:51.554200: svmem(total=99932160000, available=71413280768, percent=28.5, used=24235933696, free=26227609600, active=49294368768, inactive=21976489984, buffers=61534208, cached=49407082496, shared=2195968000, slab=1318961152) +2022-04-12 00:28:51.555222: +epoch: 220 +2022-04-12 00:30:39.849387: train loss : -0.6204 +2022-04-12 00:30:51.813840: validation loss: -0.6432 +2022-04-12 00:30:51.815958: Average global foreground Dice: [0.8798, 0.9569, 0.9737, 0.437] +2022-04-12 00:30:51.828503: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:30:52.509521: lr: 0.005915 +2022-04-12 00:30:52.510734: saving scheduled checkpoint file... +2022-04-12 00:30:52.644224: saving checkpoint... +2022-04-12 00:30:57.484642: done, saving took 4.97 seconds +2022-04-12 00:30:57.506173: done +2022-04-12 00:30:57.631576: saving checkpoint... +2022-04-12 00:31:02.514010: done, saving took 5.01 seconds +2022-04-12 00:31:02.535675: This epoch took 130.979632 s + +2022-04-12 00:31:02.537361: svmem(total=99932160000, available=71553986560, percent=28.4, used=24095084544, free=26367320064, active=49177583616, inactive=21949337600, buffers=61665280, cached=49408090112, shared=2196131840, slab=1320022016) +2022-04-12 00:31:02.538147: +epoch: 221 +2022-04-12 00:32:51.975080: train loss : -0.6102 +2022-04-12 00:33:04.336286: validation loss: -0.5962 +2022-04-12 00:33:04.340161: Average global foreground Dice: [0.8507, 0.9527, 0.9694, 0.408] +2022-04-12 00:33:04.341302: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:33:05.255138: lr: 0.005896 +2022-04-12 00:33:05.256484: saving scheduled checkpoint file... +2022-04-12 00:33:05.349961: saving checkpoint... +2022-04-12 00:33:10.601449: done, saving took 5.34 seconds +2022-04-12 00:33:10.648000: done +2022-04-12 00:33:10.704303: saving checkpoint... +2022-04-12 00:33:15.414212: done, saving took 4.76 seconds +2022-04-12 00:33:15.424614: This epoch took 132.885072 s + +2022-04-12 00:33:15.426017: svmem(total=99932160000, available=71508025344, percent=28.4, used=24139956224, free=26320449536, active=49227042816, inactive=21951053824, buffers=61751296, cached=49410002944, shared=2197131264, slab=1320251392) +2022-04-12 00:33:15.426866: +epoch: 222 +2022-04-12 00:35:03.440009: train loss : -0.6160 +2022-04-12 00:35:15.686985: validation loss: -0.6236 +2022-04-12 00:35:15.693366: Average global foreground Dice: [0.8495, 0.9523, 0.9752, 0.3801] +2022-04-12 00:35:15.699544: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:35:16.394711: lr: 0.005877 +2022-04-12 00:35:16.396012: saving scheduled checkpoint file... +2022-04-12 00:35:16.446297: saving checkpoint... +2022-04-12 00:35:21.066526: done, saving took 4.67 seconds +2022-04-12 00:35:21.081100: done +2022-04-12 00:35:21.082117: This epoch took 125.654482 s + +2022-04-12 00:35:21.086461: svmem(total=99932160000, available=71365275648, percent=28.6, used=24273100800, free=26176765952, active=49356230656, inactive=21959487488, buffers=61849600, cached=49420443648, shared=2206650368, slab=1322573824) +2022-04-12 00:35:21.087352: +epoch: 223 +2022-04-12 00:37:09.908945: train loss : -0.6154 +2022-04-12 00:37:22.202156: validation loss: -0.6343 +2022-04-12 00:37:22.212241: Average global foreground Dice: [0.866, 0.9609, 0.9801, 0.3709] +2022-04-12 00:37:22.213133: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:37:22.833310: lr: 0.005858 +2022-04-12 00:37:22.835204: saving scheduled checkpoint file... +2022-04-12 00:37:22.924563: saving checkpoint... +2022-04-12 00:37:27.768919: done, saving took 4.93 seconds +2022-04-12 00:37:27.783954: done +2022-04-12 00:37:27.785335: This epoch took 126.696918 s + +2022-04-12 00:37:27.786489: svmem(total=99932160000, available=71068917760, percent=28.9, used=24573030400, free=25879535616, active=49654370304, inactive=21955981312, buffers=61927424, cached=49417666560, shared=2203185152, slab=1322033152) +2022-04-12 00:37:27.787358: +epoch: 224 +2022-04-12 00:39:16.220421: train loss : -0.6213 +2022-04-12 00:39:29.044477: validation loss: -0.6137 +2022-04-12 00:39:29.046844: Average global foreground Dice: [0.8538, 0.9522, 0.9697, 0.4316] +2022-04-12 00:39:29.047725: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:39:29.889697: lr: 0.005839 +2022-04-12 00:39:29.891487: saving scheduled checkpoint file... +2022-04-12 00:39:29.945473: saving checkpoint... +2022-04-12 00:39:35.139974: done, saving took 5.25 seconds +2022-04-12 00:39:35.297314: done +2022-04-12 00:39:35.393584: saving checkpoint... +2022-04-12 00:39:41.259675: done, saving took 5.94 seconds +2022-04-12 00:39:41.288777: This epoch took 133.500567 s + +2022-04-12 00:39:41.290287: svmem(total=99932160000, available=71313313792, percent=28.6, used=24335069184, free=26122997760, active=49417424896, inactive=21950246912, buffers=62005248, cached=49412087808, shared=2196652032, slab=1321598976) +2022-04-12 00:39:41.290996: +epoch: 225 +2022-04-12 00:41:30.281283: train loss : -0.6361 +2022-04-12 00:41:43.211455: validation loss: -0.6245 +2022-04-12 00:41:43.214075: Average global foreground Dice: [0.8786, 0.9518, 0.9719, 0.3973] +2022-04-12 00:41:43.215788: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:41:44.019614: lr: 0.00582 +2022-04-12 00:41:44.022279: saving scheduled checkpoint file... +2022-04-12 00:41:44.134461: saving checkpoint... +2022-04-12 00:41:48.627438: done, saving took 4.60 seconds +2022-04-12 00:41:48.646457: done +2022-04-12 00:41:48.737242: saving checkpoint... +2022-04-12 00:41:53.409836: done, saving took 4.76 seconds +2022-04-12 00:41:53.428238: This epoch took 132.136356 s + +2022-04-12 00:41:53.429851: svmem(total=99932160000, available=71398805504, percent=28.6, used=24235950080, free=26207612928, active=49321734144, inactive=21961293824, buffers=62070784, cached=49426526208, shared=2210119680, slab=1321885696) +2022-04-12 00:41:53.430526: +epoch: 226 +2022-04-12 00:43:43.089658: train loss : -0.6370 +2022-04-12 00:43:55.492874: validation loss: -0.6361 +2022-04-12 00:43:55.495290: Average global foreground Dice: [0.8503, 0.9643, 0.9772, 0.4139] +2022-04-12 00:43:55.496517: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:43:56.339235: lr: 0.005801 +2022-04-12 00:43:56.340826: saving scheduled checkpoint file... +2022-04-12 00:43:56.470610: saving checkpoint... +2022-04-12 00:44:01.085070: done, saving took 4.74 seconds +2022-04-12 00:44:01.103902: done +2022-04-12 00:44:01.194448: saving checkpoint... +2022-04-12 00:44:07.029539: done, saving took 5.92 seconds +2022-04-12 00:44:07.051255: This epoch took 133.619535 s + +2022-04-12 00:44:07.053819: svmem(total=99932160000, available=71322992640, percent=28.6, used=24329240576, free=26131116032, active=49419067392, inactive=21945122816, buffers=62136320, cached=49409667072, shared=2192658432, slab=1319600128) +2022-04-12 00:44:07.054971: +epoch: 227 +2022-04-12 00:45:55.724977: train loss : -0.6286 +2022-04-12 00:46:07.461502: validation loss: -0.6268 +2022-04-12 00:46:07.489213: Average global foreground Dice: [0.8609, 0.9594, 0.9741, 0.3688] +2022-04-12 00:46:07.497035: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:46:08.582627: lr: 0.005781 +2022-04-12 00:46:08.587084: saving scheduled checkpoint file... +2022-04-12 00:46:08.708881: saving checkpoint... +2022-04-12 00:46:14.334924: done, saving took 5.73 seconds +2022-04-12 00:46:14.442960: done +2022-04-12 00:46:14.471022: This epoch took 127.415177 s + +2022-04-12 00:46:14.482572: svmem(total=99932160000, available=71419342848, percent=28.5, used=24229429248, free=26226319360, active=49324978176, inactive=21949440000, buffers=62234624, cached=49414176768, shared=2196070400, slab=1317470208) +2022-04-12 00:46:14.486888: +epoch: 228 +2022-04-12 00:48:02.372453: train loss : -0.6273 +2022-04-12 00:48:13.424181: validation loss: -0.6451 +2022-04-12 00:48:13.461634: Average global foreground Dice: [0.8542, 0.9566, 0.9813, 0.4255] +2022-04-12 00:48:13.463086: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:48:14.640541: lr: 0.005762 +2022-04-12 00:48:14.662558: saving scheduled checkpoint file... +2022-04-12 00:48:14.756029: saving checkpoint... +2022-04-12 00:48:20.969266: done, saving took 6.30 seconds +2022-04-12 00:48:20.986843: done +2022-04-12 00:48:21.036712: saving checkpoint... +2022-04-12 00:48:26.433502: done, saving took 5.45 seconds +2022-04-12 00:48:26.568069: This epoch took 132.062436 s + +2022-04-12 00:48:26.608185: svmem(total=99932160000, available=71365120000, percent=28.6, used=24279568384, free=26171265024, active=49369989120, inactive=21947281408, buffers=62332928, cached=49418993664, shared=2200121344, slab=1320251392) +2022-04-12 00:48:26.642542: +epoch: 229 +2022-04-12 00:50:13.251962: train loss : -0.6046 +2022-04-12 00:50:25.344384: validation loss: -0.6045 +2022-04-12 00:50:25.376267: Average global foreground Dice: [0.8358, 0.9508, 0.9802, 0.3927] +2022-04-12 00:50:25.389939: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:50:26.173210: lr: 0.005743 +2022-04-12 00:50:26.186078: saving scheduled checkpoint file... +2022-04-12 00:50:26.279495: saving checkpoint... +2022-04-12 00:50:31.853332: done, saving took 5.64 seconds +2022-04-12 00:50:31.891586: done +2022-04-12 00:50:31.932943: This epoch took 125.282122 s + +2022-04-12 00:50:31.937538: svmem(total=99932160000, available=71406739456, percent=28.5, used=24227225600, free=26167451648, active=49308708864, inactive=22016749568, buffers=62431232, cached=49475051520, shared=2209837056, slab=1322463232) +2022-04-12 00:50:31.942788: +epoch: 230 +2022-04-12 00:52:19.526421: train loss : -0.5978 +2022-04-12 00:52:31.440322: validation loss: -0.5513 +2022-04-12 00:52:31.477212: Average global foreground Dice: [0.8652, 0.9522, 0.9464, 0.4053] +2022-04-12 00:52:31.486246: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:52:32.178777: lr: 0.005724 +2022-04-12 00:52:32.185682: saving scheduled checkpoint file... +2022-04-12 00:52:32.259289: saving checkpoint... +2022-04-12 00:52:39.505330: done, saving took 7.31 seconds +2022-04-12 00:52:39.655968: done +2022-04-12 00:52:39.680540: This epoch took 127.700455 s + +2022-04-12 00:52:39.685163: svmem(total=99932160000, available=71351398400, percent=28.6, used=24295071744, free=26124300288, active=49359310848, inactive=22003400704, buffers=62500864, cached=49450287104, shared=2197233664, slab=1322921984) +2022-04-12 00:52:39.690676: +epoch: 231 +2022-04-12 00:54:28.068025: train loss : -0.6087 +2022-04-12 00:54:40.032825: validation loss: -0.6321 +2022-04-12 00:54:40.049228: Average global foreground Dice: [0.8426, 0.9643, 0.9783, 0.4025] +2022-04-12 00:54:40.054766: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:54:40.959014: lr: 0.005705 +2022-04-12 00:54:40.960870: saving scheduled checkpoint file... +2022-04-12 00:54:41.027506: saving checkpoint... +2022-04-12 00:54:46.130269: done, saving took 5.16 seconds +2022-04-12 00:54:46.144577: done +2022-04-12 00:54:46.145468: This epoch took 126.452316 s + +2022-04-12 00:54:46.146343: svmem(total=99932160000, available=71258591232, percent=28.7, used=24375193600, free=26049908736, active=49344241664, inactive=22092697600, buffers=62562304, cached=49444495360, shared=2209845248, slab=1321275392) +2022-04-12 00:54:46.147412: +epoch: 232 +2022-04-12 00:56:33.114539: train loss : -0.6117 +2022-04-12 00:56:44.087150: validation loss: -0.6259 +2022-04-12 00:56:44.114380: Average global foreground Dice: [0.865, 0.9571, 0.976, 0.4226] +2022-04-12 00:56:44.173571: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:56:45.198533: lr: 0.005686 +2022-04-12 00:56:45.200907: saving scheduled checkpoint file... +2022-04-12 00:56:45.307054: saving checkpoint... +2022-04-12 00:56:50.422279: done, saving took 5.22 seconds +2022-04-12 00:56:50.440195: done +2022-04-12 00:56:50.505610: saving checkpoint... +2022-04-12 00:56:55.153782: done, saving took 4.71 seconds +2022-04-12 00:56:55.193870: This epoch took 129.045798 s + +2022-04-12 00:56:55.198975: svmem(total=99932160000, available=71362191360, percent=28.6, used=24284516352, free=26132205568, active=49259442176, inactive=22102626304, buffers=62668800, cached=49452769280, shared=2196848640, slab=1320325120) +2022-04-12 00:56:55.200123: +epoch: 233 +2022-04-12 00:58:43.014325: train loss : -0.5904 +2022-04-12 00:58:54.225513: validation loss: -0.5721 +2022-04-12 00:58:54.227800: Average global foreground Dice: [0.8534, 0.9488, 0.9578, 0.3944] +2022-04-12 00:58:54.228799: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 00:58:54.944619: lr: 0.005667 +2022-04-12 00:58:54.945551: saving scheduled checkpoint file... +2022-04-12 00:58:55.076196: saving checkpoint... +2022-04-12 00:59:00.679675: done, saving took 5.73 seconds +2022-04-12 00:59:00.697285: done +2022-04-12 00:59:00.698298: This epoch took 125.497214 s + +2022-04-12 00:59:00.699281: svmem(total=99932160000, available=71491338240, percent=28.5, used=24156516352, free=26261016576, active=49190957056, inactive=22039461888, buffers=62746624, cached=49451880448, shared=2195746816, slab=1320947712) +2022-04-12 00:59:00.700090: +epoch: 234 +2022-04-12 01:00:49.520822: train loss : -0.6042 +2022-04-12 01:01:02.089809: validation loss: -0.6083 +2022-04-12 01:01:02.124649: Average global foreground Dice: [0.8604, 0.9595, 0.9687, 0.3874] +2022-04-12 01:01:02.130828: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:01:02.897929: lr: 0.005647 +2022-04-12 01:01:02.906846: saving scheduled checkpoint file... +2022-04-12 01:01:03.045404: saving checkpoint... +2022-04-12 01:01:08.957290: done, saving took 6.03 seconds +2022-04-12 01:01:09.128464: done +2022-04-12 01:01:09.134756: This epoch took 128.433907 s + +2022-04-12 01:01:09.142806: svmem(total=99932160000, available=71451017216, percent=28.5, used=24182362112, free=26220199936, active=49217093632, inactive=22052769792, buffers=62820352, cached=49466777600, shared=2210033664, slab=1321971712) +2022-04-12 01:01:09.151315: +epoch: 235 +2022-04-12 01:02:59.300516: train loss : -0.6276 +2022-04-12 01:03:11.474869: validation loss: -0.6213 +2022-04-12 01:03:11.486334: Average global foreground Dice: [0.8832, 0.9567, 0.9721, 0.4022] +2022-04-12 01:03:11.488071: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:03:12.382147: lr: 0.005628 +2022-04-12 01:03:12.383833: saving scheduled checkpoint file... +2022-04-12 01:03:12.497538: saving checkpoint... +2022-04-12 01:03:17.111881: done, saving took 4.73 seconds +2022-04-12 01:03:17.132324: done +2022-04-12 01:03:17.234086: saving checkpoint... +2022-04-12 01:03:21.795881: done, saving took 4.66 seconds +2022-04-12 01:03:21.817394: This epoch took 132.610580 s + +2022-04-12 01:03:21.819773: svmem(total=99932160000, available=71618932736, percent=28.3, used=24014434304, free=26388267008, active=49079996416, inactive=22019932160, buffers=62877696, cached=49466580992, shared=2210213888, slab=1322094592) +2022-04-12 01:03:21.820819: +epoch: 236 +2022-04-12 01:05:10.169604: train loss : -0.5782 +2022-04-12 01:05:22.391243: validation loss: -0.6008 +2022-04-12 01:05:22.393209: Average global foreground Dice: [0.8625, 0.9523, 0.9728, 0.355] +2022-04-12 01:05:22.394350: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:05:23.104537: lr: 0.005609 +2022-04-12 01:05:23.105854: saving scheduled checkpoint file... +2022-04-12 01:05:23.230109: saving checkpoint... +2022-04-12 01:05:28.078926: done, saving took 4.97 seconds +2022-04-12 01:05:28.119707: done +2022-04-12 01:05:28.140054: This epoch took 126.318309 s + +2022-04-12 01:05:28.145549: svmem(total=99932160000, available=71449976832, percent=28.5, used=24187084800, free=26218971136, active=49261780992, inactive=22012665856, buffers=62992384, cached=49463111680, shared=2206478336, slab=1321246720) +2022-04-12 01:05:28.155254: +epoch: 237 +2022-04-12 01:07:16.739676: train loss : -0.5754 +2022-04-12 01:07:29.446562: validation loss: -0.5961 +2022-04-12 01:07:29.455678: Average global foreground Dice: [0.8275, 0.9586, 0.9613, 0.3716] +2022-04-12 01:07:29.471505: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:07:30.272970: lr: 0.00559 +2022-04-12 01:07:30.308682: saving scheduled checkpoint file... +2022-04-12 01:07:30.450297: saving checkpoint... +2022-04-12 01:07:37.956879: done, saving took 7.63 seconds +2022-04-12 01:07:38.045258: done +2022-04-12 01:07:38.077393: This epoch took 129.918187 s + +2022-04-12 01:07:38.087335: svmem(total=99932160000, available=71530639360, percent=28.4, used=24116576256, free=26300989440, active=49192357888, inactive=22001950720, buffers=63066112, cached=49451528192, shared=2196242432, slab=1322262528) +2022-04-12 01:07:38.089260: +epoch: 238 +2022-04-12 01:09:26.118510: train loss : -0.5885 +2022-04-12 01:09:38.308372: validation loss: -0.6126 +2022-04-12 01:09:38.347775: Average global foreground Dice: [0.8677, 0.9563, 0.9714, 0.3857] +2022-04-12 01:09:38.379014: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:09:39.279375: lr: 0.005571 +2022-04-12 01:09:39.322120: saving scheduled checkpoint file... +2022-04-12 01:09:39.506279: saving checkpoint... +2022-04-12 01:09:44.653728: done, saving took 5.28 seconds +2022-04-12 01:09:44.677751: done +2022-04-12 01:09:44.679188: This epoch took 126.579186 s + +2022-04-12 01:09:44.680367: svmem(total=99932160000, available=71345586176, percent=28.6, used=24302055424, free=26162167808, active=49375252480, inactive=21956206592, buffers=63131648, cached=49404805120, shared=2196156416, slab=1320763392) +2022-04-12 01:09:44.681384: +epoch: 239 +2022-04-12 01:11:33.713505: train loss : -0.6020 +2022-04-12 01:11:45.933678: validation loss: -0.6227 +2022-04-12 01:11:45.936252: Average global foreground Dice: [0.8439, 0.9583, 0.9741, 0.4093] +2022-04-12 01:11:45.938338: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:11:46.963566: lr: 0.005551 +2022-04-12 01:11:46.965703: saving scheduled checkpoint file... +2022-04-12 01:11:47.069060: saving checkpoint... +2022-04-12 01:11:55.761660: done, saving took 8.79 seconds +2022-04-12 01:11:55.957889: done +2022-04-12 01:11:55.964230: This epoch took 131.281888 s + +2022-04-12 01:11:56.027898: svmem(total=99932160000, available=71523979264, percent=28.4, used=24123625472, free=26339774464, active=49204039680, inactive=21950513152, buffers=63209472, cached=49405550592, shared=2196156416, slab=1320529920) +2022-04-12 01:11:56.095219: +epoch: 240 +2022-04-12 01:13:43.858010: train loss : -0.6128 +2022-04-12 01:13:55.662485: validation loss: -0.5886 +2022-04-12 01:13:55.695032: Average global foreground Dice: [0.8645, 0.9519, 0.9664, 0.3572] +2022-04-12 01:13:55.741116: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:13:56.773326: lr: 0.005532 +2022-04-12 01:13:56.784316: saving scheduled checkpoint file... +2022-04-12 01:13:56.944123: saving checkpoint... +2022-04-12 01:14:06.265270: done, saving took 9.39 seconds +2022-04-12 01:14:06.394727: done +2022-04-12 01:14:06.401567: This epoch took 130.300265 s + +2022-04-12 01:14:06.407968: svmem(total=99932160000, available=76609462272, percent=23.3, used=19053613056, free=29976633344, active=44701888512, inactive=22925402112, buffers=63332352, cached=50838581248, shared=2181476352, slab=1321385984) +2022-04-12 01:14:06.414250: +epoch: 241 +2022-04-12 01:15:53.729928: train loss : -0.5944 +2022-04-12 01:16:04.671137: validation loss: -0.6090 +2022-04-12 01:16:04.682840: Average global foreground Dice: [0.8637, 0.953, 0.9742, 0.4113] +2022-04-12 01:16:04.684679: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:16:05.498538: lr: 0.005513 +2022-04-12 01:16:05.500607: saving scheduled checkpoint file... +2022-04-12 01:16:05.570750: saving checkpoint... +2022-04-12 01:16:11.864647: done, saving took 6.36 seconds +2022-04-12 01:16:11.951673: done +2022-04-12 01:16:11.956651: This epoch took 125.536370 s + +2022-04-12 01:16:11.966417: svmem(total=99932160000, available=76730449920, percent=23.2, used=19111813120, free=36077150208, active=44632596480, inactive=17273192448, buffers=63578112, cached=44679618560, shared=2181562368, slab=943865856) +2022-04-12 01:16:11.987205: +epoch: 242 +2022-04-12 01:17:59.137164: train loss : -0.6152 +2022-04-12 01:18:10.132414: validation loss: -0.5830 +2022-04-12 01:18:10.179940: Average global foreground Dice: [0.8601, 0.9525, 0.9543, 0.4068] +2022-04-12 01:18:10.209598: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:18:11.146727: lr: 0.005494 +2022-04-12 01:18:11.152020: saving scheduled checkpoint file... +2022-04-12 01:18:11.277499: saving checkpoint... +2022-04-12 01:18:18.809149: done, saving took 7.63 seconds +2022-04-12 01:18:18.899020: done +2022-04-12 01:18:18.902780: This epoch took 126.890796 s + +2022-04-12 01:18:18.928385: svmem(total=99932160000, available=76671881216, percent=23.3, used=19157204992, free=36018323456, active=44678156288, inactive=17287004160, buffers=63660032, cached=44692971520, shared=2194915328, slab=942395392) +2022-04-12 01:18:18.970627: +epoch: 243 +2022-04-12 01:20:06.488028: train loss : -0.6064 +2022-04-12 01:20:18.473919: validation loss: -0.6377 +2022-04-12 01:20:18.492906: Average global foreground Dice: [0.8586, 0.9594, 0.9762, 0.4105] +2022-04-12 01:20:18.494522: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:20:19.308372: lr: 0.005474 +2022-04-12 01:20:19.310138: saving scheduled checkpoint file... +2022-04-12 01:20:19.442122: saving checkpoint... +2022-04-12 01:20:24.166859: done, saving took 4.86 seconds +2022-04-12 01:20:24.187897: done +2022-04-12 01:20:24.189232: This epoch took 125.215784 s + +2022-04-12 01:20:24.190568: svmem(total=99932160000, available=76819406848, percent=23.1, used=19016065024, free=36165365760, active=44534259712, inactive=17280909312, buffers=63713280, cached=44687015936, shared=2188484608, slab=944582656) +2022-04-12 01:20:24.191426: +epoch: 244 +2022-04-12 01:22:11.819747: train loss : -0.6220 +2022-04-12 01:22:23.314888: validation loss: -0.6426 +2022-04-12 01:22:23.327983: Average global foreground Dice: [0.8843, 0.9675, 0.9721, 0.4079] +2022-04-12 01:22:23.330348: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:22:24.199916: lr: 0.005455 +2022-04-12 01:22:24.202789: saving scheduled checkpoint file... +2022-04-12 01:22:24.343374: saving checkpoint... +2022-04-12 01:22:29.074861: done, saving took 4.87 seconds +2022-04-12 01:22:29.102155: done +2022-04-12 01:22:29.103292: This epoch took 124.911006 s + +2022-04-12 01:22:29.104593: svmem(total=99932160000, available=76712087552, percent=23.2, used=19118833664, free=36057571328, active=44644257792, inactive=17285668864, buffers=63782912, cached=44691972096, shared=2192961536, slab=942411776) +2022-04-12 01:22:29.105411: +epoch: 245 +2022-04-12 01:24:18.524146: train loss : -0.6196 +2022-04-12 01:24:30.138465: validation loss: -0.6501 +2022-04-12 01:24:30.144661: Average global foreground Dice: [0.8636, 0.967, 0.9769, 0.4141] +2022-04-12 01:24:30.146680: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:24:31.252391: lr: 0.005436 +2022-04-12 01:24:31.255420: saving scheduled checkpoint file... +2022-04-12 01:24:31.347855: saving checkpoint... +2022-04-12 01:24:36.174048: done, saving took 4.92 seconds +2022-04-12 01:24:36.191655: done +2022-04-12 01:24:36.246214: saving checkpoint... +2022-04-12 01:24:41.103909: done, saving took 4.91 seconds +2022-04-12 01:24:41.119858: This epoch took 132.013824 s + +2022-04-12 01:24:41.122561: svmem(total=99932160000, available=76640256000, percent=23.3, used=19191791616, free=35985367040, active=44714807296, inactive=17284993024, buffers=63840256, cached=44691161088, shared=2191884288, slab=943226880) +2022-04-12 01:24:41.125635: +epoch: 246 +2022-04-12 01:26:30.522674: train loss : -0.5960 +2022-04-12 01:26:42.560816: validation loss: -0.6104 +2022-04-12 01:26:42.572705: Average global foreground Dice: [0.8563, 0.9579, 0.9772, 0.3723] +2022-04-12 01:26:42.574674: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:26:43.276576: lr: 0.005417 +2022-04-12 01:26:43.280074: saving scheduled checkpoint file... +2022-04-12 01:26:43.363880: saving checkpoint... +2022-04-12 01:26:48.271307: done, saving took 4.99 seconds +2022-04-12 01:26:48.387177: done +2022-04-12 01:26:48.390928: This epoch took 127.261981 s + +2022-04-12 01:26:48.392232: svmem(total=99932160000, available=76834770944, percent=23.1, used=18993774592, free=36179341312, active=44515213312, inactive=17288843264, buffers=63926272, cached=44695117824, shared=2195329024, slab=942718976) +2022-04-12 01:26:48.392901: +epoch: 247 +2022-04-12 01:28:35.920429: train loss : -0.5648 +2022-04-12 01:28:48.777902: validation loss: -0.6023 +2022-04-12 01:28:48.781815: Average global foreground Dice: [0.8364, 0.9509, 0.969, 0.3944] +2022-04-12 01:28:48.784584: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:28:49.671268: lr: 0.005397 +2022-04-12 01:28:49.673215: saving scheduled checkpoint file... +2022-04-12 01:28:49.793669: saving checkpoint... +2022-04-12 01:28:54.526848: done, saving took 4.85 seconds +2022-04-12 01:28:54.579617: done +2022-04-12 01:28:54.580471: This epoch took 126.186939 s + +2022-04-12 01:28:54.581574: svmem(total=99932160000, available=76821270528, percent=23.1, used=19021643776, free=36165496832, active=44539310080, inactive=17274724352, buffers=63971328, cached=44681048064, shared=2180956160, slab=944611328) +2022-04-12 01:28:54.583059: +epoch: 248 +2022-04-12 01:30:42.406587: train loss : -0.6034 +2022-04-12 01:30:54.130301: validation loss: -0.6219 +2022-04-12 01:30:54.135742: Average global foreground Dice: [0.8546, 0.9639, 0.975, 0.4035] +2022-04-12 01:30:54.137426: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:30:55.012748: lr: 0.005378 +2022-04-12 01:30:55.014442: saving scheduled checkpoint file... +2022-04-12 01:30:55.126308: saving checkpoint... +2022-04-12 01:31:00.041362: done, saving took 5.03 seconds +2022-04-12 01:31:00.095054: done +2022-04-12 01:31:00.096761: This epoch took 125.512941 s + +2022-04-12 01:31:00.098054: svmem(total=99932160000, available=76959584256, percent=23.0, used=18882256896, free=36303282176, active=44401852416, inactive=17275998208, buffers=64024576, cached=44682596352, shared=2182008832, slab=942759936) +2022-04-12 01:31:00.098962: +epoch: 249 +2022-04-12 01:32:48.702041: train loss : -0.5698 +2022-04-12 01:32:59.923011: validation loss: -0.5848 +2022-04-12 01:32:59.927959: Average global foreground Dice: [0.861, 0.9495, 0.9632, 0.375] +2022-04-12 01:32:59.928817: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:33:00.800026: lr: 0.005359 +2022-04-12 01:33:00.801688: saving scheduled checkpoint file... +2022-04-12 01:33:00.949661: saving checkpoint... +2022-04-12 01:33:05.721298: done, saving took 4.92 seconds +2022-04-12 01:33:05.756857: done +2022-04-12 01:33:05.758297: This epoch took 125.658148 s + +2022-04-12 01:33:05.759349: svmem(total=99932160000, available=76748734464, percent=23.2, used=19093737472, free=36092121088, active=44617109504, inactive=17276567552, buffers=64081920, cached=44682219520, shared=2181476352, slab=942559232) +2022-04-12 01:33:05.760139: +epoch: 250 +2022-04-12 01:34:55.319875: train loss : -0.5829 +2022-04-12 01:35:07.902851: validation loss: -0.5854 +2022-04-12 01:35:07.904774: Average global foreground Dice: [0.8171, 0.9446, 0.9638, 0.4037] +2022-04-12 01:35:07.905697: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:35:08.550078: lr: 0.00534 +2022-04-12 01:35:08.551750: saving scheduled checkpoint file... +2022-04-12 01:35:08.677631: saving checkpoint... +2022-04-12 01:35:13.495629: done, saving took 4.94 seconds +2022-04-12 01:35:13.518862: done +2022-04-12 01:35:13.520459: This epoch took 127.759464 s + +2022-04-12 01:35:13.521479: svmem(total=99932160000, available=76893937664, percent=23.1, used=18948247552, free=36236800000, active=44471300096, inactive=17276510208, buffers=64159744, cached=44682952704, shared=2181632000, slab=943771648) +2022-04-12 01:35:13.522238: +epoch: 251 +2022-04-12 01:37:02.309613: train loss : -0.6002 +2022-04-12 01:37:14.894649: validation loss: -0.6337 +2022-04-12 01:37:14.897342: Average global foreground Dice: [0.8453, 0.9591, 0.9744, 0.3958] +2022-04-12 01:37:14.903360: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:37:15.646077: lr: 0.00532 +2022-04-12 01:37:15.653440: saving scheduled checkpoint file... +2022-04-12 01:37:15.749891: saving checkpoint... +2022-04-12 01:37:20.741408: done, saving took 5.09 seconds +2022-04-12 01:37:20.795025: done +2022-04-12 01:37:20.796215: This epoch took 127.273317 s + +2022-04-12 01:37:20.797063: svmem(total=99932160000, available=76712742912, percent=23.2, used=19113984000, free=36055113728, active=44636958720, inactive=17292492800, buffers=64204800, cached=44698857472, shared=2197106688, slab=944115712) +2022-04-12 01:37:20.797690: +epoch: 252 +2022-04-12 01:39:08.585013: train loss : -0.6305 +2022-04-12 01:39:20.224855: validation loss: -0.6404 +2022-04-12 01:39:20.370957: Average global foreground Dice: [0.8676, 0.9601, 0.9767, 0.4101] +2022-04-12 01:39:20.531405: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:39:21.459439: lr: 0.005301 +2022-04-12 01:39:21.460568: saving scheduled checkpoint file... +2022-04-12 01:39:21.511324: saving checkpoint... +2022-04-12 01:39:26.620075: done, saving took 5.16 seconds +2022-04-12 01:39:26.633281: done +2022-04-12 01:39:26.634199: This epoch took 125.835908 s + +2022-04-12 01:39:26.635901: svmem(total=99932160000, available=76778749952, percent=23.2, used=19063001088, free=36120678400, active=44583346176, inactive=17277706240, buffers=64262144, cached=44684218368, shared=2182057984, slab=943849472) +2022-04-12 01:39:26.636807: +epoch: 253 +2022-04-12 01:41:18.552298: train loss : -0.6406 +2022-04-12 01:41:30.148856: validation loss: -0.6170 +2022-04-12 01:41:30.228526: Average global foreground Dice: [0.8564, 0.9579, 0.9675, 0.4157] +2022-04-12 01:41:30.233369: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:41:31.206988: lr: 0.005282 +2022-04-12 01:41:31.336547: saving scheduled checkpoint file... +2022-04-12 01:41:31.554303: saving checkpoint... +2022-04-12 01:41:39.526713: done, saving took 8.07 seconds +2022-04-12 01:41:39.542424: done +2022-04-12 01:41:39.544124: This epoch took 132.906462 s + +2022-04-12 01:41:39.545185: svmem(total=99932160000, available=76775215104, percent=23.2, used=19056558080, free=36116791296, active=44578365440, inactive=17287827456, buffers=64331776, cached=44694478848, shared=2191859712, slab=944701440) +2022-04-12 01:41:39.546539: +epoch: 254 +2022-04-12 01:43:27.133409: train loss : -0.5844 +2022-04-12 01:43:38.322727: validation loss: -0.6004 +2022-04-12 01:43:38.335618: Average global foreground Dice: [0.8376, 0.9546, 0.9716, 0.393] +2022-04-12 01:43:38.339115: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:43:39.415191: lr: 0.005262 +2022-04-12 01:43:39.443926: saving scheduled checkpoint file... +2022-04-12 01:43:39.569690: saving checkpoint... +2022-04-12 01:43:55.230233: done, saving took 15.75 seconds +2022-04-12 01:43:55.247268: done +2022-04-12 01:43:55.248593: This epoch took 135.701308 s + +2022-04-12 01:43:55.249675: svmem(total=99932160000, available=76843196416, percent=23.1, used=19011620864, free=36184338432, active=44529995776, inactive=17265422336, buffers=64376832, cached=44671823872, shared=2168975360, slab=944508928) +2022-04-12 01:43:55.251246: +epoch: 255 +2022-04-12 01:45:43.034116: train loss : -0.6194 +2022-04-12 01:45:55.337728: validation loss: -0.6572 +2022-04-12 01:45:55.346670: Average global foreground Dice: [0.8721, 0.9621, 0.9758, 0.4199] +2022-04-12 01:45:55.347763: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:45:56.151331: lr: 0.005243 +2022-04-12 01:45:56.152386: saving scheduled checkpoint file... +2022-04-12 01:45:56.273738: saving checkpoint... +2022-04-12 01:46:01.020770: done, saving took 4.87 seconds +2022-04-12 01:46:01.106633: done +2022-04-12 01:46:01.108083: This epoch took 125.856048 s + +2022-04-12 01:46:01.109617: svmem(total=99932160000, available=76884381696, percent=23.1, used=18972008448, free=36225110016, active=44496310272, inactive=17264218112, buffers=64466944, cached=44670574592, shared=2167410688, slab=943276032) +2022-04-12 01:46:01.110450: +epoch: 256 +2022-04-12 01:47:50.042483: train loss : -0.6328 +2022-04-12 01:48:02.453003: validation loss: -0.6015 +2022-04-12 01:48:02.455076: Average global foreground Dice: [0.8718, 0.9533, 0.9591, 0.4236] +2022-04-12 01:48:02.455898: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:48:03.325157: lr: 0.005224 +2022-04-12 01:48:03.327202: saving scheduled checkpoint file... +2022-04-12 01:48:03.453133: saving checkpoint... +2022-04-12 01:48:09.709128: done, saving took 6.38 seconds +2022-04-12 01:48:09.771679: done +2022-04-12 01:48:09.776295: This epoch took 128.665148 s + +2022-04-12 01:48:09.789237: svmem(total=99932160000, available=77031620608, percent=22.9, used=18883796992, free=36371804160, active=44407103488, inactive=17205518336, buffers=64512000, cached=44612046848, shared=2108313600, slab=942780416) +2022-04-12 01:48:09.791378: +epoch: 257 +2022-04-12 01:49:58.074685: train loss : -0.5856 +2022-04-12 01:50:09.661438: validation loss: -0.6172 +2022-04-12 01:50:09.690117: Average global foreground Dice: [0.8559, 0.9585, 0.9747, 0.3897] +2022-04-12 01:50:09.798544: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:50:10.896492: lr: 0.005204 +2022-04-12 01:50:10.927772: saving scheduled checkpoint file... +2022-04-12 01:50:11.074784: saving checkpoint... +2022-04-12 01:50:19.847872: done, saving took 8.89 seconds +2022-04-12 01:50:19.931380: done +2022-04-12 01:50:19.935402: This epoch took 130.137751 s + +2022-04-12 01:50:19.940931: svmem(total=99932160000, available=77068738560, percent=22.9, used=18846756864, free=36408500224, active=44371292160, inactive=17205809152, buffers=64565248, cached=44612337664, shared=2108252160, slab=942174208) +2022-04-12 01:50:19.947534: +epoch: 258 +2022-04-12 01:52:11.114528: train loss : -0.6119 +2022-04-12 01:52:23.897385: validation loss: -0.6624 +2022-04-12 01:52:23.915071: Average global foreground Dice: [0.8867, 0.9686, 0.9831, 0.4279] +2022-04-12 01:52:23.921795: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:52:24.675421: lr: 0.005185 +2022-04-12 01:52:24.699642: saving scheduled checkpoint file... +2022-04-12 01:52:24.810231: saving checkpoint... +2022-04-12 01:52:29.936641: done, saving took 5.22 seconds +2022-04-12 01:52:29.989118: done +2022-04-12 01:52:30.055194: saving checkpoint... +2022-04-12 01:52:38.132231: done, saving took 8.14 seconds +2022-04-12 01:52:38.207998: This epoch took 138.256784 s + +2022-04-12 01:52:38.212923: svmem(total=99932160000, available=77002993664, percent=22.9, used=18850217984, free=36342202368, active=44373565440, inactive=17268359168, buffers=64638976, cached=44675100672, shared=2170384384, slab=943443968) +2022-04-12 01:52:38.298784: +epoch: 259 +2022-04-12 01:54:26.419819: train loss : -0.6315 +2022-04-12 01:54:38.572697: validation loss: -0.6386 +2022-04-12 01:54:38.624684: Average global foreground Dice: [0.8643, 0.964, 0.9812, 0.3936] +2022-04-12 01:54:38.638802: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:54:39.614092: lr: 0.005166 +2022-04-12 01:54:39.680578: saving scheduled checkpoint file... +2022-04-12 01:54:39.835453: saving checkpoint... +2022-04-12 01:54:46.980951: done, saving took 7.22 seconds +2022-04-12 01:54:47.059478: done +2022-04-12 01:54:47.147956: saving checkpoint... +2022-04-12 01:54:55.979480: done, saving took 8.92 seconds +2022-04-12 01:54:56.000722: This epoch took 137.699982 s + +2022-04-12 01:54:56.003906: svmem(total=99932160000, available=76896985088, percent=23.1, used=18934321152, free=36235882496, active=44460277760, inactive=17290792960, buffers=64696320, cached=44697260032, shared=2192490496, slab=942125056) +2022-04-12 01:54:56.007245: +epoch: 260 +2022-04-12 01:56:44.659794: train loss : -0.6269 +2022-04-12 01:56:56.825523: validation loss: -0.6446 +2022-04-12 01:56:56.953298: Average global foreground Dice: [0.8509, 0.9656, 0.9728, 0.4308] +2022-04-12 01:56:57.167739: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:56:58.016994: lr: 0.005146 +2022-04-12 01:56:58.076312: saving scheduled checkpoint file... +2022-04-12 01:56:58.225248: saving checkpoint... +2022-04-12 01:57:11.945661: done, saving took 13.81 seconds +2022-04-12 01:57:12.020341: done +2022-04-12 01:57:12.126665: saving checkpoint... +2022-04-12 01:57:17.120885: done, saving took 5.10 seconds +2022-04-12 01:57:17.150229: This epoch took 141.140587 s + +2022-04-12 01:57:17.153636: svmem(total=99932160000, available=76726571008, percent=23.2, used=19115466752, free=36064997376, active=44634828800, inactive=17280200704, buffers=64782336, cached=44686913536, shared=2181554176, slab=943063040) +2022-04-12 01:57:17.171259: +epoch: 261 +2022-04-12 01:59:05.192772: train loss : -0.6323 +2022-04-12 01:59:17.146448: validation loss: -0.6564 +2022-04-12 01:59:17.420303: Average global foreground Dice: [0.8686, 0.9674, 0.982, 0.4246] +2022-04-12 01:59:17.603213: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 01:59:18.795066: lr: 0.005127 +2022-04-12 01:59:19.146288: saving scheduled checkpoint file... +2022-04-12 01:59:19.387647: saving checkpoint... +2022-04-12 01:59:26.920727: done, saving took 7.65 seconds +2022-04-12 01:59:27.033417: done +2022-04-12 01:59:27.151607: saving checkpoint... +2022-04-12 01:59:33.926268: done, saving took 6.89 seconds +2022-04-12 01:59:33.948786: This epoch took 136.776309 s + +2022-04-12 01:59:33.950386: svmem(total=99932160000, available=76900614144, percent=23.0, used=18941394944, free=36238499840, active=44461154304, inactive=17280700416, buffers=64843776, cached=44687421440, shared=2181672960, slab=941957120) +2022-04-12 01:59:33.951312: +epoch: 262 +2022-04-12 02:01:23.140153: train loss : -0.6301 +2022-04-12 02:01:35.958222: validation loss: -0.6616 +2022-04-12 02:01:35.963106: Average global foreground Dice: [0.8867, 0.9666, 0.978, 0.4066] +2022-04-12 02:01:35.964444: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:01:36.831674: lr: 0.005107 +2022-04-12 02:01:36.834543: saving scheduled checkpoint file... +2022-04-12 02:01:36.959328: saving checkpoint... +2022-04-12 02:01:45.083751: done, saving took 8.25 seconds +2022-04-12 02:01:45.290405: done +2022-04-12 02:01:45.447563: saving checkpoint... +2022-04-12 02:01:55.817754: done, saving took 10.48 seconds +2022-04-12 02:01:56.005449: This epoch took 142.053307 s + +2022-04-12 02:01:56.071616: svmem(total=99932160000, available=76615204864, percent=23.3, used=19227140096, free=35952570368, active=44747726848, inactive=17280954368, buffers=64897024, cached=44687552512, shared=2181390336, slab=943251456) +2022-04-12 02:01:56.098829: +epoch: 263 +2022-04-12 02:03:46.264660: train loss : -0.6072 +2022-04-12 02:03:59.186931: validation loss: -0.5898 +2022-04-12 02:03:59.211708: Average global foreground Dice: [0.8464, 0.954, 0.9717, 0.3822] +2022-04-12 02:03:59.270090: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:04:00.247558: lr: 0.005088 +2022-04-12 02:04:00.252285: saving scheduled checkpoint file... +2022-04-12 02:04:00.365064: saving checkpoint... +2022-04-12 02:04:07.209231: done, saving took 6.95 seconds +2022-04-12 02:04:07.290811: done +2022-04-12 02:04:07.334581: This epoch took 131.143096 s + +2022-04-12 02:04:07.396843: svmem(total=99932160000, available=76622340096, percent=23.3, used=19220209664, free=35959267328, active=44746137600, inactive=17281552384, buffers=64954368, cached=44687728640, shared=2181152768, slab=942723072) +2022-04-12 02:04:07.402029: +epoch: 264 +2022-04-12 02:05:55.595381: train loss : -0.6035 +2022-04-12 02:06:07.638160: validation loss: -0.6415 +2022-04-12 02:06:07.695264: Average global foreground Dice: [0.8575, 0.9575, 0.9803, 0.4121] +2022-04-12 02:06:07.737321: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:06:08.818169: lr: 0.005069 +2022-04-12 02:06:08.820731: saving scheduled checkpoint file... +2022-04-12 02:06:08.879166: saving checkpoint... +2022-04-12 02:06:19.203757: done, saving took 10.37 seconds +2022-04-12 02:06:19.244496: done +2022-04-12 02:06:19.256905: This epoch took 131.782642 s + +2022-04-12 02:06:19.261581: svmem(total=99932160000, available=76576444416, percent=23.4, used=19259547648, free=35912949760, active=44783267840, inactive=17288093696, buffers=65032192, cached=44694630400, shared=2187747328, slab=943337472) +2022-04-12 02:06:19.277480: +epoch: 265 +2022-04-12 02:08:05.569943: train loss : -0.6254 +2022-04-12 02:08:17.715368: validation loss: -0.6416 +2022-04-12 02:08:17.734661: Average global foreground Dice: [0.8833, 0.9584, 0.9749, 0.4078] +2022-04-12 02:08:17.739624: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:08:18.614548: lr: 0.005049 +2022-04-12 02:08:18.616505: saving scheduled checkpoint file... +2022-04-12 02:08:18.702059: saving checkpoint... +2022-04-12 02:08:31.639179: done, saving took 13.02 seconds +2022-04-12 02:08:31.716417: done +2022-04-12 02:08:31.740527: This epoch took 132.458270 s + +2022-04-12 02:08:31.745359: svmem(total=99932160000, available=76635615232, percent=23.3, used=19206537216, free=35971502080, active=44727066624, inactive=17282179072, buffers=65105920, cached=44689014784, shared=2181443584, slab=946012160) +2022-04-12 02:08:31.752023: +epoch: 266 +2022-04-12 02:10:19.284951: train loss : -0.6323 +2022-04-12 02:10:30.646948: validation loss: -0.6470 +2022-04-12 02:10:30.666032: Average global foreground Dice: [0.8508, 0.9643, 0.9798, 0.4062] +2022-04-12 02:10:30.669117: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:10:31.669193: lr: 0.00503 +2022-04-12 02:10:31.671993: saving scheduled checkpoint file... +2022-04-12 02:10:31.764443: saving checkpoint... +2022-04-12 02:10:36.488819: done, saving took 4.82 seconds +2022-04-12 02:10:36.530569: done +2022-04-12 02:10:36.533867: This epoch took 124.776534 s + +2022-04-12 02:10:36.536603: svmem(total=99932160000, available=76920193024, percent=23.0, used=18922020864, free=36255719424, active=44445863936, inactive=17282441216, buffers=65159168, cached=44689260544, shared=2181390336, slab=944111616) +2022-04-12 02:10:36.538872: +epoch: 267 +2022-04-12 02:12:25.806342: train loss : -0.6070 +2022-04-12 02:12:37.259786: validation loss: -0.6561 +2022-04-12 02:12:37.281733: Average global foreground Dice: [0.8971, 0.9647, 0.9739, 0.3846] +2022-04-12 02:12:37.296019: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:12:38.015842: lr: 0.00501 +2022-04-12 02:12:38.018591: saving scheduled checkpoint file... +2022-04-12 02:12:38.111447: saving checkpoint... +2022-04-12 02:12:45.708735: done, saving took 7.69 seconds +2022-04-12 02:12:45.812029: done +2022-04-12 02:12:45.816615: This epoch took 129.274233 s + +2022-04-12 02:12:45.902046: svmem(total=99932160000, available=76838830080, percent=23.1, used=18990260224, free=36173959168, active=44511920128, inactive=17295904768, buffers=65204224, cached=44702736384, shared=2194518016, slab=944758784) +2022-04-12 02:12:45.919863: +epoch: 268 +2022-04-12 02:14:33.936627: train loss : -0.5983 +2022-04-12 02:14:45.938893: validation loss: -0.6098 +2022-04-12 02:14:45.976759: Average global foreground Dice: [0.8501, 0.9547, 0.9655, 0.3977] +2022-04-12 02:14:46.027113: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:14:46.995378: lr: 0.004991 +2022-04-12 02:14:47.002222: saving scheduled checkpoint file... +2022-04-12 02:14:47.122600: saving checkpoint... +2022-04-12 02:14:56.655213: done, saving took 9.64 seconds +2022-04-12 02:14:56.754979: done +2022-04-12 02:14:56.758971: This epoch took 130.820790 s + +2022-04-12 02:14:56.791292: svmem(total=99932160000, available=76756746240, percent=23.2, used=19085496320, free=36091400192, active=44607270912, inactive=17284231168, buffers=65249280, cached=44690014208, shared=2181365760, slab=944898048) +2022-04-12 02:14:56.849302: +epoch: 269 +2022-04-12 02:16:46.590313: train loss : -0.5767 +2022-04-12 02:16:59.321425: validation loss: -0.5948 +2022-04-12 02:16:59.366679: Average global foreground Dice: [0.8707, 0.957, 0.9731, 0.366] +2022-04-12 02:16:59.375391: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:17:00.344816: lr: 0.004971 +2022-04-12 02:17:00.385109: saving scheduled checkpoint file... +2022-04-12 02:17:00.523287: saving checkpoint... +2022-04-12 02:17:09.173774: done, saving took 8.76 seconds +2022-04-12 02:17:09.395029: done +2022-04-12 02:17:09.434008: This epoch took 132.562431 s + +2022-04-12 02:17:09.485145: svmem(total=99932160000, available=76753387520, percent=23.2, used=19092299776, free=36087484416, active=44614934528, inactive=17280188416, buffers=65351680, cached=44687024128, shared=2177896448, slab=944320512) +2022-04-12 02:17:09.529504: +epoch: 270 +2022-04-12 02:18:57.388456: train loss : -0.6086 +2022-04-12 02:19:09.370556: validation loss: -0.6318 +2022-04-12 02:19:09.462454: Average global foreground Dice: [0.8848, 0.9578, 0.9715, 0.4225] +2022-04-12 02:19:09.554177: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:19:10.495092: lr: 0.004952 +2022-04-12 02:19:10.514089: saving scheduled checkpoint file... +2022-04-12 02:19:10.702081: saving checkpoint... +2022-04-12 02:19:22.245410: done, saving took 11.67 seconds +2022-04-12 02:19:22.279927: done +2022-04-12 02:19:22.284760: This epoch took 132.747671 s + +2022-04-12 02:19:22.315121: svmem(total=99932160000, available=76884733952, percent=23.1, used=18960859136, free=36218363904, active=44484050944, inactive=17280249856, buffers=65400832, cached=44687536128, shared=2178068480, slab=942157824) +2022-04-12 02:19:22.331489: +epoch: 271 +2022-04-12 02:21:10.984337: train loss : -0.6155 +2022-04-12 02:21:23.043148: validation loss: -0.6200 +2022-04-12 02:21:23.071534: Average global foreground Dice: [0.8566, 0.9565, 0.9714, 0.4148] +2022-04-12 02:21:23.072529: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:21:23.977762: lr: 0.004933 +2022-04-12 02:21:23.979940: saving scheduled checkpoint file... +2022-04-12 02:21:24.096457: saving checkpoint... +2022-04-12 02:21:32.452740: done, saving took 8.47 seconds +2022-04-12 02:21:32.588428: done +2022-04-12 02:21:32.610860: This epoch took 130.269828 s + +2022-04-12 02:21:32.684118: svmem(total=99932160000, available=76645126144, percent=23.3, used=19194187776, free=35978293248, active=44712947712, inactive=17287397376, buffers=65482752, cached=44694196224, shared=2184261632, slab=944181248) +2022-04-12 02:21:32.777361: +epoch: 272 +2022-04-12 02:23:22.609796: train loss : -0.6016 +2022-04-12 02:23:34.177458: validation loss: -0.6275 +2022-04-12 02:23:34.231224: Average global foreground Dice: [0.8673, 0.9535, 0.9716, 0.3918] +2022-04-12 02:23:34.237582: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:23:35.099742: lr: 0.004913 +2022-04-12 02:23:35.126588: saving scheduled checkpoint file... +2022-04-12 02:23:35.289443: saving checkpoint... +2022-04-12 02:23:41.602827: done, saving took 6.44 seconds +2022-04-12 02:23:41.829781: done +2022-04-12 02:23:41.836039: This epoch took 128.981096 s + +2022-04-12 02:23:41.839777: svmem(total=99932160000, available=76810534912, percent=23.1, used=19031797760, free=36143308800, active=44552593408, inactive=17284890624, buffers=65548288, cached=44691505152, shared=2181365760, slab=944828416) +2022-04-12 02:23:41.844361: +epoch: 273 +2022-04-12 02:25:30.054571: train loss : -0.6339 +2022-04-12 02:25:43.670219: validation loss: -0.6654 +2022-04-12 02:25:43.950953: Average global foreground Dice: [0.8814, 0.9702, 0.9776, 0.4262] +2022-04-12 02:25:44.160138: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:25:45.096548: lr: 0.004894 +2022-04-12 02:25:45.343065: saving scheduled checkpoint file... +2022-04-12 02:25:45.724472: saving checkpoint... +2022-04-12 02:25:52.964178: done, saving took 7.38 seconds +2022-04-12 02:25:53.017739: done +2022-04-12 02:25:53.148580: saving checkpoint... +2022-04-12 02:26:01.793860: done, saving took 8.77 seconds +2022-04-12 02:26:01.845948: This epoch took 139.986044 s + +2022-04-12 02:26:01.855084: svmem(total=99932160000, available=76519366656, percent=23.4, used=19323039744, free=35851694080, active=44841385984, inactive=17284947968, buffers=65626112, cached=44691800064, shared=2181169152, slab=944955392) +2022-04-12 02:26:01.871855: +epoch: 274 +2022-04-12 02:27:49.828042: train loss : -0.6283 +2022-04-12 02:28:01.741186: validation loss: -0.6410 +2022-04-12 02:28:01.758323: Average global foreground Dice: [0.8853, 0.9612, 0.9805, 0.3867] +2022-04-12 02:28:01.762009: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:28:02.829593: lr: 0.004874 +2022-04-12 02:28:02.848290: saving scheduled checkpoint file... +2022-04-12 02:28:02.993993: saving checkpoint... +2022-04-12 02:28:09.119761: done, saving took 6.27 seconds +2022-04-12 02:28:09.314089: done +2022-04-12 02:28:09.475215: saving checkpoint... +2022-04-12 02:28:16.186665: done, saving took 6.84 seconds +2022-04-12 02:28:16.272523: This epoch took 134.377619 s + +2022-04-12 02:28:16.297720: svmem(total=99932160000, available=76915531776, percent=23.0, used=18926592000, free=36247384064, active=44452786176, inactive=17285713920, buffers=65683456, cached=44692500480, shared=2181476352, slab=943767552) +2022-04-12 02:28:16.299132: +epoch: 275 +2022-04-12 02:30:06.300469: train loss : -0.6318 +2022-04-12 02:30:18.751838: validation loss: -0.6467 +2022-04-12 02:30:18.754642: Average global foreground Dice: [0.8634, 0.9631, 0.9807, 0.4357] +2022-04-12 02:30:18.756255: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:30:19.775915: lr: 0.004855 +2022-04-12 02:30:19.777403: saving scheduled checkpoint file... +2022-04-12 02:30:19.872713: saving checkpoint... +2022-04-12 02:30:27.643164: done, saving took 7.86 seconds +2022-04-12 02:30:27.680391: done +2022-04-12 02:30:27.758728: saving checkpoint... +2022-04-12 02:30:37.244215: done, saving took 9.54 seconds +2022-04-12 02:30:37.298159: This epoch took 140.994444 s + +2022-04-12 02:30:37.309173: svmem(total=99932160000, available=76641091584, percent=23.3, used=19201052672, free=35972407296, active=44728254464, inactive=17286131712, buffers=65753088, cached=44692946944, shared=2181443584, slab=943570944) +2022-04-12 02:30:37.310310: +epoch: 276 +2022-04-12 02:32:25.087130: train loss : -0.6443 +2022-04-12 02:32:37.077879: validation loss: -0.5851 +2022-04-12 02:32:37.152292: Average global foreground Dice: [0.8728, 0.9545, 0.9619, 0.4014] +2022-04-12 02:32:37.234500: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:32:38.141310: lr: 0.004835 +2022-04-12 02:32:38.192484: saving scheduled checkpoint file... +2022-04-12 02:32:38.260532: saving checkpoint... +2022-04-12 02:32:46.976587: done, saving took 8.78 seconds +2022-04-12 02:32:47.021660: done +2022-04-12 02:32:47.071287: This epoch took 129.759915 s + +2022-04-12 02:32:47.075260: svmem(total=99932160000, available=76701155328, percent=23.2, used=19140861952, free=36032069632, active=44662374400, inactive=17287036928, buffers=65810432, cached=44693417984, shared=2181562368, slab=944467968) +2022-04-12 02:32:47.086124: +epoch: 277 +2022-04-12 02:34:33.997485: train loss : -0.6429 +2022-04-12 02:34:45.321949: validation loss: -0.6730 +2022-04-12 02:34:45.346247: Average global foreground Dice: [0.8683, 0.9686, 0.9812, 0.4226] +2022-04-12 02:34:45.348081: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:34:46.440232: lr: 0.004816 +2022-04-12 02:34:46.463139: saving scheduled checkpoint file... +2022-04-12 02:34:46.619485: saving checkpoint... +2022-04-12 02:34:56.998444: done, saving took 10.46 seconds +2022-04-12 02:34:57.063816: done +2022-04-12 02:34:57.136510: saving checkpoint... +2022-04-12 02:35:02.623149: done, saving took 5.56 seconds +2022-04-12 02:35:02.744177: This epoch took 135.583723 s + +2022-04-12 02:35:02.786361: svmem(total=99932160000, available=76714795008, percent=23.2, used=19127283712, free=36045246464, active=44652249088, inactive=17286823936, buffers=65863680, cached=44693766144, shared=2181451776, slab=942665728) +2022-04-12 02:35:02.788881: +epoch: 278 +2022-04-12 02:36:51.744294: train loss : -0.6335 +2022-04-12 02:37:03.660681: validation loss: -0.6117 +2022-04-12 02:37:03.678162: Average global foreground Dice: [0.8913, 0.9519, 0.9617, 0.4305] +2022-04-12 02:37:03.719099: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:37:04.780017: lr: 0.004796 +2022-04-12 02:37:04.787401: saving scheduled checkpoint file... +2022-04-12 02:37:04.902468: saving checkpoint... +2022-04-12 02:37:10.821625: done, saving took 6.03 seconds +2022-04-12 02:37:10.839838: done +2022-04-12 02:37:10.924418: saving checkpoint... +2022-04-12 02:37:19.244806: done, saving took 8.40 seconds +2022-04-12 02:37:19.460886: This epoch took 136.667365 s + +2022-04-12 02:37:19.587910: svmem(total=99932160000, available=76744130560, percent=23.2, used=19093245952, free=36074106880, active=44613472256, inactive=17292091392, buffers=65929216, cached=44698877952, shared=2186235904, slab=943611904) +2022-04-12 02:37:19.626964: +epoch: 279 +2022-04-12 02:39:07.397555: train loss : -0.6338 +2022-04-12 02:39:17.990190: validation loss: -0.6733 +2022-04-12 02:39:18.077267: Average global foreground Dice: [0.8712, 0.9621, 0.983, 0.4247] +2022-04-12 02:39:18.080836: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:39:18.906037: lr: 0.004776 +2022-04-12 02:39:18.908214: saving scheduled checkpoint file... +2022-04-12 02:39:19.040990: saving checkpoint... +2022-04-12 02:39:27.961091: done, saving took 9.05 seconds +2022-04-12 02:39:28.043425: done +2022-04-12 02:39:28.148598: saving checkpoint... +2022-04-12 02:39:34.441371: done, saving took 6.40 seconds +2022-04-12 02:39:34.611315: This epoch took 134.920557 s + +2022-04-12 02:39:34.673569: svmem(total=99932160000, available=76734332928, percent=23.2, used=19104440320, free=36063821824, active=44633051136, inactive=17291091968, buffers=66002944, cached=44697894912, shared=2184818688, slab=942276608) +2022-04-12 02:39:34.699548: +epoch: 280 +2022-04-12 02:41:23.702277: train loss : -0.6590 +2022-04-12 02:41:35.897764: validation loss: -0.6693 +2022-04-12 02:41:35.923052: Average global foreground Dice: [0.8811, 0.9669, 0.9793, 0.4241] +2022-04-12 02:41:35.941381: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:41:36.815032: lr: 0.004757 +2022-04-12 02:41:36.861753: saving scheduled checkpoint file... +2022-04-12 02:41:37.022617: saving checkpoint... +2022-04-12 02:41:50.948914: done, saving took 14.06 seconds +2022-04-12 02:41:51.185622: done +2022-04-12 02:41:51.366485: saving checkpoint... +2022-04-12 02:42:04.479573: done, saving took 13.23 seconds +2022-04-12 02:42:04.709529: This epoch took 149.997227 s + +2022-04-12 02:42:04.770767: svmem(total=99932160000, available=76657958912, percent=23.3, used=19183906816, free=35986980864, active=44707614720, inactive=17288667136, buffers=66056192, cached=44695216128, shared=2181656576, slab=942620672) +2022-04-12 02:42:04.777825: +epoch: 281 +2022-04-12 02:43:53.419739: train loss : -0.6486 +2022-04-12 02:44:06.556729: validation loss: -0.6498 +2022-04-12 02:44:06.709311: Average global foreground Dice: [0.862, 0.9645, 0.9743, 0.427] +2022-04-12 02:44:06.793394: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:44:07.753308: lr: 0.004737 +2022-04-12 02:44:07.822544: saving scheduled checkpoint file... +2022-04-12 02:44:07.999138: saving checkpoint... +2022-04-12 02:44:19.554986: done, saving took 11.65 seconds +2022-04-12 02:44:19.816006: done +2022-04-12 02:44:19.887389: saving checkpoint... +2022-04-12 02:44:28.892661: done, saving took 9.05 seconds +2022-04-12 02:44:29.116528: This epoch took 144.337172 s + +2022-04-12 02:44:29.121643: svmem(total=99932160000, available=76791152640, percent=23.2, used=19064680448, free=36119617536, active=44584939520, inactive=17274712064, buffers=66129920, cached=44681732096, shared=2167676928, slab=943833088) +2022-04-12 02:44:29.197446: +epoch: 282 +2022-04-12 02:47:15.266811: train loss : -0.6437 +2022-04-12 02:49:35.098403: validation loss: -0.6220 +2022-04-12 02:49:36.238865: Average global foreground Dice: [0.8773, 0.9599, 0.9637, 0.447] +2022-04-12 02:49:36.668017: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:49:38.553023: lr: 0.004718 +2022-04-12 02:49:38.857841: saving scheduled checkpoint file... +2022-04-12 02:49:39.364491: saving checkpoint... +2022-04-12 02:49:46.524969: done, saving took 7.24 seconds +2022-04-12 02:49:46.863769: done +2022-04-12 02:49:47.059331: saving checkpoint... +2022-04-12 02:50:00.306075: done, saving took 13.34 seconds +2022-04-12 02:50:00.839125: This epoch took 331.565346 s + +2022-04-12 02:50:01.150081: svmem(total=99932160000, available=76884996096, percent=23.1, used=19250753536, free=65263423488, active=21494181888, inactive=11089223680, buffers=9986048, cached=15407996928, shared=2001289216, slab=740835328) +2022-04-12 02:50:01.472266: +epoch: 283 +2022-04-12 02:54:18.778905: train loss : -0.6466 +2022-04-12 02:54:39.266370: validation loss: -0.6494 +2022-04-12 02:54:39.456508: Average global foreground Dice: [0.8824, 0.962, 0.9769, 0.4066] +2022-04-12 02:54:39.660966: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:54:40.660843: lr: 0.004698 +2022-04-12 02:54:40.844174: saving scheduled checkpoint file... +2022-04-12 02:54:41.037805: saving checkpoint... +2022-04-12 02:54:50.958006: done, saving took 10.02 seconds +2022-04-12 02:54:51.173179: done +2022-04-12 02:54:51.387313: saving checkpoint... +2022-04-12 02:54:59.767072: done, saving took 8.45 seconds +2022-04-12 02:55:00.527280: This epoch took 298.761856 s + +2022-04-12 02:55:00.750721: svmem(total=99932160000, available=76601171968, percent=23.3, used=19543474176, free=54682136576, active=31331643392, inactive=11811102720, buffers=10477568, cached=25696071680, shared=1980088320, slab=763338752) +2022-04-12 02:55:00.920665: +epoch: 284 +2022-04-12 02:56:53.081409: train loss : -0.6456 +2022-04-12 02:57:04.568397: validation loss: -0.6343 +2022-04-12 02:57:05.184613: Average global foreground Dice: [0.8653, 0.9578, 0.9703, 0.411] +2022-04-12 02:57:05.558770: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:57:06.576223: lr: 0.004679 +2022-04-12 02:57:06.700625: saving scheduled checkpoint file... +2022-04-12 02:57:07.100899: saving checkpoint... +2022-04-12 02:57:22.041878: done, saving took 15.04 seconds +2022-04-12 02:57:22.286301: done +2022-04-12 02:57:22.371346: This epoch took 141.253835 s + +2022-04-12 02:57:22.462563: svmem(total=99932160000, available=76629053440, percent=23.3, used=19517292544, free=52041342976, active=36013559808, inactive=9763774464, buffers=10698752, cached=28362825728, shared=1976700928, slab=769187840) +2022-04-12 02:57:22.555461: +epoch: 285 +2022-04-12 02:59:11.627276: train loss : -0.6704 +2022-04-12 02:59:23.289956: validation loss: -0.6735 +2022-04-12 02:59:23.468471: Average global foreground Dice: [0.9015, 0.9665, 0.9755, 0.4072] +2022-04-12 02:59:23.472526: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 02:59:24.354140: lr: 0.004659 +2022-04-12 02:59:24.421104: saving scheduled checkpoint file... +2022-04-12 02:59:24.606392: saving checkpoint... +2022-04-12 02:59:33.588124: done, saving took 9.10 seconds +2022-04-12 02:59:33.933967: done +2022-04-12 02:59:34.158814: saving checkpoint... +2022-04-12 02:59:42.129774: done, saving took 8.08 seconds +2022-04-12 02:59:42.173545: This epoch took 139.542753 s + +2022-04-12 02:59:42.178760: svmem(total=99932160000, available=75954073600, percent=24.0, used=19905875968, free=49786232832, active=39534088192, inactive=8486944768, buffers=10883072, cached=30229168128, shared=2259980288, slab=774078464) +2022-04-12 02:59:42.183998: +epoch: 286 +2022-04-12 03:01:31.938221: train loss : -0.6653 +2022-04-12 03:01:44.110499: validation loss: -0.6651 +2022-04-12 03:01:44.132525: Average global foreground Dice: [0.8876, 0.9688, 0.981, 0.4133] +2022-04-12 03:01:44.134704: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:01:45.037420: lr: 0.004639 +2022-04-12 03:01:45.040853: saving scheduled checkpoint file... +2022-04-12 03:01:45.167248: saving checkpoint... +2022-04-12 03:01:51.486804: done, saving took 6.44 seconds +2022-04-12 03:01:51.645955: done +2022-04-12 03:01:51.765641: saving checkpoint... +2022-04-12 03:02:00.955252: done, saving took 9.30 seconds +2022-04-12 03:02:01.223527: This epoch took 139.034998 s + +2022-04-12 03:02:01.290924: svmem(total=99932160000, available=75934158848, percent=24.0, used=19925487616, free=48977014784, active=41212735488, inactive=7619538944, buffers=11042816, cached=31018614784, shared=2259881984, slab=774434816) +2022-04-12 03:02:01.351671: +epoch: 287 +2022-04-12 03:03:49.589810: train loss : -0.6337 +2022-04-12 03:04:01.547553: validation loss: -0.5689 +2022-04-12 03:04:01.558150: Average global foreground Dice: [0.8666, 0.9501, 0.9617, 0.3879] +2022-04-12 03:04:01.609215: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:04:02.620304: lr: 0.00462 +2022-04-12 03:04:02.634914: saving scheduled checkpoint file... +2022-04-12 03:04:02.749209: saving checkpoint... +2022-04-12 03:04:09.089604: done, saving took 6.44 seconds +2022-04-12 03:04:09.160842: done +2022-04-12 03:04:09.200104: This epoch took 127.846052 s + +2022-04-12 03:04:09.244999: svmem(total=99932160000, available=76033617920, percent=23.9, used=19825242112, free=48760676352, active=42066046976, inactive=6979579904, buffers=11210752, cached=31335030784, shared=2260889600, slab=774262784) +2022-04-12 03:04:09.246750: +epoch: 288 +2022-04-12 03:05:57.546199: train loss : -0.6214 +2022-04-12 03:06:09.465097: validation loss: -0.6138 +2022-04-12 03:06:09.504225: Average global foreground Dice: [0.8699, 0.9568, 0.9697, 0.3971] +2022-04-12 03:06:09.514468: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:06:10.456154: lr: 0.0046 +2022-04-12 03:06:10.512526: saving scheduled checkpoint file... +2022-04-12 03:06:10.604415: saving checkpoint... +2022-04-12 03:06:24.104128: done, saving took 13.59 seconds +2022-04-12 03:06:24.277104: done +2022-04-12 03:06:24.330595: This epoch took 135.079963 s + +2022-04-12 03:06:24.339310: svmem(total=99932160000, available=75845599232, percent=24.1, used=20012679168, free=48420179968, active=42927996928, inactive=6461063168, buffers=11464704, cached=31487836160, shared=2260705280, slab=776110080) +2022-04-12 03:06:24.392950: +epoch: 289 +2022-04-12 03:08:14.127527: train loss : -0.6206 +2022-04-12 03:08:25.413486: validation loss: -0.6284 +2022-04-12 03:08:25.477393: Average global foreground Dice: [0.8481, 0.9623, 0.9745, 0.4256] +2022-04-12 03:08:25.493589: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:08:26.392180: lr: 0.004581 +2022-04-12 03:08:26.397232: saving scheduled checkpoint file... +2022-04-12 03:08:26.547700: saving checkpoint... +2022-04-12 03:08:37.815324: done, saving took 11.33 seconds +2022-04-12 03:08:37.959164: done +2022-04-12 03:08:38.230678: This epoch took 133.810928 s + +2022-04-12 03:08:38.236425: svmem(total=99932160000, available=75813654528, percent=24.1, used=20044468224, free=48281255936, active=43303243776, inactive=6225842176, buffers=11730944, cached=31594704896, shared=2260803584, slab=775041024) +2022-04-12 03:08:38.241933: +epoch: 290 +2022-04-12 03:10:26.113810: train loss : -0.6293 +2022-04-12 03:10:37.040524: validation loss: -0.6357 +2022-04-12 03:10:37.081014: Average global foreground Dice: [0.876, 0.9668, 0.9769, 0.4265] +2022-04-12 03:10:37.104088: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:10:38.467708: lr: 0.004561 +2022-04-12 03:10:38.548242: saving scheduled checkpoint file... +2022-04-12 03:10:38.715800: saving checkpoint... +2022-04-12 03:10:56.971791: done, saving took 18.34 seconds +2022-04-12 03:10:57.050260: done +2022-04-12 03:10:57.087094: This epoch took 138.842851 s + +2022-04-12 03:10:57.111317: svmem(total=99932160000, available=75929444352, percent=24.0, used=19917332480, free=48184385536, active=42993594368, inactive=6625644544, buffers=12009472, cached=31818432512, shared=2271531008, slab=776192000) +2022-04-12 03:10:57.115330: +epoch: 291 +2022-04-12 03:12:44.628912: train loss : -0.6424 +2022-04-12 03:12:55.328509: validation loss: -0.5837 +2022-04-12 03:12:55.349520: Average global foreground Dice: [0.8915, 0.949, 0.9592, 0.4194] +2022-04-12 03:12:55.353858: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:12:56.324430: lr: 0.004541 +2022-04-12 03:12:56.328297: saving scheduled checkpoint file... +2022-04-12 03:12:56.437374: saving checkpoint... +2022-04-12 03:13:04.457686: done, saving took 8.13 seconds +2022-04-12 03:13:04.555638: done +2022-04-12 03:13:04.560683: This epoch took 127.429820 s + +2022-04-12 03:13:04.562515: svmem(total=99932160000, available=75972214784, percent=24.0, used=19882930176, free=48091312128, active=43151392768, inactive=6558281728, buffers=12242944, cached=31945674752, shared=2263654400, slab=777986048) +2022-04-12 03:13:04.579260: +epoch: 292 +2022-04-12 03:14:51.027641: train loss : -0.6284 +2022-04-12 03:15:02.078153: validation loss: -0.5975 +2022-04-12 03:15:02.141506: Average global foreground Dice: [0.8514, 0.9517, 0.9756, 0.3878] +2022-04-12 03:15:02.144017: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:15:03.091130: lr: 0.004522 +2022-04-12 03:15:03.143904: saving scheduled checkpoint file... +2022-04-12 03:15:03.268114: saving checkpoint... +2022-04-12 03:15:09.677135: done, saving took 6.53 seconds +2022-04-12 03:15:09.838724: done +2022-04-12 03:15:09.845858: This epoch took 125.240711 s + +2022-04-12 03:15:09.885297: svmem(total=99932160000, available=75955683328, percent=24.0, used=19902709760, free=48131985408, active=43236167680, inactive=6440022016, buffers=12496896, cached=31884967936, shared=2259574784, slab=777494528) +2022-04-12 03:15:09.927178: +epoch: 293 +2022-04-12 03:16:55.871760: train loss : -0.6331 +2022-04-12 03:17:08.136095: validation loss: -0.6367 +2022-04-12 03:17:08.139060: Average global foreground Dice: [0.8923, 0.9557, 0.9726, 0.4178] +2022-04-12 03:17:08.140048: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:17:09.335702: lr: 0.004502 +2022-04-12 03:17:09.338296: saving scheduled checkpoint file... +2022-04-12 03:17:09.443430: saving checkpoint... +2022-04-12 03:17:22.406420: done, saving took 13.06 seconds +2022-04-12 03:17:22.433585: done +2022-04-12 03:17:22.434633: This epoch took 132.493130 s + +2022-04-12 03:17:22.441224: svmem(total=99932160000, available=76109676544, percent=23.8, used=19742375936, free=48239886336, active=43164409856, inactive=6403694592, buffers=12759040, cached=31937138688, shared=2265993216, slab=775294976) +2022-04-12 03:17:22.442541: +epoch: 294 +2022-04-12 03:19:10.940057: train loss : -0.6359 +2022-04-12 03:19:23.384426: validation loss: -0.6321 +2022-04-12 03:19:23.401727: Average global foreground Dice: [0.869, 0.9595, 0.9743, 0.4029] +2022-04-12 03:19:23.471570: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:19:24.288198: lr: 0.004482 +2022-04-12 03:19:24.295245: saving scheduled checkpoint file... +2022-04-12 03:19:24.418355: saving checkpoint... +2022-04-12 03:19:34.254334: done, saving took 9.92 seconds +2022-04-12 03:19:34.752182: done +2022-04-12 03:19:34.815584: This epoch took 132.371887 s + +2022-04-12 03:19:34.830091: svmem(total=99932160000, available=76065501184, percent=23.9, used=19789901824, free=48148762624, active=43284828160, inactive=6373928960, buffers=13025280, cached=31980470272, shared=2261106688, slab=777187328) +2022-04-12 03:19:34.924347: +epoch: 295 +2022-04-12 03:21:23.556257: train loss : -0.6452 +2022-04-12 03:21:35.306305: validation loss: -0.5995 +2022-04-12 03:21:35.356630: Average global foreground Dice: [0.8775, 0.9506, 0.9589, 0.418] +2022-04-12 03:21:35.380790: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:21:36.405495: lr: 0.004463 +2022-04-12 03:21:36.565438: saving scheduled checkpoint file... +2022-04-12 03:21:36.727947: saving checkpoint... +2022-04-12 03:21:53.694665: done, saving took 17.05 seconds +2022-04-12 03:21:53.827448: done +2022-04-12 03:21:53.834094: This epoch took 138.726382 s + +2022-04-12 03:21:53.894736: svmem(total=99932160000, available=76139196416, percent=23.8, used=19711836160, free=48184967168, active=43252703232, inactive=6370762752, buffers=13287424, cached=32022069248, shared=2266959872, slab=777666560) +2022-04-12 03:21:53.954763: +epoch: 296 +2022-04-12 03:23:42.229277: train loss : -0.6400 +2022-04-12 03:23:53.054593: validation loss: -0.5767 +2022-04-12 03:23:53.209049: Average global foreground Dice: [0.8739, 0.9527, 0.9592, 0.419] +2022-04-12 03:23:53.214974: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:23:54.354729: lr: 0.004443 +2022-04-12 03:23:54.360777: saving scheduled checkpoint file... +2022-04-12 03:23:54.500728: saving checkpoint... +2022-04-12 03:24:05.725168: done, saving took 11.35 seconds +2022-04-12 03:24:05.875754: done +2022-04-12 03:24:05.907326: This epoch took 131.946219 s + +2022-04-12 03:24:05.916254: svmem(total=99932160000, available=76007047168, percent=23.9, used=19849617408, free=48031137792, active=43435032576, inactive=6334636032, buffers=13533184, cached=32037871616, shared=2260951040, slab=776843264) +2022-04-12 03:24:05.940827: +epoch: 297 +2022-04-12 03:25:53.010609: train loss : -0.6520 +2022-04-12 03:26:04.735868: validation loss: -0.6721 +2022-04-12 03:26:04.760981: Average global foreground Dice: [0.8805, 0.9661, 0.9792, 0.4294] +2022-04-12 03:26:04.763789: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:26:05.835217: lr: 0.004423 +2022-04-12 03:26:05.899360: saving scheduled checkpoint file... +2022-04-12 03:26:06.140442: saving checkpoint... +2022-04-12 03:26:11.624501: done, saving took 5.71 seconds +2022-04-12 03:26:11.646261: done +2022-04-12 03:26:11.647573: This epoch took 125.664218 s + +2022-04-12 03:26:11.648494: svmem(total=99932160000, available=76048138240, percent=23.9, used=19798421504, free=48055431168, active=43420119040, inactive=6332477440, buffers=13795328, cached=32064512000, shared=2271420416, slab=776912896) +2022-04-12 03:26:11.649200: +epoch: 298 +2022-04-12 03:28:00.761143: train loss : -0.6483 +2022-04-12 03:28:11.882485: validation loss: -0.6483 +2022-04-12 03:28:12.339311: Average global foreground Dice: [0.8804, 0.9604, 0.9687, 0.4005] +2022-04-12 03:28:12.568092: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:28:13.814774: lr: 0.004404 +2022-04-12 03:28:13.829570: saving scheduled checkpoint file... +2022-04-12 03:28:14.008834: saving checkpoint... +2022-04-12 03:28:21.989455: done, saving took 8.10 seconds +2022-04-12 03:28:22.105726: done +2022-04-12 03:28:22.162945: This epoch took 130.513072 s + +2022-04-12 03:28:22.167423: svmem(total=99932160000, available=76027305984, percent=23.9, used=19829686272, free=48024551424, active=43478155264, inactive=6305112064, buffers=14041088, cached=32063881216, shared=2260865024, slab=776847360) +2022-04-12 03:28:22.171265: +epoch: 299 +2022-04-12 03:30:10.522789: train loss : -0.6542 +2022-04-12 03:30:22.616546: validation loss: -0.6445 +2022-04-12 03:30:23.012022: Average global foreground Dice: [0.8928, 0.9627, 0.9695, 0.4018] +2022-04-12 03:30:23.418779: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:30:24.611412: lr: 0.004384 +2022-04-12 03:30:24.616149: saving scheduled checkpoint file... +2022-04-12 03:30:24.708005: saving checkpoint... +2022-04-12 03:30:37.755029: done, saving took 13.13 seconds +2022-04-12 03:30:37.882527: done +2022-04-12 03:30:37.926895: This epoch took 135.752957 s + +2022-04-12 03:30:37.936254: svmem(total=99932160000, available=76012916736, percent=23.9, used=19843969024, free=47990951936, active=43514290176, inactive=6295351296, buffers=14307328, cached=32082931712, shared=2260922368, slab=777662464) +2022-04-12 03:30:37.970912: +epoch: 300 +2022-04-12 03:32:27.445344: train loss : -0.6655 +2022-04-12 03:32:38.773080: validation loss: -0.6775 +2022-04-12 03:32:38.808747: Average global foreground Dice: [0.9047, 0.9677, 0.9815, 0.4129] +2022-04-12 03:32:38.812598: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:32:39.887372: lr: 0.004364 +2022-04-12 03:32:39.902395: saving scheduled checkpoint file... +2022-04-12 03:32:40.030446: saving checkpoint... +2022-04-12 03:32:48.124916: done, saving took 8.18 seconds +2022-04-12 03:32:48.263354: done +2022-04-12 03:32:48.270786: This epoch took 130.288135 s + +2022-04-12 03:32:48.272721: svmem(total=99932160000, available=76066459648, percent=23.9, used=19804499968, free=48047079424, active=43492913152, inactive=6265356288, buffers=14540800, cached=32066039808, shared=2247049216, slab=775741440) +2022-04-12 03:32:48.275406: +epoch: 301 +2022-04-12 03:34:35.353979: train loss : -0.6552 +2022-04-12 03:34:47.350765: validation loss: -0.5604 +2022-04-12 03:34:47.811173: Average global foreground Dice: [0.8826, 0.9576, 0.955, 0.3797] +2022-04-12 03:34:48.241461: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:34:49.796871: lr: 0.004344 +2022-04-12 03:34:50.286202: saving scheduled checkpoint file... +2022-04-12 03:34:50.734099: saving checkpoint... +2022-04-12 03:35:02.561511: done, saving took 11.87 seconds +2022-04-12 03:35:02.640161: done +2022-04-12 03:35:02.646536: This epoch took 134.368388 s + +2022-04-12 03:35:02.693003: svmem(total=99932160000, available=75988135936, percent=24.0, used=19858350080, free=47989985280, active=43527684096, inactive=6291025920, buffers=14794752, cached=32069029888, shared=2271334400, slab=776200192) +2022-04-12 03:35:02.748819: +epoch: 302 +2022-04-12 03:36:50.033747: train loss : -0.6626 +2022-04-12 03:37:01.825317: validation loss: -0.6251 +2022-04-12 03:37:01.957595: Average global foreground Dice: [0.899, 0.9573, 0.9672, 0.4263] +2022-04-12 03:37:02.001372: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:37:02.911274: lr: 0.004325 +2022-04-12 03:37:02.936379: saving scheduled checkpoint file... +2022-04-12 03:37:03.019791: saving checkpoint... +2022-04-12 03:37:11.960599: done, saving took 9.01 seconds +2022-04-12 03:37:12.133593: done +2022-04-12 03:37:12.137733: This epoch took 129.364197 s + +2022-04-12 03:37:12.192891: svmem(total=99932160000, available=75950366720, percent=24.0, used=19904483328, free=47949549568, active=43580301312, inactive=6279200768, buffers=15052800, cached=32063074304, shared=2262913024, slab=775233536) +2022-04-12 03:37:12.209892: +epoch: 303 +2022-04-12 03:38:59.951105: train loss : -0.6640 +2022-04-12 03:39:11.111186: validation loss: -0.6803 +2022-04-12 03:39:11.511251: Average global foreground Dice: [0.8896, 0.9629, 0.979, 0.4188] +2022-04-12 03:39:11.881349: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:39:13.553766: lr: 0.004305 +2022-04-12 03:39:13.911612: saving scheduled checkpoint file... +2022-04-12 03:39:14.359589: saving checkpoint... +2022-04-12 03:39:22.678642: done, saving took 8.41 seconds +2022-04-12 03:39:22.832423: done +2022-04-12 03:39:22.844258: This epoch took 130.581433 s + +2022-04-12 03:39:22.905586: svmem(total=99932160000, available=75927166976, percent=24.0, used=19929825280, free=47919443968, active=43611631616, inactive=6275416064, buffers=15298560, cached=32067592192, shared=2260840448, slab=775229440) +2022-04-12 03:39:22.913258: +epoch: 304 +2022-04-12 03:41:10.921521: train loss : -0.6690 +2022-04-12 03:41:21.916021: validation loss: -0.6686 +2022-04-12 03:41:23.348652: Average global foreground Dice: [0.8868, 0.9678, 0.9801, 0.4007] +2022-04-12 03:41:24.090859: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:41:26.207094: lr: 0.004285 +2022-04-12 03:41:26.976164: saving scheduled checkpoint file... +2022-04-12 03:41:27.697530: saving checkpoint... +2022-04-12 03:41:36.974995: done, saving took 9.38 seconds +2022-04-12 03:41:37.066235: done +2022-04-12 03:41:37.116699: This epoch took 134.200888 s + +2022-04-12 03:41:37.123158: svmem(total=99932160000, available=76040396800, percent=23.9, used=19805876224, free=48029028352, active=43493146624, inactive=6286020608, buffers=15597568, cached=32081657856, shared=2271358976, slab=776540160) +2022-04-12 03:41:37.188343: +epoch: 305 +2022-04-12 03:43:25.721879: train loss : -0.6532 +2022-04-12 03:43:38.012273: validation loss: -0.6135 +2022-04-12 03:43:38.523007: Average global foreground Dice: [0.8976, 0.9609, 0.9666, 0.3996] +2022-04-12 03:43:38.846750: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:43:39.957432: lr: 0.004265 +2022-04-12 03:43:40.210797: saving scheduled checkpoint file... +2022-04-12 03:43:40.522429: saving checkpoint... +2022-04-12 03:43:50.311992: done, saving took 9.89 seconds +2022-04-12 03:43:50.521806: done +2022-04-12 03:43:50.549134: This epoch took 133.356902 s + +2022-04-12 03:43:50.650113: svmem(total=99932160000, available=76170670080, percent=23.8, used=19685892096, free=48156524544, active=43377000448, inactive=6275108864, buffers=15851520, cached=32073891840, shared=2260951040, slab=776953856) +2022-04-12 03:43:50.732050: +epoch: 306 +2022-04-12 03:45:39.114276: train loss : -0.6478 +2022-04-12 03:45:50.658880: validation loss: -0.6679 +2022-04-12 03:45:51.362966: Average global foreground Dice: [0.8956, 0.9673, 0.9799, 0.3903] +2022-04-12 03:45:51.931643: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:45:53.955218: lr: 0.004245 +2022-04-12 03:45:54.844742: saving scheduled checkpoint file... +2022-04-12 03:45:55.646783: saving checkpoint... +2022-04-12 03:46:08.740692: done, saving took 13.18 seconds +2022-04-12 03:46:08.830222: done +2022-04-12 03:46:08.888731: This epoch took 138.028737 s + +2022-04-12 03:46:08.892436: svmem(total=99932160000, available=76153364480, percent=23.8, used=19715141632, free=48152612864, active=43392884736, inactive=6259322880, buffers=16138240, cached=32048267264, shared=2249064448, slab=777199616) +2022-04-12 03:46:08.911168: +epoch: 307 +2022-04-12 03:47:58.065584: train loss : -0.6446 +2022-04-12 03:48:09.877141: validation loss: -0.6195 +2022-04-12 03:48:10.027693: Average global foreground Dice: [0.8677, 0.9598, 0.9654, 0.4022] +2022-04-12 03:48:10.257308: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:48:11.826822: lr: 0.004226 +2022-04-12 03:48:12.209766: saving scheduled checkpoint file... +2022-04-12 03:48:12.648376: saving checkpoint... +2022-04-12 03:48:29.003309: done, saving took 16.46 seconds +2022-04-12 03:48:29.082565: done +2022-04-12 03:48:29.088245: This epoch took 140.130164 s + +2022-04-12 03:48:29.091156: svmem(total=99932160000, available=75867385856, percent=24.1, used=19989598208, free=47880118272, active=43649044480, inactive=6272733184, buffers=16412672, cached=32046030848, shared=2260508672, slab=778567680) +2022-04-12 03:48:29.093525: +epoch: 308 +2022-04-12 03:50:18.073313: train loss : -0.6488 +2022-04-12 03:50:29.242021: validation loss: -0.6546 +2022-04-12 03:50:29.827133: Average global foreground Dice: [0.8982, 0.9651, 0.9807, 0.4036] +2022-04-12 03:50:30.389128: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:50:32.184930: lr: 0.004206 +2022-04-12 03:50:32.650139: saving scheduled checkpoint file... +2022-04-12 03:50:33.303377: saving checkpoint... +2022-04-12 03:50:46.757590: done, saving took 13.58 seconds +2022-04-12 03:50:46.847160: done +2022-04-12 03:50:46.849428: This epoch took 137.753565 s + +2022-04-12 03:50:46.852847: svmem(total=99932160000, available=76179238912, percent=23.8, used=19677114368, free=48205709312, active=43323494400, inactive=6271959040, buffers=16658432, cached=32032677888, shared=2261139456, slab=779161600) +2022-04-12 03:50:46.856894: +epoch: 309 +2022-04-12 03:52:36.179174: train loss : -0.6565 +2022-04-12 03:52:47.821751: validation loss: -0.6333 +2022-04-12 03:52:47.903508: Average global foreground Dice: [0.8839, 0.9602, 0.9714, 0.4041] +2022-04-12 03:52:47.973301: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:52:49.306123: lr: 0.004186 +2022-04-12 03:52:49.810942: saving scheduled checkpoint file... +2022-04-12 03:52:50.381445: saving checkpoint... +2022-04-12 03:53:08.370841: done, saving took 18.12 seconds +2022-04-12 03:53:08.540555: done +2022-04-12 03:53:08.547927: This epoch took 141.666095 s + +2022-04-12 03:53:08.570626: svmem(total=99932160000, available=76182695936, percent=23.8, used=19673739264, free=48206340096, active=43324796928, inactive=6272831488, buffers=16916480, cached=32035164160, shared=2260942848, slab=776613888) +2022-04-12 03:53:08.594875: +epoch: 310 +2022-04-12 03:54:58.335818: train loss : -0.6767 +2022-04-12 03:55:10.848824: validation loss: -0.6574 +2022-04-12 03:55:11.239914: Average global foreground Dice: [0.8648, 0.9651, 0.9798, 0.4153] +2022-04-12 03:55:12.078438: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:55:13.686284: lr: 0.004166 +2022-04-12 03:55:13.945891: saving scheduled checkpoint file... +2022-04-12 03:55:14.348815: saving checkpoint... +2022-04-12 03:55:28.701863: done, saving took 14.47 seconds +2022-04-12 03:55:29.057880: done +2022-04-12 03:55:29.136566: This epoch took 140.539780 s + +2022-04-12 03:55:29.144669: svmem(total=99932160000, available=76158447616, percent=23.8, used=19706757120, free=48178249728, active=43365535744, inactive=6260424704, buffers=17223680, cached=32029929472, shared=2252029952, slab=777175040) +2022-04-12 03:55:29.208804: +epoch: 311 +2022-04-12 03:57:18.901769: train loss : -0.6738 +2022-04-12 03:57:31.561523: validation loss: -0.6788 +2022-04-12 03:57:32.328915: Average global foreground Dice: [0.9082, 0.9709, 0.982, 0.4099] +2022-04-12 03:57:33.006888: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 03:57:34.922788: lr: 0.004146 +2022-04-12 03:57:35.353418: saving scheduled checkpoint file... +2022-04-12 03:57:35.936820: saving checkpoint... +2022-04-12 03:57:54.985016: done, saving took 19.14 seconds +2022-04-12 03:57:55.060365: done +2022-04-12 03:57:55.173524: saving checkpoint... +2022-04-12 03:58:04.515643: done, saving took 9.40 seconds +2022-04-12 03:58:04.579330: This epoch took 155.297781 s + +2022-04-12 03:58:04.596337: svmem(total=99932160000, available=75852386304, percent=24.1, used=19990020096, free=47869231104, active=43651256320, inactive=6282907648, buffers=17510400, cached=32055398400, shared=2274906112, slab=777187328) +2022-04-12 03:58:04.618971: +epoch: 312 +2022-04-12 03:59:52.446055: train loss : -0.6277 +2022-04-12 04:00:03.360120: validation loss: -0.6444 +2022-04-12 04:00:03.676728: Average global foreground Dice: [0.8754, 0.9638, 0.9743, 0.4457] +2022-04-12 04:00:03.941945: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:00:05.063740: lr: 0.004127 +2022-04-12 04:00:05.087916: saving scheduled checkpoint file... +2022-04-12 04:00:05.203586: saving checkpoint... +2022-04-12 04:00:13.457546: done, saving took 8.33 seconds +2022-04-12 04:00:13.550990: done +2022-04-12 04:00:13.632852: saving checkpoint... +2022-04-12 04:00:22.709543: done, saving took 9.13 seconds +2022-04-12 04:00:22.762591: This epoch took 138.111690 s + +2022-04-12 04:00:22.768061: svmem(total=99932160000, available=75869503488, percent=24.1, used=19986690048, free=47883104256, active=43648532480, inactive=6271561728, buffers=17764352, cached=32044601344, shared=2261118976, slab=778448896) +2022-04-12 04:00:22.841360: +epoch: 313 +2022-04-12 04:02:09.378865: train loss : -0.6084 +2022-04-12 04:02:20.822227: validation loss: -0.6328 +2022-04-12 04:02:20.844131: Average global foreground Dice: [0.8755, 0.9612, 0.9719, 0.4101] +2022-04-12 04:02:20.849394: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:02:21.885974: lr: 0.004107 +2022-04-12 04:02:21.894151: saving scheduled checkpoint file... +2022-04-12 04:02:21.963118: saving checkpoint... +2022-04-12 04:02:29.226984: done, saving took 7.33 seconds +2022-04-12 04:02:29.278052: done +2022-04-12 04:02:29.280556: This epoch took 126.434015 s + +2022-04-12 04:02:29.284994: svmem(total=99932160000, available=76242673664, percent=23.7, used=19613585408, free=48253739008, active=43278278656, inactive=6273736704, buffers=18014208, cached=32046821376, shared=2261037056, slab=777678848) +2022-04-12 04:02:29.287834: +epoch: 314 +2022-04-12 04:04:17.086981: train loss : -0.5996 +2022-04-12 04:04:28.480645: validation loss: -0.6218 +2022-04-12 04:04:28.654434: Average global foreground Dice: [0.8695, 0.9583, 0.9651, 0.3841] +2022-04-12 04:04:28.706409: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:04:29.733478: lr: 0.004087 +2022-04-12 04:04:29.790074: saving scheduled checkpoint file... +2022-04-12 04:04:29.903136: saving checkpoint... +2022-04-12 04:04:44.164824: done, saving took 14.37 seconds +2022-04-12 04:04:44.294576: done +2022-04-12 04:04:44.301582: This epoch took 135.011819 s + +2022-04-12 04:04:44.351509: svmem(total=99932160000, available=75837722624, percent=24.1, used=20018167808, free=47845556224, active=43684827136, inactive=6276452352, buffers=18255872, cached=32050180096, shared=2261295104, slab=777945088) +2022-04-12 04:04:44.429163: +epoch: 315 +2022-04-12 04:06:31.586702: train loss : -0.6324 +2022-04-12 04:06:42.990304: validation loss: -0.6035 +2022-04-12 04:06:43.107879: Average global foreground Dice: [0.8767, 0.9542, 0.9651, 0.38] +2022-04-12 04:06:43.193203: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:06:44.224562: lr: 0.004067 +2022-04-12 04:06:44.266416: saving scheduled checkpoint file... +2022-04-12 04:06:44.431741: saving checkpoint... +2022-04-12 04:06:59.354766: done, saving took 15.05 seconds +2022-04-12 04:06:59.582994: done +2022-04-12 04:06:59.588118: This epoch took 135.152536 s + +2022-04-12 04:06:59.660672: svmem(total=99932160000, available=76156657664, percent=23.8, used=19699818496, free=48161546240, active=43362676736, inactive=6278909952, buffers=18534400, cached=32052260864, shared=2260729856, slab=778956800) +2022-04-12 04:06:59.739807: +epoch: 316 +2022-04-12 04:08:47.099930: train loss : -0.6075 +2022-04-12 04:08:58.890023: validation loss: -0.5974 +2022-04-12 04:08:59.422269: Average global foreground Dice: [0.8813, 0.959, 0.9664, 0.3916] +2022-04-12 04:08:59.587806: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:09:00.601734: lr: 0.004047 +2022-04-12 04:09:00.655817: saving scheduled checkpoint file... +2022-04-12 04:09:00.896385: saving checkpoint... +2022-04-12 04:09:14.291698: done, saving took 13.51 seconds +2022-04-12 04:09:14.403275: done +2022-04-12 04:09:14.421526: This epoch took 134.672439 s + +2022-04-12 04:09:14.427127: svmem(total=99932160000, available=75890925568, percent=24.1, used=19965485056, free=47893348352, active=43635294208, inactive=6280331264, buffers=18784256, cached=32054542336, shared=2260668416, slab=778506240) +2022-04-12 04:09:14.469898: +epoch: 317 +2022-04-12 04:11:02.802765: train loss : -0.5862 +2022-04-12 04:11:13.652715: validation loss: -0.6227 +2022-04-12 04:11:14.223527: Average global foreground Dice: [0.8532, 0.9558, 0.9742, 0.3942] +2022-04-12 04:11:14.886127: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:11:17.120447: lr: 0.004027 +2022-04-12 04:11:17.689165: saving scheduled checkpoint file... +2022-04-12 04:11:18.457817: saving checkpoint... +2022-04-12 04:11:32.503970: done, saving took 14.14 seconds +2022-04-12 04:11:32.708647: done +2022-04-12 04:11:32.761770: This epoch took 138.289800 s + +2022-04-12 04:11:32.834498: svmem(total=99932160000, available=75777384448, percent=24.2, used=20078526464, free=47776952320, active=43743039488, inactive=6283268096, buffers=19058688, cached=32057622528, shared=2261147648, slab=779755520) +2022-04-12 04:11:32.894468: +epoch: 318 +2022-04-12 04:13:22.051082: train loss : -0.6139 +2022-04-12 04:13:33.351761: validation loss: -0.6317 +2022-04-12 04:13:34.108902: Average global foreground Dice: [0.8624, 0.9625, 0.9758, 0.4282] +2022-04-12 04:13:35.535397: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:13:37.523262: lr: 0.004007 +2022-04-12 04:13:38.051687: saving scheduled checkpoint file... +2022-04-12 04:13:38.520904: saving checkpoint... +2022-04-12 04:13:50.740098: done, saving took 12.30 seconds +2022-04-12 04:13:52.159452: done +2022-04-12 04:13:52.815787: This epoch took 139.857919 s + +2022-04-12 04:13:53.286160: svmem(total=99932160000, available=76183191552, percent=23.8, used=19686875136, free=48179982336, active=43353903104, inactive=6271594496, buffers=19312640, cached=32045989888, shared=2247049216, slab=780267520) +2022-04-12 04:13:53.358580: +epoch: 319 +2022-04-12 04:15:40.868423: train loss : -0.6188 +2022-04-12 04:15:52.642946: validation loss: -0.6168 +2022-04-12 04:15:52.703340: Average global foreground Dice: [0.879, 0.9518, 0.961, 0.4154] +2022-04-12 04:15:52.765714: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:15:53.681841: lr: 0.003987 +2022-04-12 04:15:53.726708: saving scheduled checkpoint file... +2022-04-12 04:15:53.805819: saving checkpoint... +2022-04-12 04:16:13.628350: done, saving took 19.87 seconds +2022-04-12 04:16:15.328400: done +2022-04-12 04:16:15.418175: This epoch took 142.053504 s + +2022-04-12 04:16:15.489740: svmem(total=99932160000, available=76005810176, percent=23.9, used=19864272896, free=47999676416, active=43519438848, inactive=6272692224, buffers=19603456, cached=32048607232, shared=2246963200, slab=791277568) +2022-04-12 04:16:15.498541: +epoch: 320 +2022-04-12 04:18:03.803935: train loss : -0.6088 +2022-04-12 04:18:15.174225: validation loss: -0.6363 +2022-04-12 04:18:15.238592: Average global foreground Dice: [0.8824, 0.9566, 0.9759, 0.4024] +2022-04-12 04:18:15.242985: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:18:16.260089: lr: 0.003967 +2022-04-12 04:18:16.312432: saving scheduled checkpoint file... +2022-04-12 04:18:16.419858: saving checkpoint... +2022-04-12 04:18:29.522379: done, saving took 13.18 seconds +2022-04-12 04:18:30.461872: done +2022-04-12 04:18:30.695869: This epoch took 135.134158 s + +2022-04-12 04:18:30.911823: svmem(total=99932160000, available=76030980096, percent=23.9, used=19839123456, free=48022016000, active=43489427456, inactive=6275137536, buffers=19890176, cached=32051130368, shared=2246963200, slab=792322048) +2022-04-12 04:18:31.063281: +epoch: 321 +2022-04-12 04:20:18.929154: train loss : -0.6465 +2022-04-12 04:20:30.210606: validation loss: -0.6694 +2022-04-12 04:20:30.231306: Average global foreground Dice: [0.8894, 0.9679, 0.9793, 0.4174] +2022-04-12 04:20:30.244616: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:20:31.173008: lr: 0.003947 +2022-04-12 04:20:31.188789: saving scheduled checkpoint file... +2022-04-12 04:20:31.305767: saving checkpoint... +2022-04-12 04:20:49.310004: done, saving took 18.10 seconds +2022-04-12 04:20:49.389589: done +2022-04-12 04:20:49.422909: This epoch took 138.305446 s + +2022-04-12 04:20:49.451616: svmem(total=99932160000, available=76302987264, percent=23.6, used=19542519808, free=48291307520, active=43214761984, inactive=6301941760, buffers=20140032, cached=32078192640, shared=2271404032, slab=778608640) +2022-04-12 04:20:49.455917: +epoch: 322 +2022-04-12 04:22:38.160337: train loss : -0.6559 +2022-04-12 04:22:50.771091: validation loss: -0.6537 +2022-04-12 04:22:50.932453: Average global foreground Dice: [0.8816, 0.9641, 0.9769, 0.3947] +2022-04-12 04:22:50.999733: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:22:52.041847: lr: 0.003927 +2022-04-12 04:22:52.404862: saving scheduled checkpoint file... +2022-04-12 04:22:52.932536: saving checkpoint... +2022-04-12 04:23:12.261153: done, saving took 19.44 seconds +2022-04-12 04:23:12.434777: done +2022-04-12 04:23:12.501949: This epoch took 143.036450 s + +2022-04-12 04:23:12.575196: svmem(total=99932160000, available=76060065792, percent=23.9, used=19785519104, free=48045449216, active=43456397312, inactive=6303842304, buffers=20414464, cached=32080777216, shared=2271326208, slab=778268672) +2022-04-12 04:23:12.676827: +epoch: 323 +2022-04-12 04:25:02.324560: train loss : -0.6586 +2022-04-12 04:25:14.917442: validation loss: -0.6774 +2022-04-12 04:25:15.606900: Average global foreground Dice: [0.8896, 0.9661, 0.9792, 0.429] +2022-04-12 04:25:16.140917: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:25:18.336234: lr: 0.003907 +2022-04-12 04:25:19.308519: saving scheduled checkpoint file... +2022-04-12 04:25:20.246092: saving checkpoint... +2022-04-12 04:25:37.514095: done, saving took 17.37 seconds +2022-04-12 04:25:37.650753: done +2022-04-12 04:25:37.653094: This epoch took 144.965724 s + +2022-04-12 04:25:37.656082: svmem(total=99932160000, available=76120166400, percent=23.8, used=19736440832, free=48102502400, active=43404574720, inactive=6295474176, buffers=20742144, cached=32072474624, shared=2260267008, slab=777084928) +2022-04-12 04:25:37.687910: +epoch: 324 +2022-04-12 04:27:25.650028: train loss : -0.6349 +2022-04-12 04:27:37.447951: validation loss: -0.6367 +2022-04-12 04:27:37.493157: Average global foreground Dice: [0.8783, 0.9628, 0.976, 0.3783] +2022-04-12 04:27:37.511972: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:27:38.514754: lr: 0.003887 +2022-04-12 04:27:38.564533: saving scheduled checkpoint file... +2022-04-12 04:27:38.680303: saving checkpoint... +2022-04-12 04:27:54.349491: done, saving took 15.72 seconds +2022-04-12 04:27:55.309248: done +2022-04-12 04:27:55.661821: This epoch took 137.946062 s + +2022-04-12 04:27:56.019853: svmem(total=99932160000, available=76120788992, percent=23.8, used=19748913152, free=48100376576, active=43406327808, inactive=6284763136, buffers=21004288, cached=32061865984, shared=2247135232, slab=791740416) +2022-04-12 04:27:56.275015: +epoch: 325 +2022-04-12 04:29:44.818891: train loss : -0.6336 +2022-04-12 04:29:56.789924: validation loss: -0.6242 +2022-04-12 04:29:56.845259: Average global foreground Dice: [0.8851, 0.9575, 0.9671, 0.4193] +2022-04-12 04:29:56.871659: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:29:57.671826: lr: 0.003867 +2022-04-12 04:29:57.774826: saving scheduled checkpoint file... +2022-04-12 04:29:57.956363: saving checkpoint... +2022-04-12 04:30:10.667657: done, saving took 12.79 seconds +2022-04-12 04:30:11.464339: done +2022-04-12 04:30:11.817292: This epoch took 135.184515 s + +2022-04-12 04:30:12.168172: svmem(total=99932160000, available=76071989248, percent=23.9, used=19786231808, free=48040931328, active=43422404608, inactive=6325227520, buffers=22810624, cached=32082186240, shared=2258448384, slab=791986176) +2022-04-12 04:30:12.623521: +epoch: 326 +2022-04-12 04:32:01.003499: train loss : -0.6528 +2022-04-12 04:32:11.705382: validation loss: -0.6592 +2022-04-12 04:32:11.723604: Average global foreground Dice: [0.9005, 0.9613, 0.9774, 0.4009] +2022-04-12 04:32:11.724870: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:32:12.443680: lr: 0.003847 +2022-04-12 04:32:12.522764: saving scheduled checkpoint file... +2022-04-12 04:32:12.577013: saving checkpoint... +2022-04-12 04:32:24.144399: done, saving took 11.62 seconds +2022-04-12 04:32:25.104436: done +2022-04-12 04:32:25.737118: This epoch took 132.531523 s + +2022-04-12 04:32:26.208338: svmem(total=99932160000, available=75774705664, percent=24.2, used=20094992384, free=47740829696, active=43734224896, inactive=6315651072, buffers=23085056, cached=32073252864, shared=2246963200, slab=791179264) +2022-04-12 04:32:26.683493: +epoch: 327 +2022-04-12 04:34:15.122861: train loss : -0.6740 +2022-04-12 04:34:26.019690: validation loss: -0.6832 +2022-04-12 04:34:26.058617: Average global foreground Dice: [0.8888, 0.9696, 0.982, 0.4152] +2022-04-12 04:34:26.134544: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:34:27.133405: lr: 0.003827 +2022-04-12 04:34:27.159924: saving scheduled checkpoint file... +2022-04-12 04:34:27.416059: saving checkpoint... +2022-04-12 04:34:39.961967: done, saving took 12.64 seconds +2022-04-12 04:34:40.306642: done +2022-04-12 04:34:41.075573: This epoch took 134.052473 s + +2022-04-12 04:34:41.622190: svmem(total=99932160000, available=76068929536, percent=23.9, used=19798716416, free=48032354304, active=43439751168, inactive=6319955968, buffers=23339008, cached=32077750272, shared=2248945664, slab=792588288) +2022-04-12 04:34:42.256564: +epoch: 328 +2022-04-12 04:36:30.403928: train loss : -0.6684 +2022-04-12 04:36:42.516173: validation loss: -0.6861 +2022-04-12 04:36:42.608834: Average global foreground Dice: [0.8979, 0.9715, 0.9837, 0.4234] +2022-04-12 04:36:42.675176: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:36:43.539511: lr: 0.003807 +2022-04-12 04:36:43.770440: saving scheduled checkpoint file... +2022-04-12 04:36:43.887338: saving checkpoint... +2022-04-12 04:36:53.387671: done, saving took 9.60 seconds +2022-04-12 04:36:53.498922: done +2022-04-12 04:36:53.502172: This epoch took 130.655148 s + +2022-04-12 04:36:53.508269: svmem(total=99932160000, available=75974418432, percent=24.0, used=19895164928, free=48256909312, active=43541442560, inactive=6007037952, buffers=23621632, cached=31756464128, shared=2246877184, slab=781885440) +2022-04-12 04:36:53.570732: +epoch: 329 +2022-04-12 04:38:41.677947: train loss : -0.6761 +2022-04-12 04:38:52.568385: validation loss: -0.6149 +2022-04-12 04:38:52.593896: Average global foreground Dice: [0.8797, 0.9646, 0.9717, 0.4046] +2022-04-12 04:38:52.619650: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:38:53.695050: lr: 0.003787 +2022-04-12 04:38:53.726931: saving scheduled checkpoint file... +2022-04-12 04:38:53.871104: saving checkpoint... +2022-04-12 04:39:00.735984: done, saving took 6.94 seconds +2022-04-12 04:39:00.944990: done +2022-04-12 04:39:00.951493: This epoch took 127.375583 s + +2022-04-12 04:39:00.999982: svmem(total=99932160000, available=76350865408, percent=23.6, used=19553046528, free=48318447616, active=43202789376, inactive=6287200256, buffers=23859200, cached=32036806656, shared=2212544512, slab=777961472) +2022-04-12 04:39:01.019400: +epoch: 330 +2022-04-12 04:40:49.453986: train loss : -0.6552 +2022-04-12 04:41:02.647362: validation loss: -0.6570 +2022-04-12 04:41:02.654390: Average global foreground Dice: [0.8668, 0.9703, 0.9839, 0.4236] +2022-04-12 04:41:02.663047: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:41:03.621947: lr: 0.003767 +2022-04-12 04:41:03.623427: saving scheduled checkpoint file... +2022-04-12 04:41:03.713227: saving checkpoint... +2022-04-12 04:41:12.413011: done, saving took 8.79 seconds +2022-04-12 04:41:12.621931: done +2022-04-12 04:41:12.626180: This epoch took 131.580717 s + +2022-04-12 04:41:12.635560: svmem(total=99932160000, available=76071047168, percent=23.9, used=19782488064, free=48036155392, active=43433603072, inactive=6339760128, buffers=24109056, cached=32089407488, shared=2262974464, slab=779108352) +2022-04-12 04:41:12.721901: +epoch: 331 +2022-04-12 04:42:59.980928: train loss : -0.6586 +2022-04-12 04:43:11.656546: validation loss: -0.6553 +2022-04-12 04:43:11.726847: Average global foreground Dice: [0.8848, 0.9671, 0.9729, 0.4022] +2022-04-12 04:43:11.788766: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:43:12.719303: lr: 0.003747 +2022-04-12 04:43:12.760141: saving scheduled checkpoint file... +2022-04-12 04:43:12.852930: saving checkpoint... +2022-04-12 04:43:20.240820: done, saving took 7.44 seconds +2022-04-12 04:43:20.262981: done +2022-04-12 04:43:20.303435: This epoch took 127.570134 s + +2022-04-12 04:43:20.309480: svmem(total=99932160000, available=76029501440, percent=23.9, used=19823673344, free=47992111104, active=43472347136, inactive=6342176768, buffers=24342528, cached=32092033024, shared=2263146496, slab=778797056) +2022-04-12 04:43:20.311942: +epoch: 332 +2022-04-12 04:45:09.190392: train loss : -0.6624 +2022-04-12 04:45:19.662566: validation loss: -0.6124 +2022-04-12 04:45:19.674224: Average global foreground Dice: [0.8934, 0.9529, 0.9557, 0.4057] +2022-04-12 04:45:19.688493: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:45:20.616523: lr: 0.003727 +2022-04-12 04:45:20.619476: saving scheduled checkpoint file... +2022-04-12 04:45:20.697847: saving checkpoint... +2022-04-12 04:45:27.185748: done, saving took 6.56 seconds +2022-04-12 04:45:27.383669: done +2022-04-12 04:45:27.393806: This epoch took 127.073530 s + +2022-04-12 04:45:27.431418: svmem(total=99932160000, available=76112756736, percent=23.8, used=19744808960, free=48072675328, active=43392684032, inactive=6340292608, buffers=24616960, cached=32090058752, shared=2258870272, slab=779173888) +2022-04-12 04:45:27.503720: +epoch: 333 +2022-04-12 04:47:15.832438: train loss : -0.6489 +2022-04-12 04:47:28.008688: validation loss: -0.6826 +2022-04-12 04:47:28.048758: Average global foreground Dice: [0.8944, 0.9695, 0.9812, 0.4161] +2022-04-12 04:47:28.062824: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:47:28.734666: lr: 0.003707 +2022-04-12 04:47:28.742413: saving scheduled checkpoint file... +2022-04-12 04:47:28.836147: saving checkpoint... +2022-04-12 04:47:40.544196: done, saving took 11.79 seconds +2022-04-12 04:47:40.887762: done +2022-04-12 04:47:40.957253: This epoch took 133.449080 s + +2022-04-12 04:47:41.019776: svmem(total=99932160000, available=75798794240, percent=24.1, used=20041650176, free=47756161024, active=43695480832, inactive=6359552000, buffers=24879104, cached=32109469696, shared=2275885056, slab=778108928) +2022-04-12 04:47:41.087280: +epoch: 334 +2022-04-12 04:49:28.936810: train loss : -0.6577 +2022-04-12 04:49:40.515542: validation loss: -0.6536 +2022-04-12 04:49:40.655600: Average global foreground Dice: [0.8922, 0.9661, 0.9778, 0.386] +2022-04-12 04:49:40.659004: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:49:41.469621: lr: 0.003687 +2022-04-12 04:49:41.560672: saving scheduled checkpoint file... +2022-04-12 04:49:41.667032: saving checkpoint... +2022-04-12 04:49:54.660087: done, saving took 13.10 seconds +2022-04-12 04:49:55.010857: done +2022-04-12 04:49:55.037453: This epoch took 133.888138 s + +2022-04-12 04:49:55.062049: svmem(total=99932160000, available=76122820608, percent=23.8, used=19732336640, free=48077479936, active=43377713152, inactive=6347317248, buffers=25128960, cached=32097214464, shared=2261188608, slab=780394496) +2022-04-12 04:49:55.111843: +epoch: 335 +2022-04-12 04:51:43.716264: train loss : -0.6643 +2022-04-12 04:51:55.625228: validation loss: -0.6908 +2022-04-12 04:51:55.731863: Average global foreground Dice: [0.8944, 0.9658, 0.9857, 0.4422] +2022-04-12 04:51:55.798590: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:51:56.885508: lr: 0.003667 +2022-04-12 04:51:56.987708: saving scheduled checkpoint file... +2022-04-12 04:51:57.090473: saving checkpoint... +2022-04-12 04:52:11.626386: done, saving took 14.64 seconds +2022-04-12 04:52:11.644031: done +2022-04-12 04:52:11.692642: saving checkpoint... +2022-04-12 04:52:20.287821: done, saving took 8.64 seconds +2022-04-12 04:52:20.446620: This epoch took 145.319683 s + +2022-04-12 04:52:20.457996: svmem(total=99932160000, available=76001263616, percent=23.9, used=19840667648, free=47953199104, active=43485478912, inactive=6363136000, buffers=25399296, cached=32112893952, shared=2274463744, slab=780382208) +2022-04-12 04:52:20.521855: +epoch: 336 +2022-04-12 04:54:08.391822: train loss : -0.6838 +2022-04-12 04:54:19.713856: validation loss: -0.6949 +2022-04-12 04:54:19.739005: Average global foreground Dice: [0.9052, 0.9707, 0.9842, 0.4416] +2022-04-12 04:54:19.740811: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:54:20.628954: lr: 0.003647 +2022-04-12 04:54:20.630435: saving scheduled checkpoint file... +2022-04-12 04:54:20.683589: saving checkpoint... +2022-04-12 04:54:30.212359: done, saving took 9.58 seconds +2022-04-12 04:54:30.337020: done +2022-04-12 04:54:30.393356: saving checkpoint... +2022-04-12 04:54:35.694326: done, saving took 5.35 seconds +2022-04-12 04:54:35.768006: This epoch took 135.184421 s + +2022-04-12 04:54:35.770399: svmem(total=99932160000, available=75823153152, percent=24.1, used=20031807488, free=47772626944, active=43682226176, inactive=6352609280, buffers=25665536, cached=32102060032, shared=2261348352, slab=779616256) +2022-04-12 04:54:35.775022: +epoch: 337 +2022-04-12 04:56:23.987992: train loss : -0.6673 +2022-04-12 04:56:35.968671: validation loss: -0.6129 +2022-04-12 04:56:36.007637: Average global foreground Dice: [0.8837, 0.9585, 0.9655, 0.4295] +2022-04-12 04:56:36.010343: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:56:36.649854: lr: 0.003627 +2022-04-12 04:56:36.663693: saving scheduled checkpoint file... +2022-04-12 04:56:36.757243: saving checkpoint... +2022-04-12 04:56:44.827333: done, saving took 8.15 seconds +2022-04-12 04:56:44.971119: done +2022-04-12 04:56:44.975224: This epoch took 129.190153 s + +2022-04-12 04:56:45.002425: svmem(total=99932160000, available=75853058048, percent=24.1, used=19998253056, free=47799992320, active=43653754880, inactive=6357774336, buffers=25952256, cached=32107962368, shared=2264903680, slab=778211328) +2022-04-12 04:56:45.041124: +epoch: 338 +2022-04-12 04:58:32.992400: train loss : -0.6639 +2022-04-12 04:58:45.102154: validation loss: -0.6520 +2022-04-12 04:58:45.143482: Average global foreground Dice: [0.8834, 0.9622, 0.9744, 0.3917] +2022-04-12 04:58:45.176594: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 04:58:46.181078: lr: 0.003606 +2022-04-12 04:58:46.185860: saving scheduled checkpoint file... +2022-04-12 04:58:46.237253: saving checkpoint... +2022-04-12 04:58:56.250076: done, saving took 10.06 seconds +2022-04-12 04:58:56.388931: done +2022-04-12 04:58:56.394595: This epoch took 131.350354 s + +2022-04-12 04:58:56.420858: svmem(total=99932160000, available=75958378496, percent=24.0, used=19889598464, free=47902695424, active=43537059840, inactive=6363447296, buffers=26198016, cached=32113668096, shared=2268184576, slab=779595776) +2022-04-12 04:58:56.461412: +epoch: 339 +2022-04-12 05:00:44.199587: train loss : -0.6816 +2022-04-12 05:00:55.277692: validation loss: -0.6688 +2022-04-12 05:00:56.252632: Average global foreground Dice: [0.8979, 0.9681, 0.9833, 0.4085] +2022-04-12 05:00:56.667405: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:00:57.493316: lr: 0.003586 +2022-04-12 05:00:57.499248: saving scheduled checkpoint file... +2022-04-12 05:00:57.605328: saving checkpoint... +2022-04-12 05:01:11.974264: done, saving took 14.45 seconds +2022-04-12 05:01:12.007611: done +2022-04-12 05:01:12.040425: This epoch took 135.265842 s + +2022-04-12 05:01:12.046763: svmem(total=99932160000, available=76025831424, percent=23.9, used=19842666496, free=47967535104, active=43488677888, inactive=6345179136, buffers=26447872, cached=32095510528, shared=2247643136, slab=779161600) +2022-04-12 05:01:12.106776: +epoch: 340 +2022-04-12 05:03:00.432524: train loss : -0.6835 +2022-04-12 05:03:12.153302: validation loss: -0.6848 +2022-04-12 05:03:12.276681: Average global foreground Dice: [0.9039, 0.9709, 0.9813, 0.4062] +2022-04-12 05:03:12.283296: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:03:13.352670: lr: 0.003566 +2022-04-12 05:03:13.358166: saving scheduled checkpoint file... +2022-04-12 05:03:13.489343: saving checkpoint... +2022-04-12 05:03:25.052682: done, saving took 11.68 seconds +2022-04-12 05:03:25.087582: done +2022-04-12 05:03:25.178170: saving checkpoint... +2022-04-12 05:03:30.539290: done, saving took 5.45 seconds +2022-04-12 05:03:30.567208: This epoch took 138.456331 s + +2022-04-12 05:03:30.569150: svmem(total=99932160000, available=75936428032, percent=24.0, used=19916406784, free=47875297280, active=43568541696, inactive=6363176960, buffers=26726400, cached=32113729536, shared=2263232512, slab=780128256) +2022-04-12 05:03:30.571584: +epoch: 341 +2022-04-12 05:05:18.793197: train loss : -0.6822 +2022-04-12 05:05:31.258421: validation loss: -0.6764 +2022-04-12 05:05:31.262242: Average global foreground Dice: [0.8882, 0.9707, 0.9836, 0.4123] +2022-04-12 05:05:31.266319: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:05:32.256320: lr: 0.003546 +2022-04-12 05:05:32.259251: saving scheduled checkpoint file... +2022-04-12 05:05:32.384281: saving checkpoint... +2022-04-12 05:05:38.579936: done, saving took 6.32 seconds +2022-04-12 05:05:38.675788: done +2022-04-12 05:05:38.805440: saving checkpoint... +2022-04-12 05:05:49.493750: done, saving took 10.78 seconds +2022-04-12 05:05:49.517023: This epoch took 138.944208 s + +2022-04-12 05:05:49.519744: svmem(total=99932160000, available=76254117888, percent=23.7, used=19600404480, free=48190144512, active=43251138560, inactive=6364893184, buffers=27004928, cached=32114606080, shared=2261458944, slab=779124736) +2022-04-12 05:05:49.523467: +epoch: 342 +2022-04-12 05:07:40.521010: train loss : -0.6794 +2022-04-12 05:07:52.628748: validation loss: -0.6637 +2022-04-12 05:07:52.644434: Average global foreground Dice: [0.8767, 0.9692, 0.976, 0.4119] +2022-04-12 05:07:52.646817: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:07:53.517259: lr: 0.003526 +2022-04-12 05:07:53.612998: saving scheduled checkpoint file... +2022-04-12 05:07:53.770546: saving checkpoint... +2022-04-12 05:08:02.550786: done, saving took 8.87 seconds +2022-04-12 05:08:02.681038: done +2022-04-12 05:08:02.711245: This epoch took 133.186425 s + +2022-04-12 05:08:02.760392: svmem(total=99932160000, available=75920605184, percent=24.0, used=19932872704, free=47853674496, active=43581550592, inactive=6366265344, buffers=27254784, cached=32118358016, shared=2263007232, slab=779849728) +2022-04-12 05:08:02.780419: +epoch: 343 +2022-04-12 05:09:52.692163: train loss : -0.6512 +2022-04-12 05:10:04.422883: validation loss: -0.6128 +2022-04-12 05:10:04.468439: Average global foreground Dice: [0.8756, 0.9522, 0.9619, 0.4333] +2022-04-12 05:10:04.503349: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:10:05.445059: lr: 0.003505 +2022-04-12 05:10:05.509697: saving scheduled checkpoint file... +2022-04-12 05:10:05.744662: saving checkpoint... +2022-04-12 05:10:14.722746: done, saving took 9.10 seconds +2022-04-12 05:10:14.933048: done +2022-04-12 05:10:14.978752: This epoch took 132.157543 s + +2022-04-12 05:10:15.039253: svmem(total=99932160000, available=76067553280, percent=23.9, used=19773104128, free=47998406656, active=43422494720, inactive=6382383104, buffers=27508736, cached=32133140480, shared=2275225600, slab=779198464) +2022-04-12 05:10:15.069601: +epoch: 344 +2022-04-12 05:12:05.518416: train loss : -0.6423 +2022-04-12 05:12:18.029988: validation loss: -0.6220 +2022-04-12 05:12:18.178746: Average global foreground Dice: [0.8731, 0.9583, 0.9672, 0.3815] +2022-04-12 05:12:18.302458: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:12:19.083330: lr: 0.003485 +2022-04-12 05:12:19.130599: saving scheduled checkpoint file... +2022-04-12 05:12:19.301230: saving checkpoint... +2022-04-12 05:12:31.667235: done, saving took 12.49 seconds +2022-04-12 05:12:31.795217: done +2022-04-12 05:12:31.861400: This epoch took 136.743997 s + +2022-04-12 05:12:31.920315: svmem(total=99932160000, available=76162736128, percent=23.8, used=19692109824, free=48090726400, active=43342729216, inactive=6370754560, buffers=27774976, cached=32121548800, shared=2261041152, slab=780435456) +2022-04-12 05:12:31.999362: +epoch: 345 +2022-04-12 05:14:19.471342: train loss : -0.6558 +2022-04-12 05:14:30.984222: validation loss: -0.6615 +2022-04-12 05:14:31.000977: Average global foreground Dice: [0.8778, 0.964, 0.9763, 0.4185] +2022-04-12 05:14:31.001704: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:14:31.633285: lr: 0.003465 +2022-04-12 05:14:31.634812: saving scheduled checkpoint file... +2022-04-12 05:14:31.789881: saving checkpoint... +2022-04-12 05:14:46.805409: done, saving took 15.17 seconds +2022-04-12 05:14:46.850231: done +2022-04-12 05:14:46.868675: This epoch took 134.786023 s + +2022-04-12 05:14:46.874962: svmem(total=99932160000, available=76329820160, percent=23.6, used=19525197824, free=48255459328, active=43170942976, inactive=6372777984, buffers=28028928, cached=32123473920, shared=2260852736, slab=781029376) +2022-04-12 05:14:46.876928: +epoch: 346 +2022-04-12 05:16:35.726840: train loss : -0.6640 +2022-04-12 05:16:48.064507: validation loss: -0.6274 +2022-04-12 05:16:48.173040: Average global foreground Dice: [0.8942, 0.9624, 0.9672, 0.3792] +2022-04-12 05:16:48.178723: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:16:48.774223: lr: 0.003445 +2022-04-12 05:16:48.780836: saving scheduled checkpoint file... +2022-04-12 05:16:48.963985: saving checkpoint... +2022-04-12 05:16:59.973679: done, saving took 11.16 seconds +2022-04-12 05:17:00.000483: done +2022-04-12 05:17:00.003217: This epoch took 133.125248 s + +2022-04-12 05:17:00.017902: svmem(total=99932160000, available=75995176960, percent=24.0, used=19859181568, free=47917957120, active=43509485568, inactive=6375759872, buffers=28303360, cached=32126717952, shared=2261520384, slab=781111296) +2022-04-12 05:17:00.020711: +epoch: 347 +2022-04-12 05:18:48.448788: train loss : -0.6583 +2022-04-12 05:19:00.324297: validation loss: -0.6826 +2022-04-12 05:19:00.604858: Average global foreground Dice: [0.8979, 0.9688, 0.9795, 0.4202] +2022-04-12 05:19:00.667749: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:19:01.726232: lr: 0.003424 +2022-04-12 05:19:01.730843: saving scheduled checkpoint file... +2022-04-12 05:19:01.894586: saving checkpoint... +2022-04-12 05:19:11.928319: done, saving took 10.12 seconds +2022-04-12 05:19:12.060050: done +2022-04-12 05:19:12.105436: This epoch took 132.083159 s + +2022-04-12 05:19:12.110008: svmem(total=99932160000, available=75992076288, percent=24.0, used=19862298624, free=47912419328, active=43513683968, inactive=6377717760, buffers=28577792, cached=32128864256, shared=2261364736, slab=780058624) +2022-04-12 05:19:12.117186: +epoch: 348 +2022-04-12 05:21:00.098012: train loss : -0.6509 +2022-04-12 05:21:11.428977: validation loss: -0.6078 +2022-04-12 05:21:11.433012: Average global foreground Dice: [0.8719, 0.9587, 0.9603, 0.4093] +2022-04-12 05:21:11.451794: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:21:12.512870: lr: 0.003404 +2022-04-12 05:21:12.552074: saving scheduled checkpoint file... +2022-04-12 05:21:12.660268: saving checkpoint... +2022-04-12 05:21:24.433545: done, saving took 11.86 seconds +2022-04-12 05:21:24.479603: done +2022-04-12 05:21:24.539186: This epoch took 132.357494 s + +2022-04-12 05:21:24.542961: svmem(total=99932160000, available=75828613120, percent=24.1, used=20025552896, free=47746572288, active=43674488832, inactive=6380199936, buffers=28831744, cached=32131203072, shared=2261708800, slab=783441920) +2022-04-12 05:21:24.554942: +epoch: 349 +2022-04-12 05:23:11.987155: train loss : -0.6362 +2022-04-12 05:23:23.803149: validation loss: -0.6690 +2022-04-12 05:23:23.828247: Average global foreground Dice: [0.8846, 0.9648, 0.9824, 0.4185] +2022-04-12 05:23:23.857147: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:23:24.675017: lr: 0.003384 +2022-04-12 05:23:24.692508: saving scheduled checkpoint file... +2022-04-12 05:23:24.803391: saving checkpoint... +2022-04-12 05:23:38.518178: done, saving took 13.78 seconds +2022-04-12 05:23:38.655611: done +2022-04-12 05:23:38.674783: This epoch took 134.115408 s + +2022-04-12 05:23:38.738443: svmem(total=99932160000, available=75905331200, percent=24.0, used=19949596672, free=47820959744, active=43599822848, inactive=6381490176, buffers=29102080, cached=32132501504, shared=2260938752, slab=782082048) +2022-04-12 05:23:38.800680: +epoch: 350 +2022-04-12 05:25:26.795918: train loss : -0.6707 +2022-04-12 05:25:38.289898: validation loss: -0.6750 +2022-04-12 05:25:38.320184: Average global foreground Dice: [0.8954, 0.9672, 0.9773, 0.4033] +2022-04-12 05:25:38.331658: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:25:39.873872: lr: 0.003364 +2022-04-12 05:25:40.820300: saving scheduled checkpoint file... +2022-04-12 05:25:41.670312: saving checkpoint... +2022-04-12 05:25:56.867077: done, saving took 15.29 seconds +2022-04-12 05:25:56.991952: done +2022-04-12 05:25:57.020619: This epoch took 138.185032 s + +2022-04-12 05:25:57.057618: svmem(total=99932160000, available=76202471424, percent=23.7, used=19637948416, free=48115757056, active=43295215616, inactive=6396878848, buffers=29388800, cached=32149065728, shared=2275311616, slab=781926400) +2022-04-12 05:25:57.069324: +epoch: 351 +2022-04-12 05:27:44.516211: train loss : -0.6913 +2022-04-12 05:27:56.450079: validation loss: -0.6890 +2022-04-12 05:27:56.458521: Average global foreground Dice: [0.898, 0.9687, 0.9836, 0.4231] +2022-04-12 05:27:56.463200: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:27:57.322616: lr: 0.003343 +2022-04-12 05:27:57.324173: saving scheduled checkpoint file... +2022-04-12 05:27:57.442288: saving checkpoint... +2022-04-12 05:28:06.778984: done, saving took 9.45 seconds +2022-04-12 05:28:07.074672: done +2022-04-12 05:28:07.118739: This epoch took 130.008545 s + +2022-04-12 05:28:07.165632: svmem(total=99932160000, available=76194811904, percent=23.8, used=19668144128, free=48105771008, active=43321843712, inactive=6377459712, buffers=29634560, cached=32128610304, shared=2252812288, slab=781615104) +2022-04-12 05:28:07.167559: +epoch: 352 +2022-04-12 05:29:53.940681: train loss : -0.6592 +2022-04-12 05:30:05.167747: validation loss: -0.6580 +2022-04-12 05:30:05.184626: Average global foreground Dice: [0.8937, 0.9638, 0.9677, 0.423] +2022-04-12 05:30:05.187304: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:30:05.885252: lr: 0.003323 +2022-04-12 05:30:05.892379: saving scheduled checkpoint file... +2022-04-12 05:30:06.031203: saving checkpoint... +2022-04-12 05:30:20.247618: done, saving took 14.35 seconds +2022-04-12 05:30:21.532577: done +2022-04-12 05:30:22.038744: This epoch took 134.868944 s + +2022-04-12 05:30:22.389023: svmem(total=99932160000, available=76080033792, percent=23.9, used=19788234752, free=47988633600, active=43416977408, inactive=6374187008, buffers=29884416, cached=32125407232, shared=2247446528, slab=796573696) +2022-04-12 05:30:22.899611: +epoch: 353 +2022-04-12 05:32:12.183036: train loss : -0.6631 +2022-04-12 05:32:23.975452: validation loss: -0.6895 +2022-04-12 05:32:24.015265: Average global foreground Dice: [0.901, 0.9724, 0.9818, 0.4052] +2022-04-12 05:32:24.068841: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:32:25.158390: lr: 0.003303 +2022-04-12 05:32:25.188276: saving scheduled checkpoint file... +2022-04-12 05:32:25.303934: saving checkpoint... +2022-04-12 05:32:36.922841: done, saving took 11.70 seconds +2022-04-12 05:32:37.192688: done +2022-04-12 05:32:37.198232: This epoch took 133.936829 s + +2022-04-12 05:32:37.214770: svmem(total=99932160000, available=76311842816, percent=23.6, used=19532435456, free=48217911296, active=43184771072, inactive=6400114688, buffers=30146560, cached=32151666688, shared=2271346688, slab=781381632) +2022-04-12 05:32:37.230589: +epoch: 354 +2022-04-12 05:34:26.186599: train loss : -0.6977 +2022-04-12 05:34:37.813496: validation loss: -0.6614 +2022-04-12 05:34:37.848619: Average global foreground Dice: [0.8878, 0.9677, 0.9753, 0.3969] +2022-04-12 05:34:37.853916: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:34:38.613667: lr: 0.003282 +2022-04-12 05:34:38.663691: saving scheduled checkpoint file... +2022-04-12 05:34:38.747572: saving checkpoint... +2022-04-12 05:34:53.439439: done, saving took 14.77 seconds +2022-04-12 05:34:53.554655: done +2022-04-12 05:34:53.586857: This epoch took 136.308087 s + +2022-04-12 05:34:53.620464: svmem(total=99932160000, available=76109225984, percent=23.8, used=19745177600, free=48012967936, active=43399929856, inactive=6392373248, buffers=30396416, cached=32143618048, shared=2261323776, slab=781549568) +2022-04-12 05:34:53.622836: +epoch: 355 +2022-04-12 05:36:42.155038: train loss : -0.6850 +2022-04-12 05:36:54.366054: validation loss: -0.6972 +2022-04-12 05:36:54.379158: Average global foreground Dice: [0.9114, 0.9742, 0.9849, 0.4075] +2022-04-12 05:36:54.418541: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:36:55.134614: lr: 0.003262 +2022-04-12 05:36:55.154914: saving scheduled checkpoint file... +2022-04-12 05:36:55.292703: saving checkpoint... +2022-04-12 05:37:05.010735: done, saving took 9.81 seconds +2022-04-12 05:37:05.197941: done +2022-04-12 05:37:05.319074: This epoch took 131.621055 s + +2022-04-12 05:37:05.387498: svmem(total=99932160000, available=76102205440, percent=23.8, used=19752390656, free=48003411968, active=43405737984, inactive=6394191872, buffers=30670848, cached=32145686528, shared=2261000192, slab=782401536) +2022-04-12 05:37:05.438622: +epoch: 356 +2022-04-12 05:38:53.860384: train loss : -0.6916 +2022-04-12 05:39:05.523149: validation loss: -0.6771 +2022-04-12 05:39:05.532825: Average global foreground Dice: [0.8895, 0.9699, 0.9746, 0.4214] +2022-04-12 05:39:05.534902: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:39:06.509350: lr: 0.003241 +2022-04-12 05:39:06.514388: saving scheduled checkpoint file... +2022-04-12 05:39:06.635117: saving checkpoint... +2022-04-12 05:39:16.716313: done, saving took 10.20 seconds +2022-04-12 05:39:16.788043: done +2022-04-12 05:39:16.898029: saving checkpoint... +2022-04-12 05:39:23.839065: done, saving took 7.04 seconds +2022-04-12 05:39:24.049887: This epoch took 138.569512 s + +2022-04-12 05:39:24.056651: svmem(total=99932160000, available=76139126784, percent=23.8, used=19703803904, free=48037883904, active=43359481856, inactive=6408122368, buffers=30932992, cached=32159539200, shared=2272710656, slab=781905920) +2022-04-12 05:39:24.061681: +epoch: 357 +2022-04-12 05:41:10.858447: train loss : -0.6970 +2022-04-12 05:41:22.533164: validation loss: -0.6716 +2022-04-12 05:41:22.553436: Average global foreground Dice: [0.893, 0.9631, 0.9747, 0.4291] +2022-04-12 05:41:22.555955: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:41:23.516706: lr: 0.003221 +2022-04-12 05:41:23.519317: saving scheduled checkpoint file... +2022-04-12 05:41:23.631641: saving checkpoint... +2022-04-12 05:41:38.177965: done, saving took 14.65 seconds +2022-04-12 05:41:38.224594: done +2022-04-12 05:41:38.336646: saving checkpoint... +2022-04-12 05:41:48.678635: done, saving took 10.45 seconds +2022-04-12 05:41:48.735887: This epoch took 144.665325 s + +2022-04-12 05:41:48.741499: svmem(total=99932160000, available=76241104896, percent=23.7, used=19626795008, free=48137248768, active=43284783104, inactive=6385393664, buffers=31215616, cached=32136900608, shared=2247700480, slab=782434304) +2022-04-12 05:41:48.800902: +epoch: 358 +2022-04-12 05:43:37.983057: train loss : -0.6916 +2022-04-12 05:43:49.263158: validation loss: -0.6789 +2022-04-12 05:43:49.315669: Average global foreground Dice: [0.8921, 0.9736, 0.9814, 0.4013] +2022-04-12 05:43:49.322780: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:43:50.000987: lr: 0.003201 +2022-04-12 05:43:50.008243: saving scheduled checkpoint file... +2022-04-12 05:43:50.137939: saving checkpoint... +2022-04-12 05:43:59.440852: done, saving took 9.43 seconds +2022-04-12 05:43:59.633739: done +2022-04-12 05:43:59.768672: saving checkpoint... +2022-04-12 05:44:22.067673: done, saving took 22.43 seconds +2022-04-12 05:44:22.236599: This epoch took 153.368617 s + +2022-04-12 05:44:22.253673: svmem(total=99932160000, available=76150865920, percent=23.8, used=19716993024, free=48044437504, active=43369574400, inactive=6387597312, buffers=31494144, cached=32139235328, shared=2247614464, slab=784326656) +2022-04-12 05:44:22.267844: +epoch: 359 +2022-04-12 05:46:11.313312: train loss : -0.6978 +2022-04-12 05:46:23.344044: validation loss: -0.7068 +2022-04-12 05:46:23.437137: Average global foreground Dice: [0.9067, 0.9737, 0.9845, 0.4387] +2022-04-12 05:46:23.454314: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:46:24.408156: lr: 0.00318 +2022-04-12 05:46:24.411180: saving scheduled checkpoint file... +2022-04-12 05:46:24.495950: saving checkpoint... +2022-04-12 05:46:41.887087: done, saving took 17.47 seconds +2022-04-12 05:46:42.100941: done +2022-04-12 05:46:42.234178: saving checkpoint... +2022-04-12 05:46:55.367875: done, saving took 13.20 seconds +2022-04-12 05:46:55.470186: This epoch took 153.196071 s + +2022-04-12 05:46:55.534731: svmem(total=99932160000, available=76069388288, percent=23.9, used=19783806976, free=47960100864, active=43448336384, inactive=6404096000, buffers=31813632, cached=32156438528, shared=2262159360, slab=785457152) +2022-04-12 05:46:55.607635: +epoch: 360 +2022-04-12 05:48:43.263588: train loss : -0.6977 +2022-04-12 05:48:53.749636: validation loss: -0.6644 +2022-04-12 05:48:53.845456: Average global foreground Dice: [0.9028, 0.9712, 0.9709, 0.4075] +2022-04-12 05:48:53.902402: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:48:54.770836: lr: 0.00316 +2022-04-12 05:48:54.890981: saving scheduled checkpoint file... +2022-04-12 05:48:55.137514: saving checkpoint... +2022-04-12 05:49:02.382107: done, saving took 7.42 seconds +2022-04-12 05:49:02.473511: done +2022-04-12 05:49:02.512859: This epoch took 126.890645 s + +2022-04-12 05:49:02.549080: svmem(total=99932160000, available=75956359168, percent=24.0, used=19897331712, free=47844831232, active=43553550336, inactive=6406127616, buffers=32051200, cached=32157945856, shared=2261766144, slab=782589952) +2022-04-12 05:49:02.585814: +epoch: 361 +2022-04-12 05:50:49.864936: train loss : -0.6909 +2022-04-12 05:51:00.729325: validation loss: -0.6926 +2022-04-12 05:51:01.178680: Average global foreground Dice: [0.9071, 0.974, 0.9848, 0.4179] +2022-04-12 05:51:01.618993: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:51:03.179780: lr: 0.003139 +2022-04-12 05:51:03.445164: saving scheduled checkpoint file... +2022-04-12 05:51:03.575927: saving checkpoint... +2022-04-12 05:51:14.056853: done, saving took 10.53 seconds +2022-04-12 05:51:14.212739: done +2022-04-12 05:51:14.333516: saving checkpoint... +2022-04-12 05:51:24.269398: done, saving took 10.00 seconds +2022-04-12 05:51:24.598099: This epoch took 141.994857 s + +2022-04-12 05:51:24.684623: svmem(total=99932160000, available=76103454720, percent=23.8, used=19729342464, free=47989354496, active=43381485568, inactive=6429077504, buffers=32317440, cached=32181145600, shared=2282573824, slab=780525568) +2022-04-12 05:51:24.739473: +epoch: 362 +2022-04-12 05:53:11.745063: train loss : -0.6906 +2022-04-12 05:53:22.839715: validation loss: -0.6656 +2022-04-12 05:53:22.902000: Average global foreground Dice: [0.8957, 0.9653, 0.9644, 0.4548] +2022-04-12 05:53:22.907420: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:53:24.125593: lr: 0.003119 +2022-04-12 05:53:24.153133: saving scheduled checkpoint file... +2022-04-12 05:53:24.272055: saving checkpoint... +2022-04-12 05:53:34.872253: done, saving took 10.70 seconds +2022-04-12 05:53:34.941375: done +2022-04-12 05:53:35.043825: saving checkpoint... +2022-04-12 05:53:50.608875: done, saving took 15.65 seconds +2022-04-12 05:53:50.950901: This epoch took 146.203402 s + +2022-04-12 05:53:50.964939: svmem(total=99932160000, available=76093255680, percent=23.9, used=19760439296, free=47976583168, active=43416842240, inactive=6410502144, buffers=32600064, cached=32162537472, shared=2261602304, slab=781905920) +2022-04-12 05:53:51.043557: +epoch: 363 +2022-04-12 05:55:39.294300: train loss : -0.6942 +2022-04-12 05:55:50.546857: validation loss: -0.6570 +2022-04-12 05:55:50.635187: Average global foreground Dice: [0.8929, 0.9642, 0.9738, 0.4019] +2022-04-12 05:55:50.705132: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:55:51.746054: lr: 0.003098 +2022-04-12 05:55:51.907811: saving scheduled checkpoint file... +2022-04-12 05:55:52.046256: saving checkpoint... +2022-04-12 05:56:03.252389: done, saving took 11.30 seconds +2022-04-12 05:56:03.422230: done +2022-04-12 05:56:03.460669: This epoch took 132.352419 s + +2022-04-12 05:56:03.493949: svmem(total=99932160000, available=76009295872, percent=23.9, used=19847766016, free=47890415616, active=43502899200, inactive=6409080832, buffers=32890880, cached=32161087488, shared=2258219008, slab=783360000) +2022-04-12 05:56:03.535606: +epoch: 364 +2022-04-12 05:57:50.956855: train loss : -0.6949 +2022-04-12 05:58:02.850257: validation loss: -0.6615 +2022-04-12 05:58:02.930261: Average global foreground Dice: [0.9043, 0.9661, 0.9757, 0.4104] +2022-04-12 05:58:02.935984: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 05:58:03.831291: lr: 0.003078 +2022-04-12 05:58:03.878399: saving scheduled checkpoint file... +2022-04-12 05:58:03.995327: saving checkpoint... +2022-04-12 05:58:12.880388: done, saving took 9.00 seconds +2022-04-12 05:58:13.125141: done +2022-04-12 05:58:13.188752: This epoch took 129.641970 s + +2022-04-12 05:58:13.196012: svmem(total=99932160000, available=76054499328, percent=23.9, used=19799191552, free=47933382656, active=43456516096, inactive=6414422016, buffers=33124352, cached=32166461440, shared=2261602304, slab=781701120) +2022-04-12 05:58:13.245448: +epoch: 365 +2022-04-12 06:00:01.659606: train loss : -0.6990 +2022-04-12 06:00:13.874542: validation loss: -0.6731 +2022-04-12 06:00:13.906425: Average global foreground Dice: [0.9092, 0.9645, 0.9817, 0.4489] +2022-04-12 06:00:13.988428: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:00:15.205546: lr: 0.003057 +2022-04-12 06:00:15.212208: saving scheduled checkpoint file... +2022-04-12 06:00:15.353393: saving checkpoint... +2022-04-12 06:00:29.166896: done, saving took 13.91 seconds +2022-04-12 06:00:29.415998: done +2022-04-12 06:00:29.513327: saving checkpoint... +2022-04-12 06:00:43.747183: done, saving took 14.28 seconds +2022-04-12 06:00:43.962284: This epoch took 150.669265 s + +2022-04-12 06:00:44.042736: svmem(total=99932160000, available=75997118464, percent=24.0, used=19856535552, free=47873282048, active=43509399552, inactive=6416736256, buffers=33411072, cached=32168931328, shared=2261602304, slab=782704640) +2022-04-12 06:00:44.063655: +epoch: 366 +2022-04-12 06:02:32.041295: train loss : -0.7011 +2022-04-12 06:02:44.249717: validation loss: -0.6505 +2022-04-12 06:02:44.387365: Average global foreground Dice: [0.9068, 0.9608, 0.9628, 0.4109] +2022-04-12 06:02:44.442914: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:02:45.510870: lr: 0.003037 +2022-04-12 06:02:45.519409: saving scheduled checkpoint file... +2022-04-12 06:02:45.663955: saving checkpoint... +2022-04-12 06:02:58.784994: done, saving took 13.20 seconds +2022-04-12 06:02:59.148920: done +2022-04-12 06:02:59.237217: This epoch took 135.054892 s + +2022-04-12 06:02:59.306818: svmem(total=99932160000, available=75881771008, percent=24.1, used=19951808512, free=47755661312, active=43607056384, inactive=6439157760, buffers=33652736, cached=32191037440, shared=2281574400, slab=782622720) +2022-04-12 06:02:59.392661: +epoch: 367 +2022-04-12 06:04:46.673051: train loss : -0.6972 +2022-04-12 06:04:58.860146: validation loss: -0.7000 +2022-04-12 06:04:58.964291: Average global foreground Dice: [0.9118, 0.9722, 0.9841, 0.4279] +2022-04-12 06:04:59.018144: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:05:00.068316: lr: 0.003016 +2022-04-12 06:05:00.220305: saving scheduled checkpoint file... +2022-04-12 06:05:00.393575: saving checkpoint... +2022-04-12 06:05:12.432731: done, saving took 12.12 seconds +2022-04-12 06:05:12.898378: done +2022-04-12 06:05:12.996728: saving checkpoint... +2022-04-12 06:05:24.945500: done, saving took 12.04 seconds +2022-04-12 06:05:25.127734: This epoch took 145.647887 s + +2022-04-12 06:05:25.135114: svmem(total=99932160000, available=75919888384, percent=24.0, used=19933745152, free=47791235072, active=43587207168, inactive=6420996096, buffers=33955840, cached=32173223936, shared=2261602304, slab=783073280) +2022-04-12 06:05:25.197765: +epoch: 368 +2022-04-12 06:07:12.645473: train loss : -0.6808 +2022-04-12 06:07:24.524625: validation loss: -0.5962 +2022-04-12 06:07:24.550285: Average global foreground Dice: [0.9012, 0.9569, 0.9588, 0.4128] +2022-04-12 06:07:24.651446: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:07:25.394385: lr: 0.002996 +2022-04-12 06:07:25.464852: saving scheduled checkpoint file... +2022-04-12 06:07:25.564742: saving checkpoint... +2022-04-12 06:07:43.639116: done, saving took 18.17 seconds +2022-04-12 06:07:43.803306: done +2022-04-12 06:07:43.885671: This epoch took 138.666383 s + +2022-04-12 06:07:43.890464: svmem(total=99932160000, available=75946201088, percent=24.0, used=19906785280, free=47815106560, active=43565125632, inactive=6423764992, buffers=34213888, cached=32176054272, shared=2262028288, slab=783822848) +2022-04-12 06:07:43.986680: +epoch: 369 +2022-04-12 06:09:32.902923: train loss : -0.6883 +2022-04-12 06:09:45.485696: validation loss: -0.6863 +2022-04-12 06:09:45.564996: Average global foreground Dice: [0.9111, 0.97, 0.9834, 0.4172] +2022-04-12 06:09:45.681643: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:09:46.865639: lr: 0.002975 +2022-04-12 06:09:46.913875: saving scheduled checkpoint file... +2022-04-12 06:09:47.124272: saving checkpoint... +2022-04-12 06:10:03.604987: done, saving took 16.59 seconds +2022-04-12 06:10:03.659767: done +2022-04-12 06:10:03.703996: This epoch took 139.714351 s + +2022-04-12 06:10:03.724836: svmem(total=99932160000, available=75947257856, percent=24.0, used=19906015232, free=47813722112, active=43559387136, inactive=6425788416, buffers=34467840, cached=32177954816, shared=2261774336, slab=784261120) +2022-04-12 06:10:03.786367: +epoch: 370 +2022-04-12 06:11:52.524157: train loss : -0.6939 +2022-04-12 06:12:03.456305: validation loss: -0.6814 +2022-04-12 06:12:03.549055: Average global foreground Dice: [0.8908, 0.9726, 0.9831, 0.4134] +2022-04-12 06:12:03.577249: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:12:04.487761: lr: 0.002954 +2022-04-12 06:12:04.527975: saving scheduled checkpoint file... +2022-04-12 06:12:04.729621: saving checkpoint... +2022-04-12 06:12:19.735981: done, saving took 15.15 seconds +2022-04-12 06:12:19.863325: done +2022-04-12 06:12:19.912329: This epoch took 136.081026 s + +2022-04-12 06:12:19.948217: svmem(total=99932160000, available=76098625536, percent=23.8, used=19755016192, free=47962730496, active=43410165760, inactive=6427127808, buffers=34709504, cached=32179703808, shared=2261413888, slab=782962688) +2022-04-12 06:12:20.005927: +epoch: 371 +2022-04-12 06:14:08.867198: train loss : -0.6755 +2022-04-12 06:14:21.393265: validation loss: -0.6094 +2022-04-12 06:14:21.455411: Average global foreground Dice: [0.9085, 0.9548, 0.9536, 0.4073] +2022-04-12 06:14:21.483166: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:14:22.547200: lr: 0.002934 +2022-04-12 06:14:22.588578: saving scheduled checkpoint file... +2022-04-12 06:14:22.729678: saving checkpoint... +2022-04-12 06:14:32.851828: done, saving took 10.21 seconds +2022-04-12 06:14:32.930779: done +2022-04-12 06:14:32.992181: This epoch took 132.973399 s + +2022-04-12 06:14:33.068481: svmem(total=99932160000, available=75763363840, percent=24.2, used=20076617728, free=47625240576, active=43735781376, inactive=6442627072, buffers=34963456, cached=32195338240, shared=2274983936, slab=783073280) +2022-04-12 06:14:33.130529: +epoch: 372 +2022-04-12 06:16:20.326393: train loss : -0.6778 +2022-04-12 06:16:32.520717: validation loss: -0.6777 +2022-04-12 06:16:32.614784: Average global foreground Dice: [0.9068, 0.9685, 0.9782, 0.4184] +2022-04-12 06:16:32.691632: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:16:33.427761: lr: 0.002913 +2022-04-12 06:16:33.457238: saving scheduled checkpoint file... +2022-04-12 06:16:33.615372: saving checkpoint... +2022-04-12 06:16:46.604787: done, saving took 13.04 seconds +2022-04-12 06:16:46.712662: done +2022-04-12 06:16:46.761359: This epoch took 133.564967 s + +2022-04-12 06:16:46.818263: svmem(total=99932160000, available=75763683328, percent=24.2, used=20075360256, free=47623114752, active=43732680704, inactive=6445494272, buffers=35233792, cached=32198451200, shared=2275860480, slab=783384576) +2022-04-12 06:16:46.837850: +epoch: 373 +2022-04-12 06:18:34.673038: train loss : -0.7033 +2022-04-12 06:18:46.602112: validation loss: -0.6403 +2022-04-12 06:18:46.710267: Average global foreground Dice: [0.9095, 0.9665, 0.9747, 0.4206] +2022-04-12 06:18:46.727775: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:18:47.853407: lr: 0.002892 +2022-04-12 06:18:47.954620: saving scheduled checkpoint file... +2022-04-12 06:18:48.114438: saving checkpoint... +2022-04-12 06:19:03.900292: done, saving took 15.89 seconds +2022-04-12 06:19:05.402817: done +2022-04-12 06:19:05.450142: This epoch took 138.566591 s + +2022-04-12 06:19:05.516373: svmem(total=99932160000, available=75919323136, percent=24.0, used=19934429184, free=47776382976, active=43588653056, inactive=6433038336, buffers=35500032, cached=32185847808, shared=2261192704, slab=783437824) +2022-04-12 06:19:05.571962: +epoch: 374 +2022-04-12 06:20:54.160661: train loss : -0.6948 +2022-04-12 06:21:06.464522: validation loss: -0.6630 +2022-04-12 06:21:07.909099: Average global foreground Dice: [0.9099, 0.9661, 0.967, 0.4203] +2022-04-12 06:21:08.859836: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:21:09.821696: lr: 0.002872 +2022-04-12 06:21:09.876339: saving scheduled checkpoint file... +2022-04-12 06:21:10.051684: saving checkpoint... +2022-04-12 06:21:26.097613: done, saving took 16.11 seconds +2022-04-12 06:21:26.475104: done +2022-04-12 06:21:26.575510: This epoch took 140.944958 s + +2022-04-12 06:21:26.633142: svmem(total=99932160000, available=75878809600, percent=24.1, used=19965448192, free=47733403648, active=43623350272, inactive=6444056576, buffers=35786752, cached=32197521408, shared=2270609408, slab=783499264) +2022-04-12 06:21:26.695697: +epoch: 375 +2022-04-12 06:23:14.702555: train loss : -0.6892 +2022-04-12 06:23:25.621958: validation loss: -0.7056 +2022-04-12 06:23:26.652630: Average global foreground Dice: [0.8946, 0.9725, 0.9842, 0.4353] +2022-04-12 06:23:27.807244: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:23:30.890956: lr: 0.002851 +2022-04-12 06:23:31.093512: saving scheduled checkpoint file... +2022-04-12 06:23:31.189674: saving checkpoint... +2022-04-12 06:23:43.891652: done, saving took 12.78 seconds +2022-04-12 06:23:44.075656: done +2022-04-12 06:23:44.204355: saving checkpoint... +2022-04-12 06:23:57.276632: done, saving took 13.15 seconds +2022-04-12 06:23:57.524381: This epoch took 150.781436 s + +2022-04-12 06:23:57.608240: svmem(total=99932160000, available=75882057728, percent=24.1, used=19971301376, free=47734116352, active=43624607744, inactive=6437732352, buffers=36057088, cached=32190685184, shared=2261491712, slab=781967360) +2022-04-12 06:23:57.674555: +epoch: 376 +2022-04-12 06:25:46.675586: train loss : -0.6979 +2022-04-12 06:25:58.678216: validation loss: -0.7093 +2022-04-12 06:25:58.728121: Average global foreground Dice: [0.9101, 0.9747, 0.9821, 0.4282] +2022-04-12 06:25:58.735880: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:25:59.618288: lr: 0.00283 +2022-04-12 06:25:59.620654: saving scheduled checkpoint file... +2022-04-12 06:25:59.745915: saving checkpoint... +2022-04-12 06:26:09.264227: done, saving took 9.64 seconds +2022-04-12 06:26:09.326028: done +2022-04-12 06:26:09.463410: saving checkpoint... +2022-04-12 06:26:25.659528: done, saving took 16.30 seconds +2022-04-12 06:26:25.827775: This epoch took 148.139096 s + +2022-04-12 06:26:25.852008: svmem(total=99932160000, available=76011347968, percent=23.9, used=19841826816, free=47827931136, active=43512098816, inactive=6462500864, buffers=39325696, cached=32223076352, shared=2261561344, slab=783261696) +2022-04-12 06:26:25.859044: +epoch: 377 +2022-04-12 06:28:15.875558: train loss : -0.7011 +2022-04-12 06:28:30.529249: validation loss: -0.6832 +2022-04-12 06:28:31.288456: Average global foreground Dice: [0.9133, 0.9638, 0.9805, 0.417] +2022-04-12 06:28:32.300989: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:28:34.164771: lr: 0.00281 +2022-04-12 06:28:34.171939: saving scheduled checkpoint file... +2022-04-12 06:28:34.325802: saving checkpoint... +2022-04-12 06:28:45.882406: done, saving took 11.65 seconds +2022-04-12 06:28:46.071359: done +2022-04-12 06:28:46.193297: saving checkpoint... +2022-04-12 06:28:54.559402: done, saving took 8.41 seconds +2022-04-12 06:28:54.754206: This epoch took 148.815956 s + +2022-04-12 06:28:54.772894: svmem(total=99932160000, available=75794563072, percent=24.2, used=20048310272, free=47608524800, active=43720687616, inactive=6475100160, buffers=39583744, cached=32235741184, shared=2271821824, slab=781889536) +2022-04-12 06:28:54.870423: +epoch: 378 +2022-04-12 06:30:42.668785: train loss : -0.6970 +2022-04-12 06:30:55.490423: validation loss: -0.6755 +2022-04-12 06:30:55.994601: Average global foreground Dice: [0.9099, 0.9707, 0.9754, 0.4272] +2022-04-12 06:30:56.715704: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:30:58.400112: lr: 0.002789 +2022-04-12 06:30:58.517884: saving scheduled checkpoint file... +2022-04-12 06:30:58.647070: saving checkpoint... +2022-04-12 06:31:22.447974: done, saving took 23.88 seconds +2022-04-12 06:31:23.064412: done +2022-04-12 06:31:23.186288: saving checkpoint... +2022-04-12 06:31:36.728433: done, saving took 13.60 seconds +2022-04-12 06:31:37.053357: This epoch took 162.120326 s + +2022-04-12 06:31:37.117756: svmem(total=99932160000, available=75766222848, percent=24.2, used=20071649280, free=47577464832, active=43736170496, inactive=6482628608, buffers=39895040, cached=32243150848, shared=2276880384, slab=782970880) +2022-04-12 06:31:37.161355: +epoch: 379 +2022-04-12 06:33:25.059395: train loss : -0.7056 +2022-04-12 06:33:36.112914: validation loss: -0.6627 +2022-04-12 06:33:36.437441: Average global foreground Dice: [0.9075, 0.9669, 0.971, 0.4002] +2022-04-12 06:33:36.926660: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:33:39.258956: lr: 0.002768 +2022-04-12 06:33:39.900202: saving scheduled checkpoint file... +2022-04-12 06:33:40.560519: saving checkpoint... +2022-04-12 06:33:59.558869: done, saving took 19.08 seconds +2022-04-12 06:33:59.911632: done +2022-04-12 06:33:59.955379: This epoch took 142.707278 s + +2022-04-12 06:33:59.959302: svmem(total=99932160000, available=76027621376, percent=23.9, used=19839401984, free=47836438528, active=43513397248, inactive=6455459840, buffers=40148992, cached=32216170496, shared=2247614464, slab=783589376) +2022-04-12 06:34:00.024457: +epoch: 380 +2022-04-12 06:35:48.850251: train loss : -0.7064 +2022-04-12 06:35:59.819031: validation loss: -0.7024 +2022-04-12 06:35:59.893897: Average global foreground Dice: [0.9098, 0.9725, 0.979, 0.4475] +2022-04-12 06:35:59.973082: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:36:00.989630: lr: 0.002747 +2022-04-12 06:36:01.041115: saving scheduled checkpoint file... +2022-04-12 06:36:01.157943: saving checkpoint... +2022-04-12 06:36:34.523561: done, saving took 33.44 seconds +2022-04-12 06:36:35.692329: done +2022-04-12 06:36:35.928153: saving checkpoint... +2022-04-12 06:37:01.198867: done, saving took 25.33 seconds +2022-04-12 06:37:01.300384: This epoch took 181.183017 s + +2022-04-12 06:37:01.306098: svmem(total=99932160000, available=76001619968, percent=23.9, used=19851440128, free=47807225856, active=43523411968, inactive=6472159232, buffers=40517632, cached=32232976384, shared=2261577728, slab=785649664) +2022-04-12 06:37:01.313739: +epoch: 381 +2022-04-12 06:38:48.896669: train loss : -0.6972 +2022-04-12 06:39:01.526668: validation loss: -0.6990 +2022-04-12 06:39:01.625124: Average global foreground Dice: [0.9111, 0.9723, 0.9844, 0.4014] +2022-04-12 06:39:01.739583: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:39:02.491322: lr: 0.002727 +2022-04-12 06:39:02.497912: saving scheduled checkpoint file... +2022-04-12 06:39:02.624709: saving checkpoint... +2022-04-12 06:39:23.568074: done, saving took 20.99 seconds +2022-04-12 06:39:24.810564: done +2022-04-12 06:39:24.889739: This epoch took 143.497390 s + +2022-04-12 06:39:24.954201: svmem(total=99932160000, available=75992055808, percent=24.0, used=19874885632, free=47795425280, active=43546124288, inactive=6460243968, buffers=40783872, cached=32221065216, shared=2247614464, slab=784400384) +2022-04-12 06:39:25.047114: +epoch: 382 +2022-04-12 06:41:11.885677: train loss : -0.6872 +2022-04-12 06:41:23.272246: validation loss: -0.6102 +2022-04-12 06:41:24.256614: Average global foreground Dice: [0.9055, 0.9604, 0.9627, 0.4403] +2022-04-12 06:41:24.374207: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:41:25.191435: lr: 0.002706 +2022-04-12 06:41:25.251753: saving scheduled checkpoint file... +2022-04-12 06:41:25.417156: saving checkpoint... +2022-04-12 06:41:53.986182: done, saving took 28.66 seconds +2022-04-12 06:41:55.923933: done +2022-04-12 06:41:56.469591: This epoch took 151.347846 s + +2022-04-12 06:41:57.006665: svmem(total=99932160000, available=76084305920, percent=23.9, used=19781517312, free=47887638528, active=43437395968, inactive=6460940288, buffers=41054208, cached=32221949952, shared=2248691712, slab=797671424) +2022-04-12 06:41:57.354995: +epoch: 383 +2022-04-12 06:43:44.900289: train loss : -0.6905 +2022-04-12 06:43:56.836214: validation loss: -0.5761 +2022-04-12 06:43:57.109295: Average global foreground Dice: [0.9069, 0.9573, 0.9559, 0.4123] +2022-04-12 06:43:57.231686: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:43:58.335981: lr: 0.002685 +2022-04-12 06:43:58.554551: saving scheduled checkpoint file... +2022-04-12 06:43:58.788231: saving checkpoint... +2022-04-12 06:44:26.745419: done, saving took 28.06 seconds +2022-04-12 06:44:30.466378: done +2022-04-12 06:44:31.946412: This epoch took 154.584771 s + +2022-04-12 06:44:33.608536: svmem(total=99932160000, available=76228771840, percent=23.7, used=19623944192, free=48029413376, active=43280834560, inactive=6476103680, buffers=41345024, cached=32237457408, shared=2261577728, slab=797282304) +2022-04-12 06:44:34.817565: +epoch: 384 +2022-04-12 06:46:23.949013: train loss : -0.7050 +2022-04-12 06:46:36.001266: validation loss: -0.6726 +2022-04-12 06:46:36.060665: Average global foreground Dice: [0.9039, 0.9667, 0.9778, 0.4436] +2022-04-12 06:46:36.116187: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:46:37.032726: lr: 0.002664 +2022-04-12 06:46:37.147968: saving scheduled checkpoint file... +2022-04-12 06:46:37.384013: saving checkpoint... +2022-04-12 06:46:59.967510: done, saving took 22.70 seconds +2022-04-12 06:47:02.478814: done +2022-04-12 06:47:03.096910: This epoch took 147.237679 s + +2022-04-12 06:47:04.107913: svmem(total=99932160000, available=76130680832, percent=23.8, used=19736100864, free=47929024512, active=43389255680, inactive=6464421888, buffers=41635840, cached=32225398784, shared=2247700480, slab=797827072) +2022-04-12 06:47:04.768768: +epoch: 385 +2022-04-12 06:48:56.684855: train loss : -0.6657 +2022-04-12 06:49:09.441647: validation loss: -0.6409 +2022-04-12 06:49:09.518214: Average global foreground Dice: [0.8975, 0.9627, 0.9669, 0.4198] +2022-04-12 06:49:09.571065: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:49:10.478296: lr: 0.002643 +2022-04-12 06:49:10.544445: saving scheduled checkpoint file... +2022-04-12 06:49:10.713018: saving checkpoint... +2022-04-12 06:50:19.883122: done, saving took 69.29 seconds +2022-04-12 06:50:20.164777: done +2022-04-12 06:50:20.221884: This epoch took 194.241817 s + +2022-04-12 06:50:20.238735: svmem(total=99932160000, available=75857522688, percent=24.1, used=19984609280, free=47652392960, active=43651465216, inactive=6491983872, buffers=41988096, cached=32253169664, shared=2272329728, slab=786776064) +2022-04-12 06:50:20.338678: +epoch: 386 +2022-04-12 06:52:09.240848: train loss : -0.6703 +2022-04-12 06:52:22.744739: validation loss: -0.7097 +2022-04-12 06:52:23.717847: Average global foreground Dice: [0.9015, 0.9665, 0.9858, 0.4642] +2022-04-12 06:52:24.213768: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:52:25.897945: lr: 0.002622 +2022-04-12 06:52:26.328375: saving scheduled checkpoint file... +2022-04-12 06:52:26.903401: saving checkpoint... +2022-04-12 06:52:50.459702: done, saving took 23.69 seconds +2022-04-12 06:52:50.655701: done +2022-04-12 06:52:50.927900: saving checkpoint... +2022-04-12 06:53:09.445898: done, saving took 18.66 seconds +2022-04-12 06:53:09.785252: This epoch took 169.388982 s + +2022-04-12 06:53:09.894384: svmem(total=99932160000, available=75998228480, percent=24.0, used=19853967360, free=47790292992, active=43526455296, inactive=6484176896, buffers=42299392, cached=32245600256, shared=2262142976, slab=784461824) +2022-04-12 06:53:09.952968: +epoch: 387 +2022-04-12 06:54:57.134086: train loss : -0.7015 +2022-04-12 06:55:09.911667: validation loss: -0.6772 +2022-04-12 06:55:10.076916: Average global foreground Dice: [0.9202, 0.9698, 0.9799, 0.375] +2022-04-12 06:55:10.130428: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:55:10.946275: lr: 0.002601 +2022-04-12 06:55:11.035752: saving scheduled checkpoint file... +2022-04-12 06:55:11.237920: saving checkpoint... +2022-04-12 06:55:28.485671: done, saving took 17.37 seconds +2022-04-12 06:55:28.609185: done +2022-04-12 06:55:28.631914: This epoch took 138.579623 s + +2022-04-12 06:55:28.678944: svmem(total=99932160000, available=75988336640, percent=24.0, used=19863482368, free=47727898624, active=43531939840, inactive=6532222976, buffers=43024384, cached=32297754624, shared=2261835776, slab=786329600) +2022-04-12 06:55:28.729201: +epoch: 388 +2022-04-12 06:57:18.250926: train loss : -0.7018 +2022-04-12 06:57:31.500520: validation loss: -0.6864 +2022-04-12 06:57:32.127646: Average global foreground Dice: [0.898, 0.9735, 0.9798, 0.3975] +2022-04-12 06:57:32.241135: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:57:33.039299: lr: 0.002581 +2022-04-12 06:57:33.118319: saving scheduled checkpoint file... +2022-04-12 06:57:33.258593: saving checkpoint... +2022-04-12 06:57:50.730140: done, saving took 17.60 seconds +2022-04-12 06:57:51.018284: done +2022-04-12 06:57:51.088318: This epoch took 142.294628 s + +2022-04-12 06:57:51.125079: svmem(total=99932160000, available=75829334016, percent=24.1, used=20022546432, free=47566725120, active=43704610816, inactive=6534111232, buffers=43290624, cached=32299597824, shared=2261716992, slab=783724544) +2022-04-12 06:57:51.175222: +epoch: 389 +2022-04-12 06:59:39.952045: train loss : -0.7077 +2022-04-12 06:59:53.186447: validation loss: -0.6828 +2022-04-12 06:59:55.026191: Average global foreground Dice: [0.9025, 0.9709, 0.9731, 0.4027] +2022-04-12 06:59:56.417634: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 06:59:59.656744: lr: 0.00256 +2022-04-12 07:00:00.973659: saving scheduled checkpoint file... +2022-04-12 07:00:02.191804: saving checkpoint... +2022-04-12 07:00:20.917578: done, saving took 18.82 seconds +2022-04-12 07:00:21.131512: done +2022-04-12 07:00:21.219884: This epoch took 149.994554 s + +2022-04-12 07:00:21.306831: svmem(total=99932160000, available=76071809024, percent=23.9, used=19779887104, free=47806455808, active=43453042688, inactive=6536413184, buffers=43585536, cached=32302231552, shared=2261835776, slab=785465344) +2022-04-12 07:00:21.391629: +epoch: 390 +2022-04-12 07:02:10.238240: train loss : -0.7071 +2022-04-12 07:02:22.619834: validation loss: -0.7018 +2022-04-12 07:02:23.793103: Average global foreground Dice: [0.8877, 0.9728, 0.9852, 0.4513] +2022-04-12 07:02:24.517918: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:02:27.142043: lr: 0.002539 +2022-04-12 07:02:27.743970: saving scheduled checkpoint file... +2022-04-12 07:02:28.404157: saving checkpoint... +2022-04-12 07:02:42.345636: done, saving took 14.02 seconds +2022-04-12 07:02:43.132247: done +2022-04-12 07:02:43.376382: This epoch took 141.895636 s + +2022-04-12 07:02:43.735461: svmem(total=99932160000, available=75907657728, percent=24.0, used=19957952512, free=47639928832, active=43619905536, inactive=6524579840, buffers=43868160, cached=32290410496, shared=2247856128, slab=799821824) +2022-04-12 07:02:44.296412: +epoch: 391 +2022-04-12 07:04:33.122868: train loss : -0.7080 +2022-04-12 07:04:45.293626: validation loss: -0.7013 +2022-04-12 07:04:45.358836: Average global foreground Dice: [0.8943, 0.972, 0.9836, 0.4455] +2022-04-12 07:04:45.376808: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:04:46.289381: lr: 0.002518 +2022-04-12 07:04:46.343886: saving scheduled checkpoint file... +2022-04-12 07:04:46.464837: saving checkpoint... +2022-04-12 07:05:05.408423: done, saving took 19.02 seconds +2022-04-12 07:05:07.457769: done +2022-04-12 07:05:08.244146: This epoch took 143.615683 s + +2022-04-12 07:05:09.503740: svmem(total=99932160000, available=76073914368, percent=23.9, used=19791810560, free=47803711488, active=43454099456, inactive=6526652416, buffers=44171264, cached=32292466688, shared=2247770112, slab=800169984) +2022-04-12 07:05:10.606134: +epoch: 392 +2022-04-12 07:06:59.755653: train loss : -0.7081 +2022-04-12 07:07:12.046492: validation loss: -0.6815 +2022-04-12 07:07:12.137107: Average global foreground Dice: [0.8976, 0.971, 0.9803, 0.4401] +2022-04-12 07:07:12.192566: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:07:12.937935: lr: 0.002497 +2022-04-12 07:07:13.113620: saving scheduled checkpoint file... +2022-04-12 07:07:13.272179: saving checkpoint... +2022-04-12 07:07:24.541468: done, saving took 11.35 seconds +2022-04-12 07:07:24.972268: done +2022-04-12 07:07:25.208080: saving checkpoint... +2022-04-12 07:07:38.068627: done, saving took 12.94 seconds +2022-04-12 07:07:38.248086: This epoch took 146.514914 s + +2022-04-12 07:07:38.291863: svmem(total=99932160000, available=76030631936, percent=23.9, used=19835068416, free=47920414720, active=43505827840, inactive=6366789632, buffers=44470272, cached=32132206592, shared=2247770112, slab=792899584) +2022-04-12 07:07:38.366852: +epoch: 393 +2022-04-12 07:09:26.972041: train loss : -0.7077 +2022-04-12 07:09:38.401164: validation loss: -0.6667 +2022-04-12 07:09:38.542239: Average global foreground Dice: [0.8993, 0.9696, 0.9753, 0.4304] +2022-04-12 07:09:38.618669: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:09:39.487385: lr: 0.002476 +2022-04-12 07:09:39.560107: saving scheduled checkpoint file... +2022-04-12 07:09:39.749590: saving checkpoint... +2022-04-12 07:10:13.623752: done, saving took 33.97 seconds +2022-04-12 07:10:13.767549: done +2022-04-12 07:10:13.938403: saving checkpoint... +2022-04-12 07:10:33.969036: done, saving took 20.15 seconds +2022-04-12 07:10:35.841099: This epoch took 177.409221 s + +2022-04-12 07:10:35.888078: svmem(total=99932160000, available=76043481088, percent=23.9, used=19822182400, free=47767875584, active=43491864576, inactive=6531211264, buffers=44822528, cached=32297279488, shared=2247684096, slab=788054016) +2022-04-12 07:10:35.943543: +epoch: 394 +2022-04-12 07:12:23.989276: train loss : -0.7137 +2022-04-12 07:12:35.329493: validation loss: -0.7304 +2022-04-12 07:12:35.472144: Average global foreground Dice: [0.9233, 0.9741, 0.9869, 0.4468] +2022-04-12 07:12:35.579571: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:12:36.804199: lr: 0.002455 +2022-04-12 07:12:37.369383: saving scheduled checkpoint file... +2022-04-12 07:12:37.925566: saving checkpoint... +2022-04-12 07:12:54.975364: done, saving took 17.17 seconds +2022-04-12 07:12:55.106736: done +2022-04-12 07:12:55.234747: saving checkpoint... +2022-04-12 07:13:15.270362: done, saving took 20.15 seconds +2022-04-12 07:13:15.744670: This epoch took 159.701907 s + +2022-04-12 07:13:15.870847: svmem(total=99932160000, available=76131389440, percent=23.8, used=19720155136, free=47853035520, active=43396014080, inactive=6547623936, buffers=45129728, cached=32313839616, shared=2261733376, slab=786886656) +2022-04-12 07:13:16.011241: +epoch: 395 +2022-04-12 07:15:06.632503: train loss : -0.7155 +2022-04-12 07:15:17.729224: validation loss: -0.6907 +2022-04-12 07:15:17.820560: Average global foreground Dice: [0.9168, 0.9685, 0.9761, 0.4212] +2022-04-12 07:15:17.932408: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:15:19.044515: lr: 0.002434 +2022-04-12 07:15:19.085672: saving scheduled checkpoint file... +2022-04-12 07:15:19.199448: saving checkpoint... +2022-04-12 07:15:29.568047: done, saving took 10.47 seconds +2022-04-12 07:15:29.697968: done +2022-04-12 07:15:29.770875: saving checkpoint... +2022-04-12 07:15:52.061371: done, saving took 22.34 seconds +2022-04-12 07:15:52.695215: This epoch took 156.576321 s + +2022-04-12 07:15:52.709641: svmem(total=99932160000, available=76252835840, percent=23.7, used=19612676096, free=47971889152, active=43293306880, inactive=6535868416, buffers=45469696, cached=32302125056, shared=2247770112, slab=787574784) +2022-04-12 07:15:52.924877: +epoch: 396 +2022-04-12 07:17:40.316907: train loss : -0.6891 +2022-04-12 07:17:52.334098: validation loss: -0.6679 +2022-04-12 07:17:52.928783: Average global foreground Dice: [0.9121, 0.9721, 0.9781, 0.4065] +2022-04-12 07:17:53.014869: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:17:53.870763: lr: 0.002413 +2022-04-12 07:17:53.917030: saving scheduled checkpoint file... +2022-04-12 07:17:54.028366: saving checkpoint... +2022-04-12 07:18:14.194810: done, saving took 20.24 seconds +2022-04-12 07:18:14.580700: done +2022-04-12 07:18:14.752604: This epoch took 141.523180 s + +2022-04-12 07:18:14.805349: svmem(total=99932160000, available=76147957760, percent=23.8, used=19715575808, free=47864586240, active=43392638976, inactive=6539972608, buffers=45760512, cached=32306237440, shared=2249752576, slab=787656704) +2022-04-12 07:18:14.854582: +epoch: 397 +2022-04-12 07:20:01.247206: train loss : -0.7081 +2022-04-12 07:20:13.058248: validation loss: -0.6602 +2022-04-12 07:20:13.125330: Average global foreground Dice: [0.9188, 0.9653, 0.9628, 0.39] +2022-04-12 07:20:13.233255: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:20:14.241195: lr: 0.002391 +2022-04-12 07:20:14.310353: saving scheduled checkpoint file... +2022-04-12 07:20:14.438934: saving checkpoint... +2022-04-12 07:20:32.223104: done, saving took 17.85 seconds +2022-04-12 07:20:32.378870: done +2022-04-12 07:20:32.452333: This epoch took 137.532333 s + +2022-04-12 07:20:32.532550: svmem(total=99932160000, available=76324921344, percent=23.6, used=19540250624, free=48035811328, active=43257647104, inactive=6498734080, buffers=46800896, cached=32309297152, shared=2247729152, slab=788291584) +2022-04-12 07:20:32.639259: +epoch: 398 +2022-04-12 07:22:20.291757: train loss : -0.7158 +2022-04-12 07:22:31.710324: validation loss: -0.6478 +2022-04-12 07:22:31.840448: Average global foreground Dice: [0.9234, 0.9604, 0.9703, 0.4297] +2022-04-12 07:22:31.970406: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:22:32.854284: lr: 0.00237 +2022-04-12 07:22:32.920053: saving scheduled checkpoint file... +2022-04-12 07:22:33.055439: saving checkpoint... +2022-04-12 07:22:51.914028: done, saving took 18.93 seconds +2022-04-12 07:22:52.104554: done +2022-04-12 07:22:52.189509: This epoch took 139.498497 s + +2022-04-12 07:22:52.193964: svmem(total=99932160000, available=76200738816, percent=23.7, used=19670753280, free=47909376000, active=43388764160, inactive=6493741056, buffers=47075328, cached=32304955392, shared=2241335296, slab=789999616) +2022-04-12 07:22:52.282658: +epoch: 399 +2022-04-12 07:24:40.175260: train loss : -0.7061 +2022-04-12 07:24:52.692944: validation loss: -0.6906 +2022-04-12 07:24:52.775490: Average global foreground Dice: [0.891, 0.9705, 0.9814, 0.4175] +2022-04-12 07:24:52.846453: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:24:53.517261: lr: 0.002349 +2022-04-12 07:24:53.605669: saving scheduled checkpoint file... +2022-04-12 07:24:53.757117: saving checkpoint... +2022-04-12 07:25:13.445569: done, saving took 19.79 seconds +2022-04-12 07:25:13.569746: done +2022-04-12 07:25:13.606559: This epoch took 141.151786 s + +2022-04-12 07:25:13.645169: svmem(total=99932160000, available=76304846848, percent=23.6, used=19594915840, free=48011018240, active=43316137984, inactive=6467502080, buffers=47366144, cached=32278859776, shared=2212999168, slab=789409792) +2022-04-12 07:25:13.690610: +epoch: 400 +2022-04-12 07:27:00.471510: train loss : -0.6986 +2022-04-12 07:27:12.518146: validation loss: -0.6881 +2022-04-12 07:27:12.808062: Average global foreground Dice: [0.918, 0.9704, 0.9805, 0.4271] +2022-04-12 07:27:12.921438: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:27:13.902461: lr: 0.002328 +2022-04-12 07:27:13.980657: saving scheduled checkpoint file... +2022-04-12 07:27:14.151962: saving checkpoint... +2022-04-12 07:27:27.875479: done, saving took 13.84 seconds +2022-04-12 07:27:28.242192: done +2022-04-12 07:27:28.308179: This epoch took 134.567595 s + +2022-04-12 07:27:28.360480: svmem(total=99932160000, available=76132974592, percent=23.8, used=19717644288, free=47836868608, active=43441389568, inactive=6518784000, buffers=47636480, cached=32330010624, shared=2262032384, slab=786419712) +2022-04-12 07:27:28.414756: +epoch: 401 +2022-04-12 07:29:16.091314: train loss : -0.7114 +2022-04-12 07:29:27.858854: validation loss: -0.6452 +2022-04-12 07:29:27.917824: Average global foreground Dice: [0.9075, 0.9667, 0.9699, 0.421] +2022-04-12 07:29:27.931746: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:29:28.939070: lr: 0.002307 +2022-04-12 07:29:28.943398: saving scheduled checkpoint file... +2022-04-12 07:29:29.011629: saving checkpoint... +2022-04-12 07:29:42.171249: done, saving took 13.22 seconds +2022-04-12 07:29:42.299692: done +2022-04-12 07:29:42.354892: This epoch took 133.879998 s + +2022-04-12 07:29:42.364432: svmem(total=99932160000, available=75981840384, percent=24.0, used=19869036544, free=47683678208, active=43589681152, inactive=6520066048, buffers=47886336, cached=32331558912, shared=2261835776, slab=787660800) +2022-04-12 07:29:42.426809: +epoch: 402 +2022-04-12 07:31:31.615172: train loss : -0.7081 +2022-04-12 07:31:43.047739: validation loss: -0.6744 +2022-04-12 07:31:43.073022: Average global foreground Dice: [0.9133, 0.9703, 0.9736, 0.4203] +2022-04-12 07:31:43.148881: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:31:44.053813: lr: 0.002286 +2022-04-12 07:31:44.101971: saving scheduled checkpoint file... +2022-04-12 07:31:44.310339: saving checkpoint... +2022-04-12 07:32:05.381555: done, saving took 21.17 seconds +2022-04-12 07:32:05.556515: done +2022-04-12 07:32:05.583726: This epoch took 143.100895 s + +2022-04-12 07:32:05.633559: svmem(total=99932160000, available=75948867584, percent=24.0, used=19895291904, free=47648600064, active=43620806656, inactive=6529163264, buffers=48144384, cached=32340123648, shared=2268491776, slab=788570112) +2022-04-12 07:32:05.707662: +epoch: 403 +2022-04-12 07:33:54.117302: train loss : -0.7088 +2022-04-12 07:34:06.980239: validation loss: -0.6607 +2022-04-12 07:34:07.226796: Average global foreground Dice: [0.898, 0.9677, 0.9612, 0.4325] +2022-04-12 07:34:07.365358: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:34:08.341404: lr: 0.002264 +2022-04-12 07:34:08.411473: saving scheduled checkpoint file... +2022-04-12 07:34:08.561033: saving checkpoint... +2022-04-12 07:34:21.805224: done, saving took 13.31 seconds +2022-04-12 07:34:21.995779: done +2022-04-12 07:34:22.029491: This epoch took 136.250354 s + +2022-04-12 07:34:22.063803: svmem(total=99932160000, available=75907973120, percent=24.0, used=19918888960, free=47605542912, active=43644661760, inactive=6547693568, buffers=48398336, cached=32359329792, shared=2285744128, slab=786345984) +2022-04-12 07:34:22.091478: +epoch: 404 +2022-04-12 07:36:12.312649: train loss : -0.7087 +2022-04-12 07:36:24.813869: validation loss: -0.7204 +2022-04-12 07:36:24.861596: Average global foreground Dice: [0.9252, 0.9723, 0.9852, 0.4326] +2022-04-12 07:36:24.918464: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:36:25.876463: lr: 0.002243 +2022-04-12 07:36:25.938982: saving scheduled checkpoint file... +2022-04-12 07:36:26.152246: saving checkpoint... +2022-04-12 07:36:42.913732: done, saving took 16.88 seconds +2022-04-12 07:36:43.193446: done +2022-04-12 07:36:43.274356: This epoch took 141.134863 s + +2022-04-12 07:36:43.337310: svmem(total=99932160000, available=75916824576, percent=24.0, used=19933880320, free=47614074880, active=43657564160, inactive=6523838464, buffers=48693248, cached=32335511552, shared=2261921792, slab=788713472) +2022-04-12 07:36:43.355242: +epoch: 405 +2022-04-12 07:38:32.016853: train loss : -0.7087 +2022-04-12 07:38:44.398254: validation loss: -0.7102 +2022-04-12 07:38:44.459204: Average global foreground Dice: [0.9067, 0.9725, 0.9834, 0.4349] +2022-04-12 07:38:44.519272: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:38:45.775178: lr: 0.002222 +2022-04-12 07:38:45.881595: saving scheduled checkpoint file... +2022-04-12 07:38:46.055336: saving checkpoint... +2022-04-12 07:38:57.308284: done, saving took 11.36 seconds +2022-04-12 07:38:57.556906: done +2022-04-12 07:38:57.693444: saving checkpoint... +2022-04-12 07:39:17.190655: done, saving took 19.61 seconds +2022-04-12 07:39:17.391829: This epoch took 154.003123 s + +2022-04-12 07:39:17.421688: svmem(total=99932160000, available=75899191296, percent=24.0, used=19951792128, free=47594086400, active=43675181056, inactive=6525468672, buffers=48975872, cached=32337305600, shared=2261557248, slab=788615168) +2022-04-12 07:39:17.464138: +epoch: 406 +2022-04-12 07:41:06.621822: train loss : -0.7146 +2022-04-12 07:41:18.053239: validation loss: -0.6964 +2022-04-12 07:41:18.122765: Average global foreground Dice: [0.9269, 0.9731, 0.9789, 0.385] +2022-04-12 07:41:18.184550: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:41:19.491113: lr: 0.002201 +2022-04-12 07:41:19.554569: saving scheduled checkpoint file... +2022-04-12 07:41:19.711968: saving checkpoint... +2022-04-12 07:41:29.981616: done, saving took 10.37 seconds +2022-04-12 07:41:30.184965: done +2022-04-12 07:41:30.228738: This epoch took 132.742906 s + +2022-04-12 07:41:30.245722: svmem(total=99932160000, available=75863400448, percent=24.1, used=19987714048, free=47558119424, active=43712114688, inactive=6526808064, buffers=49233920, cached=32337092608, shared=2261422080, slab=787853312) +2022-04-12 07:41:30.300343: +epoch: 407 +2022-04-12 07:43:18.443975: train loss : -0.7061 +2022-04-12 07:43:31.476734: validation loss: -0.6528 +2022-04-12 07:43:31.588454: Average global foreground Dice: [0.9037, 0.9641, 0.9663, 0.4403] +2022-04-12 07:43:31.602039: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:43:32.824361: lr: 0.002179 +2022-04-12 07:43:32.938518: saving scheduled checkpoint file... +2022-04-12 07:43:32.999784: saving checkpoint... +2022-04-12 07:43:52.638621: done, saving took 19.69 seconds +2022-04-12 07:43:52.907912: done +2022-04-12 07:43:52.991368: This epoch took 142.683156 s + +2022-04-12 07:43:53.071617: svmem(total=99932160000, available=75757821952, percent=24.2, used=20082479104, free=47450419200, active=43808489472, inactive=6539452416, buffers=49496064, cached=32349765632, shared=2272169984, slab=787701760) +2022-04-12 07:43:53.085747: +epoch: 408 +2022-04-12 07:45:40.418283: train loss : -0.7145 +2022-04-12 07:45:52.157485: validation loss: -0.7046 +2022-04-12 07:45:52.283610: Average global foreground Dice: [0.906, 0.9768, 0.9819, 0.4234] +2022-04-12 07:45:52.332910: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:45:53.461831: lr: 0.002158 +2022-04-12 07:45:53.559303: saving scheduled checkpoint file... +2022-04-12 07:45:53.674156: saving checkpoint... +2022-04-12 07:46:08.943374: done, saving took 15.34 seconds +2022-04-12 07:46:09.148114: done +2022-04-12 07:46:09.152339: This epoch took 135.974278 s + +2022-04-12 07:46:09.211955: svmem(total=99932160000, available=76022530048, percent=23.9, used=19827998720, free=47712923648, active=43549396992, inactive=6531178496, buffers=49770496, cached=32341467136, shared=2261757952, slab=789831680) +2022-04-12 07:46:09.245410: +epoch: 409 +2022-04-12 07:47:56.666354: train loss : -0.7130 +2022-04-12 07:48:09.706958: validation loss: -0.6945 +2022-04-12 07:48:09.819788: Average global foreground Dice: [0.9085, 0.9751, 0.9862, 0.4296] +2022-04-12 07:48:09.903102: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:48:11.631889: lr: 0.002137 +2022-04-12 07:48:12.469237: saving scheduled checkpoint file... +2022-04-12 07:48:12.780058: saving checkpoint... +2022-04-12 07:48:28.848754: done, saving took 16.13 seconds +2022-04-12 07:48:29.237051: done +2022-04-12 07:48:29.459441: saving checkpoint... +2022-04-12 07:48:48.394182: done, saving took 19.01 seconds +2022-04-12 07:48:49.004974: This epoch took 159.665555 s + +2022-04-12 07:48:49.161090: svmem(total=99932160000, available=75887022080, percent=24.1, used=19963392000, free=47675219968, active=43593977856, inactive=6523908096, buffers=50073600, cached=32243474432, shared=2261823488, slab=790208512) +2022-04-12 07:48:49.286834: +epoch: 410 +2022-04-12 07:50:36.213222: train loss : -0.7045 +2022-04-12 07:50:48.210407: validation loss: -0.7056 +2022-04-12 07:50:48.410301: Average global foreground Dice: [0.9026, 0.9715, 0.9831, 0.4187] +2022-04-12 07:50:48.499031: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:50:49.493936: lr: 0.002115 +2022-04-12 07:50:49.709231: saving scheduled checkpoint file... +2022-04-12 07:50:49.954091: saving checkpoint... +2022-04-12 07:51:04.766761: done, saving took 14.93 seconds +2022-04-12 07:51:04.947917: done +2022-04-12 07:51:04.998469: This epoch took 135.608994 s + +2022-04-12 07:51:05.013055: svmem(total=99932160000, available=75983208448, percent=24.0, used=19860389888, free=47769268224, active=43492007936, inactive=6532685824, buffers=50323456, cached=32252178432, shared=2268672000, slab=789348352) +2022-04-12 07:51:05.022909: +epoch: 411 +2022-04-12 07:52:53.165936: train loss : -0.7142 +2022-04-12 07:53:05.625122: validation loss: -0.6862 +2022-04-12 07:53:05.696063: Average global foreground Dice: [0.9073, 0.9742, 0.9809, 0.4119] +2022-04-12 07:53:05.730047: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:53:06.547633: lr: 0.002094 +2022-04-12 07:53:06.607457: saving scheduled checkpoint file... +2022-04-12 07:53:06.873146: saving checkpoint... +2022-04-12 07:53:20.964807: done, saving took 14.21 seconds +2022-04-12 07:53:21.072358: done +2022-04-12 07:53:21.076121: This epoch took 136.041353 s + +2022-04-12 07:53:21.079083: svmem(total=99932160000, available=75819798528, percent=24.1, used=20030922752, free=47603650560, active=43658846208, inactive=6528466944, buffers=50569216, cached=32247017472, shared=2261532672, slab=789434368) +2022-04-12 07:53:21.081933: +epoch: 412 +2022-04-12 07:55:10.125965: train loss : -0.7159 +2022-04-12 07:55:22.184870: validation loss: -0.6814 +2022-04-12 07:55:22.392989: Average global foreground Dice: [0.9114, 0.9737, 0.9823, 0.4121] +2022-04-12 07:55:22.533912: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:55:23.798939: lr: 0.002072 +2022-04-12 07:55:24.133961: saving scheduled checkpoint file... +2022-04-12 07:55:24.375932: saving checkpoint... +2022-04-12 07:55:36.569971: done, saving took 12.28 seconds +2022-04-12 07:55:36.741660: done +2022-04-12 07:55:36.817078: This epoch took 135.732697 s + +2022-04-12 07:55:36.876791: svmem(total=99932160000, available=75746119680, percent=24.2, used=20104318976, free=47527800832, active=43736739840, inactive=6529339392, buffers=50868224, cached=32249171968, shared=2261671936, slab=790294528) +2022-04-12 07:55:36.884550: +epoch: 413 +2022-04-12 07:57:26.289050: train loss : -0.7177 +2022-04-12 07:57:39.875568: validation loss: -0.7289 +2022-04-12 07:57:39.914317: Average global foreground Dice: [0.9193, 0.9751, 0.9867, 0.424] +2022-04-12 07:57:39.962322: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 07:57:40.905927: lr: 0.002051 +2022-04-12 07:57:40.950336: saving scheduled checkpoint file... +2022-04-12 07:57:41.002850: saving checkpoint... +2022-04-12 07:57:50.986600: done, saving took 10.03 seconds +2022-04-12 07:57:51.207343: done +2022-04-12 07:57:51.286894: saving checkpoint... +2022-04-12 07:58:05.398518: done, saving took 14.18 seconds +2022-04-12 07:58:05.447441: This epoch took 148.473946 s + +2022-04-12 07:58:05.451219: svmem(total=99932160000, available=76141068288, percent=23.8, used=19709161472, free=47920267264, active=43339010048, inactive=6531649536, buffers=51142656, cached=32251588608, shared=2261843968, slab=788533248) +2022-04-12 07:58:05.452965: +epoch: 414 +2022-04-12 07:59:53.288860: train loss : -0.7222 +2022-04-12 08:00:04.964714: validation loss: -0.6637 +2022-04-12 08:00:05.936780: Average global foreground Dice: [0.8978, 0.9626, 0.9653, 0.4397] +2022-04-12 08:00:06.495990: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:00:07.891211: lr: 0.00203 +2022-04-12 08:00:07.936422: saving scheduled checkpoint file... +2022-04-12 08:00:08.089674: saving checkpoint... +2022-04-12 08:00:38.046609: done, saving took 30.06 seconds +2022-04-12 08:00:38.289643: done +2022-04-12 08:00:38.375510: This epoch took 152.920373 s + +2022-04-12 08:00:38.455075: svmem(total=99932160000, available=75910959104, percent=24.0, used=19939274752, free=47687876608, active=43572654080, inactive=6533447680, buffers=51433472, cached=32253575168, shared=2261647360, slab=790487040) +2022-04-12 08:00:38.536124: +epoch: 415 +2022-04-12 08:02:27.165164: train loss : -0.7268 +2022-04-12 08:02:43.726649: validation loss: -0.7314 +2022-04-12 08:02:43.925111: Average global foreground Dice: [0.9146, 0.9779, 0.9886, 0.4291] +2022-04-12 08:02:44.021603: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:02:45.212959: lr: 0.002008 +2022-04-12 08:02:45.307742: saving scheduled checkpoint file... +2022-04-12 08:02:45.472289: saving checkpoint... +2022-04-12 08:03:01.215554: done, saving took 15.87 seconds +2022-04-12 08:03:01.283831: done +2022-04-12 08:03:01.403505: saving checkpoint... +2022-04-12 08:03:13.149704: done, saving took 11.85 seconds +2022-04-12 08:03:13.292627: This epoch took 154.648126 s + +2022-04-12 08:03:13.339650: svmem(total=99932160000, available=75807739904, percent=24.1, used=20056293376, free=47582023680, active=43688046592, inactive=6522052608, buffers=51712000, cached=32242130944, shared=2247942144, slab=790343680) +2022-04-12 08:03:13.391557: +epoch: 416 +2022-04-12 08:05:04.636791: train loss : -0.7221 +2022-04-12 08:05:16.546427: validation loss: -0.6508 +2022-04-12 08:05:16.694091: Average global foreground Dice: [0.9065, 0.9664, 0.967, 0.4156] +2022-04-12 08:05:16.822137: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:05:17.765935: lr: 0.001987 +2022-04-12 08:05:17.847331: saving scheduled checkpoint file... +2022-04-12 08:05:18.027260: saving checkpoint... +2022-04-12 08:05:34.061926: done, saving took 16.15 seconds +2022-04-12 08:05:34.314766: done +2022-04-12 08:05:34.437645: This epoch took 140.988534 s + +2022-04-12 08:05:34.466999: svmem(total=99932160000, available=75880857600, percent=24.1, used=19967365120, free=47652835328, active=43597217792, inactive=6540328960, buffers=52002816, cached=32259956736, shared=2263732224, slab=790724608) +2022-04-12 08:05:34.568937: +epoch: 417 +2022-04-12 08:07:23.653921: train loss : -0.7233 +2022-04-12 08:07:36.172675: validation loss: -0.7191 +2022-04-12 08:07:36.310313: Average global foreground Dice: [0.9102, 0.9756, 0.9859, 0.4214] +2022-04-12 08:07:36.376138: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:07:37.323340: lr: 0.001965 +2022-04-12 08:07:37.417416: saving scheduled checkpoint file... +2022-04-12 08:07:37.612045: saving checkpoint... +2022-04-12 08:07:55.897831: done, saving took 18.42 seconds +2022-04-12 08:07:56.135241: done +2022-04-12 08:07:56.184877: This epoch took 141.549087 s + +2022-04-12 08:07:56.302224: svmem(total=99932160000, available=76083408896, percent=23.9, used=19766751232, free=47853170688, active=43399774208, inactive=6539788288, buffers=52260864, cached=32259977216, shared=2261733376, slab=791924736) +2022-04-12 08:07:56.345376: +epoch: 418 +2022-04-12 08:09:46.303713: train loss : -0.7139 +2022-04-12 08:09:58.485687: validation loss: -0.7094 +2022-04-12 08:09:58.554342: Average global foreground Dice: [0.9273, 0.977, 0.9846, 0.4023] +2022-04-12 08:09:58.670294: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:09:59.595144: lr: 0.001943 +2022-04-12 08:09:59.672703: saving scheduled checkpoint file... +2022-04-12 08:09:59.828469: saving checkpoint... +2022-04-12 08:10:11.672825: done, saving took 11.93 seconds +2022-04-12 08:10:11.843431: done +2022-04-12 08:10:11.898602: This epoch took 135.435056 s + +2022-04-12 08:10:11.971223: svmem(total=99932160000, available=76013424640, percent=23.9, used=19836502016, free=47781007360, active=43471671296, inactive=6541795328, buffers=52510720, cached=32262139904, shared=2261860352, slab=790216704) +2022-04-12 08:10:12.026153: +epoch: 419 +2022-04-12 08:11:59.459299: train loss : -0.7038 +2022-04-12 08:12:11.337075: validation loss: -0.6885 +2022-04-12 08:12:11.390260: Average global foreground Dice: [0.9093, 0.9683, 0.9765, 0.4446] +2022-04-12 08:12:11.432323: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:12:12.407770: lr: 0.001922 +2022-04-12 08:12:12.514604: saving scheduled checkpoint file... +2022-04-12 08:12:12.613527: saving checkpoint... +2022-04-12 08:12:32.014924: done, saving took 19.48 seconds +2022-04-12 08:12:32.400039: done +2022-04-12 08:12:32.477065: saving checkpoint... +2022-04-12 08:12:56.795264: done, saving took 24.38 seconds +2022-04-12 08:12:56.928990: This epoch took 164.866553 s + +2022-04-12 08:12:57.062899: svmem(total=99932160000, available=75871883264, percent=24.1, used=19975962624, free=47636811776, active=43609993216, inactive=6546055168, buffers=52822016, cached=32266563584, shared=2263912448, slab=791031808) +2022-04-12 08:12:57.112348: +epoch: 420 +2022-04-12 08:14:44.463167: train loss : -0.7169 +2022-04-12 08:14:56.006869: validation loss: -0.6791 +2022-04-12 08:14:56.067888: Average global foreground Dice: [0.9044, 0.9656, 0.9671, 0.4391] +2022-04-12 08:14:56.177652: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:14:57.115830: lr: 0.0019 +2022-04-12 08:14:57.129145: saving scheduled checkpoint file... +2022-04-12 08:14:57.249552: saving checkpoint... +2022-04-12 08:15:13.450448: done, saving took 16.26 seconds +2022-04-12 08:15:13.623425: done +2022-04-12 08:15:13.638336: This epoch took 136.436566 s + +2022-04-12 08:15:13.709268: svmem(total=99932160000, available=75928428544, percent=24.0, used=19914223616, free=47691153408, active=43550289920, inactive=6553305088, buffers=53096448, cached=32273686528, shared=2269204480, slab=789762048) +2022-04-12 08:15:13.783704: +epoch: 421 +2022-04-12 08:17:00.595994: train loss : -0.7196 +2022-04-12 08:17:12.396938: validation loss: -0.6798 +2022-04-12 08:17:12.589362: Average global foreground Dice: [0.9203, 0.9717, 0.9778, 0.4065] +2022-04-12 08:17:12.776608: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:17:13.898632: lr: 0.001879 +2022-04-12 08:17:14.109277: saving scheduled checkpoint file... +2022-04-12 08:17:14.237215: saving checkpoint... +2022-04-12 08:17:36.447659: done, saving took 22.29 seconds +2022-04-12 08:17:36.661226: done +2022-04-12 08:17:36.708455: This epoch took 142.921766 s + +2022-04-12 08:17:36.815425: svmem(total=99932160000, available=75873947648, percent=24.1, used=19975938048, free=47634419712, active=43608567808, inactive=6547869696, buffers=53391360, cached=32268410880, shared=2261827584, slab=791113728) +2022-04-12 08:17:36.882514: +epoch: 422 +2022-04-12 08:19:23.304500: train loss : -0.7200 +2022-04-12 08:19:34.600649: validation loss: -0.7004 +2022-04-12 08:19:34.666128: Average global foreground Dice: [0.9183, 0.9724, 0.9781, 0.4413] +2022-04-12 08:19:34.709274: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:19:35.609435: lr: 0.001857 +2022-04-12 08:19:35.656237: saving scheduled checkpoint file... +2022-04-12 08:19:35.787789: saving checkpoint... +2022-04-12 08:19:53.341197: done, saving took 17.67 seconds +2022-04-12 08:19:53.612977: done +2022-04-12 08:19:53.735609: saving checkpoint... +2022-04-12 08:20:05.864790: done, saving took 12.22 seconds +2022-04-12 08:20:06.041864: This epoch took 149.040636 s + +2022-04-12 08:20:06.148757: svmem(total=99932160000, available=75844444160, percent=24.1, used=19989696512, free=47602536448, active=43627839488, inactive=6565789696, buffers=53673984, cached=32286253056, shared=2277617664, slab=790290432) +2022-04-12 08:20:06.227892: +epoch: 423 +2022-04-12 08:21:54.120020: train loss : -0.7239 +2022-04-12 08:22:05.966803: validation loss: -0.6811 +2022-04-12 08:22:06.159117: Average global foreground Dice: [0.9158, 0.9663, 0.9735, 0.441] +2022-04-12 08:22:06.219156: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:22:07.315097: lr: 0.001835 +2022-04-12 08:22:07.358390: saving scheduled checkpoint file... +2022-04-12 08:22:07.513688: saving checkpoint... +2022-04-12 08:22:21.664251: done, saving took 14.25 seconds +2022-04-12 08:22:21.934873: done +2022-04-12 08:22:22.123416: saving checkpoint... +2022-04-12 08:22:32.886665: done, saving took 10.87 seconds +2022-04-12 08:22:33.177750: This epoch took 146.834070 s + +2022-04-12 08:22:33.268232: svmem(total=99932160000, available=75888078848, percent=24.1, used=19960885248, free=47643951104, active=43600723968, inactive=6552928256, buffers=53964800, cached=32273358848, shared=2262827008, slab=789172224) +2022-04-12 08:22:33.309693: +epoch: 424 +2022-04-12 08:24:22.623488: train loss : -0.7299 +2022-04-12 08:24:34.506198: validation loss: -0.7052 +2022-04-12 08:24:34.571774: Average global foreground Dice: [0.9163, 0.9702, 0.9784, 0.4392] +2022-04-12 08:24:34.609314: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:24:35.924356: lr: 0.001813 +2022-04-12 08:24:36.029296: saving scheduled checkpoint file... +2022-04-12 08:24:36.152819: saving checkpoint... +2022-04-12 08:24:52.360070: done, saving took 16.31 seconds +2022-04-12 08:24:52.595646: done +2022-04-12 08:24:52.710639: saving checkpoint... +2022-04-12 08:25:10.914165: done, saving took 18.26 seconds +2022-04-12 08:25:11.080873: This epoch took 157.726001 s + +2022-04-12 08:25:11.151434: svmem(total=99932160000, available=75741036544, percent=24.2, used=20109045760, free=47494340608, active=43740950528, inactive=6554451968, buffers=54272000, cached=32274501632, shared=2261655552, slab=789815296) +2022-04-12 08:25:11.167641: +epoch: 425 +2022-04-12 08:26:59.802057: train loss : -0.7221 +2022-04-12 08:27:12.103130: validation loss: -0.7260 +2022-04-12 08:27:12.135071: Average global foreground Dice: [0.935, 0.9779, 0.9856, 0.436] +2022-04-12 08:27:12.163254: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:27:12.955579: lr: 0.001792 +2022-04-12 08:27:13.048479: saving scheduled checkpoint file... +2022-04-12 08:27:13.160346: saving checkpoint... +2022-04-12 08:27:28.641854: done, saving took 15.57 seconds +2022-04-12 08:27:28.902067: done +2022-04-12 08:27:29.009027: saving checkpoint... +2022-04-12 08:27:47.105392: done, saving took 18.15 seconds +2022-04-12 08:27:47.370549: This epoch took 156.110431 s + +2022-04-12 08:27:47.594869: svmem(total=99932160000, available=75654262784, percent=24.3, used=20186873856, free=47405117440, active=43819884544, inactive=6564917248, buffers=54575104, cached=32285593600, shared=2270543872, slab=791085056) +2022-04-12 08:27:47.713620: +epoch: 426 +2022-04-12 08:29:36.648303: train loss : -0.7227 +2022-04-12 08:29:48.256639: validation loss: -0.6787 +2022-04-12 08:29:48.350878: Average global foreground Dice: [0.914, 0.9698, 0.9709, 0.4371] +2022-04-12 08:29:48.419147: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:29:49.438136: lr: 0.00177 +2022-04-12 08:29:49.557539: saving scheduled checkpoint file... +2022-04-12 08:29:49.662966: saving checkpoint... +2022-04-12 08:30:07.810368: done, saving took 18.20 seconds +2022-04-12 08:30:08.024308: done +2022-04-12 08:30:08.147808: This epoch took 140.321820 s + +2022-04-12 08:30:08.264678: svmem(total=99932160000, available=75783671808, percent=24.2, used=20066246656, free=47531892736, active=43726180352, inactive=6530351104, buffers=55042048, cached=32278978560, shared=2261733376, slab=792588288) +2022-04-12 08:30:08.315742: +epoch: 427 +2022-04-12 08:31:56.861831: train loss : -0.7106 +2022-04-12 08:32:08.572088: validation loss: -0.6904 +2022-04-12 08:32:08.819698: Average global foreground Dice: [0.8963, 0.967, 0.9725, 0.4408] +2022-04-12 08:32:08.873187: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:32:10.202828: lr: 0.001748 +2022-04-12 08:32:10.495707: saving scheduled checkpoint file... +2022-04-12 08:32:11.061768: saving checkpoint... +2022-04-12 08:32:29.245443: done, saving took 18.27 seconds +2022-04-12 08:32:29.568432: done +2022-04-12 08:32:29.640189: This epoch took 141.262947 s + +2022-04-12 08:32:29.782411: svmem(total=99932160000, available=75798274048, percent=24.2, used=20051230720, free=47544066048, active=43705819136, inactive=6532575232, buffers=55320576, cached=32281542656, shared=2262056960, slab=792264704) +2022-04-12 08:32:29.855636: +epoch: 428 +2022-04-12 08:34:18.798039: train loss : -0.7203 +2022-04-12 08:34:31.208051: validation loss: -0.7187 +2022-04-12 08:34:31.292652: Average global foreground Dice: [0.9202, 0.9745, 0.9865, 0.427] +2022-04-12 08:34:31.460967: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:34:32.504235: lr: 0.001726 +2022-04-12 08:34:32.598295: saving scheduled checkpoint file... +2022-04-12 08:34:32.807335: saving checkpoint... +2022-04-12 08:34:51.424142: done, saving took 18.70 seconds +2022-04-12 08:34:51.712628: done +2022-04-12 08:34:51.764691: This epoch took 141.849548 s + +2022-04-12 08:34:51.770003: svmem(total=99932160000, available=75725942784, percent=24.2, used=20123738112, free=47469654016, active=43786031104, inactive=6534295552, buffers=55578624, cached=32283189248, shared=2261860352, slab=791134208) +2022-04-12 08:34:51.861224: +epoch: 429 +2022-04-12 08:36:39.095435: train loss : -0.7141 +2022-04-12 08:36:51.917355: validation loss: -0.6995 +2022-04-12 08:36:51.989319: Average global foreground Dice: [0.9237, 0.9706, 0.9792, 0.4363] +2022-04-12 08:36:52.039206: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:36:53.037199: lr: 0.001704 +2022-04-12 08:36:53.087289: saving scheduled checkpoint file... +2022-04-12 08:36:53.240926: saving checkpoint... +2022-04-12 08:37:09.962202: done, saving took 16.81 seconds +2022-04-12 08:37:10.335476: done +2022-04-12 08:37:10.509698: saving checkpoint... +2022-04-12 08:37:24.941263: done, saving took 14.53 seconds +2022-04-12 08:37:25.199063: This epoch took 153.247899 s + +2022-04-12 08:37:25.278919: svmem(total=99932160000, available=75839414272, percent=24.1, used=20020150272, free=47580700672, active=43684442112, inactive=6522654720, buffers=55885824, cached=32275423232, shared=2252058624, slab=790585344) +2022-04-12 08:37:25.330322: +epoch: 430 +2022-04-12 08:39:14.320306: train loss : -0.7226 +2022-04-12 08:39:25.942804: validation loss: -0.6980 +2022-04-12 08:39:26.046374: Average global foreground Dice: [0.9114, 0.9753, 0.984, 0.4102] +2022-04-12 08:39:26.092272: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:39:27.229352: lr: 0.001682 +2022-04-12 08:39:27.289294: saving scheduled checkpoint file... +2022-04-12 08:39:27.470494: saving checkpoint... +2022-04-12 08:39:43.578172: done, saving took 16.21 seconds +2022-04-12 08:39:43.771440: done +2022-04-12 08:39:43.778064: This epoch took 138.384658 s + +2022-04-12 08:39:43.848668: svmem(total=99932160000, available=75891658752, percent=24.1, used=19944103936, free=47630680064, active=43608829952, inactive=6552199168, buffers=56135680, cached=32301240320, shared=2275737600, slab=791810048) +2022-04-12 08:39:43.930737: +epoch: 431 +2022-04-12 08:41:33.408612: train loss : -0.7267 +2022-04-12 08:41:45.480803: validation loss: -0.6974 +2022-04-12 08:41:45.513395: Average global foreground Dice: [0.9178, 0.9728, 0.9814, 0.4326] +2022-04-12 08:41:45.551709: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:41:46.504861: lr: 0.00166 +2022-04-12 08:41:46.564548: saving scheduled checkpoint file... +2022-04-12 08:41:46.706178: saving checkpoint... +2022-04-12 08:41:57.283906: done, saving took 10.67 seconds +2022-04-12 08:41:57.584727: done +2022-04-12 08:41:57.618770: This epoch took 133.573741 s + +2022-04-12 08:41:57.657279: svmem(total=99932160000, available=76036767744, percent=23.9, used=19813081088, free=47773597696, active=43482873856, inactive=6539964416, buffers=56389632, cached=32289091584, shared=2261561344, slab=790704128) +2022-04-12 08:41:57.683614: +epoch: 432 +2022-04-12 08:43:46.836541: train loss : -0.7194 +2022-04-12 08:44:00.568874: validation loss: -0.7273 +2022-04-12 08:44:01.325452: Average global foreground Dice: [0.9231, 0.9779, 0.987, 0.4394] +2022-04-12 08:44:01.422438: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:44:02.414572: lr: 0.001638 +2022-04-12 08:44:02.471614: saving scheduled checkpoint file... +2022-04-12 08:44:02.625179: saving checkpoint... +2022-04-12 08:44:20.032609: done, saving took 17.50 seconds +2022-04-12 08:44:20.185973: done +2022-04-12 08:44:20.308466: saving checkpoint... +2022-04-12 08:44:38.322353: done, saving took 18.10 seconds +2022-04-12 08:44:38.519641: This epoch took 160.782323 s + +2022-04-12 08:44:38.526892: svmem(total=99932160000, available=75920371712, percent=24.0, used=19929362432, free=47654776832, active=43598757888, inactive=6542172160, buffers=56688640, cached=32291332096, shared=2261647360, slab=790982656) +2022-04-12 08:44:38.551836: +epoch: 433 +2022-04-12 08:46:25.960160: train loss : -0.7327 +2022-04-12 08:46:38.898290: validation loss: -0.7262 +2022-04-12 08:46:39.008913: Average global foreground Dice: [0.9023, 0.977, 0.9864, 0.4655] +2022-04-12 08:46:39.026778: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:46:39.903403: lr: 0.001616 +2022-04-12 08:46:40.008807: saving scheduled checkpoint file... +2022-04-12 08:46:40.166529: saving checkpoint... +2022-04-12 08:46:53.207040: done, saving took 13.15 seconds +2022-04-12 08:46:53.342865: done +2022-04-12 08:46:53.513644: saving checkpoint... +2022-04-12 08:47:06.169699: done, saving took 12.76 seconds +2022-04-12 08:47:06.582408: This epoch took 147.960511 s + +2022-04-12 08:47:06.731596: svmem(total=99932160000, available=76015153152, percent=23.9, used=19816861696, free=47747072000, active=43489210368, inactive=6557638656, buffers=56983552, cached=32311242752, shared=2279456768, slab=792047616) +2022-04-12 08:47:07.015622: +epoch: 434 +2022-04-12 08:48:57.581575: train loss : -0.7225 +2022-04-12 08:49:09.116674: validation loss: -0.7240 +2022-04-12 08:49:09.180126: Average global foreground Dice: [0.9255, 0.977, 0.9848, 0.4179] +2022-04-12 08:49:09.211374: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:49:10.134618: lr: 0.001594 +2022-04-12 08:49:10.225920: saving scheduled checkpoint file... +2022-04-12 08:49:10.368708: saving checkpoint... +2022-04-12 08:49:26.740801: done, saving took 16.50 seconds +2022-04-12 08:49:27.083288: done +2022-04-12 08:49:27.263847: saving checkpoint... +2022-04-12 08:49:43.520302: done, saving took 16.37 seconds +2022-04-12 08:49:43.803084: This epoch took 156.627681 s + +2022-04-12 08:49:43.864318: svmem(total=99932160000, available=76062920704, percent=23.9, used=19772882944, free=47792541696, active=43438002176, inactive=6560403456, buffers=57278464, cached=32309456896, shared=2275635200, slab=791597056) +2022-04-12 08:49:43.900490: +epoch: 435 +2022-04-12 08:51:33.031802: train loss : -0.7176 +2022-04-12 08:51:44.690886: validation loss: -0.6878 +2022-04-12 08:51:45.192113: Average global foreground Dice: [0.8939, 0.9712, 0.9771, 0.4366] +2022-04-12 08:51:45.273075: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:51:46.230864: lr: 0.001572 +2022-04-12 08:51:46.337024: saving scheduled checkpoint file... +2022-04-12 08:51:46.573476: saving checkpoint... +2022-04-12 08:52:03.160916: done, saving took 16.71 seconds +2022-04-12 08:52:03.345755: done +2022-04-12 08:52:03.400614: This epoch took 139.465389 s + +2022-04-12 08:52:03.452367: svmem(total=99932160000, available=75984519168, percent=24.0, used=19859263488, free=47711776768, active=43524804608, inactive=6554288128, buffers=57536512, cached=32303583232, shared=2267602944, slab=790888448) +2022-04-12 08:52:03.497710: +epoch: 436 +2022-04-12 08:53:53.232079: train loss : -0.7297 +2022-04-12 08:54:06.574861: validation loss: -0.7174 +2022-04-12 08:54:06.651271: Average global foreground Dice: [0.9009, 0.9768, 0.9865, 0.4428] +2022-04-12 08:54:06.776293: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:54:08.065629: lr: 0.00155 +2022-04-12 08:54:08.291307: saving scheduled checkpoint file... +2022-04-12 08:54:08.575108: saving checkpoint... +2022-04-12 08:54:29.213929: done, saving took 20.73 seconds +2022-04-12 08:54:29.818989: done +2022-04-12 08:54:29.866362: This epoch took 146.275351 s + +2022-04-12 08:54:29.894240: svmem(total=99932160000, available=75920191488, percent=24.0, used=19929444352, free=47645167616, active=43598540800, inactive=6550593536, buffers=57815040, cached=32299732992, shared=2261835776, slab=790638592) +2022-04-12 08:54:29.936911: +epoch: 437 +2022-04-12 08:56:18.188893: train loss : -0.7243 +2022-04-12 08:56:30.272674: validation loss: -0.6959 +2022-04-12 08:56:30.370159: Average global foreground Dice: [0.9208, 0.9717, 0.9815, 0.4234] +2022-04-12 08:56:30.484063: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:56:31.422231: lr: 0.001528 +2022-04-12 08:56:31.579908: saving scheduled checkpoint file... +2022-04-12 08:56:31.942852: saving checkpoint... +2022-04-12 08:56:51.448309: done, saving took 19.58 seconds +2022-04-12 08:56:51.910040: done +2022-04-12 08:56:52.036742: This epoch took 142.053796 s + +2022-04-12 08:56:52.098798: svmem(total=99932160000, available=75970420736, percent=24.0, used=19878977536, free=47693074432, active=43549974528, inactive=6552600576, buffers=58105856, cached=32302002176, shared=2261942272, slab=791359488) +2022-04-12 08:56:52.162168: +epoch: 438 +2022-04-12 08:58:40.370168: train loss : -0.7213 +2022-04-12 08:58:52.244781: validation loss: -0.7195 +2022-04-12 08:58:52.445314: Average global foreground Dice: [0.9221, 0.9764, 0.9833, 0.4246] +2022-04-12 08:58:52.491658: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 08:58:53.647496: lr: 0.001506 +2022-04-12 08:58:53.761276: saving scheduled checkpoint file... +2022-04-12 08:58:53.930666: saving checkpoint... +2022-04-12 08:59:11.666822: done, saving took 17.82 seconds +2022-04-12 08:59:11.955301: done +2022-04-12 08:59:11.998886: This epoch took 139.716067 s + +2022-04-12 08:59:12.007241: svmem(total=99932160000, available=75872989184, percent=24.1, used=19976589312, free=47593488384, active=43646169088, inactive=6554816512, buffers=58347520, cached=32303734784, shared=2261757952, slab=790310912) +2022-04-12 08:59:12.066800: +epoch: 439 +2022-04-12 09:00:59.618268: train loss : -0.7328 +2022-04-12 09:01:11.042207: validation loss: -0.7153 +2022-04-12 09:01:11.272531: Average global foreground Dice: [0.9262, 0.9734, 0.9823, 0.4294] +2022-04-12 09:01:11.542288: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:01:12.692647: lr: 0.001483 +2022-04-12 09:01:12.820620: saving scheduled checkpoint file... +2022-04-12 09:01:12.925470: saving checkpoint... +2022-04-12 09:01:28.596912: done, saving took 15.72 seconds +2022-04-12 09:01:29.092242: done +2022-04-12 09:01:29.237797: saving checkpoint... +2022-04-12 09:01:43.061429: done, saving took 13.90 seconds +2022-04-12 09:01:43.230232: This epoch took 151.132009 s + +2022-04-12 09:01:43.355756: svmem(total=99932160000, available=75794722816, percent=24.2, used=20054478848, free=47512854528, active=43723931648, inactive=6556528640, buffers=58642432, cached=32306184192, shared=2262016000, slab=790687744) +2022-04-12 09:01:43.425123: +epoch: 440 +2022-04-12 09:03:32.309922: train loss : -0.7295 +2022-04-12 09:03:43.416359: validation loss: -0.7331 +2022-04-12 09:03:43.555949: Average global foreground Dice: [0.9249, 0.9779, 0.9868, 0.4339] +2022-04-12 09:03:43.591070: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:03:44.597364: lr: 0.001461 +2022-04-12 09:03:44.666065: saving scheduled checkpoint file... +2022-04-12 09:03:44.825899: saving checkpoint... +2022-04-12 09:03:58.618578: done, saving took 13.89 seconds +2022-04-12 09:03:58.940703: done +2022-04-12 09:03:59.052882: saving checkpoint... +2022-04-12 09:04:15.736168: done, saving took 16.77 seconds +2022-04-12 09:04:15.851315: This epoch took 152.331179 s + +2022-04-12 09:04:15.904025: svmem(total=99932160000, available=75934937088, percent=24.0, used=19914293248, free=47650684928, active=43582791680, inactive=6558654464, buffers=58912768, cached=32308269056, shared=2261979136, slab=791879680) +2022-04-12 09:04:15.939495: +epoch: 441 +2022-04-12 09:06:03.394294: train loss : -0.7228 +2022-04-12 09:06:14.630226: validation loss: -0.7254 +2022-04-12 09:06:14.733232: Average global foreground Dice: [0.9226, 0.9774, 0.9844, 0.4479] +2022-04-12 09:06:14.806300: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:06:15.711198: lr: 0.001439 +2022-04-12 09:06:15.762929: saving scheduled checkpoint file... +2022-04-12 09:06:15.980348: saving checkpoint... +2022-04-12 09:06:26.111818: done, saving took 10.25 seconds +2022-04-12 09:06:26.372335: done +2022-04-12 09:06:26.511956: saving checkpoint... +2022-04-12 09:06:39.207165: done, saving took 12.81 seconds +2022-04-12 09:06:39.288734: This epoch took 143.329099 s + +2022-04-12 09:06:39.325174: svmem(total=99932160000, available=75931291648, percent=24.0, used=19918274560, free=47644790784, active=43581403136, inactive=6560866304, buffers=59211776, cached=32309882880, shared=2261671936, slab=791502848) +2022-04-12 09:06:39.326994: +epoch: 442 +2022-04-12 09:08:28.477420: train loss : -0.7107 +2022-04-12 09:08:41.469383: validation loss: -0.6289 +2022-04-12 09:08:41.747205: Average global foreground Dice: [0.9086, 0.9619, 0.9715, 0.4034] +2022-04-12 09:08:41.872363: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:08:43.051309: lr: 0.001416 +2022-04-12 09:08:43.103318: saving scheduled checkpoint file... +2022-04-12 09:08:43.318264: saving checkpoint... +2022-04-12 09:09:01.741207: done, saving took 18.53 seconds +2022-04-12 09:09:02.028252: done +2022-04-12 09:09:02.098819: This epoch took 142.712630 s + +2022-04-12 09:09:02.217738: svmem(total=99932160000, available=75737870336, percent=24.2, used=20111745024, free=47448985600, active=43779092480, inactive=6562443264, buffers=59473920, cached=32311955456, shared=2261401600, slab=792219648) +2022-04-12 09:09:02.270908: +epoch: 443 +2022-04-12 09:10:50.019163: train loss : -0.7189 +2022-04-12 09:11:00.746909: validation loss: -0.7156 +2022-04-12 09:11:00.824185: Average global foreground Dice: [0.927, 0.9734, 0.9873, 0.4216] +2022-04-12 09:11:00.877424: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:11:02.260562: lr: 0.001394 +2022-04-12 09:11:02.769300: saving scheduled checkpoint file... +2022-04-12 09:11:03.058189: saving checkpoint... +2022-04-12 09:11:16.730470: done, saving took 13.75 seconds +2022-04-12 09:11:16.880742: done +2022-04-12 09:11:16.949114: This epoch took 134.601568 s + +2022-04-12 09:11:16.986860: svmem(total=99932160000, available=75972460544, percent=24.0, used=19874430976, free=47681396736, active=43548254208, inactive=6566871040, buffers=59715584, cached=32316616704, shared=2264297472, slab=791945216) +2022-04-12 09:11:17.039990: +epoch: 444 +2022-04-12 09:13:04.495157: train loss : -0.7135 +2022-04-12 09:13:15.340828: validation loss: -0.7270 +2022-04-12 09:13:15.438465: Average global foreground Dice: [0.9257, 0.9785, 0.9814, 0.4343] +2022-04-12 09:13:15.529243: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:13:16.582367: lr: 0.001372 +2022-04-12 09:13:16.649197: saving scheduled checkpoint file... +2022-04-12 09:13:16.762054: saving checkpoint... +2022-04-12 09:13:33.220217: done, saving took 16.51 seconds +2022-04-12 09:13:33.355212: done +2022-04-12 09:13:33.447813: This epoch took 136.352504 s + +2022-04-12 09:13:33.495391: svmem(total=99932160000, available=75731230720, percent=24.2, used=20107927552, free=47438024704, active=43780608000, inactive=6576443392, buffers=59973632, cached=32326234112, shared=2271993856, slab=791773184) +2022-04-12 09:13:33.539513: +epoch: 445 +2022-04-12 09:15:21.958538: train loss : -0.7271 +2022-04-12 09:15:33.414079: validation loss: -0.6887 +2022-04-12 09:15:33.440882: Average global foreground Dice: [0.9217, 0.9693, 0.9784, 0.4223] +2022-04-12 09:15:33.462904: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:15:34.421995: lr: 0.001349 +2022-04-12 09:15:34.434016: saving scheduled checkpoint file... +2022-04-12 09:15:34.564423: saving checkpoint... +2022-04-12 09:15:46.585684: done, saving took 12.09 seconds +2022-04-12 09:15:46.775351: done +2022-04-12 09:15:46.883919: This epoch took 133.306448 s + +2022-04-12 09:15:46.894265: svmem(total=99932160000, available=75852533760, percent=24.1, used=19996864512, free=47557169152, active=43663855616, inactive=6567936000, buffers=60252160, cached=32317874176, shared=2261647360, slab=792956928) +2022-04-12 09:15:46.948358: +epoch: 446 +2022-04-12 09:17:34.322901: train loss : -0.7335 +2022-04-12 09:17:47.021041: validation loss: -0.6909 +2022-04-12 09:17:47.104306: Average global foreground Dice: [0.9376, 0.9724, 0.9728, 0.4104] +2022-04-12 09:17:47.161574: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:17:47.947348: lr: 0.001327 +2022-04-12 09:17:47.975527: saving scheduled checkpoint file... +2022-04-12 09:17:48.047338: saving checkpoint... +2022-04-12 09:18:02.642092: done, saving took 14.65 seconds +2022-04-12 09:18:02.930656: done +2022-04-12 09:18:02.988919: This epoch took 136.014056 s + +2022-04-12 09:18:03.035162: svmem(total=99932160000, available=75805474816, percent=24.1, used=20043796480, free=47507931136, active=43712761856, inactive=6569918464, buffers=60514304, cached=32319918080, shared=2261749760, slab=791973888) +2022-04-12 09:18:03.093600: +epoch: 447 +2022-04-12 09:19:50.770708: train loss : -0.7347 +2022-04-12 09:20:01.496588: validation loss: -0.6926 +2022-04-12 09:20:01.560873: Average global foreground Dice: [0.9191, 0.9708, 0.9788, 0.4491] +2022-04-12 09:20:01.738617: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:20:02.852683: lr: 0.001304 +2022-04-12 09:20:03.162911: saving scheduled checkpoint file... +2022-04-12 09:20:03.547251: saving checkpoint... +2022-04-12 09:20:19.433458: done, saving took 15.99 seconds +2022-04-12 09:20:19.607899: done +2022-04-12 09:20:19.619870: This epoch took 136.421373 s + +2022-04-12 09:20:19.673663: svmem(total=99932160000, available=75830046720, percent=24.1, used=20019187712, free=47530393600, active=43687809024, inactive=6571892736, buffers=60768256, cached=32321810432, shared=2261843968, slab=791703552) +2022-04-12 09:20:19.743532: +epoch: 448 +2022-04-12 09:22:09.380721: train loss : -0.7375 +2022-04-12 09:22:22.271918: validation loss: -0.7146 +2022-04-12 09:22:22.441715: Average global foreground Dice: [0.9241, 0.9733, 0.9809, 0.4299] +2022-04-12 09:22:22.607977: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:22:23.608985: lr: 0.001282 +2022-04-12 09:22:23.648621: saving scheduled checkpoint file... +2022-04-12 09:22:23.818040: saving checkpoint... +2022-04-12 09:22:36.018351: done, saving took 12.27 seconds +2022-04-12 09:22:36.129290: done +2022-04-12 09:22:36.175060: This epoch took 136.418530 s + +2022-04-12 09:22:36.302677: svmem(total=99932160000, available=75965644800, percent=24.0, used=19883360256, free=47663636480, active=43551121408, inactive=6574215168, buffers=61038592, cached=32324124672, shared=2261843968, slab=792530944) +2022-04-12 09:22:36.375471: +epoch: 449 +2022-04-12 09:24:24.361999: train loss : -0.7319 +2022-04-12 09:24:36.187558: validation loss: -0.6630 +2022-04-12 09:24:36.293822: Average global foreground Dice: [0.9326, 0.9652, 0.9706, 0.418] +2022-04-12 09:24:36.536480: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:24:37.471580: lr: 0.001259 +2022-04-12 09:24:37.593341: saving scheduled checkpoint file... +2022-04-12 09:24:37.736726: saving checkpoint... +2022-04-12 09:24:50.841134: done, saving took 13.18 seconds +2022-04-12 09:24:50.957578: done +2022-04-12 09:24:50.987040: This epoch took 134.563320 s + +2022-04-12 09:24:51.014042: svmem(total=99932160000, available=75913142272, percent=24.0, used=19922006016, free=47609094144, active=43593424896, inactive=6589341696, buffers=61292544, cached=32339767296, shared=2275684352, slab=793341952) +2022-04-12 09:24:51.058468: +epoch: 450 +2022-04-12 09:26:39.492741: train loss : -0.7319 +2022-04-12 09:26:51.553733: validation loss: -0.7182 +2022-04-12 09:26:51.664146: Average global foreground Dice: [0.9218, 0.9781, 0.9849, 0.434] +2022-04-12 09:26:51.720802: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:26:52.936802: lr: 0.001236 +2022-04-12 09:26:53.119856: saving scheduled checkpoint file... +2022-04-12 09:26:53.252932: saving checkpoint... +2022-04-12 09:27:06.723229: done, saving took 13.54 seconds +2022-04-12 09:27:06.864691: done +2022-04-12 09:27:06.883732: This epoch took 135.792138 s + +2022-04-12 09:27:06.919940: svmem(total=99932160000, available=75900657664, percent=24.0, used=19948089344, free=47594344448, active=43624058880, inactive=6578073600, buffers=61575168, cached=32328151040, shared=2262118400, slab=791879680) +2022-04-12 09:27:06.929168: +epoch: 451 +2022-04-12 09:28:54.869331: train loss : -0.7283 +2022-04-12 09:29:07.376362: validation loss: -0.6459 +2022-04-12 09:29:07.533085: Average global foreground Dice: [0.9172, 0.965, 0.9664, 0.4496] +2022-04-12 09:29:07.631538: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:29:08.735639: lr: 0.001214 +2022-04-12 09:29:08.836457: saving scheduled checkpoint file... +2022-04-12 09:29:08.988385: saving checkpoint... +2022-04-12 09:29:24.292319: done, saving took 15.36 seconds +2022-04-12 09:29:24.490999: done +2022-04-12 09:29:24.551545: This epoch took 137.557793 s + +2022-04-12 09:29:24.618258: svmem(total=99932160000, available=75733471232, percent=24.2, used=20115255296, free=47424983040, active=43786240000, inactive=6579376128, buffers=61816832, cached=32330104832, shared=2261995520, slab=791900160) +2022-04-12 09:29:24.633568: +epoch: 452 +2022-04-12 09:31:13.518015: train loss : -0.7304 +2022-04-12 09:31:25.755717: validation loss: -0.7234 +2022-04-12 09:31:25.848887: Average global foreground Dice: [0.9304, 0.9781, 0.9839, 0.4048] +2022-04-12 09:31:25.922416: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:31:26.972297: lr: 0.001191 +2022-04-12 09:31:27.082155: saving scheduled checkpoint file... +2022-04-12 09:31:27.357669: saving checkpoint... +2022-04-12 09:31:46.453151: done, saving took 19.15 seconds +2022-04-12 09:31:46.555055: done +2022-04-12 09:31:46.647132: This epoch took 141.948046 s + +2022-04-12 09:31:46.696552: svmem(total=99932160000, available=75864567808, percent=24.1, used=19998285824, free=47553736704, active=43670011904, inactive=6567407616, buffers=62095360, cached=32318042112, shared=2247884800, slab=791830528) +2022-04-12 09:31:46.877307: +epoch: 453 +2022-04-12 09:33:33.288096: train loss : -0.7416 +2022-04-12 09:33:44.761114: validation loss: -0.7141 +2022-04-12 09:33:44.789392: Average global foreground Dice: [0.9261, 0.9738, 0.9782, 0.4559] +2022-04-12 09:33:44.795635: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:33:45.766158: lr: 0.001168 +2022-04-12 09:33:45.780339: saving scheduled checkpoint file... +2022-04-12 09:33:45.917364: saving checkpoint... +2022-04-12 09:33:59.437148: done, saving took 13.60 seconds +2022-04-12 09:33:59.595234: done +2022-04-12 09:33:59.630200: This epoch took 132.680154 s + +2022-04-12 09:33:59.671064: svmem(total=99932160000, available=75770339328, percent=24.2, used=20078477312, free=47457464320, active=43751399424, inactive=6583066624, buffers=62337024, cached=32333881344, shared=2261762048, slab=793985024) +2022-04-12 09:33:59.726398: +epoch: 454 +2022-04-12 09:35:48.454865: train loss : -0.7294 +2022-04-12 09:36:03.106407: validation loss: -0.7253 +2022-04-12 09:36:03.151459: Average global foreground Dice: [0.9253, 0.9789, 0.9866, 0.4259] +2022-04-12 09:36:03.210624: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:36:04.299792: lr: 0.001145 +2022-04-12 09:36:04.350743: saving scheduled checkpoint file... +2022-04-12 09:36:04.469813: saving checkpoint... +2022-04-12 09:36:16.422089: done, saving took 12.03 seconds +2022-04-12 09:36:16.573746: done +2022-04-12 09:36:16.690680: saving checkpoint... +2022-04-12 09:36:29.974120: done, saving took 13.33 seconds +2022-04-12 09:36:30.236506: This epoch took 150.451733 s + +2022-04-12 09:36:30.309546: svmem(total=99932160000, available=75705028608, percent=24.2, used=20129112064, free=47389650944, active=43793588224, inactive=6599778304, buffers=62648320, cached=32350748672, shared=2276421632, slab=792883200) +2022-04-12 09:36:30.322833: +epoch: 455 +2022-04-12 09:38:18.037973: train loss : -0.7235 +2022-04-12 09:38:29.468757: validation loss: -0.7015 +2022-04-12 09:38:29.765684: Average global foreground Dice: [0.9321, 0.9718, 0.9762, 0.4486] +2022-04-12 09:38:30.042465: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:38:32.258335: lr: 0.001122 +2022-04-12 09:38:32.375389: saving scheduled checkpoint file... +2022-04-12 09:38:32.630469: saving checkpoint... +2022-04-12 09:38:46.841628: done, saving took 14.28 seconds +2022-04-12 09:38:47.039700: done +2022-04-12 09:38:47.099702: saving checkpoint... +2022-04-12 09:39:06.058823: done, saving took 19.01 seconds +2022-04-12 09:39:06.593312: This epoch took 156.209806 s + +2022-04-12 09:39:06.721301: svmem(total=99932160000, available=75796492288, percent=24.2, used=20052066304, free=47478796288, active=43722620928, inactive=6587297792, buffers=62922752, cached=32338374656, shared=2261934080, slab=793722880) +2022-04-12 09:39:06.726192: +epoch: 456 +2022-04-12 09:40:53.119483: train loss : -0.7371 +2022-04-12 09:41:06.469173: validation loss: -0.6912 +2022-04-12 09:41:06.580370: Average global foreground Dice: [0.9161, 0.9738, 0.9795, 0.4345] +2022-04-12 09:41:06.637275: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:41:07.796156: lr: 0.001099 +2022-04-12 09:41:07.967195: saving scheduled checkpoint file... +2022-04-12 09:41:08.067331: saving checkpoint... +2022-04-12 09:41:18.047277: done, saving took 10.03 seconds +2022-04-12 09:41:18.540649: done +2022-04-12 09:41:18.694265: This epoch took 131.921668 s + +2022-04-12 09:41:18.853734: svmem(total=99932160000, available=75768553472, percent=24.2, used=20066156544, free=47448629248, active=43739910144, inactive=6603104256, buffers=63164416, cached=32354209792, shared=2275803136, slab=793862144) +2022-04-12 09:41:18.966742: +epoch: 457 +2022-04-12 09:43:08.276174: train loss : -0.7327 +2022-04-12 09:43:19.846682: validation loss: -0.7014 +2022-04-12 09:43:19.890496: Average global foreground Dice: [0.9136, 0.9754, 0.9737, 0.4503] +2022-04-12 09:43:19.943429: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:43:20.806633: lr: 0.001076 +2022-04-12 09:43:20.858913: saving scheduled checkpoint file... +2022-04-12 09:43:20.985108: saving checkpoint... +2022-04-12 09:43:36.026044: done, saving took 15.11 seconds +2022-04-12 09:43:36.262995: done +2022-04-12 09:43:36.376135: This epoch took 137.295647 s + +2022-04-12 09:43:36.441874: svmem(total=99932160000, available=75798319104, percent=24.2, used=20037988352, free=47476244480, active=43711418368, inactive=6603350016, buffers=63434752, cached=32354492416, shared=2274152448, slab=793751552) +2022-04-12 09:43:36.495872: +epoch: 458 +2022-04-12 09:45:25.339580: train loss : -0.7346 +2022-04-12 09:45:36.843314: validation loss: -0.7165 +2022-04-12 09:45:37.075548: Average global foreground Dice: [0.9242, 0.9767, 0.9844, 0.3831] +2022-04-12 09:45:37.179444: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:45:37.971718: lr: 0.001053 +2022-04-12 09:45:38.127411: saving scheduled checkpoint file... +2022-04-12 09:45:38.405198: saving checkpoint... +2022-04-12 09:45:56.305751: done, saving took 18.02 seconds +2022-04-12 09:45:57.226625: done +2022-04-12 09:45:57.270326: This epoch took 140.706207 s + +2022-04-12 09:45:57.306838: svmem(total=99932160000, available=75838111744, percent=24.1, used=20011683840, free=47513755648, active=43687952384, inactive=6592180224, buffers=63709184, cached=32343011328, shared=2260635648, slab=792973312) +2022-04-12 09:45:57.354815: +epoch: 459 +2022-04-12 09:47:45.051096: train loss : -0.7361 +2022-04-12 09:47:56.343373: validation loss: -0.6990 +2022-04-12 09:47:56.477606: Average global foreground Dice: [0.9255, 0.9769, 0.9815, 0.4131] +2022-04-12 09:47:56.531779: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:47:57.692458: lr: 0.00103 +2022-04-12 09:47:57.802530: saving scheduled checkpoint file... +2022-04-12 09:47:57.962656: saving checkpoint... +2022-04-12 09:48:10.337637: done, saving took 12.47 seconds +2022-04-12 09:48:10.530798: done +2022-04-12 09:48:10.635281: This epoch took 133.214955 s + +2022-04-12 09:48:10.683580: svmem(total=99932160000, available=75723825152, percent=24.2, used=20125200384, free=47397384192, active=43800559616, inactive=6594211840, buffers=63954944, cached=32345620480, shared=2261266432, slab=794058752) +2022-04-12 09:48:10.732435: +epoch: 460 +2022-04-12 09:49:59.939216: train loss : -0.7337 +2022-04-12 09:50:12.252161: validation loss: -0.7071 +2022-04-12 09:50:12.405601: Average global foreground Dice: [0.9283, 0.9707, 0.9756, 0.4513] +2022-04-12 09:50:12.591442: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:50:13.700126: lr: 0.001007 +2022-04-12 09:50:13.748197: saving scheduled checkpoint file... +2022-04-12 09:50:13.891637: saving checkpoint... +2022-04-12 09:50:27.668377: done, saving took 13.88 seconds +2022-04-12 09:50:27.864362: done +2022-04-12 09:50:27.910148: This epoch took 137.119896 s + +2022-04-12 09:50:27.998738: svmem(total=99932160000, available=75945828352, percent=24.0, used=19899789312, free=47617175552, active=43572858880, inactive=6599622656, buffers=64221184, cached=32350973952, shared=2264735744, slab=791728128) +2022-04-12 09:50:28.066930: +epoch: 461 +2022-04-12 09:52:17.552459: train loss : -0.7375 +2022-04-12 09:52:30.552992: validation loss: -0.7171 +2022-04-12 09:52:30.674760: Average global foreground Dice: [0.9132, 0.9741, 0.9812, 0.4382] +2022-04-12 09:52:30.773262: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:52:31.662918: lr: 0.000983 +2022-04-12 09:52:31.699104: saving scheduled checkpoint file... +2022-04-12 09:52:31.862401: saving checkpoint... +2022-04-12 09:52:48.348759: done, saving took 16.56 seconds +2022-04-12 09:52:48.638058: done +2022-04-12 09:52:48.740691: This epoch took 140.610401 s + +2022-04-12 09:52:48.795928: svmem(total=99932160000, available=76076437504, percent=23.9, used=19772358656, free=47745589248, active=43448373248, inactive=6598881280, buffers=64475136, cached=32349736960, shared=2261540864, slab=793538560) +2022-04-12 09:52:48.861862: +epoch: 462 +2022-04-12 09:54:37.278308: train loss : -0.7336 +2022-04-12 09:54:49.217543: validation loss: -0.6977 +2022-04-12 09:54:49.464401: Average global foreground Dice: [0.9224, 0.9729, 0.9713, 0.4745] +2022-04-12 09:54:49.605628: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:54:50.480913: lr: 0.00096 +2022-04-12 09:54:50.536806: saving scheduled checkpoint file... +2022-04-12 09:54:50.728664: saving checkpoint... +2022-04-12 09:55:10.000117: done, saving took 19.35 seconds +2022-04-12 09:55:10.206656: done +2022-04-12 09:55:10.355181: saving checkpoint... +2022-04-12 09:55:26.639640: done, saving took 16.38 seconds +2022-04-12 09:55:26.838002: This epoch took 157.920793 s + +2022-04-12 09:55:26.893682: svmem(total=99932160000, available=76065488896, percent=23.9, used=19783487488, free=47732027392, active=43455741952, inactive=6600404992, buffers=64806912, cached=32351838208, shared=2261377024, slab=795176960) +2022-04-12 09:55:26.989249: +epoch: 463 +2022-04-12 09:57:16.165903: train loss : -0.7424 +2022-04-12 09:57:28.371253: validation loss: -0.7409 +2022-04-12 09:57:28.468155: Average global foreground Dice: [0.9248, 0.9782, 0.9873, 0.466] +2022-04-12 09:57:28.598446: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 09:57:29.810778: lr: 0.000937 +2022-04-12 09:57:29.922365: saving scheduled checkpoint file... +2022-04-12 09:57:30.159452: saving checkpoint... +2022-04-12 09:57:41.664268: done, saving took 11.63 seconds +2022-04-12 09:57:41.844622: done +2022-04-12 09:57:41.923900: saving checkpoint... +2022-04-12 09:57:59.980846: done, saving took 18.10 seconds +2022-04-12 09:58:01.547452: This epoch took 154.465000 s + +2022-04-12 09:58:01.647844: svmem(total=99932160000, available=76029636608, percent=23.9, used=19822239744, free=47693840384, active=43495694336, inactive=6599421952, buffers=65081344, cached=32350998528, shared=2258378752, slab=794394624) +2022-04-12 09:58:01.655665: +epoch: 464 +2022-04-12 09:59:51.299855: train loss : -0.7422 +2022-04-12 10:00:03.927652: validation loss: -0.7118 +2022-04-12 10:00:04.083706: Average global foreground Dice: [0.9159, 0.973, 0.9798, 0.4378] +2022-04-12 10:00:04.133990: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:00:05.120420: lr: 0.000913 +2022-04-12 10:00:05.181067: saving scheduled checkpoint file... +2022-04-12 10:00:05.320617: saving checkpoint... +2022-04-12 10:00:15.969663: done, saving took 10.74 seconds +2022-04-12 10:00:16.094818: done +2022-04-12 10:00:16.100306: This epoch took 134.363095 s + +2022-04-12 10:00:16.120254: svmem(total=99932160000, available=75895463936, percent=24.1, used=19942948864, free=47557439488, active=43616641024, inactive=6614724608, buffers=65331200, cached=32366440448, shared=2271760384, slab=794619904) +2022-04-12 10:00:16.134569: +epoch: 465 +2022-04-12 10:02:06.600398: train loss : -0.7403 +2022-04-12 10:02:20.846993: validation loss: -0.6785 +2022-04-12 10:02:21.011609: Average global foreground Dice: [0.9242, 0.971, 0.9742, 0.4267] +2022-04-12 10:02:21.082988: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:02:21.954835: lr: 0.00089 +2022-04-12 10:02:22.048851: saving scheduled checkpoint file... +2022-04-12 10:02:22.179241: saving checkpoint... +2022-04-12 10:02:36.898954: done, saving took 14.84 seconds +2022-04-12 10:02:37.238407: done +2022-04-12 10:02:37.365920: This epoch took 141.221666 s + +2022-04-12 10:02:37.491571: svmem(total=99932160000, available=75888037888, percent=24.1, used=19960512512, free=47547953152, active=43635003392, inactive=6606540800, buffers=65601536, cached=32358092800, shared=2261676032, slab=793219072) +2022-04-12 10:02:37.537219: +epoch: 466 +2022-04-12 10:04:29.079238: train loss : -0.7435 +2022-04-12 10:04:40.683316: validation loss: -0.7110 +2022-04-12 10:04:41.172819: Average global foreground Dice: [0.9355, 0.9713, 0.9775, 0.4418] +2022-04-12 10:04:41.231489: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:04:42.548090: lr: 0.000866 +2022-04-12 10:04:42.660049: saving scheduled checkpoint file... +2022-04-12 10:04:42.792992: saving checkpoint... +2022-04-12 10:05:02.074845: done, saving took 19.36 seconds +2022-04-12 10:05:02.324797: done +2022-04-12 10:05:02.388388: This epoch took 144.802532 s + +2022-04-12 10:05:02.448916: svmem(total=99932160000, available=75894976512, percent=24.1, used=19953434624, free=47552487424, active=43629158400, inactive=6609133568, buffers=65859584, cached=32360378368, shared=2261753856, slab=792748032) +2022-04-12 10:05:02.471430: +epoch: 467 +2022-04-12 10:06:49.510880: train loss : -0.7415 +2022-04-12 10:07:01.471202: validation loss: -0.7446 +2022-04-12 10:07:01.511301: Average global foreground Dice: [0.9377, 0.9819, 0.9875, 0.4538] +2022-04-12 10:07:01.568114: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:07:02.740752: lr: 0.000842 +2022-04-12 10:07:02.872512: saving scheduled checkpoint file... +2022-04-12 10:07:03.147127: saving checkpoint... +2022-04-12 10:07:19.419890: done, saving took 16.36 seconds +2022-04-12 10:07:19.569721: done +2022-04-12 10:07:19.668808: saving checkpoint... +2022-04-12 10:07:30.986574: done, saving took 11.37 seconds +2022-04-12 10:07:31.046074: This epoch took 148.515426 s + +2022-04-12 10:07:31.074231: svmem(total=99932160000, available=75962802176, percent=24.0, used=19885363200, free=47617888256, active=43560361984, inactive=6610812928, buffers=66166784, cached=32362741760, shared=2261934080, slab=795181056) +2022-04-12 10:07:31.109576: +epoch: 468 +2022-04-12 10:09:17.019556: train loss : -0.7531 +2022-04-12 10:09:30.756897: validation loss: -0.7231 +2022-04-12 10:09:30.896399: Average global foreground Dice: [0.9176, 0.9722, 0.9826, 0.4325] +2022-04-12 10:09:31.074211: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:09:32.094864: lr: 0.000819 +2022-04-12 10:09:32.215349: saving scheduled checkpoint file... +2022-04-12 10:09:32.336195: saving checkpoint... +2022-04-12 10:09:54.521610: done, saving took 22.23 seconds +2022-04-12 10:09:54.694478: done +2022-04-12 10:09:54.708329: This epoch took 143.586025 s + +2022-04-12 10:09:54.761478: svmem(total=99932160000, available=75758505984, percent=24.2, used=20089765888, free=47411392512, active=43766009856, inactive=6612733952, buffers=66416640, cached=32364584960, shared=2261839872, slab=794722304) +2022-04-12 10:09:54.814025: +epoch: 469 +2022-04-12 10:11:42.371332: train loss : -0.7277 +2022-04-12 10:11:53.889098: validation loss: -0.7080 +2022-04-12 10:11:53.950198: Average global foreground Dice: [0.9288, 0.9766, 0.986, 0.4198] +2022-04-12 10:11:53.964695: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:11:54.903595: lr: 0.000795 +2022-04-12 10:11:54.967114: saving scheduled checkpoint file... +2022-04-12 10:11:55.054469: saving checkpoint... +2022-04-12 10:12:06.095825: done, saving took 11.12 seconds +2022-04-12 10:12:06.275192: done +2022-04-12 10:12:06.328635: This epoch took 131.508958 s + +2022-04-12 10:12:06.366826: svmem(total=99932160000, available=76000628736, percent=23.9, used=19833765888, free=47651352576, active=43511304192, inactive=6628372480, buffers=66662400, cached=32380379136, shared=2275725312, slab=794214400) +2022-04-12 10:12:06.405722: +epoch: 470 +2022-04-12 10:13:55.175676: train loss : -0.7361 +2022-04-12 10:14:06.326052: validation loss: -0.6935 +2022-04-12 10:14:06.366320: Average global foreground Dice: [0.9204, 0.9735, 0.9777, 0.4526] +2022-04-12 10:14:06.391293: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:14:07.495027: lr: 0.000771 +2022-04-12 10:14:07.556028: saving scheduled checkpoint file... +2022-04-12 10:14:07.699986: saving checkpoint... +2022-04-12 10:14:27.783147: done, saving took 20.18 seconds +2022-04-12 10:14:28.080675: done +2022-04-12 10:14:28.190654: This epoch took 141.686689 s + +2022-04-12 10:14:28.297903: svmem(total=99932160000, available=75740753920, percent=24.2, used=20107243520, free=47389220864, active=43785875456, inactive=6616821760, buffers=66928640, cached=32368766976, shared=2262061056, slab=792756224) +2022-04-12 10:14:28.352538: +epoch: 471 +2022-04-12 10:16:16.551161: train loss : -0.7388 +2022-04-12 10:16:27.647624: validation loss: -0.7179 +2022-04-12 10:16:27.724787: Average global foreground Dice: [0.9311, 0.9767, 0.9857, 0.4226] +2022-04-12 10:16:27.768021: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:16:28.961949: lr: 0.000747 +2022-04-12 10:16:29.018092: saving scheduled checkpoint file... +2022-04-12 10:16:29.139645: saving checkpoint... +2022-04-12 10:16:42.760566: done, saving took 13.72 seconds +2022-04-12 10:16:42.982419: done +2022-04-12 10:16:43.026558: This epoch took 134.607765 s + +2022-04-12 10:16:43.073727: svmem(total=99932160000, available=75886673920, percent=24.1, used=19954663424, free=47533072384, active=43628670976, inactive=6625120256, buffers=67203072, cached=32377221120, shared=2268643328, slab=794611712) +2022-04-12 10:16:43.078482: +epoch: 472 +2022-04-12 10:18:30.424066: train loss : -0.7390 +2022-04-12 10:18:41.818417: validation loss: -0.7234 +2022-04-12 10:18:41.907471: Average global foreground Dice: [0.932, 0.978, 0.9856, 0.4193] +2022-04-12 10:18:41.974816: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:18:43.083082: lr: 0.000723 +2022-04-12 10:18:43.103856: saving scheduled checkpoint file... +2022-04-12 10:18:43.226811: saving checkpoint... +2022-04-12 10:18:59.594344: done, saving took 16.45 seconds +2022-04-12 10:18:59.665649: done +2022-04-12 10:18:59.671654: This epoch took 136.532143 s + +2022-04-12 10:18:59.678870: svmem(total=99932160000, available=75763335168, percent=24.2, used=20070883328, free=47407538176, active=43746365440, inactive=6633914368, buffers=67473408, cached=32386265088, shared=2275614720, slab=796155904) +2022-04-12 10:18:59.684455: +epoch: 473 +2022-04-12 10:20:47.467829: train loss : -0.7512 +2022-04-12 10:20:58.974046: validation loss: -0.7219 +2022-04-12 10:20:59.082453: Average global foreground Dice: [0.9311, 0.9774, 0.9835, 0.4363] +2022-04-12 10:20:59.146513: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:21:00.325634: lr: 0.000699 +2022-04-12 10:21:00.468678: saving scheduled checkpoint file... +2022-04-12 10:21:00.704763: saving checkpoint... +2022-04-12 10:21:15.220548: done, saving took 14.57 seconds +2022-04-12 10:21:15.448875: done +2022-04-12 10:21:15.457743: This epoch took 135.759813 s + +2022-04-12 10:21:15.484921: svmem(total=99932160000, available=75827077120, percent=24.1, used=20011118592, free=47506120704, active=43661385728, inactive=6622629888, buffers=67723264, cached=32347197440, shared=2271711232, slab=795996160) +2022-04-12 10:21:15.532644: +epoch: 474 +2022-04-12 10:23:05.116215: train loss : -0.7499 +2022-04-12 10:23:16.548289: validation loss: -0.7149 +2022-04-12 10:23:16.611385: Average global foreground Dice: [0.9266, 0.9737, 0.9768, 0.4496] +2022-04-12 10:23:16.730975: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:23:17.841797: lr: 0.000675 +2022-04-12 10:23:17.905842: saving scheduled checkpoint file... +2022-04-12 10:23:18.019475: saving checkpoint... +2022-04-12 10:23:34.021592: done, saving took 16.09 seconds +2022-04-12 10:23:34.125267: done +2022-04-12 10:23:34.215074: saving checkpoint... +2022-04-12 10:23:47.497777: done, saving took 13.36 seconds +2022-04-12 10:23:47.720149: This epoch took 152.182531 s + +2022-04-12 10:23:47.810920: svmem(total=99932160000, available=76103852032, percent=23.8, used=19745984512, free=47780421632, active=43408863232, inactive=6601850880, buffers=68022272, cached=32337731584, shared=2260029440, slab=793645056) +2022-04-12 10:23:47.964742: +epoch: 475 +2022-04-12 10:25:36.045556: train loss : -0.7398 +2022-04-12 10:25:47.251657: validation loss: -0.7290 +2022-04-12 10:25:47.335207: Average global foreground Dice: [0.9239, 0.9769, 0.986, 0.4372] +2022-04-12 10:25:47.402994: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:25:48.486155: lr: 0.00065 +2022-04-12 10:25:48.546523: saving scheduled checkpoint file... +2022-04-12 10:25:48.689489: saving checkpoint... +2022-04-12 10:26:03.282146: done, saving took 14.68 seconds +2022-04-12 10:26:03.503598: done +2022-04-12 10:26:03.745271: saving checkpoint... +2022-04-12 10:26:17.402146: done, saving took 13.75 seconds +2022-04-12 10:26:17.584207: This epoch took 149.546830 s + +2022-04-12 10:26:17.639129: svmem(total=99932160000, available=75916214272, percent=24.0, used=19931787264, free=47590416384, active=43598483456, inactive=6595026944, buffers=68325376, cached=32341630976, shared=2261848064, slab=795467776) +2022-04-12 10:26:17.643606: +epoch: 476 +2022-04-12 10:28:04.685069: train loss : -0.7377 +2022-04-12 10:28:16.927709: validation loss: -0.6912 +2022-04-12 10:28:16.991297: Average global foreground Dice: [0.9292, 0.9731, 0.9793, 0.4151] +2022-04-12 10:28:17.103217: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:28:18.016479: lr: 0.000626 +2022-04-12 10:28:18.160943: saving scheduled checkpoint file... +2022-04-12 10:28:18.360830: saving checkpoint... +2022-04-12 10:28:35.817873: done, saving took 17.56 seconds +2022-04-12 10:28:36.104481: done +2022-04-12 10:28:36.132291: This epoch took 138.429183 s + +2022-04-12 10:28:36.243550: svmem(total=99932160000, available=75812782080, percent=24.1, used=20042174464, free=47484891136, active=43712749568, inactive=6590480384, buffers=68595712, cached=32336498688, shared=2254909440, slab=795148288) +2022-04-12 10:28:36.390750: +epoch: 477 +2022-04-12 10:30:23.233731: train loss : -0.7483 +2022-04-12 10:30:34.783620: validation loss: -0.7223 +2022-04-12 10:30:35.014043: Average global foreground Dice: [0.9205, 0.9785, 0.9871, 0.4377] +2022-04-12 10:30:35.127574: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:30:36.275351: lr: 0.000601 +2022-04-12 10:30:36.428828: saving scheduled checkpoint file... +2022-04-12 10:30:36.669377: saving checkpoint... +2022-04-12 10:30:52.734897: done, saving took 16.17 seconds +2022-04-12 10:30:52.908796: done +2022-04-12 10:30:52.975674: This epoch took 136.536427 s + +2022-04-12 10:30:53.043866: svmem(total=99932160000, available=75840577536, percent=24.1, used=20010901504, free=47510441984, active=43684425728, inactive=6595260416, buffers=68845568, cached=32341970944, shared=2258292736, slab=792858624) +2022-04-12 10:30:53.047973: +epoch: 478 +2022-04-12 10:32:42.416729: train loss : -0.7424 +2022-04-12 10:32:54.480680: validation loss: -0.7251 +2022-04-12 10:32:54.533396: Average global foreground Dice: [0.9275, 0.9776, 0.9863, 0.4228] +2022-04-12 10:32:54.603340: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:32:55.707791: lr: 0.000577 +2022-04-12 10:32:55.824898: saving scheduled checkpoint file... +2022-04-12 10:32:55.945239: saving checkpoint... +2022-04-12 10:33:14.289615: done, saving took 18.45 seconds +2022-04-12 10:33:14.613801: done +2022-04-12 10:33:14.737222: This epoch took 141.670458 s + +2022-04-12 10:33:14.803915: svmem(total=99932160000, available=75864862720, percent=24.1, used=19997040640, free=47532470272, active=43671035904, inactive=6586736640, buffers=69099520, cached=32333549568, shared=2247774208, slab=794177536) +2022-04-12 10:33:14.915397: +epoch: 479 +2022-04-12 10:35:03.236755: train loss : -0.7365 +2022-04-12 10:35:14.584960: validation loss: -0.7231 +2022-04-12 10:35:14.726038: Average global foreground Dice: [0.9301, 0.977, 0.9848, 0.4147] +2022-04-12 10:35:14.791849: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:35:15.776785: lr: 0.000552 +2022-04-12 10:35:15.913175: saving scheduled checkpoint file... +2022-04-12 10:35:16.079901: saving checkpoint... +2022-04-12 10:35:37.495983: done, saving took 21.49 seconds +2022-04-12 10:35:37.533703: done +2022-04-12 10:35:37.537576: This epoch took 142.570935 s + +2022-04-12 10:35:37.541843: svmem(total=99932160000, available=75934339072, percent=24.0, used=19903152128, free=47599640576, active=43575697408, inactive=6613483520, buffers=69390336, cached=32359976960, shared=2272169984, slab=794804224) +2022-04-12 10:35:37.550262: +epoch: 480 +2022-04-12 10:37:25.938748: train loss : -0.7430 +2022-04-12 10:37:37.936327: validation loss: -0.7157 +2022-04-12 10:37:38.059472: Average global foreground Dice: [0.8982, 0.9788, 0.9857, 0.4504] +2022-04-12 10:37:38.118932: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:37:38.774821: lr: 0.000527 +2022-04-12 10:37:38.833771: saving scheduled checkpoint file... +2022-04-12 10:37:38.911064: saving checkpoint... +2022-04-12 10:37:56.628518: done, saving took 17.79 seconds +2022-04-12 10:37:56.823307: done +2022-04-12 10:37:56.887369: This epoch took 139.328502 s + +2022-04-12 10:37:57.049905: svmem(total=99932160000, available=75798339584, percent=24.2, used=20049432576, free=47461363712, active=43722702848, inactive=6604550144, buffers=69640192, cached=32351723520, shared=2261839872, slab=794599424) +2022-04-12 10:37:57.075105: +epoch: 481 +2022-04-12 10:39:45.555611: train loss : -0.7376 +2022-04-12 10:39:57.226888: validation loss: -0.7277 +2022-04-12 10:39:57.392372: Average global foreground Dice: [0.9347, 0.9761, 0.9862, 0.4306] +2022-04-12 10:39:57.461741: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:39:58.550288: lr: 0.000502 +2022-04-12 10:39:58.611647: saving scheduled checkpoint file... +2022-04-12 10:39:58.743588: saving checkpoint... +2022-04-12 10:40:13.296847: done, saving took 14.62 seconds +2022-04-12 10:40:13.522438: done +2022-04-12 10:40:13.701769: This epoch took 136.550683 s + +2022-04-12 10:40:13.837896: svmem(total=99932160000, available=75761430528, percent=24.2, used=20100427776, free=47422320640, active=43776143360, inactive=6592942080, buffers=69898240, cached=32339513344, shared=2247774208, slab=794394624) +2022-04-12 10:40:13.843281: +epoch: 482 +2022-04-12 10:42:02.396822: train loss : -0.7364 +2022-04-12 10:42:13.968488: validation loss: -0.7017 +2022-04-12 10:42:14.030048: Average global foreground Dice: [0.925, 0.9738, 0.9783, 0.4182] +2022-04-12 10:42:14.091205: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:42:15.131613: lr: 0.000477 +2022-04-12 10:42:15.177938: saving scheduled checkpoint file... +2022-04-12 10:42:15.309199: saving checkpoint... +2022-04-12 10:42:26.434548: done, saving took 11.21 seconds +2022-04-12 10:42:26.689111: done +2022-04-12 10:42:26.738357: This epoch took 132.822539 s + +2022-04-12 10:42:26.783415: svmem(total=99932160000, available=75925684224, percent=24.0, used=19906973696, free=47584497664, active=43577888768, inactive=6623313920, buffers=70148096, cached=32370540544, shared=2276909056, slab=794935296) +2022-04-12 10:42:26.818545: +epoch: 483 +2022-04-12 10:44:14.815975: train loss : -0.7480 +2022-04-12 10:44:25.520386: validation loss: -0.7139 +2022-04-12 10:44:25.597458: Average global foreground Dice: [0.9359, 0.9737, 0.9806, 0.4267] +2022-04-12 10:44:25.651311: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:44:26.700716: lr: 0.000451 +2022-04-12 10:44:26.794854: saving scheduled checkpoint file... +2022-04-12 10:44:26.998321: saving checkpoint... +2022-04-12 10:44:39.959474: done, saving took 13.08 seconds +2022-04-12 10:44:40.204751: done +2022-04-12 10:44:40.248900: This epoch took 133.340266 s + +2022-04-12 10:44:40.312222: svmem(total=99932160000, available=75900022784, percent=24.0, used=19947933696, free=47556694016, active=43622944768, inactive=6610042880, buffers=70385664, cached=32357146624, shared=2261676032, slab=795521024) +2022-04-12 10:44:40.369317: +epoch: 484 +2022-04-12 10:46:28.936426: train loss : -0.7554 +2022-04-12 10:46:41.094431: validation loss: -0.7248 +2022-04-12 10:46:41.437357: Average global foreground Dice: [0.9342, 0.9793, 0.9873, 0.4338] +2022-04-12 10:46:41.982833: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:46:42.903485: lr: 0.000426 +2022-04-12 10:46:42.951181: saving scheduled checkpoint file... +2022-04-12 10:46:43.132458: saving checkpoint... +2022-04-12 10:47:12.180407: done, saving took 29.14 seconds +2022-04-12 10:47:12.345167: done +2022-04-12 10:47:12.426937: This epoch took 152.013952 s + +2022-04-12 10:47:12.473036: svmem(total=99932160000, available=75861536768, percent=24.1, used=19965505536, free=47515893760, active=43638710272, inactive=6632890368, buffers=70684672, cached=32380076032, shared=2282491904, slab=796651520) +2022-04-12 10:47:12.557004: +epoch: 485 +2022-04-12 10:48:59.888911: train loss : -0.7492 +2022-04-12 10:49:14.733922: validation loss: -0.7030 +2022-04-12 10:49:14.773610: Average global foreground Dice: [0.928, 0.9734, 0.9782, 0.4424] +2022-04-12 10:49:14.839529: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:49:16.722657: lr: 0.0004 +2022-04-12 10:49:16.773250: saving scheduled checkpoint file... +2022-04-12 10:49:16.921305: saving checkpoint... +2022-04-12 10:52:07.542117: done, saving took 170.67 seconds +2022-04-12 10:52:07.911385: done +2022-04-12 10:52:07.962831: This epoch took 295.370147 s + +2022-04-12 10:52:07.970104: svmem(total=99932160000, available=76124569600, percent=23.8, used=19943579648, free=47774404608, active=43618332672, inactive=6396846080, buffers=71229440, cached=32142946304, shared=2041458688, slab=797380608) +2022-04-12 10:52:07.987705: +epoch: 486 +2022-04-12 10:53:54.262419: train loss : -0.7521 +2022-04-12 10:54:06.238555: validation loss: -0.6970 +2022-04-12 10:54:06.469235: Average global foreground Dice: [0.9317, 0.9714, 0.9784, 0.4505] +2022-04-12 10:54:06.652825: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:54:07.620991: lr: 0.000375 +2022-04-12 10:54:07.705072: saving scheduled checkpoint file... +2022-04-12 10:54:07.864785: saving checkpoint... +2022-04-12 10:54:24.650208: done, saving took 16.85 seconds +2022-04-12 10:54:24.813073: done +2022-04-12 10:54:24.865421: This epoch took 136.841509 s + +2022-04-12 10:54:24.877491: svmem(total=99932160000, available=75967635456, percent=24.0, used=19881226240, free=47615238144, active=43552624640, inactive=6616850432, buffers=71491584, cached=32364204032, shared=2260582400, slab=796286976) +2022-04-12 10:54:24.937773: +epoch: 487 +2022-04-12 10:56:13.496329: train loss : -0.7555 +2022-04-12 10:56:24.945206: validation loss: -0.7350 +2022-04-12 10:56:25.071752: Average global foreground Dice: [0.9276, 0.9789, 0.9857, 0.4266] +2022-04-12 10:56:25.079562: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:56:26.127922: lr: 0.000348 +2022-04-12 10:56:26.142408: saving scheduled checkpoint file... +2022-04-12 10:56:26.320219: saving checkpoint... +2022-04-12 10:56:41.095742: done, saving took 14.88 seconds +2022-04-12 10:56:41.427248: done +2022-04-12 10:56:41.573344: This epoch took 136.582990 s + +2022-04-12 10:56:41.714426: svmem(total=99932160000, available=76017643520, percent=23.9, used=19840458752, free=47663042560, active=43513569280, inactive=6609260544, buffers=71774208, cached=32356884480, shared=2251210752, slab=793825280) +2022-04-12 10:56:41.780061: +epoch: 488 +2022-04-12 10:58:30.772829: train loss : -0.7313 +2022-04-12 10:58:41.717411: validation loss: -0.7221 +2022-04-12 10:58:41.795269: Average global foreground Dice: [0.9185, 0.9758, 0.9797, 0.4374] +2022-04-12 10:58:41.802145: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 10:58:42.840141: lr: 0.000322 +2022-04-12 10:58:42.966120: saving scheduled checkpoint file... +2022-04-12 10:58:43.119485: saving checkpoint... +2022-04-12 10:58:55.842929: done, saving took 12.83 seconds +2022-04-12 10:58:55.894000: done +2022-04-12 10:58:55.905283: This epoch took 134.066374 s + +2022-04-12 10:58:55.921976: svmem(total=99932160000, available=75910623232, percent=24.0, used=19950673920, free=47553839104, active=43620859904, inactive=6608011264, buffers=72019968, cached=32355627008, shared=2248073216, slab=795955200) +2022-04-12 10:58:55.936008: +epoch: 489 +2022-04-12 11:00:46.456132: train loss : -0.7498 +2022-04-12 11:00:58.196643: validation loss: -0.7005 +2022-04-12 11:00:58.354907: Average global foreground Dice: [0.9344, 0.9706, 0.9716, 0.4175] +2022-04-12 11:00:58.423638: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:00:59.440229: lr: 0.000296 +2022-04-12 11:00:59.549676: saving scheduled checkpoint file... +2022-04-12 11:00:59.704535: saving checkpoint... +2022-04-12 11:01:10.237988: done, saving took 10.64 seconds +2022-04-12 11:01:10.372004: done +2022-04-12 11:01:10.385924: This epoch took 134.431898 s + +2022-04-12 11:01:10.438147: svmem(total=99932160000, available=75873165312, percent=24.1, used=19974295552, free=47514308608, active=43647107072, inactive=6623559680, buffers=72269824, cached=32371286016, shared=2261848064, slab=795324416) +2022-04-12 11:01:10.461028: +epoch: 490 +2022-04-12 11:02:59.145484: train loss : -0.7552 +2022-04-12 11:03:12.255452: validation loss: -0.7030 +2022-04-12 11:03:12.407466: Average global foreground Dice: [0.9264, 0.9727, 0.9797, 0.4274] +2022-04-12 11:03:12.484313: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:03:13.205738: lr: 0.000269 +2022-04-12 11:03:13.282536: saving scheduled checkpoint file... +2022-04-12 11:03:13.447520: saving checkpoint... +2022-04-12 11:03:28.881238: done, saving took 15.52 seconds +2022-04-12 11:03:29.227443: done +2022-04-12 11:03:29.303053: This epoch took 138.800620 s + +2022-04-12 11:03:29.345150: svmem(total=99932160000, available=76044464128, percent=23.9, used=19801034752, free=47683309568, active=43476942848, inactive=6627590144, buffers=72527872, cached=32375287808, shared=2263838720, slab=795324416) +2022-04-12 11:03:29.395054: +epoch: 491 +2022-04-12 11:05:16.901134: train loss : -0.7528 +2022-04-12 11:05:29.225668: validation loss: -0.6922 +2022-04-12 11:05:29.352616: Average global foreground Dice: [0.9349, 0.9689, 0.9754, 0.4213] +2022-04-12 11:05:29.409050: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:05:30.404454: lr: 0.000242 +2022-04-12 11:05:30.579298: saving scheduled checkpoint file... +2022-04-12 11:05:30.663580: saving checkpoint... +2022-04-12 11:05:45.176186: done, saving took 14.58 seconds +2022-04-12 11:05:45.695335: done +2022-04-12 11:05:45.799490: This epoch took 136.351767 s + +2022-04-12 11:05:45.902970: svmem(total=99932160000, available=75752452096, percent=24.2, used=20091179008, free=47389106176, active=43770486784, inactive=6631026688, buffers=72806400, cached=32379068416, shared=2265513984, slab=796348416) +2022-04-12 11:05:45.962017: +epoch: 492 +2022-04-12 11:07:32.849026: train loss : -0.7569 +2022-04-12 11:07:44.743767: validation loss: -0.7076 +2022-04-12 11:07:44.840177: Average global foreground Dice: [0.9316, 0.9751, 0.9772, 0.4243] +2022-04-12 11:07:44.963940: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:07:45.833643: lr: 0.000215 +2022-04-12 11:07:45.894632: saving scheduled checkpoint file... +2022-04-12 11:07:45.999854: saving checkpoint... +2022-04-12 11:08:14.165754: done, saving took 28.21 seconds +2022-04-12 11:08:14.582094: done +2022-04-12 11:08:14.632879: This epoch took 148.622969 s + +2022-04-12 11:08:14.696645: svmem(total=99932160000, available=75921965056, percent=24.0, used=19925291008, free=47556366336, active=43598807040, inactive=6629400576, buffers=73064448, cached=32377438208, shared=2261848064, slab=796090368) +2022-04-12 11:08:14.746538: +epoch: 493 +2022-04-12 11:10:02.197973: train loss : -0.7454 +2022-04-12 11:10:14.554211: validation loss: -0.7376 +2022-04-12 11:10:14.639730: Average global foreground Dice: [0.9451, 0.9792, 0.986, 0.4218] +2022-04-12 11:10:14.691273: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:10:15.654755: lr: 0.000187 +2022-04-12 11:10:15.715584: saving scheduled checkpoint file... +2022-04-12 11:10:15.895759: saving checkpoint... +2022-04-12 11:10:27.846974: done, saving took 12.02 seconds +2022-04-12 11:10:28.258668: done +2022-04-12 11:10:28.329816: This epoch took 133.514771 s + +2022-04-12 11:10:28.399531: svmem(total=99932160000, available=75948781568, percent=24.0, used=19885633536, free=47581028352, active=43562074112, inactive=6644580352, buffers=73334784, cached=32392163328, shared=2274672640, slab=796282880) +2022-04-12 11:10:28.454755: +epoch: 494 +2022-04-12 11:12:16.050130: train loss : -0.7454 +2022-04-12 11:12:27.785103: validation loss: -0.7200 +2022-04-12 11:12:27.899321: Average global foreground Dice: [0.9306, 0.9737, 0.9796, 0.4297] +2022-04-12 11:12:27.938340: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:12:29.040729: lr: 0.000158 +2022-04-12 11:12:29.096250: saving scheduled checkpoint file... +2022-04-12 11:12:29.259053: saving checkpoint... +2022-04-12 11:12:45.361369: done, saving took 16.21 seconds +2022-04-12 11:12:45.827841: done +2022-04-12 11:12:45.893777: This epoch took 137.265621 s + +2022-04-12 11:12:46.004236: svmem(total=99932160000, available=75830067200, percent=24.1, used=20017061888, free=47460261888, active=43693510656, inactive=6633119744, buffers=73576448, cached=32381259776, shared=2261901312, slab=796925952) +2022-04-12 11:12:46.056141: +epoch: 495 +2022-04-12 11:14:34.319885: train loss : -0.7384 +2022-04-12 11:14:46.450962: validation loss: -0.7277 +2022-04-12 11:14:46.521986: Average global foreground Dice: [0.933, 0.9768, 0.9888, 0.4088] +2022-04-12 11:14:46.570058: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:14:47.631684: lr: 0.00013 +2022-04-12 11:14:47.712332: saving scheduled checkpoint file... +2022-04-12 11:14:47.879297: saving checkpoint... +2022-04-12 11:14:59.836626: done, saving took 12.07 seconds +2022-04-12 11:15:00.193197: done +2022-04-12 11:15:00.336887: This epoch took 134.214861 s + +2022-04-12 11:15:00.409401: svmem(total=99932160000, available=75702972416, percent=24.2, used=20144283648, free=47330766848, active=43817857024, inactive=6634991616, buffers=73822208, cached=32383287296, shared=2261737472, slab=798056448) +2022-04-12 11:15:00.471343: +epoch: 496 +2022-04-12 11:16:49.990234: train loss : -0.7531 +2022-04-12 11:17:02.329904: validation loss: -0.7214 +2022-04-12 11:17:02.427763: Average global foreground Dice: [0.9196, 0.9812, 0.9869, 0.4581] +2022-04-12 11:17:02.578523: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:17:03.579842: lr: 0.0001 +2022-04-12 11:17:03.747918: saving scheduled checkpoint file... +2022-04-12 11:17:03.886935: saving checkpoint... +2022-04-12 11:17:26.329800: done, saving took 22.53 seconds +2022-04-12 11:17:26.472908: done +2022-04-12 11:17:26.489335: This epoch took 145.951811 s + +2022-04-12 11:17:26.533285: svmem(total=99932160000, available=75855294464, percent=24.1, used=19991896064, free=47480836096, active=43664601088, inactive=6636937216, buffers=74145792, cached=32385282048, shared=2261762048, slab=796868608) +2022-04-12 11:17:26.548366: +epoch: 497 +2022-04-12 11:19:15.534265: train loss : -0.7552 +2022-04-12 11:19:28.401506: validation loss: -0.7063 +2022-04-12 11:19:28.424167: Average global foreground Dice: [0.9132, 0.9744, 0.9825, 0.3843] +2022-04-12 11:19:28.430353: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:19:29.409965: lr: 6.9e-05 +2022-04-12 11:19:29.475420: saving scheduled checkpoint file... +2022-04-12 11:19:29.582071: saving checkpoint... +2022-04-12 11:19:44.579304: done, saving took 15.05 seconds +2022-04-12 11:19:44.942950: done +2022-04-12 11:19:45.079659: This epoch took 138.475892 s + +2022-04-12 11:19:45.139363: svmem(total=99932160000, available=75643609088, percent=24.3, used=20203438080, free=47266840576, active=43876933632, inactive=6638919680, buffers=74395648, cached=32387485696, shared=2261860352, slab=797237248) +2022-04-12 11:19:45.200031: +epoch: 498 +2022-04-12 11:21:32.721307: train loss : -0.7505 +2022-04-12 11:21:45.900546: validation loss: -0.7191 +2022-04-12 11:21:47.199168: Average global foreground Dice: [0.9369, 0.9716, 0.9829, 0.4063] +2022-04-12 11:21:48.396574: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:21:51.124655: lr: 3.7e-05 +2022-04-12 11:21:52.119935: saving scheduled checkpoint file... +2022-04-12 11:21:53.143687: saving checkpoint... +2022-04-12 11:22:35.610309: done, saving took 42.54 seconds +2022-04-12 11:22:35.737709: done +2022-04-12 11:22:35.793891: This epoch took 170.535305 s + +2022-04-12 11:22:35.800825: svmem(total=99932160000, available=75808391168, percent=24.1, used=20063526912, free=47428972544, active=43742478336, inactive=6614986752, buffers=74764288, cached=32364896256, shared=2236981248, slab=796540928) +2022-04-12 11:22:35.814619: +epoch: 499 +2022-04-12 11:24:25.649872: train loss : -0.7493 +2022-04-12 11:25:51.654438: validation loss: -0.7206 +2022-04-12 11:26:12.578899: Average global foreground Dice: [0.9266, 0.977, 0.9844, 0.3996] +2022-04-12 11:26:18.964076: (interpret this as an estimate for the Dice of the different classes. This is not exact.) +2022-04-12 11:26:45.269011: lr: 0.0 +2022-04-12 11:26:52.726048: saving scheduled checkpoint file... +2022-04-12 11:27:11.949045: saving checkpoint... +2022-04-12 11:30:26.645684: done, saving took 194.81 seconds +2022-04-12 11:30:29.256842: done +2022-04-12 11:30:29.372894: This epoch took 473.527990 s + +2022-04-12 11:30:29.647146: saving checkpoint... +2022-04-12 11:30:46.461553: done, saving took 16.92 seconds +2022-04-12 11:41:39.567918: finished prediction +2022-04-12 11:41:39.700278: evaluation of raw predictions