Hugging Face's logo

Spaces:
ho11laqe
/
nnUNet_calvingfront_detection
Sleeping

App Files Community
2
nnUNet_calvingfront_detection / Task500_Glacier_zonefronts /fold_0 /training_log_2022_4_6_13_42_45.txt
ho11laqe's picture
ho11laqe
Upload 35 files
a602f58
raw history blame
No virus
33.4 kB
Starting...
2022-04-06 13:42:45.151882: Creating new 5-fold cross-validation split...
2022-04-06 13:42:45.161848: Desired fold for training: 0
2022-04-06 13:42:45.162639: This split has 447 training and 112 validation cases.
2022-04-06 13:46:30.086602: TRAINING KEYS:
odict_keys(['Crane_2002-11-09_ERS_20_2_061', 'Crane_2002-11-18_ERS_20_3_195', 'Crane_2003-01-18_ERS_20_1_061', 'Crane_2006-11-27_ERS_20_1_195', 'Crane_2007-02-05_ERS_20_1_195', 'Crane_2007-03-31_ENVISAT_20_3_467', 'Crane_2007-05-05_ENVISAT_20_1_467', 'Crane_2007-06-09_ENVISAT_20_1_467', 'Crane_2007-07-14_ENVISAT_20_1_467', 'Crane_2007-11-21_PALSAR_17_2_135', 'Crane_2008-01-05_ENVISAT_20_2_467', 'Crane_2008-10-13_TSX_7_3_034', 'Crane_2008-10-24_TSX_7_2_034', 'Crane_2008-11-04_TSX_7_1_034', 'Crane_2008-11-23_PALSAR_17_2_135', 'Crane_2009-09-29_PALSAR_17_1_137', 'Crane_2009-11-02_TSX_7_3_034', 'Crane_2009-11-13_TSX_7_1_034', 'Crane_2010-05-29_ENVISAT_20_3_467', 'Crane_2010-07-03_ENVISAT_20_2_467', 'Crane_2010-09-15_PALSAR_17_4_138', 'Crane_2010-10-14_PALSAR_17_2_135', 'Crane_2010-11-05_TSX_7_2_110', 'Crane_2010-11-16_TSX_7_2_110', 'Crane_2010-12-08_TSX_7_1_110', 'Crane_2010-12-14_TSX_7_1_034', 'Crane_2010-12-19_TSX_7_1_110', 'Crane_2010-12-25_TSX_7_1_034', 'Crane_2011-06-02_TSX_7_2_110', 'Crane_2011-07-16_TSX_7_1_110', 'Crane_2011-08-13_TSX_7_1_034', 'Crane_2011-09-04_TSX_7_1_034', 'Crane_2011-10-18_TSX_7_1_034', 'Crane_2011-12-01_TSX_7_1_034', 'Crane_2011-12-12_TSX_7_2_034', 'Crane_2012-05-14_TSX_7_2_034', 'Crane_2012-06-16_TSX_7_2_034', 'Crane_2012-06-27_TSX_7_2_034', 'Crane_2012-07-03_TSX_7_1_125', 'Crane_2012-07-30_TSX_7_2_034', 'Crane_2012-08-21_TSX_7_3_034', 'Crane_2012-09-01_TSX_7_1_034', 'Crane_2012-10-04_TSX_7_3_034', 'Crane_2013-02-24_TSX_7_2_034', 'Crane_2013-07-17_TSX_7_3_034', 'Crane_2013-07-22_TSX_7_3_110', 'Crane_2013-08-02_TSX_7_3_110', 'Crane_2013-08-19_TSX_7_4_034', 'Crane_2013-11-04_TSX_7_4_034', 'Crane_2013-11-15_TSX_7_3_034', 'Crane_2013-11-26_TSX_7_4_034', 'Crane_2013-12-07_TSX_7_5_034', 'Crane_2013-12-18_TSX_7_4_034', 'Crane_2013-12-29_TSX_7_4_034', 'Crane_2014-01-20_TSX_7_2_034', 'DBE_1995-11-16_ERS_20_1_152', 'DBE_1997-01-29_ERS_20_2_424', 'DBE_1997-03-05_ERS_20_1_424', 'DBE_1998-02-18_ERS_20_1_424', 'DBE_1999-11-10_ERS_20_4_424', 'DBE_2000-10-04_RSAT_20_4_282', 'DBE_2000-10-06_RSAT_12_3_306', 'DBE_2000-10-13_RSAT_20_4_063', 'DBE_2002-02-27_ERS_20_1_424', 'DBE_2002-11-16_ERS_20_3_161', 'DBE_2002-12-21_ERS_20_4_161', 'DBE_2003-01-08_ERS_20_4_424', 'DBE_2003-02-12_ERS_20_3_424', 'DBE_2003-11-19_ERS_20_3_424', 'DBE_2004-03-03_ENVISAT_20_3_424', 'DBE_2004-03-03_ERS_20_3_424', 'DBE_2004-05-12_ENVISAT_20_3_424', 'DBE_2004-08-25_ENVISAT_20_3_424', 'DBE_2004-09-13_RSAT_12_3_282', 'DBE_2004-09-29_ENVISAT_20_3_424', 'DBE_2004-10-07_RSAT_12_4_282', 'DBE_2004-10-16_RSAT_12_3_063', 'DBE_2004-11-03_ENVISAT_20_4_424', 'DBE_2004-11-24_RSAT_12_3_282', 'DBE_2004-12-11_RSAT_12_4_182', 'DBE_2004-12-18_RSAT_12_3_282', 'DBE_2005-01-12_ERS_20_6_424', 'DBE_2005-02-16_ENVISAT_20_1_424', 'DBE_2005-03-23_ENVISAT_20_1_424', 'DBE_2005-05-11_RSAT_12_3_282', 'DBE_2005-06-01_ENVISAT_20_2_424', 'DBE_2005-07-06_ENVISAT_20_3_424', 'DBE_2006-01-06_RSAT_12_4_282', 'DBE_2006-02-23_RSAT_12_3_282', 'DBE_2006-03-08_ENVISAT_20_1_424', 'DBE_2006-03-19_RSAT_12_4_282', 'DBE_2006-04-12_RSAT_12_1_282', 'DBE_2006-05-06_RSAT_12_3_282', 'DBE_2006-05-18_PALSAR_17_2_135', 'DBE_2006-07-03_PALSAR_17_3_135', 'DBE_2006-07-26_ENVISAT_20_3_424', 'DBE_2006-08-06_PALSAR_17_2_137', 'DBE_2006-08-10_RSAT_20_4_282', 'DBE_2006-08-30_ENVISAT_20_2_424', 'DBE_2006-09-03_RSAT_20_5_282', 'DBE_2006-10-04_ENVISAT_20_3_424', 'DBE_2006-11-06_PALSAR_17_3_137', 'DBE_2006-11-08_ENVISAT_20_3_424', 'DBE_2007-01-01_RSAT_12_5_282', 'DBE_2007-01-25_RSAT_12_5_282', 'DBE_2007-02-18_RSAT_12_4_282', 'DBE_2007-03-14_RSAT_12_3_282', 'DBE_2007-05-01_RSAT_12_4_282', 'DBE_2007-11-11_PALSAR_17_4_129', 'DBE_2007-12-03_RSAT_12_4_282', 'DBE_2007-12-27_PALSAR_17_5_129', 'DBE_2007-12-27_RSAT_12_5_282', 'DBE_2008-01-20_RSAT_12_4_282', 'DBE_2008-03-30_TSX_7_3_049', 'DBE_2008-09-28_PALSAR_17_5_129', 'DBE_2008-10-25_TSX_7_2_049', 'DBE_2008-11-05_TSX_7_2_049', 'DBE_2008-11-13_PALSAR_17_5_129', 'DBE_2009-02-13_PALSAR_17_5_129', 'DBE_2009-11-03_TSX_7_2_049', 'DBE_2009-12-03_PALSAR_17_5_132', 'DBE_2010-01-18_PALSAR_17_5_132', 'DBE_2010-03-17_ENVISAT_20_6_424', 'DBE_2010-10-10_TSX_7_2_049', 'DBE_2010-10-21_PALSAR_17_4_132', 'DBE_2010-11-19_PALSAR_17_4_129', 'DBE_2010-11-23_TSX_7_5_049', 'DBE_2010-12-01_PALSAR_17_4_127', 'DBE_2011-01-04_PALSAR_17_4_129', 'DBE_2011-01-16_PALSAR_17_4_127', 'DBE_2011-01-17_TSX_7_4_049', 'DBE_2011-01-28_TSX_7_2_049', 'DBE_2011-07-05_TSX_7_1_110', 'DBE_2011-07-27_TSX_7_1_110', 'DBE_2011-09-09_TSX_7_1_110', 'DBE_2011-09-20_TSX_7_1_110', 'DBE_2011-10-23_TSX_7_1_110', 'DBE_2011-11-14_TSX_7_1_110', 'DBE_2011-12-06_TSX_7_5_110', 'DBE_2011-12-28_TSX_7_2_110', 'DBE_2012-04-05_TSX_7_1_110', 'DBE_2012-05-08_TSX_7_1_110', 'DBE_2012-06-21_TSX_7_1_110', 'DBE_2012-08-04_TSX_7_1_110', 'DBE_2012-10-31_TSX_7_1_110', 'DBE_2012-11-11_TSX_7_1_110', 'DBE_2013-03-12_TSX_7_1_110', 'DBE_2013-03-23_TSX_7_1_110', 'DBE_2013-04-03_TSX_7_1_110', 'DBE_2013-07-11_TSX_7_1_110', 'DBE_2013-09-04_TSX_7_1_110', 'DBE_2013-10-07_TSX_7_1_110', 'DBE_2013-12-01_TSX_7_2_110', 'DBE_2013-12-23_TSX_7_2_110', 'DBE_2014-01-03_TSX_7_5_110', 'DBE_2014-02-05_TSX_7_1_110', 'DBE_2014-03-21_TSX_7_2_110', 'JAC_2009-04-16_TSX_6_1_005', 'JAC_2009-05-08_TSX_6_1_005', 'JAC_2009-05-19_TSX_6_1_005', 'JAC_2009-05-30_TSX_6_1_005', 'JAC_2009-06-21_TSX_6_1_005', 'JAC_2009-07-02_TSX_6_1_005', 'JAC_2009-07-13_TSX_6_1_005', 'JAC_2009-07-24_TSX_6_1_005', 'JAC_2009-08-04_TSX_6_1_005', 'JAC_2009-08-15_TSX_6_1_005', 'JAC_2009-08-26_TSX_6_1_005', 'JAC_2009-09-06_TSX_6_1_005', 'JAC_2009-09-17_TSX_6_1_005', 'JAC_2009-09-28_TSX_6_1_005', 'JAC_2009-10-09_TSX_6_1_005', 'JAC_2009-10-20_TSX_6_1_005', 'JAC_2009-11-11_TSX_6_1_005', 'JAC_2009-11-22_TSX_6_1_005', 'JAC_2009-12-14_TSX_6_1_005', 'JAC_2009-12-25_TSX_6_1_005', 'JAC_2010-01-05_TSX_6_1_005', 'JAC_2010-01-16_TSX_6_1_005', 'JAC_2010-02-07_TSX_6_1_005', 'JAC_2010-02-18_TSX_6_1_005', 'JAC_2010-03-12_TSX_6_1_005', 'JAC_2010-03-23_TSX_6_1_005', 'JAC_2010-04-14_TSX_6_1_005', 'JAC_2010-04-25_TSX_6_1_005', 'JAC_2010-05-06_TSX_6_1_005', 'JAC_2010-05-17_TSX_6_1_005', 'JAC_2010-06-19_TSX_6_1_005', 'JAC_2010-06-30_TSX_6_1_005', 'JAC_2010-07-11_TSX_6_1_005', 'JAC_2010-07-22_TSX_6_1_005', 'JAC_2010-09-04_TSX_6_1_005', 'JAC_2010-10-07_TSX_6_1_005', 'JAC_2010-10-18_TSX_6_1_005', 'JAC_2010-11-09_TSX_6_1_005', 'JAC_2010-11-20_TSX_6_1_005', 'JAC_2011-01-03_TSX_6_1_005', 'JAC_2011-01-25_TSX_6_1_005', 'JAC_2011-02-16_TSX_6_1_005', 'JAC_2011-03-10_TSX_6_1_005', 'JAC_2011-03-21_TSX_6_1_005', 'JAC_2011-04-12_TSX_6_1_005', 'JAC_2011-04-23_TSX_6_1_005', 'JAC_2011-05-26_TSX_6_1_005', 'JAC_2011-07-20_TSX_6_1_005', 'JAC_2011-08-23_TSX_6_1_027', 'JAC_2011-09-03_TSX_6_1_027', 'JAC_2011-09-14_TSX_6_1_027', 'JAC_2011-10-05_TSX_6_1_005', 'JAC_2011-10-16_TSX_6_1_005', 'JAC_2011-11-08_TSX_6_1_027', 'JAC_2011-11-18_TSX_6_1_005', 'JAC_2011-12-10_TSX_6_1_005', 'JAC_2011-12-21_TSX_6_1_005', 'JAC_2012-01-13_TSX_6_1_027', 'JAC_2012-01-23_TSX_6_1_005', 'JAC_2012-02-14_TSX_6_1_005', 'JAC_2012-02-25_TSX_6_1_005', 'JAC_2012-03-29_TSX_6_1_005', 'JAC_2012-06-03_TSX_6_1_005', 'JAC_2012-06-14_TSX_6_1_005', 'JAC_2012-07-07_TSX_6_1_027', 'JAC_2012-07-18_TSX_6_1_027', 'JAC_2012-08-09_TSX_6_1_027', 'JAC_2012-09-21_TSX_6_1_005', 'JAC_2012-10-03_TSX_6_1_027', 'JAC_2012-10-14_TSX_6_1_027', 'JAC_2012-11-04_TSX_6_1_005', 'JAC_2012-11-15_TSX_6_1_005', 'JAC_2012-12-07_TSX_6_1_005', 'JAC_2012-12-19_TSX_6_1_027', 'JAC_2013-01-09_TSX_6_1_005', 'JAC_2013-01-20_TSX_6_1_005', 'JAC_2013-02-11_TSX_6_1_005', 'JAC_2013-02-22_TSX_6_1_005', 'JAC_2013-03-06_TSX_6_1_027', 'JAC_2013-03-17_TSX_6_1_027', 'JAC_2013-04-18_TSX_6_1_005', 'JAC_2013-05-10_TSX_6_1_005', 'JAC_2013-06-24_TSX_6_1_027', 'JAC_2013-07-05_TSX_6_1_027', 'JAC_2013-07-27_TSX_6_1_027', 'JAC_2013-08-17_TSX_6_1_005', 'JAC_2013-08-28_TSX_6_1_005', 'JAC_2013-09-20_TSX_6_1_027', 'JAC_2013-10-12_TSX_6_1_027', 'JAC_2013-10-23_TSX_6_1_027', 'JAC_2013-11-14_TSX_6_1_027', 'JAC_2013-11-25_TSX_6_1_027', 'JAC_2013-12-06_TSX_6_1_027', 'JAC_2014-03-03_TSX_6_1_005', 'JAC_2014-03-14_TSX_6_1_005', 'JAC_2014-03-25_TSX_6_1_005', 'JAC_2014-04-05_TSX_6_1_005', 'JAC_2014-04-27_TSX_6_1_005', 'JAC_2014-06-10_TSX_6_1_005', 'JAC_2014-06-21_TSX_6_1_005', 'JAC_2014-07-02_TSX_6_1_005', 'JAC_2014-07-14_TSX_6_1_027', 'JAC_2014-08-04_TSX_6_1_005', 'JAC_2014-09-18_TSX_6_1_027', 'JAC_2014-09-28_TSX_6_1_005', 'JAC_2014-10-09_TSX_6_1_005', 'JAC_2014-10-20_TSX_6_1_005', 'JAC_2014-11-11_TSX_6_1_005', 'JAC_2014-12-03_TSX_6_1_005', 'JAC_2014-12-14_TSX_6_1_005', 'JAC_2015-01-16_TSX_6_1_005', 'JAC_2015-01-27_TSX_6_1_005', 'JAC_2015-02-07_TSX_6_1_005', 'JAC_2015-02-19_TSX_6_1_027', 'JAC_2015-03-12_TSX_6_1_005', 'JAC_2015-03-23_TSX_6_1_005', 'JAC_2015-04-14_TSX_6_1_005', 'JAC_2015-04-25_TSX_6_1_005', 'JAC_2015-05-17_TSX_6_1_005', 'JAC_2015-06-19_TSX_6_1_005', 'JAC_2015-07-22_TSX_6_1_005', 'JAC_2015-08-02_TSX_6_1_005', 'JAC_2015-08-13_TSX_6_1_005', 'JAC_2015-09-04_TSX_6_1_005', 'JAC_2015-09-26_TSX_6_1_005', 'JAC_2015-10-18_TSX_6_1_005', 'JAC_2015-10-29_TSX_6_1_005', 'JAC_2015-11-09_TSX_6_1_005', 'JAC_2015-12-23_TSX_6_1_005', 'Jorum_2006-11-27_ERS_20_3_195', 'Jorum_2007-02-05_ERS_20_1_195', 'Jorum_2007-03-31_ENVISAT_20_2_467', 'Jorum_2007-07-14_ENVISAT_20_2_467', 'Jorum_2007-09-22_ENVISAT_20_3_467', 'Jorum_2007-10-27_ENVISAT_20_3_467', 'Jorum_2007-11-21_PALSAR_17_2_135', 'Jorum_2008-01-05_ENVISAT_20_2_467', 'Jorum_2009-12-25_PALSAR_17_1_136', 'Jorum_2010-03-20_ENVISAT_20_2_467', 'Jorum_2010-05-29_ENVISAT_20_2_467', 'Jorum_2010-07-03_ENVISAT_20_2_467', 'Jorum_2010-09-15_PALSAR_17_2_138', 'Jorum_2010-10-14_PALSAR_17_2_135', 'Jorum_2010-11-05_TSX_7_1_110', 'Jorum_2010-11-16_TSX_7_2_110', 'Jorum_2010-11-27_TSX_7_2_110', 'Jorum_2010-11-29_PALSAR_17_1_135', 'Jorum_2010-12-14_TSX_7_1_034', 'Jorum_2010-12-30_TSX_7_1_110', 'Jorum_2011-06-02_TSX_7_3_110', 'Jorum_2011-06-24_TSX_7_2_110', 'Jorum_2011-07-16_TSX_7_3_110', 'Jorum_2011-08-13_TSX_7_2_034', 'Jorum_2011-09-04_TSX_7_4_034', 'Jorum_2011-10-18_TSX_7_3_034', 'Jorum_2011-12-01_TSX_7_1_034', 'Jorum_2011-12-12_TSX_7_3_034', 'Jorum_2012-06-27_TSX_7_2_034', 'Jorum_2012-07-03_TSX_7_2_125', 'Jorum_2012-07-25_TSX_7_2_125', 'Jorum_2012-08-10_TSX_7_2_034', 'Jorum_2012-08-21_TSX_7_2_034', 'Jorum_2012-09-01_TSX_7_2_034', 'Jorum_2012-10-04_TSX_7_2_034', 'Jorum_2012-11-06_TSX_7_2_034', 'Jorum_2013-03-07_TSX_7_2_034', 'Jorum_2013-03-29_TSX_7_2_034', 'Jorum_2013-06-30_TSX_7_2_110', 'Jorum_2013-07-17_TSX_7_1_034', 'Jorum_2013-08-19_TSX_7_2_034', 'Jorum_2013-11-04_TSX_7_2_034', 'Jorum_2013-11-15_TSX_7_2_034', 'Jorum_2013-12-07_TSX_7_5_034', 'Jorum_2013-12-18_TSX_7_5_034', 'Jorum_2013-12-29_TSX_7_4_034', 'Jorum_2014-01-20_TSX_7_2_034', 'Jorum_2016-07-03_S1_20_2_009', 'Jorum_2016-09-01_S1_20_2_009', 'Jorum_2016-12-30_S1_20_2_009', 'Jorum_2017-09-08_S1_20_2_009', 'Jorum_2018-01-06_S1_20_3_009', 'Jorum_2018-03-07_S1_20_3_009', 'Jorum_2018-06-11_S1_20_2_009', 'Jorum_2018-09-03_S1_20_3_009', 'Jorum_2019-01-01_S1_20_3_009', 'Jorum_2019-03-02_S1_20_3_009', 'Jorum_2019-09-22_S1_20_3_009', 'Jorum_2020-01-08_S1_20_3_009', 'Jorum_2020-03-08_S1_20_3_009', 'SI_1995-11-13_ERS_20_3_109', 'SI_1995-11-14_ERS_20_4_118', 'SI_1995-11-15_ERS_20_3_118', 'SI_1995-11-16_ERS_20_4_152', 'SI_1996-01-23_ERS_20_5_109', 'SI_1996-01-24_ERS_20_4_118', 'SI_1996-02-12_ERS_20_2_390', 'SI_1996-02-29_ERS_20_2_152', 'SI_1996-03-18_ERS_20_2_390', 'SI_1997-01-27_ERS_20_4_390', 'SI_2000-09-10_RSAT_20_5_282', 'SI_2000-09-19_RSAT_20_5_063', 'SI_2000-10-04_RSAT_20_5_282', 'SI_2000-10-13_RSAT_20_5_063', 'SI_2002-01-21_ERS_20_2_390', 'SI_2002-02-25_ERS_20_3_390', 'SI_2002-11-12_ERS_20_4_109', 'SI_2002-12-21_ERS_20_3_161', 'SI_2003-12-05_ENVISAT_20_4_152', 'SI_2004-07-02_ENVISAT_20_4_152', 'SI_2004-08-06_ENVISAT_20_4_152', 'SI_2004-09-13_RSAT_12_5_282', 'SI_2004-10-15_ERS_20_5_152', 'SI_2004-11-09_RSAT_12_5_063', 'SI_2004-11-19_ERS_20_4_152', 'SI_2004-11-24_RSAT_12_4_282', 'SI_2004-12-18_RSAT_12_4_282', 'SI_2004-12-24_ENVISAT_20_5_152', 'SI_2004-12-24_ERS_20_5_152', 'SI_2005-01-28_ERS_20_5_152', 'SI_2005-02-28_RSAT_12_5_282', 'SI_2005-03-04_ENVISAT_20_3_152', 'SI_2005-04-08_ENVISAT_20_2_152', 'SI_2005-04-17_RSAT_12_3_282', 'SI_2005-05-11_RSAT_12_2_282', 'SI_2005-07-22_ENVISAT_20_2_152', 'SI_2005-08-26_ENVISAT_20_2_152', 'SI_2006-01-06_RSAT_12_2_282', 'SI_2006-02-23_RSAT_12_1_282', 'SI_2006-03-05_ENVISAT_20_2_381', 'SI_2006-03-19_RSAT_12_2_282', 'SI_2006-04-12_RSAT_12_1_282', 'SI_2006-05-06_RSAT_12_3_282', 'SI_2006-07-03_PALSAR_17_3_135', 'SI_2006-07-23_ENVISAT_20_4_381', 'SI_2006-08-10_RSAT_20_3_282', 'SI_2006-09-03_RSAT_12_2_282', 'SI_2006-09-03_RSAT_20_2_282', 'SI_2006-10-18_ERS_20_4_118', 'SI_2006-11-21_ERS_20_3_109', 'SI_2007-01-01_RSAT_12_3_282', 'SI_2007-01-14_ENVISAT_20_3_381', 'SI_2007-03-14_RSAT_12_2_282', 'SI_2007-03-25_ENVISAT_20_3_381', 'SI_2007-05-01_RSAT_12_2_282', 'SI_2007-06-03_ENVISAT_20_3_381', 'SI_2007-07-08_ENVISAT_20_3_381', 'SI_2007-09-16_ENVISAT_20_3_381', 'SI_2007-10-21_ENVISAT_20_3_381', 'SI_2007-12-03_RSAT_12_4_282', 'SI_2007-12-27_RSAT_12_2_282', 'SI_2008-01-20_RSAT_12_2_282', 'SI_2008-04-13_ENVISAT_20_2_381', 'SI_2008-05-18_ENVISAT_20_3_381', 'SI_2008-07-27_ENVISAT_20_4_381', 'SI_2008-08-31_ENVISAT_20_4_381', 'SI_2009-02-22_ENVISAT_20_2_381', 'SI_2009-03-29_ENVISAT_20_4_381', 'SI_2009-07-12_ENVISAT_20_3_381', 'SI_2009-08-16_ENVISAT_20_3_381', 'SI_2010-01-13_PALSAR_17_2_127', 'SI_2010-01-22_ERS_20_3_152', 'SI_2010-02-26_ERS_20_2_152', 'SI_2010-04-02_ERS_20_2_152', 'SI_2010-10-16_PALSAR_17_3_127', 'SI_2010-12-01_PALSAR_17_3_127', 'SI_2011-01-16_PALSAR_17_2_127', 'SI_2011-03-03_PALSAR_17_2_127', 'SI_2011-06-14_TSX_7_1_125', 'SI_2011-07-06_TSX_7_1_125', 'SI_2011-08-19_TSX_7_1_125', 'SI_2011-09-10_TSX_7_1_125', 'SI_2011-09-21_TSX_7_1_125', 'SI_2011-12-29_TSX_7_1_125', 'SI_2012-01-09_TSX_7_1_125', 'SI_2012-03-04_TSX_7_1_125', 'SI_2012-03-15_TSX_7_1_125', 'SI_2012-03-26_TSX_7_1_125', 'SI_2012-05-20_TSX_7_1_125', 'SI_2012-06-22_TSX_7_1_125', 'SI_2012-08-05_TSX_7_1_125', 'SI_2012-08-16_TSX_7_1_125', 'SI_2013-07-12_TSX_7_1_125', 'SI_2013-08-14_TSX_7_1_125', 'SI_2013-09-16_TSX_7_1_125', 'SI_2013-10-19_TSX_7_1_125', 'SI_2013-10-28_TSX_7_1_104', 'SI_2013-11-21_TSX_7_1_125', 'SI_2014-01-04_TSX_7_2_125', 'SI_2014-04-02_TSX_7_1_125', 'SI_2014-12-11_TSX_7_4_125'])
2022-04-06 13:46:30.089200: VALIDATION KEYS:
odict_keys(['Crane_2007-09-22_ENVISAT_20_1_467', 'Crane_2007-09-24_PALSAR_17_1_137', 'Crane_2007-10-27_ENVISAT_20_2_467', 'Crane_2010-11-27_TSX_7_1_110', 'Crane_2010-11-29_PALSAR_17_1_135', 'Crane_2010-12-30_TSX_7_1_110', 'Crane_2011-06-24_TSX_7_1_110', 'Crane_2012-07-25_TSX_7_2_125', 'Crane_2012-08-10_TSX_7_3_034', 'Crane_2012-11-06_TSX_7_2_034', 'Crane_2013-03-07_TSX_7_2_034', 'Crane_2013-03-29_TSX_7_2_034', 'Crane_2013-06-30_TSX_7_2_110', 'Crane_2014-02-11_TSX_7_3_034', 'DBE_1996-01-25_ERS_20_1_152', 'DBE_2000-09-10_RSAT_20_3_282', 'DBE_2004-04-07_ENVISAT_20_3_424', 'DBE_2004-06-16_ENVISAT_20_3_424', 'DBE_2004-07-21_ENVISAT_20_3_424', 'DBE_2004-11-09_RSAT_12_3_063', 'DBE_2004-12-08_ENVISAT_20_5_424', 'DBE_2004-12-08_ERS_20_5_424', 'DBE_2005-04-27_ENVISAT_20_2_424', 'DBE_2005-08-10_ENVISAT_20_2_424', 'DBE_2006-04-12_ENVISAT_20_1_424', 'DBE_2006-06-21_ENVISAT_20_2_424', 'DBE_2006-06-21_PALSAR_17_2_137', 'DBE_2007-08-15_ENVISAT_20_5_424', 'DBE_2007-09-19_ENVISAT_20_6_424', 'DBE_2008-10-14_TSX_7_3_049', 'DBE_2008-12-29_PALSAR_17_5_129', 'DBE_2009-10-23_TSX_7_2_049', 'DBE_2010-01-22_ERS_20_6_152', 'DBE_2010-04-21_ENVISAT_20_6_424', 'DBE_2010-10-16_PALSAR_17_4_127', 'DBE_2010-11-12_TSX_7_4_049', 'DBE_2010-12-06_PALSAR_17_3_132', 'DBE_2012-03-14_TSX_7_1_110', 'DBE_2012-06-10_TSX_7_1_110', 'DBE_2012-08-15_TSX_7_1_110', 'DBE_2013-01-16_TSX_7_5_110', 'DBE_2013-05-28_TSX_7_1_110', 'DBE_2013-06-19_TSX_7_1_110', 'DBE_2013-11-09_TSX_7_1_110', 'DBE_2013-12-12_TSX_7_3_110', 'JAC_2009-04-27_TSX_6_1_005', 'JAC_2009-06-10_TSX_6_1_005', 'JAC_2009-10-31_TSX_6_1_005', 'JAC_2009-12-03_TSX_6_1_005', 'JAC_2010-09-15_TSX_6_1_005', 'JAC_2010-12-01_TSX_6_1_005', 'JAC_2010-12-23_TSX_6_1_005', 'JAC_2011-05-15_TSX_6_1_005', 'JAC_2011-06-17_TSX_6_1_005', 'JAC_2012-03-07_TSX_6_1_005', 'JAC_2012-04-20_TSX_6_1_005', 'JAC_2012-05-24_TSX_6_1_027', 'JAC_2012-08-20_TSX_6_1_027', 'JAC_2012-09-11_TSX_6_1_027', 'JAC_2013-04-07_TSX_6_1_005', 'JAC_2013-05-22_TSX_6_1_027', 'JAC_2013-06-13_TSX_6_1_027', 'JAC_2013-09-08_TSX_6_1_005', 'JAC_2013-12-17_TSX_6_1_027', 'JAC_2014-01-29_TSX_6_1_005', 'JAC_2014-04-16_TSX_6_1_005', 'JAC_2014-07-03_TSX_6_1_027', 'JAC_2014-07-25_TSX_6_1_027', 'JAC_2014-12-25_TSX_6_1_005', 'JAC_2015-01-05_TSX_6_1_005', 'JAC_2015-05-06_TSX_6_1_005', 'JAC_2015-06-08_TSX_6_1_005', 'JAC_2015-07-11_TSX_6_1_005', 'JAC_2015-11-20_TSX_6_1_005', 'JAC_2015-12-12_TSX_6_1_005', 'Jorum_2003-01-27_ERS_20_1_195', 'Jorum_2007-05-05_ENVISAT_20_2_467', 'Jorum_2007-06-09_ENVISAT_20_2_467', 'Jorum_2008-11-23_PALSAR_17_2_135', 'Jorum_2010-12-08_TSX_7_2_110', 'Jorum_2010-12-19_TSX_7_1_110', 'Jorum_2010-12-25_TSX_7_1_034', 'Jorum_2012-05-14_TSX_7_2_034', 'Jorum_2012-06-16_TSX_7_2_034', 'Jorum_2012-07-30_TSX_7_2_034', 'Jorum_2013-02-24_TSX_7_3_034', 'Jorum_2013-07-22_TSX_7_1_110', 'Jorum_2013-08-02_TSX_7_2_110', 'Jorum_2013-11-26_TSX_7_5_034', 'Jorum_2014-02-11_TSX_7_2_034', 'Jorum_2017-06-16_S1_20_2_009', 'Jorum_2020-06-12_S1_20_3_009', 'SI_1996-01-22_ERS_20_3_109', 'SI_1997-03-03_ERS_20_4_390', 'SI_1999-11-07_ERS_20_2_381', 'SI_2000-02-20_ERS_20_3_381', 'SI_2003-01-25_ERS_20_3_161', 'SI_2004-02-13_ENVISAT_20_4_152', 'SI_2004-12-03_RSAT_12_5_063', 'SI_2006-01-29_ENVISAT_20_3_381', 'SI_2006-05-18_PALSAR_17_2_135', 'SI_2006-08-27_ENVISAT_20_3_381', 'SI_2007-01-25_RSAT_12_1_282', 'SI_2011-07-28_TSX_7_1_125', 'SI_2011-11-26_TSX_7_2_125', 'SI_2012-04-28_TSX_7_1_125', 'SI_2012-07-03_TSX_7_1_125', 'SI_2013-11-08_TSX_7_2_104', 'SI_2013-12-02_TSX_7_1_125', 'SI_2013-12-13_TSX_7_2_125', 'SI_2014-03-22_TSX_7_1_125', 'SI_2014-12-22_TSX_7_2_125'])
2022-04-06 13:48:05.212889: lr: 0.01
2022-04-06 13:48:13.483644: Unable to plot network architecture:
2022-04-06 13:48:13.507257: No module named 'hiddenlayer'
2022-04-06 13:48:13.519965:
printing the network instead:
2022-04-06 13:48:13.530723: Generic_UNet(
(conv_blocks_localization): ModuleList(
(0): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
(1): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
(2): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
(3): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(960, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
(4): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(512, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
(5): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(256, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
(6): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(128, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
(7): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(64, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
)
(conv_blocks_context): ModuleList(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(1, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
(1): ConvDropoutNormNonlin(
(conv): Conv2d(32, 32, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(32, 64, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
(instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
(1): ConvDropoutNormNonlin(
(conv): Conv2d(64, 64, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(2): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(64, 128, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
(instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
(1): ConvDropoutNormNonlin(
(conv): Conv2d(128, 128, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(3): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(128, 256, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
(instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
(1): ConvDropoutNormNonlin(
(conv): Conv2d(256, 256, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(4): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(256, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
(1): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(5): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
(1): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(6): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
(1): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(7): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
(1): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(8): Sequential(
(0): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=[2, 2], padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
(1): StackedConvLayers(
(blocks): Sequential(
(0): ConvDropoutNormNonlin(
(conv): Conv2d(480, 480, kernel_size=[3, 3], stride=(1, 1), padding=[1, 1])
(instnorm): InstanceNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
(lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
)
)
)
)
)
(td): ModuleList()
(tu): ModuleList(
(0): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
(1): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
(2): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
(3): ConvTranspose2d(480, 480, kernel_size=[2, 2], stride=[2, 2], bias=False)
(4): ConvTranspose2d(480, 256, kernel_size=[2, 2], stride=[2, 2], bias=False)
(5): ConvTranspose2d(256, 128, kernel_size=[2, 2], stride=[2, 2], bias=False)
(6): ConvTranspose2d(128, 64, kernel_size=[2, 2], stride=[2, 2], bias=False)
(7): ConvTranspose2d(64, 32, kernel_size=[2, 2], stride=[2, 2], bias=False)
)
(seg_outputs): ModuleList(
(0): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
(2): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
(3): Conv2d(480, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
(4): Conv2d(256, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
(5): Conv2d(128, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
(6): Conv2d(64, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
(7): Conv2d(32, 5, kernel_size=(1, 1), stride=(1, 1), bias=False)
)
)
2022-04-06 13:48:13.561954:
2022-04-06 13:48:13.574286: svmem(total=99932160000, available=89355911168, percent=10.6, used=7974862848, free=74463080448, active=7446126592, inactive=15933480960, buffers=14180352, cached=17480036352, shared=786059264, slab=754434048)
2022-04-06 13:48:13.587833:
epoch: 0
2022-04-06 13:49:16.764398: train loss : 1.2657
2022-04-06 13:49:29.162044: validation loss: 1.0054
2022-04-06 13:49:29.184389: Average global foreground Dice: [0.0002, 0.6649, 0.0923, 0.0013]
2022-04-06 13:49:29.186826: (interpret this as an estimate for the Dice of the different classes. This is not exact.)
2022-04-06 13:49:30.284652: lr: 0.0
2022-04-06 13:49:30.286498: saving scheduled checkpoint file...
2022-04-06 13:49:30.449905: saving checkpoint...
2022-04-06 13:49:35.022912: done, saving took 4.73 seconds
2022-04-06 13:49:35.052222: done
2022-04-06 13:49:35.057490: This epoch took 81.447241 s
2022-04-06 13:49:35.205750: saving checkpoint...
2022-04-06 13:49:39.567125: done, saving took 4.51 seconds
2022-04-06 13:59:38.934049: finished prediction
2022-04-06 13:59:38.935330: evaluation of raw predictions