Can we use deep learning to predict when deep learning works? Our results suggest the affirmative. We created a dataset by training 13,500 neural networks with different architectures, on different variations of spiral datasets, and using different optimization parameters. We used this dataset to train task-independent and architectureindependent generalization gap predictors for those neural networks. We extend Jiang et al. (2018) to also use DNNs and RNNs and show that they outperform the linear model, obtaining R 2 = 0.965. We also show results for architecture-independent, task-independent, and out-of-distribution generalization gap prediction tasks. Both DNNs and RNNs consistently and significantly outperform linear models, with RNNs obtaining R 2 = 0.584.
REF expand upon this work by training a large number of small fully-connected networks on different variations of a generated spiral dataset.
Towards Task and Architecture-Independent Generalization Gap Predictors
ArXiv
ArXiv
Mathematics Computer Science