  Recent observations have advanced our understanding of the neural network
optimization landscape, revealing the existence of (1) paths of high accuracy
containing diverse solutions and (2) wider minima offering improved
performance. Previous methods observing diverse paths require multiple training
runs. In contrast we aim to leverage both property (1) and (2) with a single
method and in a single training run. With a similar computational cost as
training one model, we learn lines, curves, and simplexes of high-accuracy
neural networks. These neural network subspaces contain diverse solutions that
can be ensembled, approaching the ensemble performance of independently trained
networks without the training cost. Moreover, using the subspace midpoint
boosts accuracy, calibration, and robustness to label noise, outperforming
Stochastic Weight Averaging.
