  We revisit and extend the experiments of Goodfellow et al. (2014), who showed
that - for then state-of-the-art networks - "the objective function has a
simple, approximately convex shape" along the linear path between
initialization and the trained weights. We do not find this to be the case for
modern networks on CIFAR-10 and ImageNet. Instead, although loss is roughly
monotonically non-increasing along this path, it remains high until close to
the optimum. In addition, training quickly becomes linearly separated from the
optimum by loss barriers. We conclude that, although Goodfellow et al.'s
findings describe the "relatively easy to optimize" MNIST setting, behavior is
qualitatively different in modern settings.
