  Designing neural network architectures is a challenging task and knowing
which specific layers of a model must be adapted to improve the performance is
almost a mystery. In this paper, we introduce a novel methodology to identify
layers that decrease the test accuracy of trained models. Conflicting layers
are detected as early as the beginning of training. In the worst-case scenario,
we prove that such a layer could lead to a network that cannot be trained at
all. A theoretical analysis is provided on what is the origin of those layers
that result in a lower overall network performance, which is complemented by
our extensive empirical evaluation. More precisely, we identified those layers
that worsen the performance because they would produce what we name conflicting
training bundles. We will show that around 60% of the layers of trained
residual networks can be completely removed from the architecture with no
significant increase in the test-error. We will further present a novel
neural-architecture-search (NAS) algorithm that identifies conflicting layers
at the beginning of the training. Architectures found by our auto-tuning
algorithm achieve competitive accuracy values when compared against more
complex state-of-the-art architectures, while drastically reducing memory
consumption and inference time for different computer vision tasks. The source
code is available on https://github.com/peerdavid/conflicting-bundles
