  Multi-objective optimization (MOO) is a prevalent challenge for Deep
Learning, however, there exists no scalable MOO solution for truly deep neural
networks. Prior work either demand optimizing a new network for every point on
the Pareto front, or induce a large overhead to the number of trainable
parameters by using hyper-networks conditioned on modifiable preferences. In
this paper, we propose to condition the network directly on these preferences
by augmenting them to the feature space. Furthermore, we ensure a well-spread
Pareto front by penalizing the solutions to maintain a small angle to the
preference vector. In a series of experiments, we demonstrate that our Pareto
fronts achieve state-of-the-art quality despite being computed significantly
faster. Furthermore, we showcase the scalability as our method approximates the
full Pareto front on the CelebA dataset with an EfficientNet network at a tiny
training time overhead of 7% compared to a simple single-objective
optimization. We make our code publicly available at
https://github.com/ruchtem/cosmos.
