  Self-supervised pre-training using so-called "pretext" tasks has recently
shown impressive performance across a wide range of modalities. In this work,
we advance self-supervised learning from permutations, by pre-training a model
to reorder shuffled parts of the spectrogram of an audio signal, to improve
downstream classification performance. We make two main contributions. First,
we overcome the main challenges of integrating permutation inversions into an
end-to-end training scheme, using recent advances in differentiable ranking.
This was heretofore sidestepped by casting the reordering task as
classification, fundamentally reducing the space of permutations that can be
exploited. Our experiments validate that learning from all possible
permutations improves the quality of the pre-trained representations over using
a limited, fixed set. Second, we show that inverting permutations is a
meaningful pretext task for learning audio representations in an unsupervised
fashion. In particular, we improve instrument classification and pitch
estimation of musical notes by reordering spectrogram patches in the
time-frequency space.
