  Deep learning is very data hungry, and supervised learning especially
requires massive labeled data to work well. Machine listening research often
suffers from limited labeled data problem, as human annotations are costly to
acquire, and annotations for audio are time consuming and less intuitive.
Besides, models learned from labeled dataset often embed biases specific to
that particular dataset. Therefore, unsupervised learning techniques become
popular approaches in solving machine listening problems. Particularly, a
self-supervised learning technique utilizing reconstructions of multiple
hand-crafted audio features has shown promising results when it is applied to
speech domain such as emotion recognition and automatic speech recognition
(ASR). In this paper, we apply self-supervised and multi-task learning methods
for pre-training music encoders, and explore various design choices including
encoder architectures, weighting mechanisms to combine losses from multiple
tasks, and worker selections of pretext tasks. We investigate how these design
choices interact with various downstream music classification tasks. We find
that using various music specific workers altogether with weighting mechanisms
to balance the losses during pre-training helps improve and generalize to the
downstream tasks.
