  Multimodal affect recognition constitutes an important aspect for enhancing
interpersonal relationships in human-computer interaction. However, relevant
data is hard to come by and notably costly to annotate, which poses a
challenging barrier to build robust multimodal affect recognition systems.
Models trained on these relatively small datasets tend to overfit and the
improvement gained by using complex state-of-the-art models is marginal
compared to simple baselines. Meanwhile, there are many different multimodal
affect recognition datasets, though each may be small. In this paper, we
propose to leverage these datasets using weakly-supervised multi-task learning
to improve the generalization performance on each of them. Specifically, we
explore three multimodal affect recognition tasks: 1) emotion recognition; 2)
sentiment analysis; and 3) sarcasm recognition. Our experimental results show
that multi-tasking can benefit all these tasks, achieving an improvement up to
2.9% accuracy and 3.3% F1-score. Furthermore, our method also helps to improve
the stability of model performance. In addition, our analysis suggests that
weak supervision can provide a comparable contribution to strong supervision if
the tasks are highly correlated.
