  Deep learning models need large amounts of data for training. In video
recognition and classification, significant advances were achieved with the
introduction of new large databases. However, the creation of large-databases
for training is infeasible in several scenarios. Thus, existing or small
collected databases are typically joined and amplified to train these models.
Nevertheless, training neural networks on limited data is not straightforward
and comes with a set of problems. In this paper, we explore the effects of
stacking databases, model initialization, and data amplification techniques
when training with limited data on deep learning models' performance. We
focused on the problem of Facial Expression Recognition from videos. We
performed an extensive study with four databases at a different complexity and
nine deep-learning architectures for video classification. We found that (i)
complex training sets translate better to more stable test sets when trained
with transfer learning and synthetically generated data, but their performance
yields a high variance; (ii) training with more detailed data translates to
more stable performance on novel scenarios (albeit with lower performance);
(iii) merging heterogeneous data is not a straightforward improvement, as the
type of augmentation and initialization is crucial; (iv) classical data
augmentation cannot fill the holes created by joining largely separated
datasets; and (v) inductive biases help to bridge the gap when paired with
synthetic data, but this data is not enough when working with standard
initialization techniques.
