  During the last years, there have been substantial increases in the
prediction performances of natural language processing models on text-based
supervised learning tasks. Especially deep learning models that are based on
the Transformer architecture (Vaswani et al., 2017) and are used in a transfer
learning setting have contributed to this development. As Transformer-based
models for transfer learning have the potential to achieve higher prediction
accuracies with relatively few training data instances, they are likely to
benefit social scientists that seek to have as accurate as possible text-based
measures but only have limited resources for annotating training data. To
enable social scientists to leverage these potential benefits for their
research, this paper explains how these methods work, why they might be
advantageous, and what their limitations are. Additionally, three
Transformer-based models for transfer learning, BERT (Devlin et al., 2019),
RoBERTa (Liu et al., 2019), and the Longformer (Beltagy et al., 2020), are
compared to conventional machine learning algorithms on three social science
applications. Across all evaluated tasks, textual styles, and training data set
sizes, the conventional models are consistently outperformed by transfer
learning with Transformer-based models, thereby demonstrating the potential
benefits these models can bring to text-based social science research.
