  In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained
language models for downstream tasks. We introduce UDALM, a fine-tuning
procedure, using a mixed classification and Masked Language Model loss, that
can adapt to the target domain distribution in a robust and sample efficient
manner. Our experiments show that performance of models trained with the mixed
loss scales with the amount of available target data and the mixed loss can be
effectively used as a stopping criterion during UDA training. Furthermore, we
discuss the relationship between A-distance and the target error and explore
some limitations of the Domain Adversarial Training approach. Our method is
evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset,
yielding $91.74\%$ accuracy, which is an $1.11\%$ absolute improvement over the
state-of-the-art.
