  Pretrained language models are now of widespread use in Natural Language
Processing. Despite their success, applying them to Low Resource languages is
still a huge challenge. Although Multilingual models hold great promise,
applying them to specific low-resource languages e.g. Roman Urdu can be
excessive. In this paper, we show how the code-switching property of languages
may be used to perform cross-lingual transfer learning from a corresponding
high resource language. We also show how this transfer learning technique
termed Bilingual Language Modeling can be used to produce better performing
models for Roman Urdu. To enable training and experimentation, we also present
a collection of novel corpora for Roman Urdu extracted from various sources and
social networking sites, e.g. Twitter. We train Monolingual, Multilingual, and
Bilingual models of Roman Urdu - the proposed bilingual model achieves 23%
accuracy compared to the 2% and 11% of the monolingual and multilingual models
respectively in the Masked Language Modeling (MLM) task.
