On the Universality of Deep Contextual Language Models (2021)

Abstract: Deep Contextual Language Models (LMs) like ELMO, BERT, and their successors dominate the landscape of Natural Language Processing due to their ability to scale across multiple tasks rapidly by pre-training a single model, followed by task-specific fine-tuning. Furthermore, multilingual versions of such models like XLM-R and mBERT have given promising results in zero-shot cross-lingual transfer, potentially enabling NLP applications in many under-served and under-resourced languages. Due to this initial success, pre-trained models are being used as ‘Universal Language Models’ as the starting point across diverse tasks, domains, and languages. This work explores the notion of ‘Universality’ by identifying seven dimensions across which a universal model should be able to scale, that is, perform equally well or reasonably well, to be useful across diverse settings. We outline the current theoretical and empirical results that support model performance across these dimensions, along with extensions that may help address some of their current limitations. Through this survey, we lay the foundation for understanding the capabilities and limitations of massive contextual language models and help discern research gaps and directions for future work to make these LMs inclusive and fair to diverse applications, users, and linguistic phenomena.

## introduction language models (lms) have evolved considerably in the past decade, starting from the introduction of word2vec @xcite to the more recent transformer-based deep models like bert @xcite and its successors. when finetuned with task-specific data, pre-trained lms can be adapted to several different settings, i.e., tasks, domains, and even languages, as these lms have * work done while at microsoft research, india been extended to multiple languages in the multilingual versions like m-bert and derivatives. these models can be thought of as 'universal' because of their potential to be utilized 'universally' in several different application scenarios. 1 the merits of transfer learning or pre-training word representations have been known for a long time. moreover, the recent advancements in largescale deep learning have pushed the boundaries of intensive computation and tremendous amounts of data that can be used to pre-train lms. however, pre-training is resource-intensive and is not carried out for specific scenarios. instead, massive lms are deployed into downstream applications with potentially billions of users around the world. this makes 'universality' a vital characteristic as the models must be inclusive towards a variety of language usage. the key contributions of this paper are: our definition of universality spans seven dimensions. these are: language, multilingualism, task, domain, medium of expression, demography and geography, and time period. we selected these dimensions to cover a broad spectrum of language usage and a diverse set of nlp applications. ideally, a truly universal model should perform strongly, or at least reasonably, across them. we present a detailed analysis of the capabilities and limitations of lms in these dimensions in the subsequent sections. this is followed by extensions, which are techniques that can be leveraged to overcome the limitations in the particular dimension. ## language there are over 7000 languages in the world. there is an increased demand for multilingual systems as information technologies penetrate more lives globally. the largest available lms include mbert @xcite , xlm-r @xcite , and mt5 @xcite serve 104, 100, and 101 languages respectively. it is clear that they are far from universal in terms of language coverage compared to the number of languages in the world. further, there is an expectation that massive multilingual lms will perform equally, or at least reasonably, well on all the languages they serve. the limited availability of evaluation benchmarks is a major bottleneck in knowing how lms perform across the languages they are pre-trained on. there is a great disparity in performance across the languages that are tested through these benchmarks. a general observation is that the performance of low resource languages continues to be lower than high resource languages. the extent to which cross-lingual transfer helps improve performance varies across languages. studies that empirically support these claims are: cross-script transfer is not equally good across languages in mbert. @xcite find that cross-script cross-lingual transfer is effective in the case of hindi and urdu, whereas this is not observed between english and japanese. word order differences across language leads to worse cross-lingual transfer @xcite . the correlation between word ordering distance and cross-lingual transfer is found to be high in the experiments by @xcite also find that word order has a significant bearing on transfer. contrary to common intuition, k et al. 2020 find that shared vocabulary does not affect universality or generalization across languages considerably. @xcite also find that 'effective vocabulary size per language' affects cross-lingual performance rather than joint or disjoint vocabulary of multiple languages. in massively multilingual lms, where typically a joint vocabulary across languages is used, languages tend to compete for the allocations in the shared vocabulary. @xcite show that increasing number of languages may worsen performance compared to models with fewer languages. this is similar to the findings of wu and dredze 2019. thus, limiting pre-training to only the required languages needed for the downstream tasks may be more beneficial. @xcite coined the term "curse of multilinguality" for this phenomenon and pointed to the trade-off between model performance and language coverage. this result is also shown in muril, a bert model trained on 17 indian languages, which outperforms mbert on the xtreme benchmark significantly across all languages @xcite . similarly, clustering languages and using different multilingual model for each group, rather than one massive model, gives better performance in neural machine translation @xcite . wu and dredze 2019 observe that mbert does not transfer well between distant languages. further, they conclude that while mbert may perform very well in cross-lingual transfer compared to other models, it still falls short of models that have been trained with cross-lingual signals like bitext, bilingual dictionaries, or limited target language supervision. answering whether all languages in mbert are equally well represented, wu and dredze 2020 find that mbert does not learn high-quality representations for all languages, especially for low resource languages. the bottom 30% languages in terms of data size perform even worse than a non-bert model for ner. for low resource languages, the combined effect of less data for pre-training and annotated data for fine-tuning compounds together leading to worsening of their performance. on the other end of the spectrum, the top 10% languages are hurt by joint training as mbert performs worse than monolingual baselines of ner. to summarize, universality in the language dimension has three levels. at the highest level, the largest models available today span only around 100 of the 7000+ languages globally and thus are far from universal in terms of language coverage. secondly, out of the languages that these models are trained on, not all of them are evaluated, implying that we do not have enough information to make generalized claims of universality in performance for the languages that the lms support. finally, at the lowest level, the performance is not uniform across the tested languages. the performance of lower resourced languages is lower than that of higher-resourced languages. increasing the number of languages hurts performance at both ends of the spectrum, and cross-lingual transfer is non-uniform and dependent on many factors. extensions: monolingual models learn generalizable representations and can be adapted to new languages without joint training. @xcite show that the representations learned by monolingual models without any shared vocabulary align with each other and can be adapted to a new language. similarly, artetxe et al. 2020b study the transfer of monolingual representations to new languages without using shared vocabulary or joint training. they propose a zero-shot cross-lingual transfer technique where the resultant model is a monolingual lm adapted to a new language. tela et al. 2020 study adaptation to the extremely low resourced language, tigrinya. they find that english xlnet generalizes better than bert and mbert, which is surprising given that mbert is trained in multiple languages. thus, the adaptation of monolingual models may help in extending lms to new low-resource languages. wang et al. 2020 enlarge mbert's vocabulary and continue pre-training on 27 target languages, out of which 11 are new. they observe performance improvement in zero-shot cross-lingual ner for all 27 languages. the extension benefits both the existing and newly added languages. the drawback is that the base model (mbert) is biased towards the target languages, downgrading performance on non-target languages. the data and compute cost of training lms from scratch poses a major limitation, especially for lowresourced languages. tran 2020 propose a data and compute efficient technique to circumvent the need of training language-specific models from scratch. they learn target language word-embeddings from an english lm while keeping the pre-trained encoder layer fixed. the english and target language lms are then both fine-tuned to obtain a bilingual lm. this technique performs better than mbert and xlm-r on xnli in 5 of 6 languages with different amounts of resources. chi et al. 2020 combine the cross-lingual transfer of a multilingual lm with a task-specific monolingual lm to improve zero-shot cross-lingual classification. the source-language monolingual 'teacher' model provides supervision for the downstream task, and the multilingual model acts as a 'student'. the method outperforms direct multilingual finetuning for zero-shot cross-lingual sentiment analysis and xnli in most of the languages. @xcite propose an adaptor based modular framework that mitigates the curse of multilinguality and adapts a multilingual model to arbitrary tasks and languages using language and task-specific adaptors. their method gives stateof-the-art results for cross-lingual transfer among typologically diverse languages across tasks including ner, causal commonsense reasoning, and qa. ## multilingualism in multilingual communities, several linguistic phenomena lead to variation in language usage. while some of these are well-known and studied, others do not get enough attention. universal lms should be able to deal with these phenomena as we can expect them to encounter such forms of language when deployed in user-facing applications. code-mixing, or using two more languages in a single utterance, is common in multilingual communities. lms may not perform optimally in the presence of such code-mixing. multilingual models like mbert are not pre-trained with mixed language data, which leaves the model under-prepared for code-switched settings resulting in sub-optimal performance @xcite . this can be overcome to a certain extent by training using other data, such as social media, but it is unlikely to cover all the forms of code-switching produced by multilinguals. romanization of languages to the latin script has increased with the advent of digitization of communication. while some models like xlm-r @xcite and muril (muril, 2021) use romanized versions of some languages in training, the effectiveness of these lms on romanized (or more generally transliterated) text is still unclear. diglossia is a kind of multilingualism where a single community uses a substantially different language dialect in different communication settings @xcite . since data from the internet is used in training, it is likely that lms cannot handle diglossia. however, there are no studies that concretely prove (or disprove) this. to summarize, apart from code-mixing, there has been very little work in recognizing, understanding, or improving lms for different phenomena arising due to multilingualism, making the dimension under-represented in the study of lms. extensions: efforts have been made to improve lms, particularly mbert to deal with code mixed data. @xcite present a modified version of mbert which performs better than standard mbert in english-hindi and english-spanish code mixed data using synthetically generated code-mixed data for continued pre-training. like languages, the extent of evaluation on different nlp tasks is constrained by the availability of benchmarks that span various tasks. as shown in universal language model fine-tuning for text classification (ulmfit) uses discriminative finetuning, gradual unfreezing of layers, and slanted triangular learning rates for target-specific finetuning, giving better performance on multiple tasks @xcite . this work also explicitly defines the term 'universal', in their context as referring to -applicable to all tasks in text classification, using a single training process and architecture, usable without feature-engineering, and not requiring additional in-domain data. masked language modeling (mlm) is the most generalizable pre-training objective for the extent of transfer among twelve pre-training objectives for nine target tasks @xcite . data size is important in effective pre-training of lms @xcite but transfer gains between source and target tasks are also possible with smaller source datasets @xcite . similarity between source and target tasks is important for transfer gains. @xcite find that closeness in pre-training objective and target task is important for transfer. peters et al. 2019 find that while feature extraction and fine-tuning of lms give similar performance, exceptions occur when the source and target tasks are either very similar or very dissimilar. @xcite also find that similarity is important, especially in lowresource scenarios, but, exceptions of transfer gains between dissimilar tasks are possible. to summarize, lms are universal in the task dimension owing to task-specific architectures or fine-tuning. however, the success of lms is often associated with few benchmarks which cover limited tasks. the similarity between tasks, data size, and pre-training objectives are keys to transfer gains, which are important for universality. extensions: pattern exploiting training (pet) @xcite reformulates tasks as cloze questions, 4 making them the same as the mlm objective, requiring less data with no additional fine-tuning or a task-specific architecture to achieve remarkable zero-shot and few-shot perfor-mance. using pet with albert few-shot performance competitive to gpt-3, which is 780 times larger in terms of the number of parameters compared to albert, is obtained @xcite . the recently introduced t5 model @xcite leverages a text-to-text framework to enabling a single architecture to perform multiple tasks and achieving state-of-art results. these are concrete steps to enable universality towards tasks, as a single model can be built to generalize across solving multiple tasks. ## domain nlp models are applied to many real-world applications in different domains like medical, scientific, legal, financial etc. a universal model in this dimension should adapt to different domains or scenarios without loss of performance. processing domain-specific language often requires the processing of specialized vocabulary and language usage. even though lms learn some implicit clusters of domains @xcite , this may not be enough and specialized domain-specific lms are needed @xcite @xcite @xcite . despite the success of general domain lms, it is found that pre-training lm on in-domain data improves performance across high and low resource settings @xcite . lee et al. 2020 introduced biobert, learned using continued pre-training of bert on medical text, which performed better than bert on biomedical tasks. in contrast, @xcite introduce pubmedbert and challenge the benefits of out-of-domain data in pre-training by showing that pre-training lm from scratch on in-domain data (if available) is better than mixed or continual pre-training. @xcite propose the clinicalbert model that is pre-trained on clinical notes text corpora, which learns better relationships among medical concepts and outperforms general domain lms in clinical tasks. chalkidis et al. 2020 introduce legal-bert and observe that continual pre-training of bert or training it from scratch with legal data both perform similarly and significantly better than using bert off the shelf. @xcite release scibert, trained from scratch on scientific publications giving better performance on scientific tasks. araci 2019 use domain adaptation and transfer learning to develop finbert, achieving state-of-the-art performance in the financial domain. performance of domain-specific lm can degrade on general domain tasks @xcite @xcite . this phenomenon is also known as catastrophic forgetting and prevents the lm from being truly universal. to summarize, lms are not universal in the domain dimension, and different domain-specific lms have been introduced to cater to this requirement. domain-specific lms are either trained from scratch or by mixed or continual pre-training of existing lms. while none of these techniques are clear winners, performance degradation in general domain tasks is observed in many cases. extensions: vu et al. 2020a study adversarial masking strategies to learn specific target domain vocabulary along with continual pre-training by carefully selecting tokens to be masked, leading to better domain adaptation performance across multiple source and target domains. mutspad (multi-task supervised pretraining and adaptation) @xcite leverages hierarchical learning of a multi-task model on highresource domain followed by fine-tuning on multiple tasks on the low-resource target domain. ben-david et al. 2020 extend the pivot-based transfer learning to transformer-based lms by developing perl (pivot-based encoder representation of language), that uses continual pre-training with mlm to learn representations that reduce the gap between source and target domains followed by fine-tuning for the downstream classification task. the pivot is selected such that the source and target domain labels have greater mutual information to facilitate a good transfer. jiang and zhai 2007 propose several heuristics like removing misleading examples from the source domain, assigning more weight to target domain instances, and augmenting target training instances with predicted labels for better domain adaptation from a distributional perspective. various methods that are computationally efficient @xcite , use more effective adversarial training @xcite , and reduce the requirement of annotated data in low-resource settings @xcite have been proposed for computation and data efficient domain adaptation. rongali et al. 2020 overcome catastrophic forgetting, out-performing domain-specific lms while maintaining performance on general domain tasks. ## medium of expression language varies with the medium of expression. there are syntactic and semantic alterations like the use of ill-formed sentence or grammatical structure, inflections, slang, compressions, and abbreviations due to limited space, familiarity with the audience, and communicative intent. universal lms should be robust to such variations. often these specialized settings are simply treated as a different domain @xcite . however, in this work, we treat 'domain' as specialized fields of expertise. the current discussion pertains to the medium in which language is used. language in social media and texting may not follow conventions of written language. sentences are often shorter or grammatically ill-formed and may not be coherent enough for lms that rely on contextual information @xcite @xcite . lms perform sub-optimally on non-standard language as compared to specialized lms. bertweet @xcite , a roberta-based lm trained on english tweets, outperforms both roberta and xlm-r base models even though roberta and xlm-r use 2 times and 3.75 times more data, respectively. tweetbert (qudar and mago, 2020), another lm trained on tweets outperforms seven general domain lms. to summarize, the language used in different media of expression is substantially different. the limited amount of investigation in this direction indicates that lms are not universal to these variations and perform sub-optimally on the language used in social media and texting. extensions: @xcite propose costeffective training by appropriate selection of additional data for training a lm from a twitter corpus. ## geography and demography standard and non-standard dialects, both social and regional, lead to varied word, and language use @xcite @xcite @xcite . regional dialect refers to the varied usage of the same language across different places. for example, the use of 'wicked' can refer to bad or evil ("he is a wicked man"), or as an intensifier to adverbs ("my son is wicked smart") @xcite . sociolects, or social dialects, similar to regional dialects, are language dialects dependant on social variables like age, race, gender, socio-economic status, ethnicity, etc. @xcite . geographic and demographic variations stem from grammatical, phonological, syntactic, lexical, semantic features, or any combinations, making it difficult to capture and evaluate. since lms are trained on standard language dialects, non-standard dialects spoken by millions of people are largely ignored. such a system can result in bias against specific cultural or geographical communities in user-facing applications leading to ethical implications in building fair nlp systems. a universal lm should be sensitive to semantic shifts arising from its users' demographic or geographical diversity. taking these variations into consideration has resulted in improved performance and personalization of applications like conversational agents, sentiment analysis, word prediction, cyber-bullying detection, and machine translation @xcite @xcite @xcite @xcite . @xcite present a novel approach to quantify semantic shift that is statistically significant across geographical regions and propose a new measure of dialect similarity to establish how close the language in two regions is. demszky et al. 2020 focus on indian english and show that dialect features can be learned given very limited data with strong performance. in the 2020 vardial evaluation task for romanian dialect identification, an svm ensemble based on word and character n-grams outperformed fine-tuned romanian bert model. these results are consistent with the earlier evaluation results of vardial where shallow models outperformed deep models. in social media variety geolocation, predicting geolocation (coordinates) from text, the best performance was obtained by a bert architecture with a double regression classification output. in contrast, the next two best models were both shallow. @xcite . demographic features like age, gender have been predicted from language usage @xcite , whereas social class or ethnicity receive less attention (mohammady ardehaly and culotta, 2015). prediction of demographic features from language use quantifies the correlation between social variables and social dialects. while individuals may intentionally or naturally digress from such conventions, these statistical patterns are a cornerstone for studying the interaction between society and language in computational sociolinguistics @xcite . there are some worrisome findings of bias of performance across age, ethnicity, or gender in contextual lms. @xcite find that a pos tagger trained on the english penn treebank performed better on texts written by older authors. @xcite show bias against nonstandard english (in this case singaporean english) in bert. @xcite expose gender bias in bert by showing that the model assigns lower (or higher) scores consistently for sentences that contain words indicating gender in cases where gender information should have no bearing. to summarize, the influence of geography and demography on language usage is well known, and lms must be sensitive and inclusive of such variation. however, there has been limited, albeit now growing, attention to these factors. in some cases, shallow models have outperformed deep models in recognizing semantic shifts, and there is evidence of bias against particular social groups. extensions: bamman et al. 2014b learn language representations that take geographical situations or variations into account by enriching vector space word representations (word2vec) with geographical knowledge from metadata about authors. hovy and purschke 2018 employ retrofitting for including geographic information to capture regional variation in continuous regional distribution and at a fine-grained level using online posts in german and the corresponding cities of their authors as labels to create document embeddings. while these techniques do not involve any contextual lms, such representations and retrofitting can be extended to contextual lms. @xcite propose an adversarial approach to make models like bert more robust to nonstandard forms of english using inflectional morphology perturbations. debiasing techniques such as the ones studied in @xcite can remove gender stereotypes from pretrained word embeddings. the intensity of the change of meaning of different words is different -some are more subtly changed than the others. @xcite trace a period from 1900-2009, obtain year-specific word embeddings on the google books n-grams corpus, and pinpoint the extent and time-period of occurrence of semantic shift. hamilton et al. 2016 evaluate static word embeddings for known historical changes using corpora spanning four languages and two historical periods. they create diachronic embeddings by learning separate representations across the time periods followed by alignment over different time scales. dsc can be detected by clustering word sensed. @xcite organize words in a time-period specific graph where its nearest neighbors are cooccurring words, and word-senses are clustered. the shift in word sense or the emergence of a new word sense can be identified by the change of cluster for a particular word. giulianelli et al. 2020 perform clustering over usage types in bert and use the contextual property of lm to quantify semantic change instead of relying on a specific set of word senses. dsc evaluation lacks standardization. schlechtweg et al. 2019 perform a large scale evaluation on german, revealing the best set of parameters for optimal performance, compare various state-of-the-art methods, and outline improvements for better performance. shallow models can outperform contextual lms in identifying semantic shift. @xcite show that a shallow skip-gram model with negative sampling, orthogonal alignment, and co-sine distance performs best in identifying dsc in german. @xcite reconfirm this by using a similar model to obtain the first position in the diacr-ita shared task @xcite on italian dsc. similar findings of only limited success in contextual lms are reported in the shared task on unsupervised lexical semantic change detection in 5 languages hosted at semeval 2020 @xcite . while the success in identifying these shifts may be limited, @xcite find that dsc identified by contextual lms can have a strong correlation with human judgment of change. within contextual lms, bert and elmo perform similarly for russian. @xcite show that neither bert nor elmo outperform each other when fine-tuned using historical text in russian to detect semantic change. moreover, results from the shared task on unsupervised lexical semantic change detection in 5 languages hosted at semeval 2020 @xcite show that systems performing well over one language may not perform as well for other languages. to summarize, time period is under-studied and there is little understanding of whether contextual lms can handle such nuanced language variation. for the closely related task of dsc, shallow models can outperform deep lms, and performance can vary greatly across languages. extensions: rudolph and blei 2017 develop dynamic word-embeddings with an attribute of time that captures the semantic shifts in word meanings in sequential historical data on top of bernoulli embeddings such that representations are shared within specific time periods rather than throughout the corpus. similarly, bamler and mandt 2017 use timestamped data to build static probabilistic representation for tracing semantic change. to mitigate the problem of using different representations of words over different time periods, @xcite propose a framework for tracking and representing word senses by leveraging pre-trained bert embeddings and oxford dictionary data to learn fine-grained senses. ## conclusion deep contextual lms are being applied today to various different applications due to their perceived 'universality'. in this work, we attempt to holistically define 'universality' to encompass a wide variety of scenarios and linguistic phenomena. we define universality using seven dimensions: language, multilingualism, task, domain, medium of expression, geography and demography, and time period. these dimensions result in unique variations in language usage that are commonly encountered in real-life scenarios. we aim for this definition to be sound rather than complete. that is, a model should strive to achieve universality in these dimensions, but they are in no way a complete, exhaustive list of everything the model needs to be capable of. we survey research across all the dimensions and find that: first, while dimensions like language, task, and domain are more widely studied, other dimensions, especially multilingualism, geography and demography, and time period receive less attention. second, limited evaluation benchmarks constrain the complete understanding of capabilities even in the more studied dimensions. third, language variation arising in specific scenarios of demography, geography, time period, multilingualism, and medium of expression is often studied in an isolated manner. the dimensions we survey are a starting point that lms can aim to be inclusive towards in order to serve a diverse set of users and scenarios. large contextual lms may not be the optimal choice for all scenarios, with shallow, task-specific models sometimes leading to better outcomes. overall, 'universality' is yet to be fully understood, studied, and achieved. we hope that this work will lay the foundation to understanding the capabilities and limitations of lms and spur further research into making models more inclusive and fair.