Life after BERT : What do Other Muppets Understand about Language? (2022)

Abstract: Existing pre-trained transformer analysis works usually focus only on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. In our work, we utilize the oLMpics bench- mark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. Additionally, we adapt the oLMpics zero-shot setup for autoregres- sive models and evaluate GPT networks of different sizes. Our findings show that none of these models can resolve compositional questions in a zero-shot fashion, suggesting that this skill is not learnable using existing pre-training objectives. Furthermore, we find that global model decisions such as architecture, directionality, size of the dataset, and pre-training objective are not predictive of a model’s linguistic capabilities.

## introduction after the initial success of transfer learning in natural language processing @xcite , the number of pre-trained models in nlp has increased dramatically @xcite @xcite @xcite @xcite . however, there is a limited understanding of why certain models perform better than others and what linguistic capabilities they acquire through pre-training. while a lot of work has been done to evaluate these models on general natural language understanding datasets @xcite @xcite , such datasets do not allow researchers to identify the specific linguistic capabilities of a model. furthermore the performance on these datasets results from a combination of pre-trained knowledge and task-specific information learned through fine-tuning. probing tasks @xcite @xcite give a promising solution to this problem, as they evaluate specific capabilities of pre-trained models, and in many cases, these tasks are designed for zeroshot evaluation, which reveals the knowledge that models have actually learned purely through the upstream task. currently, most in-depth analysis studies focus on one or two model families. many analysis papers only probe bert and similar models @xcite @xcite @xcite @xcite @xcite . fortunately, this trend is changing and now we see more papers that probe models such as albert, t5 or bart @xcite @xcite . however, only a small number of analysis papers have probed multiple (three or more) model families @xcite . in our work, we test 8 families of models on olmpics tasks @xcite and 6 families on psycholinguistic tasks from @xcite . these models differ in size, architecture, pretraining objective, dataset size, and have other small yet important differences. such a diverse set of models provides a broader view of what linguistic capabilities are affected by the change of any of these properties. we also include several distilled models in our analysis. we find that different models excel in different symbolic reasoning tasks, suggesting that slight differences related to optimization or masking strategy might be more important than the pre-training approach, dataset size, or architecture. furthermore, in contrast to @xcite , we find that for olmpics tasks, model size rarely correlates with the model performance. in addition, we observe that all models fail on composition tasks when evaluated in a zero-shot fashion. ## related work pre-trained model analysis is a rapidly growing area in nlp today. there exists a number of methods for analyzing internal representations of a model, including structured head and fcn pruning @xcite @xcite , residual connection and layernormalization analysis @xcite , and analyzing attention patterns @xcite . compared to these methods, probing tasks @xcite provide a more direct way to evaluate what a model can and cannot accomplish. while it is possible to probe embeddings or hidden representations directly @xcite , the adoption of pre-trained language models has made it possible to evaluate such models by framing probing tasks close to the original model objective @xcite @xcite . however, when a research area moves this quickly, it can be hard to keep up with many new models. most of the existing research @xcite @xcite ) papers compare only one or two model families. even some of the most recent works only probe bert or very similar models @xcite . only a small number of analysis papers have probed multiple (three or more) model families @xcite . in contrast to existing work, we perform a largescale probing of 29 models across 8 different model families. we apply the existing probing benchmarks, namely, olmpics @xcite and psycholinguistic datasets @xcite , to models that differ in the pre-training objective, datasets, size, architecture, and directionality. ## experiments we evaluate eight models families on the olmpics (29 models in total) and six families on psycholinguistic data (17 models). this extends the @xcite results with six new model families and ettinger (2020) with four. ## conclusion in this work, we apply a large and diverse set of models to olmpics and psycholinguistic tasks. the variety of models allows us to investigate the performance of different architectures and pre-training methods on a variety of linguistic tasks. contrary to received wisdom, we find that parameter count within a given model family does not correlate with model performance on these tasks. we find that none of the models, even the 2.8bsized ones, can resolve multi-hop composition and always-never tasks in a zero-shot manner, suggesting that the existing pre-training methods cannot learn such tasks. finally, we find that different models excel in different symbolic reasoning tasks, suggesting that slight differences related to optimization or masking strategy might be more important than the pre-training approach, dataset size, or architecture.