# Model Details

## Model Description

This model card contains the COCO-LM model (base++ version) pretrained models on GLUE and SQuAD 2.0 benchmarks.

• Developed by: Microsoft
• Shared by [Optional]: HuggingFace
• Model type: Language model
• Language(s) (NLP): en
• GitHub Repo
• Associated Paper

# Uses

## Direct Use

Correcting and Contrasting Text Sequences for Language Model Pretraining

## Out-of-Scope Use

The model should not be used to intentionally create hostile or alienating environments for people.

# Bias, Risks, and Limitations

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.

## Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recomendations.

# Training Details

## Training Data

See the associated dataset card for further details

## Training Procedure

### Preprocessing

The model devloeprs note in the associated paper:

We employ three standard settings, base, base++, and large++. Base is the BERTBase training configuration: Pretraining on Wikipedia and BookCorpus (16 GB of texts) for 256 million samples on 512 token sequences (125K batches with 2048 batch size). We use the same corpus and 32, 768 uncased BPE vocabulary as with TUPE. Base++ trains the base size model with larger corpora and/or more training steps. Following recent research, we add in OpenWebText, CC-News , and STORIES, to a total of 160 GB texts, and train for 4 billion (with 2048 batch size) samples. We follow the prepossessing of UniLMV2 and use 64, 000 cased BPE vocabulary. Large++ uses the same training corpora as base++ and pretrains for 4 billion samples (2048 batch size). Its Transformer configuration is the same with BERTLarge.

# Evaluation

## Testing Data, Factors & Metrics

### Factors

All results are single-task, single-model fine-tuning.

## Results

### GLUE Fine-Tuning Results

The General Language Understanding Evaluation (GLUE) benchmark is a collection of sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.

GLUE dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds):

Model MNLI-m/mm QQP QNLI SST-2 CoLA RTE MRPC STS-B AVG
COCO-LM base++ 90.2/90.0 92.2 94.2 94.6 67.3 87.4 91.2 91.8 88.6
COCO-LM large++ 91.4/91.6 92.8 95.7 96.9 73.9 91.0 92.2 92.7 90.8

GLUE test set results of COCO-LM base++ and large++ models are as follows (no ensemble, task-specific tricks, etc.):

Model MNLI-m/mm QQP QNLI SST-2 CoLA RTE MRPC STS-B AVG
COCO-LM base++ 89.8/89.3 89.8 94.2 95.6 68.6 82.3 88.5 90.3 87.4
COCO-LM large++ 91.6/91.1 90.5 95.8 96.7 70.5 89.2 88.4 91.8 89.3

Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.

SQuAD 2.0 dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds):

Model EM F1
COCO-LM base++ 85.4 88.1
COCO-LM large++ 88.2 91.0

# Model Examination

The model delevopers note in the associated paper:

Architecture. Removing relative position encoding (Rel-Pos) leads to better numbers on some tasks but significantly hurts MNLI. Using a shallow auxiliary network and keeping the same hidden dimension (768) is more effective than ELECTRA’s 12-layer but 256-hidden dimension generator.

One limitation of this work is that the contrastive pairs are constructed by simple cropping and MLM replacements. Recent studies have shown the effectiveness of advanced data augmentation techniques in fine-tuning language models [16, 38, 51]. A future research direction is to explore better ways to construct contrastive pairs in language model pretraining.

# Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

# Technical Specifications [optional]

## Model Architecture and Objective

The model delevopers note in the associated paper:

Model Architecture. Our base/base++ model uses the BERTBase architecture: 12 layer Transformer, 768 hidden size, plus T5 relative position encoding. Our large++ model is the same with BERTLarge, 24 layer and 1024 hidden size, plus T5 relative position encoding. Our auxiliary network uses the same hidden size but a shallow 4-layer Transformer in base/base++ and a 6-layer one in large++. When generating XMLM we disable dropout in the auxiliary model

# Citation

BibTeX:

If you find the code and models useful for your research, please cite the following paper:

@inproceedings{meng2021cocolm,
title={{COCO-LM}: Correcting and contrasting text sequences for language model pretraining},
author={Meng, Yu and Xiong, Chenyan and Bajaj, Payal and Tiwary, Saurabh and Bennett, Paul and Han, Jiawei and Song, Xia},
booktitle={Conference on Neural Information Processing Systems},
year={2021}
}


# Model Card Authors [optional]

Microsoft in collaboration with Ezi Ozoani and the HuggingFace team

# How to Get Started with the Model

Use the code below to get started with the model.

Click to expand

from transformers import AutoModel

model = AutoModel.from_pretrained("microsoft/cocolm-base")