| # Model Card for COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining | |
| # Model Details | |
| ## Model Description | |
| This model card contains the COCO-LM model (**base++** version) pretrained models on GLUE and SQuAD 2.0 benchmarks. | |
| - **Developed by:** Microsoft | |
| - **Shared by [Optional]:** HuggingFace | |
| - **Model type:** Language model | |
| - **Language(s) (NLP):** en | |
| - **License:** MIT | |
| - **Related Models:** More information needed | |
| - **Parent Model:** More information needed | |
| - **Resources for more information:** | |
| - [GitHub Repo](https://github.com/microsoft/COCO-LM) | |
| - [Associated Paper](https://arxiv.org/abs/2102.08473) | |
| # Uses | |
| ## Direct Use | |
| Correcting and Contrasting Text Sequences for Language Model Pretraining | |
| ## Downstream Use [Optional] | |
| More information needed. | |
| ## Out-of-Scope Use | |
| The model should not be used to intentionally create hostile or alienating environments for people. | |
| # Bias, Risks, and Limitations | |
| Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. | |
| ## Recommendations | |
| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recomendations. | |
| # Training Details | |
| ## Training Data | |
| See the associated [dataset card]({0}) for further details | |
| ## Training Procedure | |
| ### Preprocessing | |
| The model devloeprs note in the [associated paper](https://arxiv.org/abs/2102.08473): | |
| >We employ three standard settings, base, base++, and large++. Base is the BERTBase training configuration: Pretraining on Wikipedia and BookCorpus (16 GB of texts) for 256 million samples on 512 token sequences (125K batches with 2048 batch size). We use the same corpus and 32, 768 uncased BPE vocabulary as with TUPE. Base++ trains the base size model with larger corpora and/or more training steps. Following recent research, we add in OpenWebText, CC-News , and STORIES, to a total of 160 GB texts, and train for 4 billion (with 2048 batch size) samples. We follow the prepossessing of UniLMV2 and use 64, 000 cased BPE vocabulary. Large++ uses the same training corpora as base++ and pretrains for 4 billion samples (2048 batch size). Its Transformer configuration is the same with BERTLarge. | |
| ### Speeds, Sizes, Times | |
| More information needed | |
| # Evaluation | |
| ## Testing Data, Factors & Metrics | |
| ### Testing Data | |
| GLUE | |
| SQuAD 2.0 | |
| ### Factors | |
| All results are single-task, single-model fine-tuning. | |
| ### Metrics | |
| More information needed | |
| ## Results | |
| ### GLUE Fine-Tuning Results | |
| The [General Language Understanding Evaluation (GLUE)](https://gluebenchmark.com/) benchmark is a collection of sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems. | |
| GLUE dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds): | |
| | Model | MNLI-m/mm | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | AVG | | |
| | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | | |
| | COCO-LM base++ | 90.2/90.0 | 92.2 | 94.2 | 94.6 | 67.3 | 87.4 | 91.2 | 91.8 | 88.6 | | |
| | COCO-LM large++ | 91.4/91.6 | 92.8 | 95.7 | 96.9 | 73.9 | 91.0 | 92.2 | 92.7 | 90.8 | | |
| GLUE test set results of COCO-LM base++ and large++ models are as follows (no ensemble, task-specific tricks, etc.): | |
| | Model | MNLI-m/mm | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | AVG | | |
| | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | | |
| | COCO-LM base++ | 89.8/89.3 | 89.8 | 94.2 | 95.6 | 68.6 | 82.3 | 88.5 | 90.3 | 87.4 | | |
| | COCO-LM large++ | 91.6/91.1 | 90.5 | 95.8 | 96.7 | 70.5 | 89.2 | 88.4 | 91.8 | 89.3 | | |
| ### SQuAD 2.0 Fine-Tuning Results | |
| [Stanford Question Answering Dataset (SQuAD)](https://rajpurkar.github.io/SQuAD-explorer/) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. | |
| SQuAD 2.0 dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds): | |
| | Model | EM | F1 | | |
| | ------ | ------ | ------ | | |
| | COCO-LM base++ | 85.4 | 88.1 | | |
| | COCO-LM large++ | 88.2 | 91.0 | | |
| # Model Examination | |
| The model delevopers note in the [associated paper](https://arxiv.org/abs/2102.08473): | |
| >Architecture. Removing relative position encoding (Rel-Pos) leads to better numbers on some tasks but significantly hurts MNLI. Using a shallow auxiliary network and keeping the same hidden dimension (768) is more effective than ELECTRA’s 12-layer but 256-hidden dimension generator. | |
| >One limitation of this work is that the contrastive pairs are constructed by simple cropping and MLM replacements. Recent studies have shown the effectiveness of advanced data augmentation techniques in fine-tuning language models [16, 38, 51]. A future research direction is to explore better ways to construct contrastive pairs in language model pretraining. | |
| # Environmental Impact | |
| <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> | |
| Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). | |
| - **Hardware Type:** More information needed | |
| - **Hours used:** More information needed | |
| - **Cloud Provider:** More information needed | |
| - **Compute Region:** More information needed | |
| - **Carbon Emitted:** More information needed} | |
| # Technical Specifications [optional] | |
| ## Model Architecture and Objective | |
| The model delevopers note in the [associated paper](https://arxiv.org/abs/2102.08473): | |
| >Model Architecture. Our base/base++ model uses the BERTBase architecture: 12 layer Transformer, 768 hidden size, plus T5 relative position encoding. Our large++ model is the same with BERTLarge, 24 layer and 1024 hidden size, plus T5 relative position encoding. Our auxiliary network uses the same hidden size but a shallow 4-layer Transformer in base/base++ and a 6-layer one in large++. When generating XMLM we disable dropout in the auxiliary model | |
| ## Compute Infrastructure | |
| More information needed | |
| ### Hardware | |
| More information needed | |
| ### Software | |
| More information needed | |
| # Citation | |
| **BibTeX:** | |
| If you find the code and models useful for your research, please cite the following paper: | |
| ``` | |
| @inproceedings{meng2021cocolm, | |
| title={{COCO-LM}: Correcting and contrasting text sequences for language model pretraining}, | |
| author={Meng, Yu and Xiong, Chenyan and Bajaj, Payal and Tiwary, Saurabh and Bennett, Paul and Han, Jiawei and Song, Xia}, | |
| booktitle={Conference on Neural Information Processing Systems}, | |
| year={2021} | |
| } | |
| ``` | |
| # Glossary [optional] | |
| More information needed | |
| # More Information [optional] | |
| More information needed | |
| # Model Card Authors [optional] | |
| Microsoft in collaboration with Ezi Ozoani and the HuggingFace team | |
| # Model Card Contact | |
| More information needed | |
| # How to Get Started with the Model | |
| Use the code below to get started with the model. | |
| <details> | |
| <summary> Click to expand </summary> | |
| ```python | |
| from transformers import AutoModel | |
| model = AutoModel.from_pretrained("microsoft/cocolm-base") | |
| ``` | |
| </details> | |