README.md

---
tags:
- text-generation

---
# Model Card for gpt2-base-gedi-detoxification
 
# Model Details
 
## Model Description
 
- **Developed by:** SkolkovoInstitute
- **Shared by [Optional]:** SkolkovoInstitute
- **Model type:** Text Generation
- **Language(s) (NLP):** More information needed
- **License:** More information needed
- **Related Models:**
  - **Parent Model:** GPT-2
- **Resources for more information:** 
	- [Associated GeDI Paper](https://arxiv.org/pdf/2009.06367.pdf)
	- [Blog Post](​​https://blog.salesforceairesearch.com/gedi/)
 
# Uses
 
 
## Direct Use
 
This model can be used for the task of Text Generation or fine-tune it to a downstream task.
 
## Downstream Use [Optional]
 
More information needed
 
## Out-of-Scope Use
 
The model should not be used to intentionally create hostile or alienating environments for people.
OpenAI note in their [model card](https://github.com/openai/gpt-2/blob/master/model_card.md)
> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true.
 
 
# Bias, Risks, and Limitations
The training data used for this model has not been released as a dataset one can browse. We know it contains a lot of
unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their
[model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases):
 
> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases
> that require the generated text to be true.
>
> Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do
> not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a
> study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race,
> and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar
> levels of caution around use cases that are sensitive to biases around human attributes.
 
See the [GPT-2 model card](https://huggingface.co/gpt2?text=My+name+is+Merve+and+my+favorite) for examples of how the model can have biased predictions 
 
 *The [GeDi Blog post](https://blog.salesforceairesearch.com/gedi/) notes*
 
>We use smaller language models as generative classifiers to guide generation from larger language models. We show that this method can make generations friendlier, reduce bias and toxicity, and achieve zero-shot controllable generation of unseen topics.
 
## Recommendations
 
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
 
# Training Details
 
## Training Data
 
The OpenAI team wanted to train this model on a corpus as large as possible. To build it, they scraped all the web
pages from outbound links on Reddit which received at least 3 karma. Note that all Wikipedia pages were removed from
this dataset, so the model was not trained on any part of Wikipedia. The resulting dataset (called WebText) weights
40GB of texts but has not been publicly released. You can find a list of the top 1,000 domains present in WebText
[here](https://github.com/openai/gpt-2/blob/master/domains.txt).
 
 
## Training Procedure
 
 
### Preprocessing
 
More information needed
 
### Speeds, Sizes, Times
 
More information needed
 
# Evaluation
 
 
## Testing Data, Factors & Metrics
 
### Testing Data
 
More information needed
 
### Factors
 
 
### Metrics
 
More information needed
## Results 
 
The [GeDi Blog post](https://blog.salesforceairesearch.com/gedi/) notes the following results 
 
| Model            | Toxicity | Linguistic Quality  |
|------------------|----------|---------------------|
| GPT-2            | 1.45     | 3.23                |
| GeDi-guided GPT2 | 1.17     | 3.44                |
 
 
# Model Examination
 
More information needed
 
# Environmental Impact
 
 
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
- **Hardware Type:** More information needed
- **Hours used:** More information needed
- **Cloud Provider:** More information needed
- **Compute Region:** More information needed
- **Carbon Emitted:** More information needed
 
# Technical Specifications [optional]
 
## Model Architecture and Objective
 
More information needed
 
## Compute Infrastructure
 
More information needed
 
### Hardware
 
More information needed
 
### Software
More information needed
 
# Citation
 
 
**BibTeX:**
 ```
@article{radford2019language,
  title={Language Models are Unsupervised Multitask Learners},
  author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
  year={2019}
}
```
 
 ```
 @article{KrauseGeDi2020,
title={{GeDi: Generative Discriminator Guided Sequence Generation}},
author={Krause, Ben and Gotmare, Akhilesh Deepak and McCann, Bryan and Keskar, Nitish Shirish and Joty, Shafiq and Socher, Richard and Rajani, Nazneen Fatema},
journal={arXiv preprint arXiv:2009.06367},
year={2020}
```
 
 
# Glossary [optional]
More information needed
 
# More Information [optional]
 
More information needed
 
# Model Card Authors [optional]
 
 
SkolkovoInstitute in collaboration with Ezi Ozoani and the Hugging Face team
 
# Model Card Contact
 
More information needed
 
# How to Get Started with the Model
 
Use the code below to get started with the model.
 
<details>
<summary> Click to expand </summary>

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
 
tokenizer = AutoTokenizer.from_pretrained("SkolkovoInstitute/gpt2-base-gedi-detoxification")
 
model = AutoModelForCausalLM.from_pretrained("SkolkovoInstitute/gpt2-base-gedi-detoxification")
 
```
</details>