model documentation

README.md

@@ -0,0 +1,192 @@
+# Model Card for gpt2-base-gedi-detoxification
+ 
+# Model Details
+ 
+## Model Description
+ 
+- **Developed by:** SkolkovoInstitute
+- **Shared by [Optional]:** SkolkovoInstitute
+- **Model type:** Text Generation
+- **Language(s) (NLP):** More information needed
+- **License:** More information needed
+- **Related Models:**
+  - **Parent Model:** GPT-2
+- **Resources for more information:** 
+	- [Associated GeDI Paper](https://arxiv.org/pdf/2009.06367.pdf)
+	- [Blog Post](​​https://blog.salesforceairesearch.com/gedi/)
+ 
+# Uses
+ 
+ 
+## Direct Use
+ 
+This model can be used for the task of Text Generation or fine-tune it to a downstream task.
+ 
+## Downstream Use [Optional]
+ 
+More information needed
+ 
+## Out-of-Scope Use
+ 
+The model should not be used to intentionally create hostile or alienating environments for people.
+OpenAI note in their [model card](https://github.com/openai/gpt-2/blob/master/model_card.md)
+> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true.
+ 
+ 
+# Bias, Risks, and Limitations
+The training data used for this model has not been released as a dataset one can browse. We know it contains a lot of
+unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their
+[model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases):
+ 
+> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases
+> that require the generated text to be true.
+>
+> Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do
+> not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a
+> study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race,
+> and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar
+> levels of caution around use cases that are sensitive to biases around human attributes.
+ 
+See the [GPT-2 model card](https://huggingface.co/gpt2?text=My+name+is+Merve+and+my+favorite) for examples of how the model can have biased predictions 
+ 
+ *The [GeDi Blog post](https://blog.salesforceairesearch.com/gedi/) notes*
+ 
+>We use smaller language models as generative classifiers to guide generation from larger language models. We show that this method can make generations friendlier, reduce bias and toxicity, and achieve zero-shot controllable generation of unseen topics.
+ 
+## Recommendations
+ 
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+ 
+ 
+# Training Details
+ 
+## Training Data
+ 
+The OpenAI team wanted to train this model on a corpus as large as possible. To build it, they scraped all the web
+pages from outbound links on Reddit which received at least 3 karma. Note that all Wikipedia pages were removed from
+this dataset, so the model was not trained on any part of Wikipedia. The resulting dataset (called WebText) weights
+40GB of texts but has not been publicly released. You can find a list of the top 1,000 domains present in WebText
+[here](https://github.com/openai/gpt-2/blob/master/domains.txt).
+ 
+ 
+## Training Procedure
+ 
+ 
+### Preprocessing
+ 
+More information needed
+ 
+### Speeds, Sizes, Times
+ 
+More information needed
+ 
+# Evaluation
+ 
+ 
+## Testing Data, Factors & Metrics
+ 
+### Testing Data
+ 
+More information needed
+ 
+### Factors
+ 
+ 
+### Metrics
+ 
+More information needed
+## Results 
+ 
+The [GeDi Blog post](https://blog.salesforceairesearch.com/gedi/) notes the following results 
+ 
+| Model            | Toxicity | Linguistic Quality  |
+|------------------|----------|---------------------|
+| GPT-2            | 1.45     | 3.23                |
+| GeDi-guided GPT2 | 1.17     | 3.44                |
+ 
+ 
+# Model Examination
+ 
+More information needed
+ 
+# Environmental Impact
+ 
+ 
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+ 
+- **Hardware Type:** More information needed
+- **Hours used:** More information needed
+- **Cloud Provider:** More information needed
+- **Compute Region:** More information needed
+- **Carbon Emitted:** More information needed
+ 
+# Technical Specifications [optional]
+ 
+## Model Architecture and Objective
+ 
+More information needed
+ 
+## Compute Infrastructure
+ 
+More information needed
+ 
+### Hardware
+ 
+More information needed
+ 
+### Software
+More information needed
+ 
+# Citation
+ 
+ 
+**BibTeX:**
+ ```
+@article{radford2019language,
+  title={Language Models are Unsupervised Multitask Learners},
+  author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
+  year={2019}
+}
+```
+ 
+ ```
+ @article{KrauseGeDi2020,
+title={{GeDi: Generative Discriminator Guided Sequence Generation}},
+author={Krause, Ben and Gotmare, Akhilesh Deepak and McCann, Bryan and Keskar, Nitish Shirish and Joty, Shafiq and Socher, Richard and Rajani, Nazneen Fatema},
+journal={arXiv preprint arXiv:2009.06367},
+year={2020}
+```
+ 
+ 
+# Glossary [optional]
+More information needed
+ 
+# More Information [optional]
+ 
+More information needed
+ 
+# Model Card Authors [optional]
+ 
+ 
+SkolkovoInstitute in collaboration with Ezi Ozoani and the Hugging Face team
+ 
+# Model Card Contact
+ 
+More information needed
+ 
+# How to Get Started with the Model
+ 
+Use the code below to get started with the model.
+ 
+<details>
+<summary> Click to expand </summary>
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+ 
+tokenizer = AutoTokenizer.from_pretrained("SkolkovoInstitute/gpt2-base-gedi-detoxification")
+ 
+model = AutoModelForCausalLM.from_pretrained("SkolkovoInstitute/gpt2-base-gedi-detoxification")
+ 
+```
+</details>
+