gpt2-base-gedi-detoxification / README.md

model documentation

fa2532d over 1 year ago

No virus

6.06 kB

	---
	tags:
	- text-generation

	---
	# Model Card for gpt2-base-gedi-detoxification

	# Model Details

	## Model Description

	- Developed by: SkolkovoInstitute
	- Shared by [Optional]: SkolkovoInstitute
	- Model type: Text Generation
	- Language(s) (NLP): More information needed
	- License: More information needed
	- Related Models:
	- Parent Model: GPT-2
	- Resources for more information:
	- [Associated GeDI Paper](https://arxiv.org/pdf/2009.06367.pdf)
	- [Blog Post](https://blog.salesforceairesearch.com/gedi/)

	# Uses


	## Direct Use

	This model can be used for the task of Text Generation or fine-tune it to a downstream task.

	## Downstream Use [Optional]

	More information needed

	## Out-of-Scope Use

	The model should not be used to intentionally create hostile or alienating environments for people.
	OpenAI note in their [model card](https://github.com/openai/gpt-2/blob/master/model_card.md)
	> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true.


	# Bias, Risks, and Limitations
	The training data used for this model has not been released as a dataset one can browse. We know it contains a lot of
	unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their
	[model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases):

	> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases
	> that require the generated text to be true.
	>
	> Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do
	> not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a
	> study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race,
	> and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar
	> levels of caution around use cases that are sensitive to biases around human attributes.

	See the [GPT-2 model card](https://huggingface.co/gpt2?text=My+name+is+Merve+and+my+favorite) for examples of how the model can have biased predictions

	The [GeDi Blog post](https://blog.salesforceairesearch.com/gedi/) notes

	>We use smaller language models as generative classifiers to guide generation from larger language models. We show that this method can make generations friendlier, reduce bias and toxicity, and achieve zero-shot controllable generation of unseen topics.

	## Recommendations

	Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.


	# Training Details

	## Training Data

	The OpenAI team wanted to train this model on a corpus as large as possible. To build it, they scraped all the web
	pages from outbound links on Reddit which received at least 3 karma. Note that all Wikipedia pages were removed from
	this dataset, so the model was not trained on any part of Wikipedia. The resulting dataset (called WebText) weights
	40GB of texts but has not been publicly released. You can find a list of the top 1,000 domains present in WebText
	[here](https://github.com/openai/gpt-2/blob/master/domains.txt).


	## Training Procedure


	### Preprocessing

	More information needed

	### Speeds, Sizes, Times

	More information needed

	# Evaluation


	## Testing Data, Factors & Metrics

	### Testing Data

	More information needed

	### Factors


	### Metrics

	More information needed
	## Results

	The [GeDi Blog post](https://blog.salesforceairesearch.com/gedi/) notes the following results

	\| Model \| Toxicity \| Linguistic Quality \|
	\|------------------\|----------\|---------------------\|
	\| GPT-2 \| 1.45 \| 3.23 \|
	\| GeDi-guided GPT2 \| 1.17 \| 3.44 \|


	# Model Examination

	More information needed

	# Environmental Impact


	Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

	- Hardware Type: More information needed
	- Hours used: More information needed
	- Cloud Provider: More information needed
	- Compute Region: More information needed
	- Carbon Emitted: More information needed

	# Technical Specifications [optional]

	## Model Architecture and Objective

	More information needed

	## Compute Infrastructure

	More information needed

	### Hardware

	More information needed

	### Software
	More information needed

	# Citation


	BibTeX:
	```
	@article{radford2019language,
	title={Language Models are Unsupervised Multitask Learners},
	author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
	year={2019}
	}
	```

	```
	@article{KrauseGeDi2020,
	title={{GeDi: Generative Discriminator Guided Sequence Generation}},
	author={Krause, Ben and Gotmare, Akhilesh Deepak and McCann, Bryan and Keskar, Nitish Shirish and Joty, Shafiq and Socher, Richard and Rajani, Nazneen Fatema},
	journal={arXiv preprint arXiv:2009.06367},
	year={2020}
	```


	# Glossary [optional]
	More information needed

	# More Information [optional]

	More information needed

	# Model Card Authors [optional]


	SkolkovoInstitute in collaboration with Ezi Ozoani and the Hugging Face team

	# Model Card Contact

	More information needed

	# How to Get Started with the Model

	Use the code below to get started with the model.

	<details>
	<summary> Click to expand </summary>

	```python
	from transformers import AutoTokenizer, AutoModelForCausalLM

	tokenizer = AutoTokenizer.from_pretrained("SkolkovoInstitute/gpt2-base-gedi-detoxification")

	model = AutoModelForCausalLM.from_pretrained("SkolkovoInstitute/gpt2-base-gedi-detoxification")

	```
	</details>