Update README.md

README.md

@@ -50,14 +50,16 @@ output = model(encoded_input)
 ```
 
 ## Limitations and bias 
-The training data used for this model has not been released as a dataset one can browse. We know it contains a lot of unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their [model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases):
+The training data used for this model are Indonesian websites of [OSCAR](https://oscar-corpus.com/) and [mc4](https://huggingface.co/datasets/mc4). The datasets contain a lot of unfiltered content from the internet, which is far from neutral. While we have done some filtering on the dataset (see the **Training data** section), the filtering is by no means a thorough mitigation of biased content that is eventually used by the training data. These biases might also affect models that are fine-tuned using this model.
+
+As the openAI team themselves point out in their [model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases):
 
 > Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true.
 
 > Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race, and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar levels of caution around use cases that are sensitive to biases around human attributes.
 
 ## Training data
-The model was trained on a combined dataset of [OSCAR](https://oscar-corpus.com/) and [mc4](https://huggingface.co/datasets/mc4) for the Indonesian language, with 29GB of data in total. The mc4 dataset was cleaned using [this script](https://github.com/Wikidepia/indonesian_datasets/blob/master/dump/mc4/cleanup.py) and we also only included links that were cited by IDWiki.
+The model was trained on a combined dataset of [OSCAR](https://oscar-corpus.com/) and [mc4](https://huggingface.co/datasets/mc4) for the Indonesian language. We have filtered the dataset so that we end up with 29 GB of data in total. The mc4 dataset was cleaned using [this filtering script](https://github.com/Wikidepia/indonesian_datasets/blob/master/dump/mc4/cleanup.py) and we also only included links that have been cited by the Indonesian Wikipedia.
 
 ## Training procedure 
 The model was trained on a TPUv3-8 VM provided by the Google Cloud team. The training duration was `6d 3h 7m 26s`.