Modified readme data and examples

About/intro.md

@@ -1,6 +1,6 @@
 # RoBERTa base model for Hindi language
 
-[Pretrained model on Hindi language](https://huggingface.co/flax-community/roberta-hindi)  using a masked language modeling (MLM) objective. 
+[Pretrained model on Hindi language](https://huggingface.co/flax-community/roberta-hindi)  using a masked language modeling (MLM) objective. Model is able to achieve competitive accuracy compared to pre-existing models on downstream tasks like NamedEntityRecognition and Classification. There are some MLM examples which show that there is a visible room for improvement, but this should serve well as a good base model for hindi languages & could be fine-tuned on specific datasets.
 
 > This is part of the
 [Flax/Jax Community Week](https://discuss.huggingface.co/t/pretrain-roberta-from-scratch-in-hindi/7091), organized by [HuggingFace](https://huggingface.co/) and TPU usage sponsored by Google.

About/model_description.md

@@ -1,3 +1,3 @@
 ## Model description
 
-It is a monolingual transformers model pretrained on a large corpus of Hindi data in a self-supervised fashion. 
+It is a monolingual transformers model pretrained on a large corpus of Hindi data (100GB+) in a self-supervised fashion. 

About/results.md

@@ -2,7 +2,7 @@
 
 RoBERTa Hindi is evaluated on downstream tasks. The results are summarized below.
 
-| Task                    | Task Type            | IndicBERT | HindiBERTa | Indic Transformers Hindi BERT | RoBERTa Hindi Guj San | RoBERTa Hindi |
+| Task                    | Task Type            | IndicBERT | HindiBERTa | Indic Transformers Hindi BERT | RoBERTa Hindi Guj San | RoBERTa Hindi(ours) |
 |-------------------------|----------------------|-----------|------------|-------------------------------|-----------------------|---------------|
 | BBC News Classification | Genre Classification | **76.44**     | 66.86      | **77.6**                          | 64.9                  | 73.67         |
 | WikiNER                 | Token Classification | -         | 90.68      | **95.09**                         | 89.61                 | **92.76**         |

About/training_data.md

@@ -1,8 +1,8 @@
 ## Training data
 
-The RoBERTa model was pretrained on the reunion of the following datasets:
-- [OSCAR](https://huggingface.co/datasets/oscar) is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture.
+The RoBERTa model was pretrained on the union, followed by a random shuffle of the following datasets:
 - [mC4](https://huggingface.co/datasets/mc4) is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus.
+- [OSCAR](https://huggingface.co/datasets/oscar) is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture.
 - [IndicGLUE](https://indicnlp.ai4bharat.org/indic-glue/) is a natural language understanding benchmark.
 - [Samanantar](https://indicnlp.ai4bharat.org/samanantar/) is a parallel corpora collection for Indic language.
 - [Hindi Wikipedia Articles - 172k](https://www.kaggle.com/disisbig/hindi-wikipedia-articles-172k) is a dataset with cleaned 172k Wikipedia articles.

About/training_procedure.md

@@ -1,18 +1,11 @@
 ## Training procedure
 ### Preprocessing
-
-The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
-the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
-with `<s>` and the end of one by `</s>`
-The details of the masking procedure for each sentence are the following:
-- 15% of the tokens are masked.
-- In 80% of the cases, the masked tokens are replaced by `<mask>`.
-- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
-- In the 10% remaining cases, the masked tokens are left as is.
-Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).
+The texts are tokenized using Byte-Pair Encoding (BPE) and a vocabulary size of 50265.
+- `mc4` and `oscar` datasets were available in `datasets` library. For rest of the datasets, we wrote our own loading scripts [available here](https://github.com/amankhandelia/roberta_hindi/blob/master/test_custom_datasets.py)
+- It was slightly challenging to run `mc4` dataset(104GB+) and perform preprocessing and use it in non-streaming mode. `datasets` library had a lot of helper functions which allowed us to merge & shuffle the datasets with ease.
+- We had to perform cleanup of mC4 and oscar datasets by removing all non hindi(non Devanagiri) characters from the datasets, as the datasets are relatively somewhat noisy.
+- We attempted to filter out evaluation set of [WikiNER of IndicGlue](https://indicnlp.ai4bharat.org/indic-glue/) benchmark by manual lablelling where the actual labels were not correct and modified the downstream evaluation dataset. Code & manually labelled file are also present in our [github repo](https://github.com/amankhandelia/roberta_hindi)
 
 ### Pretraining
-The model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores) **8 v3 TPU cores** for 42K steps with a batch size of 128 and a sequence length of 128. The
-optimizer used is Adam with a learning rate of 6e-4, \\(\beta_{1} = 0.9\\), \\(\beta_{2} = 0.98\\) and
-\\(\epsilon = 1e-6\\), a weight decay of 0.01, learning rate warmup for 24,000 steps and linear decay of the learning
-rate after.
+The model was trained on Google Cloud Engine TPUv3-8 machine (with 335 GB of RAM, 1000 GB of hard drive, 96 CPU cores).A randomized shuffle of combined dataset of mC4, oscar and other datasets listed above was used to train the model. Downstream training logs are present in [wandb](https://wandb.ai/wandb/hf-flax-roberta-hindi).
+

About/use.md

@@ -5,26 +5,34 @@ You can use this model directly with a pipeline for masked language modeling:
 ```python
 >>> from transformers import pipeline
 >>> unmasker = pipeline('fill-mask', model='flax-community/roberta-hindi')
->>> unmasker("मुझे उनसे बात करना <mask> अच्छा लगा")
+>>> unmasker("हम आपके सुखद <mask> की कामना करते हैं")
+[{'score': 0.3310680091381073,
+  'sequence': 'हम आपके सुखद सफर की कामना करते हैं',
+  'token': 1349,
+  'token_str': ' सफर'},
+ {'score': 0.15317578613758087,
+  'sequence': 'हम आपके सुखद पल की कामना करते हैं',
+  'token': 848,
+  'token_str': ' पल'},
+ {'score': 0.07826550304889679,
+  'sequence': 'हम आपके सुखद समय की कामना करते हैं',
+  'token': 453,
+  'token_str': ' समय'},
+ {'score': 0.06304813921451569,
+  'sequence': 'हम आपके सुखद पहल की कामना करते हैं',
+  'token': 404,
+  'token_str': ' पहल'},
+ {'score': 0.058322224766016006,
+  'sequence': 'हम आपके सुखद अवसर की कामना करते हैं',
+  'token': 857,
+  'token_str': ' अवसर'}]
+```
 
-[{'score': 0.2096337080001831,
-  'sequence': 'मुझे उनसे बात करना एकदम अच्छा लगा',
-  'token': 1462,
-  'token_str': ' एकदम'},
- {'score': 0.17915162444114685,
-  'sequence': 'मुझे उनसे बात करना तब अच्छा लगा',
-  'token': 594,
-  'token_str': ' तब'},
- {'score': 0.15887945890426636,
-  'sequence': 'मुझे उनसे बात करना और अच्छा लगा',
-  'token': 324,
-  'token_str': ' और'},
- {'score': 0.12024253606796265,
-  'sequence': 'मुझे उनसे बात करना लगभग अच्छा लगा',
-  'token': 743,
-  'token_str': ' लगभग'},
- {'score': 0.07114479690790176,
-  'sequence': 'मुझे उनसे बात करना कब अच्छा लगा',
-  'token': 672,
-  'token_str': ' कब'}]
+Alternatively, the model could be loaded with AutoModels (& could then be fine-tuned on downstream tasks):
 ```
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+  
+tokenizer = AutoTokenizer.from_pretrained("flax-community/roberta-hindi")
+
+model = AutoModelForMaskedLM.from_pretrained("flax-community/roberta-hindi")
+```

apps/mlm.py

@@ -56,8 +56,8 @@ def app():
         masked_text = st.text_area("Please type a masked sentence to fill", select_text)
 
     st.sidebar.markdown(
-        "### Models\n"
-        "- [RoBERTa Hindi](https://huggingface.co/flax-community/roberta-hindi)\n"
+        "### MLM Models for comparison:\n"
+        "- [RoBERTa Hindi](https://huggingface.co/flax-community/roberta-hindi) (ours)\n"
         "- [Indic Transformers Hindi](https://huggingface.co/neuralspace-reverie/indic-transformers-hi-bert)\n"
         "- [HindiBERTa](https://huggingface.co/mrm8488/HindiBERTa)\n"
         "- [RoBERTa Hindi Guj San](https://huggingface.co/surajp/RoBERTa-hindi-guj-san)"

config.json

@@ -1,6 +1,6 @@
 {
     "models": {
-        "RoBERTa Hindi": "flax-community/roberta-hindi",
+        "RoBERTa Hindi (ours)": "flax-community/roberta-hindi",
         "Indic Transformers Hindi": "neuralspace-reverie/indic-transformers-hi-bert",
         "HindiBERTa": "mrm8488/HindiBERTa",
         "RoBERTa Hindi Guj San": "surajp/RoBERTa-hindi-guj-san"

mlm_custom/mlm_targeted_text.csv

@@ -1,18 +1,16 @@
 user_id,text,output,multi
-dk-crazydiv,हम आपके <mask> यात्रा की कामना करते हैं,सुखद,
-dk-crazydiv,मुझे उनसे बात करना बहुत <mask> लगा,अच्छा,
-dk-crazydiv,"बार बार देखो, हज़ार बार देखो, ये देखने की <mask> है","[""चीज़"",""बात""]",TRUE
-dk-crazydiv,ट्रंप कल अहमदाबाद में प्रधानमंत्री मोदी से <mask> करने जा रहे हैं,"[""मुलाकात"",""मिल्ने""]",TRUE
-dk-crazydiv,बॉम्बे से बैंगलोर की <mask> 500 किलोमीटर है,दूरी,
-dk-crazydiv,कहने को साथ अपने ये <mask> चलती है,दुनिया,
-dk-crazydiv,"ये इश्क़ नहीं आसान बस इतना समझ लीजिये, एक आग का दरिया है और <mask> के जाना है",डूब,
-prateekagrawal,आपका दिन <mask> हो,"[""शुभ"",""अच्छा""]",TRUE
-prateekagrawal,हिंदी भारत में <mask> जाने वाली भाषाओं में से एक है,"[""बोली"",""सिखाई"",""आधिकारिक""]",TRUE
-prateekagrawal,शुभ <mask>,"[""प्रभात"",रात्रि"",""यात्रा"",""अवसर""]",TRUE
-prateekagrawal,इंसान को कभी बुरा नहीं <mask> चाहिए,"[""बोलना"",""देखना"",""सुनाना"",""करना""]",TRUE
-hassiahk,बात ये है कि आप इसे <mask> से ही जानते हैं,पहले,
-hassiahk,<mask> पूर्व में उगता है,सूरज,
-hassiahk,"जल्दी सोना और जल्दी उठना इंसान को स्वस्थ ,समृद्ध और बुद्धिमान <mask> है",बनाता,
-hassiahk,"रोज एक सेब खाओ, <mask> से दूर रहो",डॉक्टर,
-hassiahk,किसी पुस्तक को उसके <mask> से मत आंकिए,आवरण,
-hassiahk,सभी <mask> चीजों का एक अंत होता है,"[""अच्छी"", ""बुरी""]",TRUE
+z,"ताजमहल की <mask> को देखने के लिए, देश दुनिया से लोग दूर दूर से आते है",सुन्दरता,
+z,गौर से देखिए इस मासूम <mask> को,बच्ची,
+z,"बात तय थी, लेकिन ऐन मौके पर उसके मुकर जाने से सारा काम <mask> में पड़ गया",खटाई,
+z,"कारखाने के तैयार माल को पैक करने की अनेक <mask> मशीनें आज मिलती हैं",z,
+z,"प्रत्येक टीम में अधिकतम <mask> खिलाड़ी होते हैं",z,
+z,"बार बार देखो, हज़ार बार देखो, ये देखने की <mask> है","[""चीज़"",""बात""]",TRUE
+z,ट्रंप कल अहमदाबाद में प्रधानमंत्री मोदी से <mask> करने जा रहे हैं,"[""मुलाकात"",""मिल्ने""]",TRUE
+z,कहने को साथ अपने ये <mask> चलती है,दुनिया,
+z,"ये इश्क़ नहीं आसान बस इतना समझ लीजिये, एक आग का दरिया है और <mask> के जाना है",डूब,
+z,आपका दिन <mask> हो,"[""शुभ"",""अच्छा""]",TRUE
+z,शुभ <mask>,"[""प्रभात"",रात्रि"",""यात्रा"",""अवसर""]",TRUE
+z,बात ये है कि आप इसे <mask> से ही जानते हैं,पहले,
+z,<mask> पूर्व में उगता है,सूरज,
+z,"जल्दी सोना और जल्दी उठना इंसान को स्वस्थ ,समृद्ध और बुद्धिमान <mask> है",बनाता,
+z,सभी <mask> चीजों का एक अंत होता है,"[""अच्छी"", ""बुरी""]",TRUE