|
# Releasing Hindi ELECTRA model |
|
|
|
This is a first attempt at a Hindi language model trained with Google Research's [ELECTRA](https://github.com/google-research/electra). **I don't modify ELECTRA until we get into finetuning** |
|
|
|
Tokenization and training CoLab: https://colab.research.google.com/drive/1R8TciRSM7BONJRBc9CBZbzOmz39FTLl_ |
|
|
|
Blog post: https://medium.com/@mapmeld/teaching-hindi-to-electra-b11084baab81 |
|
|
|
I was greatly influenced by: https://huggingface.co/blog/how-to-train |
|
|
|
## Corpus |
|
|
|
Download: https://drive.google.com/drive/u/1/folders/1WikYHHMI72hjZoCQkLPr45LDV8zm9P7p |
|
|
|
The corpus is two files: |
|
- Hindi CommonCrawl deduped by OSCAR https://traces1.inria.fr/oscar/ |
|
- latest Hindi Wikipedia ( https://dumps.wikimedia.org/hiwiki/20200420/ ) + WikiExtractor to txt |
|
|
|
Bonus notes: |
|
- Adding English wiki text or parallel corpus could help with cross-lingual tasks and training |
|
|
|
## Vocabulary |
|
|
|
https://drive.google.com/file/d/1-02Um-8ogD4vjn4t-wD2EwCE-GtBjnzh/view?usp=sharing |
|
|
|
Bonus notes: |
|
- Created with HuggingFace Tokenizers; could be longer or shorter, review ELECTRA vocab_size param |
|
|
|
## Pretrain TF Records |
|
|
|
[build_pretraining_dataset.py](https://github.com/google-research/electra/blob/master/build_pretraining_dataset.py) splits the corpus into training documents |
|
|
|
Set the ELECTRA model size and whether to split the corpus by newlines. This process can take hours on its own. |
|
|
|
https://drive.google.com/drive/u/1/folders/1--wBjSH59HSFOVkYi4X-z5bigLnD32R5 |
|
|
|
Bonus notes: |
|
- I am not sure of the meaning of the corpus newline split (what is the alternative?) and given this corpus, which creates the better training docs |
|
|
|
## Training |
|
|
|
Structure your files, with data-dir named "trainer" here |
|
|
|
``` |
|
trainer |
|
- vocab.txt |
|
- pretrain_tfrecords |
|
-- (all .tfrecord... files) |
|
- models |
|
-- modelname |
|
--- checkpoint |
|
--- graph.pbtxt |
|
--- model.* |
|
``` |
|
|
|
CoLab notebook gives examples of GPU vs. TPU setup |
|
|
|
[configure_pretraining.py](https://github.com/google-research/electra/blob/master/configure_pretraining.py) |
|
|
|
Model https://drive.google.com/drive/folders/1cwQlWryLE4nlke4OixXA7NK8hzlmUR0c?usp=sharing |
|
|
|
## Using this model with Transformers |
|
|
|
Sample movie reviews classifier: https://colab.research.google.com/drive/1mSeeSfVSOT7e-dVhPlmSsQRvpn6xC05w |
|
|
|
Slightly outperforms Multilingual BERT on these Hindi Movie Reviews from https://github.com/sid573/Hindi_Sentiment_Analysis |
|
|