Back to all models

Unable to determine this model’s pipeline type. Check the docs .

Monthly model downloads

monsoon-nlp/hindi-bert monsoon-nlp/hindi-bert
310 downloads
last 30 days

pytorch

tf

Contributed by

monsoon-nlp Nick Doiron
4 models

How to use this model directly from the 🤗/transformers library:

			
Copy to clipboard
from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("monsoon-nlp/hindi-bert") model = AutoModel.from_pretrained("monsoon-nlp/hindi-bert")
Uploaded in S3

Releasing Hindi ELECTRA model

This is a first attempt at a Hindi language model trained with Google Research's ELECTRA. I don't modify ELECTRA until we get into finetuning

Tokenization and training CoLab

Blog post

I was greatly influenced by: https://huggingface.co/blog/how-to-train

Corpus

Download: https://drive.google.com/drive/folders/1SXzisKq33wuqrwbfp428xeu_hDxXVUUu?usp=sharing

The corpus is two files:

Bonus notes:

  • Adding English wiki text or parallel corpus could help with cross-lingual tasks and training

Vocabulary

https://drive.google.com/file/d/1-6tXrii3tVxjkbrpSJE9MOG_HhbvP66V/view?usp=sharing

Bonus notes:

  • Created with HuggingFace Tokenizers; could be longer or shorter, review ELECTRA vocab_size param

Training

Structure your files, with data-dir named "trainer" here

trainer
- vocab.txt
- pretrain_tfrecords
-- (all .tfrecord... files)
- models
-- modelname
--- checkpoint
--- graph.pbtxt
--- model.*

CoLab notebook gives examples of GPU vs. TPU setup

configure_pretraining.py

Using this model with Transformers

Sample movie reviews classifier: https://colab.research.google.com/drive/1mSeeSfVSOT7e-dVhPlmSsQRvpn6xC05w

Slightly outperforms Multilingual BERT on these Hindi Movie Reviews from https://github.com/sid573/Hindi_Sentiment_Analysis