i NLTK : Natural Language Toolkit for Indic Languages (2020)

Abstract: We present iNLTK, an open-source NLP library consisting of pre-trained language models and out-of-the-box support for Data Augmentation, Textual Similarity, Sentence Embeddings, Word Embeddings, Tokenization and Text Generation in 13 Indic Languages. By using pre-trained models from iNLTK for text classification on publicly available datasets, we significantly outperform previously reported results. On these datasets, we also show that by using pre-trained models and data augmentation from iNLTK, we can achieve more than 95% of the previous best performance by using less than 10% of the training data. iNLTK is already being widely used by the community and has 40,000+ downloads, 600+ stars and 100+ forks on GitHub. The library is available at https://github.com/goru001/inltk .

## introduction deep learning offers a way to harness large amounts of computation and data with little engineering by hand @xcite . with distributed representation, various deep models have become the new state-of-the-art methods for nlp problems. pre-trained language models @xcite can model syntactic/semantic relations between words and reduce feature engineering. these pre-trained models are useful for initialization and/or transfer learning for nlp tasks. pre-trained models are typically learned using unsupervised approaches from large, diverse monolingual corpora @xcite . while we have seen exciting progress across many tasks in natural language processing over the last years, most such results have been achieved in english and a small set of other high-resource languages @xcite . indic languages, widely spoken by more than a billion speakers, lack pre-trained deep language models, trained on a large corpus, which can provide a headstart for downstream tasks using transfer learning. availability of such models is critical to build a system that can achieve good results in "lowresource" settings -where labeled data is scarce and computation is expensive, which is the biggest challenge for working on nlp in indic languages. additionally, there's lack of indic languages support in nlp libraries like spacy foot_0 , nltk foot_1 -creating a barrier to entry for working with indic languages. ## inltk api to evaluate the effectiveness of using data augmentation from inltk in low resource settings, we prepare 15 reduced train sets of publicly available text-classification datasets by picking first n examples from the full train set 16 , where n is equal to size of reduced train set and compare accuracy of the classifier trained with vs without data augmen-14 https://inltk.readthedocs.io/en/latest/api docs.html#getsimilar-sentences 15 notebooks to prepare reduced datasets are accessible from the github repository of the library 16 labels in publicly available full train sets were not grouped together, instead were randomly shuffled tation. semantic textual similarity (sts) assesses the degree to which the underlying semantics of two segments of text are equivalent to each other @xcite . inltk compares 18 sentence embeddings of the two segments of text, obtained from pre-trained language model's encoder, using a comparison function, to evaluate semantic textual similarity. cosine similarity between sentence embeddings is used as the default comparison function. distributed representations are the cornerstone of modern nlp, which have led to significant advances in many nlp tasks. inltk provides utilities to obtain distributed representations for words 19 , sentences and documents 20 obtained 17 refer github repository of the library for instructions to reproduce results on full and reduced dataset 18 https://inltk.readthedocs.io/en/latest/api docs.html#getsentence-similarity 19 https://inltk.readthedocs.io/en/latest/api docs.html#getembedding-vectors 20 https://inltk.readthedocs.io/en/latest/api docs.html#get-from embedding layer and encoder output of pretrained language models, respectively. additionally, inltk provides utilities to generate text 21 given a prompt, using pre-trained language models, tokenize 22 text using sentencepiece tokenization models described in section 2.2, identify 23 which one of the supported indic languages is given text in and remove tokens of a foreign language 24 from given text. ## related work nlp and ml communities have a strong culture of building open-source tools. there are lots of easyto-use, user-facing libraries for general-purpose nlp like nltk @xcite , stanford corenlp @xcite , spacy (honnibal and montani, 2017), allennlp @xcite , flair @xcite , stanza @xcite and huggingface transformers @xcite . but most of these libraries have limited or no support for indic languages, creating a barrier to entry for working with indic languages. additionally, for many indic languages word embeddings have been trained, but they still lack richer pretrained representations from deep language models @xcite . inltk tries to solve these problems by providing pre-trained language models and out-of-the-box support for a variety of nlp tasks in 13 indic languages. ## conclusion and future work we are working on expanding the supported languages in inltk to include other indic languages like telugu, maithili; code mixed languages like hinglish (hindi and english), manglish (malayalam and english) and tanglish (tamil and english); expanding supported model architectures to include bert. additionally, we want to mitigate any possible unwarranted biases which might exist in pre-trained language models @xcite , because of training data, which might propagate into downstream systems using these models. while these tasks are work in progress, we hope this library will accelerate nlp research and development in indic languages.