I ndi S ocial FT : Multilingual Word Representation for I ndian languages in code-mixed environment (2023)

Abstract: The increasing number of Indian language users on the internet necessitates the development of Indian language technologies. In response to this demand, our paper presents a generalized representation vector for diverse text characteristics, including native scripts, transliterated text, multilingual, code-mixed, and social media-related attributes. We gather text from both social media and well-formed sources and utilize the FastText model to create the “IndiSocialFT” embedding. Through intrinsic and extrinsic evaluation methods, we compare IndiSocialFT with three popular pretrained embeddings trained over Indian languages. Our findings show that the proposed embedding surpasses the baselines in most cases and languages, demonstrating its suitability for various NLP applications.

## introduction considering the growing interest of developing language technologies (with indian languages support) due to vase indian language base internet users (expected to cross 650 millions 1 ), development of pre-trained indian language text representation suitable for various nlp applications is becoming an important task. though india is home to a vast linguistic landscapes, encompassing 1,369 rationalized languages and dialects @xcite , the majority of the indian language related nlp research focus primarily on few scheduled languages (only 22 languages are scheduled). further, in the context of developing indian language technology, the widespread use of transliterated text, creative acronyms, multilingual, code-mixed text etc. on social media should also be taken into account. to mitigate the above challenges, few studies @xcite @xcite have initiated the development of word embedding for indian languages. however, these 1 statistica-2022 www.statista.com studies mostly focus on monolingual corpora with limited reach to social media setups. motivated by these observations, this paper focuses on developing a more generalized representation vector suitable for the text with different characteristicswritten in native scripts, transliterated text, multilingual, code-mixed, and other social media related characteristics. with a target to incorporate diverse characteristics of user-generated contents on social media and also well formed text, we consider texts collected from two forms of sources -social media text (twitter, facebook and youtube), and well-formed text collected from samanantar dataset @xcite , dakshina dataset @xcite , manipuri-english comparable corpus @xcite and wikipedia (including 20 schedule languages written in respective native scripts). due to computing resource constraints at our end, the embedding of the words present in the corpus has been built using fasttext @xcite model named indisocialft foot_0 . to evaluate the quality of obtained embedding, we compare it with three popular publicly available works for indian languages namely facebook's fasttext (wiki+commoncrawl) @xcite , two models by indicnlpsuite (kakwani et al., 2020)-indicft and indicbert , and google's muril @xcite using both intrinsic and extrinsic evaluation methods. from various experimental observations, the proposed embedding outperforms all the baseline embedding for almost all the cases and languages. ## related work facebook fasttext project @xcite has published monolingual word embedding of 157 languages. their fasttext-based approach utilized wiki+commoncrawl data to create high-quality native multilingual code-mixed #lang(in) #tokens sources ft-wc ✓ --17 -common crawl and wiki indicft ✓ --11 8.8 b news crawls indicbert ✓ ✓ -12 8.8 b news crawls muril ✓ ✓ ✓ 16 11.0 b oscar corpus, wiki, pmindia, dakshina indisocialft ✓ ✓ ✓ 20+ 11.0 b social media, samanantar and dakshina dataset, wiki expanding on the growing interest in indian language representation, authors of indicnlpsuite @xcite have developed two different pre-trained models, indicft -a set of 11 monolingual pre-trained fasttext embedding models, and indicbert -a multilingual albert model trained on their corpora, referred to as indiccorp. to address the code-mixed cross-lingual transfer tasks, a few works like @xcite use transliterated data in training, but limit to including naturally present web crawl data. a more recent development in this field is the muril @xcite , which focuses on multilingual representations for 17 indian languages. muril is a pre-trained multilingual language model based on the bert framework, specifically built for indian languages. its effectiveness has been demonstrated across a wide range of nlp tasks for indian languages. ## conclusion and future work in this paper, we have addressed the challenge of representing text in a multilingual code-mixed social media environment by developing a fasttextbased embedding model. our model is trained on a diverse dataset collected from various social media platforms, supplemented with native script text from publicly available corpora to ensure balance. we have assessed the performance of our trained embeddings in both monolingual native script and code-mixed multilingual environments, employing a range of intrinsic and extrinsic evaluation techniques. the results demonstrate the effectiveness of our model's embedding space in capturing the semantic and syntactic information of text in both native monolingual and code-mixed multilingual contexts. this trained embedding model can be utilized to address various nlp challenges in the social media context in the indian region. as for future work, we plan to further improve the quality of our embeddings by incorporating additional data sources and exploring transformerbased pre-training techniques. we also aim to extend the applicability of our embeddings to a wider range of nlp tasks and evaluate their performance in more diverse linguistic scenarios.