A Federated Approach to Predicting Emojis in H indi Tweets (2022)

Abstract: The use of emojis affords a visual modality to, often private, textual communication.The task of predicting emojis however provides a challenge for machine learning as emoji use tends to cluster into the frequently used and the rarely used emojis.Much of the machine learning research on emoji use has focused on high resource languages and has conceptualised the task of predicting emojis around traditional server-side machine learning approaches.However, traditional machine learning approaches for private communication can introduce privacy concerns, as these approaches require all data to be transmitted to a central storage.In this paper, we seek to address the dual concerns of emphasising high resource languages for emoji prediction and risking the privacy of people’s data.We introduce a new dataset of 118k tweets (augmented from 25k unique tweets) for emoji prediction in Hindi, and propose a modification to the federated learning algorithm, CausalFedGSD, which aims to strike a balance between model performance and user privacy. We show that our approach obtains comparative scores with more complex centralised models while reducing the amount of data required to optimise the models and minimising risks to user privacy.

## introduction since the creation of emojis around the turn of the millennium @xcite , they have become of a staple of informal textual communication, expressing emotion and intent in written text @xcite . this development in communication style has prompted research into emoji analysis and prediction for english (e.g. @xcite @xcite . comparatively little research attention has been given to the low resource languages. emoji-prediction has posed a challenge for the research community because emojis express multiple modalities, contain visual semantics and the ability to stand in place for words @xcite . the challenge is further compounded by the quantity of emojis sent and the imbalanced distribution of emoji use @xcite . machine learning (ml) for emoji analysis and prediction has traditionally relied on traditional server-side architectures. however, training such models risks leaking sensitive information that may co-occur with emojis or be expressed through them. this can lead to potential breaches of data privacy regulation (e.g. the european general data protection regulation and the california consumer privacy act). in contrast, federated learning (fl) @xcite approaches the task of training machine learning models by emphasising privacy of data. such privacy is ensured by training models locally and sharing weight updates, rather than the data, with a central server (see figure 1 ). the fl approach assumes that some client-updates may be corrupted during transmission. fl therefore aims to retain predictive performance while emphasising user privacy in scenarios with potential data loss. motivated by prior work in privacy preserving ml (e.g. @xcite and emoji prediction for low resource languages (e.g. @xcite , we examine the application of fl to emoji topic prediction for hindi. specifically, we collect an imbalanced dataset of 118, 030 tweets in hindi which contain 700 unique emojis that we classify into 10 pre-defined categories of emojis. the dataset contains 700 unique emojis, that we classify into 10 pre-defined categories of emojis. 2 we further examine the impact of two different data balancing strategies on federated and server-side, centralised model performance. specifically, we examine: re-sampling and cost-sensitive re-weighting. we consider 6 centralised models which form our baselines: bi-directional lstm @xcite , indicbert @xcite , hindibert, foot_2 hindi-electra, foot_3 mbert @xcite , and xlm-r @xcite ; and lstms trained using two fl algorithms: fedprox @xcite and a modified version of causalfedgsd @xcite . we show that lstms trained using fl perform competitively with more complex, centralised models in spite of only using up to 50% of the data. ## prior work federated learning federated learning @xcite is a training procedure that distributes training of models onto a number of client devices. each client device locally computes weight updates on the basis of local data, and transmits the updated weights to a central server. in this way, fl can help prevent computational bottlenecks when training models on a large corpus while simultaneously preserving privacy by not transmitting raw data. this training approach has previously been applied for on-device token prediction on mobile phones for english. in a study of the quality of mobile keyboard suggestions, @xcite show that fl improves the quality of suggested words. addressing emoji-prediction in english, @xcite use the fed-eratedaveraging algorithm, to improve on traditional server-based models on user devices. we diverge from @xcite by using the causalfedgsd and fedprox algorithms on hindi tweets. fedprox develops on the federatedaveraging algorithm by introducing a regularization constant to it @xcite . in related work, @xcite seek to address the question of fl for emoji prediction for low resource languages. however, the dataset that they use, @xcite relies on emojis that are frequently used in english text and therefore may not be representative of emoji use in other, low resource languages. centralised training in efforts to extend emoji prediction, @xcite experiment with a bert-based model on a new english dataset that includes a large set of emojis for multi label prediction. addressing the issue of low resource languages, @xcite train a bidirectional lstm-based siamese network, jointly training their model with high resource and low resource languages. a number of studies on emoji prediction have been conducted in lower-resourced languages than english (e.g. @xcite @xcite @xcite . common to these approaches is the use of centralised ml models, which increase risks of breaches of privacy. in our experiments, we study using fl for emoji topic classification in low resource settings. ## data we collect our dataset for emoji topic prediction by scraping ∼1m tweets. we only keep the 24, 794 tweets that are written in hindi and contain at least one emoji. we duplicate all tweets that contain multiple emojis by the number of emojis contained, assigning a single emoji to each copy, resulting in a dataset of 118, 030 tweets with 700 unique emojis. due to the imbalanced distribution of emojis in our dataset (see figure 2 ), we assign emojis into 10 coarse-grained categories. this reduction i.e., from multi-label to multi-class and unique emojis into categories, risks losing the semantic meaning of emojis. our decision is motivated by how challenging emoji prediction is without such reductions @xcite . we pre-process our data to limit the risk of overfitting to rare tokens and platform specific tokens. for instance, we lower-case all text and remove numbers, punctuation, and retweet markers. we replace mentions, urls, and hashtags with specific tokens to avoid issues of over-fitting to these. ## experiments we conduct experiments with pytorch @xcite and transformers @xcite on google colab using a nvidia tesla v100 gpu with 26gb of ram. we create train, validation, and test (80/10/10) splits of the dataset, and measure performances using precision, recall, and weighted f1. all models are trained and evaluated on the imbalanced data and the two data balancing methods (see §3.1). for the fl setting, we conduct experiments manipulating the independent and identically distributed (i.i.d.) data assumption on client nodes. ## conclusion emoji topic prediction in user-generated text is a task which can contains highly private data. it is therefore important to consider privacy-preserving methods for the task. here, we presented a new dataset for the task for hindi and compared a privacy preserving approach, federated learning, with the centralised server-trained method. we present a modification to the causalfedgsd algorithm, and find that it converges faster than our other experimetnal models. in our experiments with different data balancing methods and simulations of i.i.d. and non-i.i.d. settings, we find that using fl affords comparable performances to the more complex fine-tuned language models that are trained centrally, while ensuring privacy. in future work, we plan to extend this work to multi-label emoji topic prediction and investigate strategies for dealing with decay of the model vocabulary.