Decentralized W ord2 V ec Using Gossip Learning (2021)

Abstract: Advanced NLP models require huge amounts of data from various domains to produce high-quality representations. It is useful then for a few large public and private organizations to join their corpora during training. However, factors such as legislation and user emphasis on data privacy may prevent centralized orchestration and data sharing among these organizations. Therefore, for this specific scenario, we investigate how gossip learning, a massively-parallel, data-private, decentralized protocol, compares to a shared-dataset solution. We find that the application of Word2Vec in a gossip learning framework is viable. Without any tuning, the results are comparable to a traditional centralized setting, with a loss of quality as low as 4.3%. Furthermore, the results are up to 54.8% better than independent local training.

## introduction machine learning models, and especially deep learning models @xcite used to represent complex systems, require huge amounts of data. this is also the case with large-scale natural language processing (nlp) models. moreover, these models benefit from merging various sources of text from different domains to obtain a more complete representation of the language. for this reason, a small number of separate organizations (for example, government agencies) may want to train a complex nlp model using the combined data of their corpora to overcome the limitations of each single corpus. however, the typical solution in which all data is moved to a centralized system to perform the training may not be viable, as that could potentially violate privacy laws or data collection agreements and would require all organization to trust the owner of the system with access to their data. this problem can potentially be solved using massively-parallel, data-private, decentralized approaches -that is, distributed approaches where training is done directly on the machines that produce and hold the data, without having to share or transfer it and without any central coordinationsuch as gossip learning @xcite . therefore, we seek to investigate, in the scenario of a small group of large organizations, how models that are produced from the corpus of each node on a decentralized, fully-distributed, data-private configuration, i.e. gossip learning, compare to models trained using a traditional centralized approach where all the data are moved from local machines to a data center. furthermore, we investigate how these models compare to models trained locally using local data only, without any cooperation. our results show that the word2vec @xcite ) models trained by our implementation of gossip learning are close to models produced by its centralized counterpart setting, in terms of quality of the generated embeddings, and vastly better than what simple local training can produce. ## background and related work the main technique for massively-parallel, dataprivate training is federated learning @xcite , a centralized approach where each worker node calculates an update of the model based on local data. this gradient is then sent back to the central node which aggregates all these gradients to produce an updated global model which is sent back to the workers. this approach, however, suffers from issues such as the presence of a central node which may act as a privileged "gatekeeper", as well as reliability issue on the account of that central node. unlike centralized approaches, with decentralized machine learning all the nodes in the network execute the same protocols with the same level of privileges, mitigating chances of exploitation by malicious actors. furthermore, with a peer-to-peer network protocol, decentralized machine learning can virtually scale up to unlimited sizes and be more fault-tolerant, as the network traffic is spread out across multiple links, and not all directed to a single central location. one such approach is the gossip learning protocol @xcite . the gossip communication approach refers to a set of decentralized communication protocols inspired by the behaviour of the spread of gossip socially among people @xcite . first introduced for the purpose of efficiently synchronizing distributed servers @xcite , it has also applied to various problems, such as data aggregation @xcite and failure detection @xcite . ## gossip learning gossip learning is an asynchronous, data-parallel, decentralized averaging approach based on gossip communications. it has been shown to be effective when applied to various ml techniques, including binary classification with support vector machines @xcite , k-means clustering @xcite and low-rank matrix decomposition @xcite . however, these implementations of gossip learning are limited to simple scenarios, where each node holds a single data point and network communications are unrestricted. @xcite showed that the gossip protocol can be extended to a wider range of more realistic conditions. however, they identify issues with certain conditions that appear in some real-world scenarios, such as bias towards the data stored with faster communication speeds and the impact of network topologies on the convergence speed of models. algorithm 1 shows the general structure of gossip learning as introduced by @xcite . intuitively, models perform random walks over the network, merging with each other and training on local data at each node visited. we simulate gossip learning on a single machine, using synchronous iterations. this approximation works well under the assumption that all nodes have similar speeds. if that is not the case, additional measures must be taken to ensure correct model behaviour @xcite . ## methodology while gossip learning could be applied to most nlp algorithms, in this work we use word2vec @xcite because it is simple, small, and fast, thus allowing us to perform larger experiments on limited hardware resources. additionally, it is a well-known, well-understood technique, allowing us to more easily interpret the results. the dataset used is the wikipedia articles dump (wikimedia foundation, 2020) of more than 16gb, which contains over 6 million articles and in wikitext format with embedded xml metadata. from this dump we extract the articles belonging to the following 5 wikipedia categories of similar size: science, politics, business, humanities and history. to measure the quality of the word embeddings produced by a specific model, we collect the @xmath0. we then assign to each of these words a score based on their ground-truth cosine similarity to w t . we repeat this process for a set of (contextually ambiguous) target words w t (|w t | = 23) and use the total sum as the quality of the model. we estimate the ground-truth word similarities using a high-quality reference model, more specifically a state of the art word2vec model trained on the google news dataset, which uses a similar embedding size (@xmath1, 2013). ## experimental results to establish the baseline to compare to, the first experiment is in the traditional non-distributed, centralized configuration of word2vec. the baseline w2v sim value is 64.479, as shown in figure 1 . we simulate gossip learning with 10 nodes, with three different data distributions. in the r-balanced distribution, the corpora of the nodes have similar sizes and are randomly drawn from the dataset. in the r-imbalanced distribution, the corpora are similarly drawn at random, but have skewed sizes (up to a 4:1 ratio). finally, in the topicwise distribution, the dataset is divided between the nodes based on the 5 wikipedia categories, with two nodes splitting each category. the intuition behind dividing the texts by topic is that often times the corpora of organizations are limited to a specific domain. and setting imbalanced content sizes in one of the distributions can provide insights into how the learning is affected when some nodes have significantly bigger corpora than others. both these configurations are very relevant to the practical applicability of this work, as they both reflect common real-world scenarios. the formulation of gossip learning presented in section 3 requires the nodes to exchange their models after every local batch update. as complex nlp models can require millions of training batches, the communication overheads can quickly add up. we thus investigate the effect of reducing the exchange frequency while still maintaining the same number of training batches. more precisely, we repeat the same tests but limit the nodes to exchange the models every 50 batch updates, thus reducing overall communication by a factor of 50. figure 2 shows the evolution of the trained models for all combinations of exchange frequency and data distribution. we perform one more experiment, in which each node independently trains a model on its local data only, using the topicwise distribution. the w2v sim values do not converge as quickly and range from 41.657 to 56.570 (see figure 3 ). this underscores the importance for different organizations to collaborate to overcome the specificity of local corpora, as this can increase model quality by as much as 54.8%. ## limitations and future work although the experimental setup of this research takes into account parameters and conditions which simulate real-world scenarios, it is still limited in scope. for instance, the network conditions were assumed to be perfect. furthermore, security and privacy considerations in the area of networking were not taken into account. although they were not the focus of this research, their significance cannot be overlooked. investigating the behaviour of the proposed solution in more realistic network conditions is therefore a possible avenue of research. a single, simple nlp algorithm (word2vec) was evaluated in this work. this is due to the purpose of this research, which was to test the viability of gossip learning and compare it to a centralized solution in a specific scenario. evaluating more recent, contextualized nlp models, such as bert @xcite would be an interesting research direction, as these can better capture the different meanings of the same words in multiple domains. finally, the experiments were run without extensive hyperparameter optimization. given the satisfactory results obtained, it is likely that a proper tuning, based on state of the art distributed training research @xcite , could lead to gossip learning matching or even surpassing the quality of traditional centralized training. ## conclusions motivated by the scenario where various organizations wish to jointly train a large, high-quality nlp model without disclosing their own sensitive data, the goal of this work was to test whether word2vec could be implemented on top of gossip learning, a massively-parallel, decentralized, data-private framework. the quality of the word embeddings produced using gossip learning is close to what can be achieved in a traditional centralized configuration using the same parameters, with a loss of quality as low as 4.3%, a gap that might be closed with more advance tuning. the frequency of model exchange, which affects bandwidth requirements, has also been reduced 50 times without negative effects. finally, gossip learning can achieve up to 54.8% better quality than local training alone, motivating the need for joint training among organizations. the results of this work therefore show that gossip learning is a viable solution for large-scale, dataprivate nlp training in real-world applications.