Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification (2021)

Abstract: Semi-supervised learning through deep generative models and multi-lingual pretraining techniques have orchestrated tremendous success across different areas of NLP. Nonetheless, their development has happened in isolation, while the combination of both could potentially be effective for tackling task-specific labelled data shortage. To bridge this gap, we combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. Compared to strong supervised learning baselines, our semi-supervised classification framework is highly competitive and outperforms the state-of-the-art counterparts in low-resource settings across several languages.

## introduction multi-lingual pretraining has been shown to effectively use unlabelled data through learning shared representations across languages that can be transferred to downstream tasks @xcite @xcite . nonetheless, the lack of labelled data still leads to inferior performance of the same model compared to those trained in languages with more labelled data such as english @xcite . semi-supervised learning is another appealing paradigm that supplements the labelled data with unlabelled data which is easy to acquire @xcite @xcite . in particular, deep generative models (dgms) such as variational autoencoder (vae; kingma and welling (2014)) are capable of capturing complex data distributions at scale with rich latent representations, and they have been used cambridgeltl/mling_sdgms. for semi-supervised learning in various tasks in nlp @xcite @xcite , as well as inducing crosslingual word embeddings @xcite , and representation learning in combination with transformers via pretraining @xcite . to leverage the benefits of both worlds, we propose a pipeline method by combining semisupervised dgms (sdgms) based on m1+m2 model @xcite with multi-lingual pretraining. the pretrained model serves as multilingual encoder, and sdgms can operate on top of it independently of encoding architecture. to highlight such independence, we experiment with two pretraining settings: (1) our lstm-based crosslingual vae, and (2) the current stat-of-the-art (sota) multi-lingual bert @xcite . our experiments on document classification in several languages show promising results via the sdgm framework with different encoders, outperforming the sota supervised counterparts. we also illustrate that the end-to-end training of m1+m2 that was previously considered too unstable to train @xcite is possible with a reformulation of the objective function. as our sdgm is independent of the encoding architecture, we use different pretrained multi-lingual models to obtain h, µ φ (h), and σ 2 φ (h), described in §3. the second layer computes the posterior distribution of z 2 , conditioned on sampled z 1 from q φ (z 1 |x) and a class variable y. when we use labelled data, i.e. y is observed, q φ (z 2 |z 1 , y) can be directly obtained. with unlabelled data, we calculate the posterior q φ (z 2 , y|z 1 ) = q φ (y|z 1 )q φ (z 2 |z 1 , y) by inferring y with the classifier q φ (y|z 1 ), and integrate over all possible values of y. therefore, the elbo for the labelled data s @xmath0, y} is l(x, y): the objective function maximises both the labelled and unlabelled elbos while training directly the classifier with the labelled data as well: due to its training difficulty, m1+m2 is trained layer-wise in @xcite , where the first layer is trained according to eq. 1 and fixed, before the second layer is trained on top. however, in our experiments ( §4.1) we found that m1+m2 is easier to train end-to-end. we attribute this to our mathematical reformulation of the objective functions, giving rise to a more stable optimisation schedule. ## sdgms with multi-lingual pretraining lstm-based encoder with vae pretraining. our pretraining is based on the framework of @xcite , in which they pretrain a crosslingual vae with parallel corpus as input. however, the parallel corpus is expensive to obtain, and only the resulting cross-lingual embeddings rather than the whole encoder could be used due to the parallel input limitation of the model. to address these shortcomings, we propose non-parallel crosslingual vae (nxvae), which has the same graphical model as the vanilla vae. each language i is associated with its own word embedding matrix, and its input sequence x i is processed via a two layer bilstm @xcite ) shared across languages. we use the concatenation of the bilstm last hidden states as h, and compute q φ (z|x i ) with eq. 2, so that z becomes the joint cross-lingual semantic space. a language specific bag-of-word decoder (bow; @xcite ) is then used to reconstruct the input sequence. additionally, we optimise a language discriminator as an adversary @xcite to encourage the mixing of different language representations and keep the shared encoder language-agnostic. after pretraining nxvae, we transfer the whole encoder, including µ φ (h) and σ 2 φ (h), directly into our sdgm framework and treat it as q φ (z 1 |x) component of the model ( §4.1). ## experiments we perform document classification on the class balanced multilingual document classification corpus (mldoc; @xcite ). each document is assigned to one of the four news topic classes: corporate/industrial (c), economics (e), government/social (g), and markets (m). we experiment with five representative languages: en, de, fr, ru, zh, and use 1k instance training set along with the standard development and test set. for experiments with varying labelled data size, the rest training data from 1k corpus is used as unlabelled data. the full statistics are shown in for all experiments, we use adam (kingma and ba, 2015) as optimiser, but with different learning rates for both settings and pretraining. we implemented the model with pytorch 3 1.10 @xcite , and use geforce gtx 1080ti gpus. see the appendix for details about model configurations and training. ## conclusion we bridged between multi-lingual pretraining and deep generative models to form a semi-supervised learning framework for document classification. while outperforming sota supervised models in several settings, we showed that the benefits of sdgms are orthogonal to the encoding architecture or pretraining procedure. it opens up a new avenue for sdgms in low-resource nlp by incorporating unlabelled data potentially from different domains and languages. our preliminary results in crosslingual zero-shot setting with sdgms+nxvae are promising, and we will continue the exploration in this direction as future work. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n. gomez, lukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems 30: annual conference on neural information processing systems 2017, december 4-9, 2017, long beach, ca, usa, pages 5998-6008. liang-chen wei and zhi-hong deng. 2017. a variational autoencoding approach for inducing crosslingual word embeddings. in proceedings of the twenty-sixth international joint conference on artificial intelligence, ijcai 2017, melbourne, australia, august 19-25, 2017, pages 4165-4171. ijcai.org. shijie wu and mark dredze. 2019. beto, bentz, becas: the surprising cross-lingual effectiveness of bert. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, emnlp-ijcnlp 2019, hong kong, china, november 3-7, 2019, pages 833-844. association for computational linguistics. zhongbin xie and shuai ma. 2019. dual-view variational autoencoders for semi-supervised text matching. in proceedings of the twenty-eighth international joint conference on artificial intelligence, ij-cai 2019, macao, china, august 10-16, 2019, pages 5306-5312. ijcai.org. weidi xu, haoze sun, chao deng, and ying tan. 2017. variational autoencoder for semi-supervised text classification. in proceedings of the thirty-first aaai conference on artificial intelligence, february 4-9, 2017, san francisco, california, usa, pages 3358-3364. aaai press. pengcheng yin, chunting zhou, junxian he, and graham neubig. 2018. structvae: tree-structured latent variable models for semi-supervised semantic parsing. in proceedings of the 56th annual meeting of the association for computational linguistics, acl 2018, melbourne, australia, july 15-20, 2018, volume 1: long papers, pages 754-765. association for computational linguistics. daniel zeman, jan hajic, martin popel, martin potthast, milan straka, filip ginter, joakim nivre, and slav petrov. 2018. conll 2018 shared task: multilingual parsing from raw text to universal dependencies. in proceedings of the conll 2018 shared task: multilingual parsing from raw text to universal dependencies, brussels, belgium, october 31 -november 1, 2018, pages 1-21. association for computational linguistics. zhi-hua zhou and ming li. 2005. tri-training: exploiting unlabeled data using three classifiers. ieee trans. knowl. data eng., 17(11):1529-1541. yi zhu, benjamin heinzerling, ivan vulic, michael strube, roi reichart, and anna korhonen. 2019. on the importance of subword information for morphological tasks in truly low-resource languages. in proceedings of the 23rd conference on computational natural language learning, conll 2019, hong kong, china, november 3-4, 2019, pages 216-226. association for computational linguistics.