Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages (2021)

Abstract: Most combinations of NLP tasks and language varieties lack in-domain examples for supervised training because of the paucity of annotated data. How can neural models make sample-efficient generalizations from task–language combinations with available data to low-resource ones? In this work, we propose a Bayesian generative model for the space of neural parameters. We assume that this space can be factorized into latent variables for each language and each task. We infer the posteriors over such latent variables based on data from seen task–language combinations through variational inference. This enables zero-shot classification on unseen combinations at prediction time. For instance, given training data for named entity recognition (NER) in Vietnamese and for part-of-speech (POS) tagging in Wolof, our model can perform accurate predictions for NER in Wolof. In particular, we experiment with a typologically diverse sample of 33 languages from 4 continents and 11 families, and show that our model yields comparable or better results than state-of-the-art, zero-shot cross-lingual transfer methods. Our code is available at github.com/cambridgeltl/parameter-factorization.

## introduction the annotation efforts in nlp have achieved impressive feats, such as the universal dependencies (ud) project @xcite , which now includes 83 languages. but even ud covers only a meager subset of the world's estimated 8,506 languages @xcite . what is more, the association for computational linguistics wiki 1 lists 24 separate nlp tasks. labeled data, which is both costly and labor-intensive, is missing for many of such task-language combinations. this shortage hinders the development of computational models for the majority of the world's languages @xcite . a common solution is transferring knowledge across domains, such as tasks and languages @xcite , which holds promise to mitigate the lack of training data inherent to a large spectrum of nlp applications @xcite @xcite @xcite . in the most extreme scenario, zero-shot learning, no annotated examples are available for the target domain. in particular, zero-shot transfer across languages implies a change in the data domain, and leverages information from resource-rich languages to tackle the same task in a previously unseen target language @xcite @xcite . zero-shot transfer across tasks within the same language @xcite , on the other hand, implies a change in the space of labels. as our main contribution, we propose a bayesian generative model of the neural parameter space. we assume this to be structured, and for this reason factorizable into task-and languagespecific latent variables. 2 by performing transfer of knowledge from both related tasks and related languages (i.e., from seen combinations), our model allows for zero-shot prediction on unseen task-language combinations. for instance, the availability of annotated data for part-of-speech (pos) tagging in wolof and for named-entity recognition (ner) in vietnamese supplies plenty of information to infer a task-agnostic representation for wolof and a language-agnostic representation for ner. conditioning on these, the appropriate neural parameters for wolof ner can be generated at evaluation time. while this idea superficially resembles matrix completion for collaborative filtering @xcite , the neural parameters are latent and are non-identifiable. rather than recovering missing entries from partial observations, in our approach we reserve latent variables to each language and each task to tie together neural parameters for combinations that have either of them in common. we adopt a bayesian perspective towards inference. the posterior distribution over the model's latent variables is approximated through stochastic variational inference @xcite . given the enormous number of parameters, we also explore a memory-efficient inference scheme based on a diagonal plus low-rank approximation of the covariance matrix. this guarantees that our model remains both expressive and tractable. we evaluate the model on two sequence labeling tasks: pos tagging and ner, relying on a typologically representative sample of 33 languages from 4 continents and 11 families. the results clearly indicate that our generative model surpasses standard baselines based on crosslingual transfer 1) from the (typologically) nearest source language; 2) from the source language with the most abundant in-domain data (english); and 3) from multiple source languages, in the form of either a multi-task, multi-lingual model with parameter sharing @xcite or an ensemble of task-and language-specific models @xcite . ## bayesian generative model in this work, we propose a bayesian generative model for multi-task, multi-lingual nlp. we train a single bayesian neural network for several tasks and languages jointly. formally, we consider a set @xmath0, . . . , t n } of n tasks and a set @xmath1, . . . , l m } of m languages. the core modeling assumption we make is that the parameter space of the neural network is structured: specifically, we posit that certain parameters correspond to tasks and others correspond to languages. this structure assumption allows us to generalize to unseen task-language pairs. in this regard, the model is reminiscent of matrix factorization as applied to collaborative filtering @xcite . we now describe our generative model in three steps that match the nesting level of the plates in the diagram in figure 1 . equivalently, the reader can follow the nesting level of the for loops in algorithm 1 for an algorithmic illustration of the generative story. while we present the model in its most general form, we take µ t @xmath2. consider the cartesian product of all tasks and languages t × l. we can decompose this product into seen task-language pairs s and unseen task-language pairs u, i.e., t × @xmath3. naturally, we are only able to train our model on the seen task-language pairs s. however, as we estimate all task-language parameter vectors θ ij jointly, our model allows us to draw inferences about the parameters for pairs in u as well. the intuition for why this should work is as follows: by observing multiple pairs where the task (language) is the same but the language (task) varies, the model learns to distill the relevant knowledge for zero-shot learning because our generative model structurally enforces a disentangled representations -separating representations for the tasks from the representations for the languages rather than lumping them together into a single entangled representation @xcite . furthermore, the neural networks f ψ and f φ mapping the task-and language-specific latent variables to neural parameters are shared, allowing the model to generalize across task-language pairs. ## variational inference exact computation of the posterior over the latent variables p(θ, t, l | x) is intractable. thus, we need to resort to an approximation. in this work, we consider variational inference as our approximate inference scheme. variational inference finds an approximate posterior over the latent variables by minimizing the variational gap, which may be expressed as the kullback-leibler (kl) divergence between the variational approximation q(θ, t, l) and the true posterior p(θ, t, l | x). in our work, we employ the following variational distributions: we note the unusual choice to tie parameters between the generative model and the variational 412 in general, the covariance matrices s t and s l in equation ( 1 ) and equation (2) will require o(h 2 ) space to store. as h is often very large, it is impractical to materialize either matrix in its entirety. thus, in this work, we experiment with smaller matrices that have a reduced memory footprint; specifically, we consider a diagonal covariance matrix and a diagonal plus low-rank covariance structure. a diagonal covariance matrix makes computation feasible with a complexity of o( ); this, however, comes at the cost of not letting parameters influence each other, and thus failing to capture their complex interactions. to allow for a more expressive variational family, we also consider a covariance matrix that is the sum of a diagonal matrix and a low-rank matrix: ## related work our approach builds on ideas from several different fields: cross-lingual transfer in nlp, with a particular focus on sequence labeling tasks, as well as matrix factorization, contextual parameter generation, and neural bayesian methods. ## conclusion the main contribution of our work is a bayesian generative model for multiple nlp tasks and languages. at its core lies the idea that the space of neural weights can be factorized into latent variables for each task and each language. while training data are available only for a meager subset of task-language combinations, our model opens up the possibility to perform prediction in novel, undocumented combinations at evaluation time. we performed inference through stochastic variational methods, and ran experiments on zeroshot named entity recognition (ner) and partof-speech (pos) tagging in a typologically diverse set of 33 languages. based on the reported results, we conclude that leveraging the information from tasks and languages simultaneously is superior to model transfer from english (relying on more abundant in-task data in the source language), from the most typologically similar language (relying on prior information on language relatedness), or from multiple source languages. moreover, we found that the entropy of predictive posterior distributions obtained through bayesian model averaging correlates almost perfectly with the error rate in the prediction. as a consequence, our approach holds promise to alleviating data paucity issues for a wide spectrum of languages and tasks, and to make knowledge transfer more robust to uncertainty. finally, we remark that our model is amenable to be extended to multilingual tasks beyond sequence labeling-such as natural language inference @xcite and question answering @xcite @xcite )-and to zero-shot transfer across combinations of multiple modalities (e.g., speech, text, and vision) with tasks and languages. we leave these exciting research threads for future research. by substituting @xmath4, it is trivial to obtain equation ( 9 ).