Multichannel G enerative L anguage M odel: L earning A ll P ossible F actorizations W ithin and A cross C hannels (2020)

Abstract: A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning, but through two separate channels corresponding to their languages. In this work, we present the Multichannel Generative Language Model (MGLM). MGLM is a generative joint distribution model over channels. MGLM marginalizes over all possible factorizations within and across all channels. MGLM endows flexible inference, including unconditional generation, conditional generation (where 1 channel is observed and other channels are generated), and partially observed generation (where incomplete observations are spread across all the channels). We experiment with the Multi30K dataset containing English, French, Czech, and German. We demonstrate experiments with unconditional, conditional, and partially conditional generation. We provide qualitative samples sampled unconditionally from the generative joint distribution. We also quantitatively analyze the quality-diversity trade-offs and find MGLM outperforms traditional bilingual discriminative models.

## introduction a natural way to consider two parallel sentences in different languages is that each language expresses the same underlying meaning from a different viewpoint. each language can be thought of as a transformation that maps an underlying concept into a view that we collectively agree is determined as 'english' or 'french'. similarly, an image of a cat and the word 'cat' are expressing two views of the same underlying concept. in this case, the image corresponds to a high bandwidth channel and the word 'cat' to a low bandwidth channel. this way of conceptualizing parallel viewpoints naturally * work done during an internship at google brain. in this work, we present a general framework for modeling the joint distribution p(x 1 , ..., x k ) over k channels by marginalizing over all possible factorizations across the channels and within each channel. this formulation allows our framework to perform: 1) unconditional generation, 2) fully conditional generation (source channels are fully observed and fixed), and 3) partial conditional generation (source channels contain incomplete sequences). the key contributions in this work are: we highlight that while we focus on languages as a specific instantiation of a channel, our framework can generalize to any arbitrary specification, such as other types of tasks (e.g., question-answering) or other modalities (e.g., image captioning). ## multichannel generative language model in multichannel generative language modeling, our goal is to learn a generative model given a dataset consisting of a set of sequences {x ## related work mglm was inspired and influenced by prior work on conditional and unconditional language modeling. insertion transformer @xcite and xlnet @xcite also marginalize over all possible factorizations. however, their work is focused on the conditional distribution p(y|x), and they do not marginalize over all possible factorizations of the joint distribution. mglm can be viewed as an extension and generalization of kermit @xcite . kermit is a generative joint distribution model that also learns all possible factorizations. however, kermit is constrained to two languages, while mglm is a generative joint distribution model across any/all languages/text while learning all possible factorizations of the joint distribution. mglm follows from prior works on crosslingual language models, which aim to learn shared representation across languages. xlm @xcite is closely related to our work and also concatenate source and target sequences from different languages; however, their work is limited to bilingual concatenation, is not fully generative, and requires length conditioning. mglm is not limited to two languages and generalizes to multiple channels/languages, is fully generative, and our insertion-based approach (as opposed to maskingbased approach) does not require length conditioning. multilingual neural language model (wada and iwata, 2018) uses a shared encoder and language-dependent decoders to generate word embeddings and evaluate word alignment tasks. in contrast, our work unifies the neural architecture with a straightforward stack of self-attention layers. finally, @xcite explored multi-task learning for machine translation with an autoregressive network. the key difference between our work and other prior work on multi-target or multitask learning is that mglm models all possible factorizations of the joint distribution across all channels, instead of just the left-to-right factorization. this difference licenses mglm to perform any form of sampling (conditional, unconditional, partially-conditional) without any rigid left-to-right restrictions. evaluation of text generative models remain a challenge @xcite . quality versus diversity plots have been used to compare the trade-off at different output softmax temperatures, as such in stochastic beam search @xcite , which used a simpler n-gram diversity instead of self-bleu @xcite . however, we are the first to characterize the q-d behaviour of insertion based models versus exist-ing left-to-right language models. other metrics summarize the quality and diversity trade-off as a single number, such as fréchet bert distance @xcite inspired by the fid score @xcite used in computer vision, or take into account human evaluation @xcite . ## experiments we experiment on a multilingual dataset to demonstrate that we can learn mglm. we perform both qualitative and quantitative experiments. we highlight the model's capabilities ranging from conditional generation (i.e., machine translation) to unconditional sampling of the joint distribution over multiple languages. we experiment on the multi30k foot_0 @xcite @xcite , a multilingual dataset which consists of 29,000 parallel training sentences in english (en), french (fr), czech (cs), and german (de) sentences. we use multi30k because multiple high-quality channels (multilingual translations) are readily available to highlight our framework. we implement mglm as a base transformer decoder, without any causal masking, with 6 hidden layers and 1024 dimensional hidden representation. we concatenate all 4 language raw text training examples and use sen-tencepiece @xcite to learn a universal subword unigram @xcite tokenizer with a shared 32k vocabulary size. we follow a similar training set up to bert @xcite , using adam @xcite optimizer with a learning rate of 1e-4, warm up over the first 10% of the total training iterations varying between 10k to 50k iterations. we can train 3 different variants of mglm by altering the sampling ratio of training data seen by the model: ## conclusion and future work in this paper, we presented the multichannel generative language model (mglm). mglm is a gen-erative joint distribution model that marginalizes over all possible factorizations within and across channels. mglm endows flexible inference, including unconditional, conditional, and partially observed generation. we experimented with those inference modes using the multi30k dataset containing english, french, czech, and german. we provide qualitative samples sampled unconditionally from the generative joint distribution. we also quantitatively analyze the quality-diversity tradeoffs and find mglm outperform traditional bilingual discriminative models. our work focused on a specific instantiation of channels as languages. however, mglm is not limited to only languages and can generalize to other notions of channels. in future work, we will consider other textual channels, such as paraphrases, premises and hypotheses, questions and answers, and multimodal channels, such as images. another direction can investigate scaling mglm to dozens/hundreds of channels. fully generative models still often lag behind purely discriminative counterparts in performance, but we hope our work motivates future research on building generative joint distribution models of the world. 10 20 30 40 50 60 70 80 90 self-bleu 10 15 20 25 30 35 40 45 50 english / french 10 20 30 40 50 60 70 80 90 self-bleu 10 15 20 25 30 35 40 45 50 english / german quality-diversity bleu (test_2017_mscoco) joint en->* joint *->en multitarget en->* multitarget *->en bilingual en->* bilingual *->en figure 10: quality-diversity bleu curve for several mglm models (bilingual, multitarget, joint) on the multi30k text 2017 mscoco test set. the dotted diagonal line signifies bleu equals self-bleu. points indicate different temperatures, from 0.1 (low diversity, left in the graph) to 1.0 (high diversity, right in the graph)