Original: T OWARDS L ANGUAGE A GNOSTIC U NIVERSAL R EPRESENTATIONS
Revised: T OWARDS L ANGUAGE A GNOSTIC U NIVERSAL R EPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both languages the student is ﬂuent in, even if the math lessons were only taught in one language.
Revised: When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both languages the student is ﬂuent in, even if the math lessons were only taught in one language.

Original: However, current representations in machine learning are language dependent.
Revised: However, current representations in machine learning are language dependent.

Original: In this work, we present a method to decouple the language from the problem by learning language agnostic representations and therefore allowing training a model in one language and applying to a different one in a zero shot fashion.
Revised: In this work, we present a method to decouple the language from the problem by learning language agnostic representations and therefore allowing training a model in one language and applying to a different one in a zero shot fashion.

Original: We learn these representations by taking inspiration from linguistics and formalizing Universal Grammar as an optimization process (Chomsky, 2014; Montague, 1970).
Revised: We learn these representations by taking inspiration from linguistics, speciﬁcally the Universal Grammar hypothesis and learn universal latent representations that are language agnostic (Chomsky, 2014; Montague, 1970).

Original: We demonstrate the capabilities of these representations by showing thatthe models trained on a single language using language agnostic representations achieve very similar accuracies in other languages.
Revised: We demonstrate the capabilities of these representations by showing that models trained on a single language using language agnostic representations achieve very similar accuracies in other languages.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Anecdotally speaking, ﬂuent bilingual speakers rarely face trouble translating a task learned in one language to another.
Revised: Anecdotally speaking, ﬂuent bilingual speakers rarely face trouble translating a task learned in one language to another.

Original: For example, a bilingual speaker who is taught a math problem in English will trivially generalize to other known languages.
Revised: For example, a bilingual speaker who is taught a math problem in English will trivially generalize to other known languages.

Original: Furthermore there is a large collection of evidence in linguistics arguing that although separate lexicons exist in multilingual speakers the core representations of concepts and theories are shared in memory (Altarriba, 1992; Mitchel, 2005; Bentin et al., 1985).
Revised: Furthermore there is a large collection of evidence in linguistics arguing that although separate lexicons exist in multilingual speakers the core representations of concepts and theories are shared in memory (Altarriba, 1992; Mitchel, 2005; Bentin et al., 1985).

Original: The fundamental question we’re interested in answering is on the learnability of these shared representations within a statistical framework.
Revised: The fundamental question we’re interested in answering is on the learnability of these shared representations within a statistical framework.

Original: We approached this problem from a linguistics perspective.
Revised: We approached this problem from a linguistics perspective.

Original: Languages have vastly varying syntactic features and rules.
Revised: Languages have vastly varying syntactic features and rules.

Original: Linguistic Relativity studies the impact of these syntactic variations on the formations of concepts and theories (Au, 1983).
Revised: Linguistic Relativity studies the impact of these syntactic variations on the formations of concepts and theories (Au, 1983).

Original: Within this framework of study, the two schools of thoughts are linguistic determinism and weak linguistic inﬂuence.
Revised: Within this framework of study, the two schools of thoughts are linguistic determinism and weak linguistic inﬂuence.

Original: Linguistic determinism argues that language entirely forms the range of cognitive processes, including the creation of various concepts, but is generally agreed to be false (Hoijer, 1954; Au, 1983).
Revised: Linguistic determinism argues that language entirely forms the range of cognitive processes, including the creation of various concepts, but is generally agreed to be false (Hoijer, 1954; Au, 1983).

Original: Although there exists some weak linguistic inﬂuence, it is by no means fundamental (Ahearn, 2016).
Revised: Although there exists some weak linguistic inﬂuence, it is by no means fundamental (Ahearn, 2016).

Original: The superﬂuous nature of syntactic variations across languages brings forward the argument of principles and parameters (PnP) which hypothesizes the existence of a small distributed parameter representation that captures the syntactic variance between languages denoted by parameters (e.g. head-ﬁrst or head-ﬁnal syntax), as well as common principles shared across all languages (Culicover, 1997).
Revised: The superﬂuous nature of syntactic variations across languages brings forward the argument of principles and parameters (PnP) which hypothesizes the existence of a small distributed parameter representation that captures the syntactic variance between languages denoted by parameters (e.g. head-ﬁrst or head-ﬁnal syntax), as well as common principles shared across all languages (Culicover, 1997).

Original: Universal Grammar (UG) is the study of principles and the parameters that are universal across languages (Montague, 1970).
Revised: Universal Grammar (UG) is the study of principles and the parameters that are universal across languages (Montague, 1970).

Original: The ability to learn these universalities would allow us to learn representations of language that are fundamentally agnostic of the speciﬁc language itself.
Revised: The ability to learn these universalities would allow us to learn representations of language that are fundamentally agnostic of the speciﬁc language itself.

Original: Doing so would allow us to learn a task in one language and reap the beneﬁts of all other languages without needing multilingual datasets.
Revised: Doing so would allow us to learn a task in one language and reap the beneﬁts of all other languages without needing multilingual datasets.

Original: Our attempt to learn these representations begins by taking inspiration from linguistics and formalizing UG as an optimization problem.
Revised: We take inspiration from the UG hypothesis and learn latent representations that are language agnostic which allow us to solve downstream problems in new languages without the need of any language-speciﬁc training data.

Original: We do not make any claims about the Universal Grammar hypothesis, but simply take inspiration from it.
Revised: We do not make any claims about the Universal Grammar hypothesis, but simply take inspiration from it.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work attempts to unite universal (task agnostic) representations with multilingual (language agnostic) representations (Peters et al., 2018;
Revised: Our work attempts to unite universal (task agnostic) representations with multilingual (language agnostic) representations (Peters et al., 2018;

Original: McCann et al., 2017).
Revised: McCann et al., 2017).

Original: The recent trend in universal representations has been moving away from context-less unsupervised word embeddings to contextrich representations.
Revised: The recent trend in universal representations has been moving away from context-less unsupervised word embeddings to contextrich representations.

Original: Deep contextualized word representations (ELMo) trains an unsupervised language model on a large corpus of data and applies it to a large set of auxiliary tasks (Peters et al., 2018).
Revised: Deep contextualized word representations (ELMo) trains an unsupervised language model on a large corpus of data and applies it to a large set of auxiliary tasks (Peters et al., 2018).

Original: These unsupervised representations boosted the performance of models on a wide array of tasks.
Revised: These unsupervised representations boosted the performance of models on a wide array of tasks.

Original: Along the same lines McCann et al.
Revised: Along the same lines McCann et al.

Original: (2017) showed the power of using latent representations of translation models as features across other non-translation tasks.
Revised: (2017) showed the power of using latent representations of translation models as features across other non-translation tasks.

Original: In general, initializing models with pre-trained language models shows promise against the standard initialization with word embeddings.
Revised: In general, initializing models with pre-trained language models shows promise against the standard initialization with word embeddings.

Original: Even further, Radford et al.
Revised: Even further, Radford et al.

Original: (2017) show that an unsupervised language model trained on a large corpus will contain a neuron that strongly correlates with sentiment without ever training on a sentiment task implying that unsupervised language models maybe picking up informative and structured signals.
Revised: (2017) show that an unsupervised language model trained on a large corpus will contain a neuron that strongly correlates with sentiment without ever training on a sentiment task implying that unsupervised language models maybe picking up informative and structured signals.

Original: In the ﬁeld of multilingual representations, a fair bit of work has been done on multilingual word embeddings.
Revised: In the ﬁeld of multilingual representations, a fair bit of work has been done on multilingual word embeddings.

Original: Ammar et al.
Revised: Ammar et al.

Original: (2016) explored the possibility of training massive amounts of word embeddings utilizing either parallel data or bilingual dictionaries via the SkipGram paradigm.
Revised: (2016) explored the possibility of training massive amounts of word embeddings utilizing either parallel data or bilingual dictionaries via the SkipGram paradigm.

Original: Later on an unsupervised approach to multilingual word representations was proposed by Chen & Cardie (2018) which utilized an adversarial training regimen to place word embeddings into a shared latent space.
Revised: Later on an unsupervised approach to multilingual word representations was proposed by Chen & Cardie (2018) which utilized an adversarial training regimen to place word embeddings into a shared latent space.

Original: Although word embeddings show great utility, they fall behind methods which exploit sentence structure as well as words.
Revised: Although word embeddings show great utility, they fall behind methods which exploit sentence structure as well as words.

Original: Less work has been done on multilingual sentence representations.
Revised: Less work has been done on multilingual sentence representations.

Original: Most notably both Schwenk & Douze (2017) and Artetxe et al.
Revised: Most notably both Schwenk & Douze (2017) and Artetxe et al.

Original: (2017) propose a way to learn multilingual sentence representation through a translation task.
Revised: (2017) propose a way to learn multilingual sentence representation through a translation task.

Original: We train downstream models using language agnostic universal representations on a set of tasks and show the ability for the downstream models to generalize to languages that we did not train on.
Revised: We train downstream models using language agnostic universal representations on a set of tasks and show the ability for the downstream models to generalize to languages that we did not train on.

Original: U NIVERSAL G RAMMAR AS AN  O PTIMIZATION P ROBLEM
Revised:  O PTIMIZATION P ROBLEM

Original: Statistical language models approximate the probability distribution of a series of words by predicting the next word given a sequence of previous words.
Revised: Statistical language models approximate the probability distribution of a series of words by predicting the next word given a sequence of previous words.

Original: [Equation]
Revised: [Equation]

Original: where w i are indices representing words in an arbitrary vocabulary.
Revised: where w i are indices representing words in an arbitrary vocabulary.

Original: Learning grammar is equivalent to language modeling, as the support of p will represent the set of all grammatically correct sentences.
Revised: Learning grammar is equivalent to language modeling, as the support of p will represent the set of all grammatically correct sentences.

Original: Furthermore, let j α represent a particular language.
Revised: Furthermore, let j α represent a particular language.

Original: Let p
Revised: Let p

Original: j α ( · ) represent the language model for the j α th language and w j α represents a word from the j α th language.
Revised: j α ( · ) represent the language model for the j α th language and w j α represents a word from the j α th language.

Original: Let k j α represent a distributed representation of a speciﬁc language along the lines of the PnP argument (Culicover, 1997).
Revised: Let k j α represent a distributed representation of a speciﬁc language along the lines of the PnP argument (Culicover, 1997).

Original: UG, through the lens of statistical language modeling, hypothesizes the existence of a factorization of p j α ( · ) containing a language agnostic segment.
Revised: UG, through the lens of statistical language modeling, hypothesizes the existence of a factorization of p j α ( · ) containing a language agnostic segment.

Original: The factorization used throughout this paper is the following ( ◦ denotes function composition):
Revised: The factorization used throughout this paper is the following ( ◦ denotes function composition):

Original: [Equation]
Revised: [Equation]

Original: The distribution matching constraint d , insures that the representations across languages are common as hypothesized by the UG argument.
Revised: The distribution matching constraint d , insures that the representations across languages are common as hypothesized by the UG argument.

Original: Function e j α : N i
Revised: Function e j α : N i

Original: → R i × d is a language speciﬁc function which takes an ordered set of integers representing tokens (token meaning w j α i ) and outputs a vector of size d per token.
Revised: → R i × d is a language speciﬁc function which takes an ordered set of integers representing tokens (token meaning w j α i ) and outputs a vector of size d per token.

Original: Function u :
Revised: Function u :

Original: R i ×
Revised: R i ×

Original: d
Revised: d

Original: → R i × d takes the language speciﬁc representation and attempts to embed into a language agnostic representation.
Revised: → R i × d takes the language speciﬁc representation and attempts to embed into a language agnostic representation.

Original: Function h : ( R i × d , R f )
Revised: Function h : ( R i × d , R f )

Original: → R i × d takes the universal representation as well
Revised: → R i × d takes the universal representation as well

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: as a distributed representation of the language of size f and returns a language speciﬁc decoded representation.
Revised: as a distributed representation of the language of size f and returns a language speciﬁc decoded representation.

Original: e − 1 j α maps our decoded representation back to the token space.
Revised: e − 1 j α maps our decoded representation back to the token space.

Original: For the purposes of distribution matching we utilize the GAN framework.
Revised: For the purposes of distribution matching we utilize the GAN framework.

Original: Following recent successes we use Wasserstein-1 as our distance function d (Arjovsky et al., 2017).
Revised: Following recent successes we use Wasserstein-1 as our distance function d (Arjovsky et al., 2017).

Original: Given two languages j α and j β the distribution of the universal representations should be within (cid:15) with respect to the W 1 of each other.
Revised: Given two languages j α and j β the distribution of the universal representations should be within (cid:15) with respect to the W 1 of each other.

Original: Using the Kantarovich-Rubenstein duality (Arjovsky et al., 2017; Villani, 2008) we deﬁne
Revised: Using the Kantarovich-Rubenstein duality (Arjovsky et al., 2017; Villani, 2008) we deﬁne

Original: [Equation]
Revised: [Equation]

Original: where L is the Lipschitz constant of f .
Revised: where L is the Lipschitz constant of f .

Original: Throughout this paper we satisfy the Lipschitz constraint by clamping the parameters to a compact space, as done in the original WGAN paper (Arjovsky et al., 2017).
Revised: Throughout this paper we satisfy the Lipschitz constraint by clamping the parameters to a compact space, as done in the original WGAN paper (Arjovsky et al., 2017).

Original: Therefore the complete loss function for m languages each containing N documents becomes:
Revised: Therefore the complete loss function for m languages each containing N documents becomes:

Original: [Equation]
Revised: [Equation]

Original: λ is a scaling factor for the distribution constraint loss.
Revised: λ is a scaling factor for the distribution constraint loss.

Original:  UG-WGAN
Revised:  UG-WGAN

Original: Our speciﬁc implementation of our factorization and optimization problem we denote as UGWGAN.
Revised: Our speciﬁc implementation of our factorization and optimization problem we denote as UGWGAN.

Original: 4. A RCHITECTURE
Revised: 4. A RCHITECTURE

Original: Each function described in the previous section we implement using neural networks.
Revised: Each function described in the previous section we implement using neural networks.

Original: For e j α in equation 1 we use a language speciﬁc embedding table followed by a LSTM (Hochreiter & Schmidhuber, 1997).
Revised: For e j α in equation 1 we use a language speciﬁc embedding table followed by a LSTM (Hochreiter & Schmidhuber, 1997).

Original: Function u in equation 1 is simply stacked LSTM’s.
Revised: Function u in equation 1 is simply stacked LSTM’s.

Original: Function h in equation 2 takes input from u as well as a PnP representation of the language via an embedding table.
Revised: Function h in equation 2 takes input from u as well as a PnP representation of the language via an embedding table.

Original: Calculating the real inverse of e − 1 j α is non trivial therefore we use another language speciﬁc LSTM whose outputs we multiply by the transpose of the embedding table of e j α to obtain token probabilities.
Revised: Calculating the real inverse of e − 1 j α is non trivial therefore we use another language speciﬁc LSTM whose outputs we multiply by the transpose of the embedding table of e j α to obtain token probabilities.

Original: For regularization we utilized standard dropout after the embedding layers and layer-wise locked dropout after each LSTM’s layer (Srivastava et al., 2014; Gal & Ghahramani, 2016).
Revised: For regularization we utilized standard dropout after the embedding layers and layer-wise locked dropout after each LSTM’s layer (Srivastava et al., 2014; Gal & Ghahramani, 2016).

Original: The critic, adopting the terminology from Arjovsky et al.
Revised: The critic, adopting the terminology from Arjovsky et al.

Original: (2017), takes the input from u , feeds it through a stacked LSTM, aggregates the hidden states using linear sequence attention as described in DrQA (Chen et al., 2017).
Revised: (2017), takes the input from u , feeds it through a stacked LSTM, aggregates the hidden states using linear sequence attention as described in DrQA (Chen et al., 2017).

Original: Once we have the aggregated state we map to a m × m matrix from where we can compute the total Wasserstein loss.
Revised: Once we have the aggregated state we map to a m × m matrix from where we can compute the total Wasserstein loss.

Original: A Batch Normalization layer is appended to the end of the critic (Ioffe & Szegedy, 2015).
Revised: A Batch Normalization layer is appended to the end of the critic (Ioffe & Szegedy, 2015).

Original: The α, β th index in the matrix correspond to the function output of f in calculating W 1 ( p ( b | j α )
Revised: The α, β th index in the matrix correspond to the function output of f in calculating W 1 ( p ( b | j α )

Original: || p ( b | j β )) .
Revised: || p ( b | j β )) .

Original: 4. T RAINING
Revised: 4. T RAINING

Original: We trained UG-WGAN with a variety of languages depending on the downstream task.
Revised: We trained UG-WGAN with a variety of languages depending on the downstream task.

Original: For each language we utilized the respective Wikipedia dump.
Revised: For each language we utilized the respective Wikipedia dump.

Original: From the wikipedia dump we extract all pages using the wiki2text 1 utility and build language speciﬁc vocabularies consisting of 16k BPE tokens (Sennrich et al., 2015).
Revised: From the wikipedia dump we extract all pages using the wiki2text 1 utility and build language speciﬁc vocabularies consisting of 16k BPE tokens (Sennrich et al., 2015).

Original: During each batch we uniform sample random documents from our set of languages which are approximately the same length, therefore a batch will be mixed with respect to language.
Revised: During each batch we uniform sample random documents from our set of languages which are approximately the same length, therefore a batch will be mixed with respect to language.

Original: We train our language model via BPTT where the truncation length progressively grows from 15 to 50 throughout training.
Revised: We train our language model via BPTT where the truncation length progressively grows from 15 to 50 throughout training.

Original: The critic is updated 10 times for every update of the language model.
Revised: The critic is updated 10 times for every update of the language model.

Original: We trained each language model for 14 days on a NVidia Titan X. For each language model we would do a sweep over λ , but in general we have found that λ = 0 .
Revised: We trained each language model for 14 days on a NVidia Titan X. For each language model we would do a sweep over λ , but in general we have found that λ = 0 .

Original: 1 works sufﬁciently well for minimizing both perplexity and Wasserstein
Revised: 1 works sufﬁciently well for minimizing both perplexity and Wasserstein

Original: distance.
Revised: distance.

Original: 4. E XPLORATION
Revised: 4. E XPLORATION

Original: A couple of interesting questions arise from the described training procedure.
Revised: A couple of interesting questions arise from the described training procedure.

Original: Is the distribution matching constraint necessary or will simple joint language model training exhibit the properties we’re interested in?
Revised: Is the distribution matching constraint necessary or will simple joint language model training exhibit the properties we’re interested in?

Original: Can this optimization process fundamentally learn individual languages grammar while being constrained by a universal channel?
Revised: Can this optimization process fundamentally learn individual languages grammar while being constrained by a universal channel?

Original: What commonalities between languages can we learn and are they informative enough to be exploited?
Revised: What commonalities between languages can we learn and are they informative enough to be exploited?

Original: We can test out the usefulness of the distribution matching constraint by running an ablation study on the λ hyper-parameter.
Revised: We can test out the usefulness of the distribution matching constraint by running an ablation study on the λ hyper-parameter.

Original: We trained UG-WGAN on English, Spanish and Arabic wikidumps following the procedure described above.
Revised: We trained UG-WGAN on English, Spanish and Arabic wikidumps following the procedure described above.

Original: We kept all the hyper-parameters consistent apart for augmenting λ from 0 to 10.
Revised: We kept all the hyper-parameters consistent apart for augmenting λ from 0 to 10.

Original: The results are shown in Figure 2.
Revised: The results are shown in Figure 2.

Original: Without any weight on the distribution matching term the critic trivially learns to separate the various languages and no further training reduces the wasserstein distance.
Revised: Without any weight on the distribution matching term the critic trivially learns to separate the various languages and no further training reduces the wasserstein distance.

Original: The joint language model internally learns individual language models who are partitioned in the latent space.
Revised: The joint language model internally learns individual language models who are partitioned in the latent space.

Original: We can see this by running a t-SNE plot on the universal ( u ( · ) ) representation of our model and seeing existence of clusters of the same language as we did in Figure 3 (Maaten & Hinton, 2008).
Revised: We can see this by running a t-SNE plot on the universal ( u ( · ) ) representation of our model and seeing existence of clusters of the same language as we did in Figure 3 (Maaten & Hinton, 2008).

Original: An universal model satisfying the distribution matching constrain would mix all languages uniformly within it’s latent space.
Revised: An universal model satisfying the distribution matching constrain would mix all languages uniformly within it’s latent space.

Original: To test the universality of UG-WGAN representations we will apply them to a set of orthogonal
Revised: To test the universality of UG-WGAN representations we will apply them to a set of orthogonal

Original: NLP tasks.
Revised: NLP tasks.

Original: We will leave the discussion on the learnability of grammar to the Discussion section of this paper.
Revised: We will leave the discussion on the learnability of grammar to the Discussion section of this paper.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: By introducing a universal channel in our language model we reduced a representations dependence on a single language.
Revised: By introducing a universal channel in our language model we reduced a representations dependence on a single language.

Original: Therefore we can utilize an arbitrary set of languages in training an auxiliary task over UG encodings.
Revised: Therefore we can utilize an arbitrary set of languages in training an auxiliary task over UG encodings.

Original: For example we can train a downstream model only on one languages data and transfer the model trivially to any other language that UG-WGAN was trained on.
Revised: For example we can train a downstream model only on one languages data and transfer the model trivially to any other language that UG-WGAN was trained on.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 5. S ENTIMENT A NALYSIS
Revised: 5. S ENTIMENT A NALYSIS

Original: To test the universality of UG-WGAN representation we ﬁrst trained UG-WGAN in English, Chinese and German following the procedure described in Section 4.
Revised: To test the universality of UG-WGAN representation we ﬁrst trained UG-WGAN in English, Chinese and German following the procedure described in Section 4.

Original: The embedding size of the table was 300 and the internal LSTM hidden size was 512.
Revised: The embedding size of the table was 300 and the internal LSTM hidden size was 512.

Original: A dropout rate of 0 .
Revised: A dropout rate of 0 .

Original: 1 was used and trained with the ADAM optimization method (Kingma & Ba, 2014).
Revised: 1 was used and trained with the ADAM optimization method (Kingma & Ba, 2014).

Original: Since we are interested in the zeroshot capabilities of our representation, we trained our sentiment analysis model only on the English
Revised: Since we are interested in the zeroshot capabilities of our representation, we trained our sentiment analysis model only on the English

Original: IMDB Large Movie Review dataset and tested it on the Chinese ChnSentiCorp dataset and German
Revised: IMDB Large Movie Review dataset and tested it on the Chinese ChnSentiCorp dataset and German

Original: SB-10K (Maas et al., 2011; Tan & Zhang, 2008).
Revised: SB-10K (Maas et al., 2011; Tan & Zhang, 2008).

Original: We binarize the label’s for all the datasets.
Revised: We binarize the label’s for all the datasets.

Original: Our sentiment analysis model ran a bi-directional LSTM on top of ﬁxed UG representations from where we took the last hidden state and computed a logistic regression.
Revised: Our sentiment analysis model ran a bi-directional LSTM on top of ﬁxed UG representations from where we took the last hidden state and computed a logistic regression.

Original: This was trained using standard SGD with momentum.
Revised: This was trained using standard SGD with momentum.

Original: [Table]
Revised: [Table]

Original: We also compare against encodings learned as a by-product of multi-encoder and decoder neural machine translation as a baseline (Klein et al., 2017).
Revised: We also compare against encodings learned as a by-product of multi-encoder and decoder neural machine translation as a baseline (Klein et al., 2017).

Original: We see that UG representations are useful in situations when there is a lack of data in an speciﬁc language.
Revised: We see that UG representations are useful in situations when there is a lack of data in an speciﬁc language.

Original: The language agnostics properties of
Revised: The language agnostics properties of

Original: UG embeddings allows us to do successful zero-shot learning without needing any parallel corpus, furthermore the ability to generalize from language modeling to sentiment attests for the universal properties of these representations.
Revised: UG embeddings allows us to do successful zero-shot learning without needing any parallel corpus, furthermore the ability to generalize from language modeling to sentiment attests for the universal properties of these representations.

Original: Although we aren’t able to improve over the state of the art in a single language we are able to learn a model that does surprisingly well on a set of languages without multilingual data.
Revised: Although we aren’t able to improve over the state of the art in a single language we are able to learn a model that does surprisingly well on a set of languages without multilingual data.

Original: 5. NLI
Revised: 5. NLI

Original: A natural language inference task consists of two sentences; a premise and a hypothesis which are either contradictions, entailments or neutral.
Revised: A natural language inference task consists of two sentences; a premise and a hypothesis which are either contradictions, entailments or neutral.

Original: Learning a NLI task takes a certain nuanced understanding of language.
Revised: Learning a NLI task takes a certain nuanced understanding of language.

Original: Therefore it is of interest whether or not UG-WGAN captures the necessary linguistic features.
Revised: Therefore it is of interest whether or not UG-WGAN captures the necessary linguistic features.

Original: For this task we use the Stanford NLI (sNLI) dataset as our training data in English (Bowman et al., 2015).
Revised: For this task we use the Stanford NLI (sNLI) dataset as our training data in English (Bowman et al., 2015).

Original: To test the zero-shot learning capabilities we created a russian sNLI test set by random sampling 400 sNLI test samples and having a native russian speaker translate both premise and hypothesis to russian.
Revised: To test the zero-shot learning capabilities we created a russian sNLI test set by random sampling 400 sNLI test samples and having a native russian speaker translate both premise and hypothesis to russian.

Original: The label was kept the same.
Revised: The label was kept the same.

Original: For this experiment we trained UG-WGAN on the English and Russian language following the procedure described in Section 4.
Revised: For this experiment we trained UG-WGAN on the English and Russian language following the procedure described in Section 4.

Original: We kept the hyper-parameters equivalent to the Sentiment Analysis experiment.
Revised: We kept the hyper-parameters equivalent to the Sentiment Analysis experiment.

Original: All of the NLI model tested were run over the ﬁxed UG embeddings.
Revised: All of the NLI model tested were run over the ﬁxed UG embeddings.

Original: We trained two different models from literature, Densely-Connected Recurrent and Co-Attentive Network by Kim et al.
Revised: We trained two different models from literature, Densely-Connected Recurrent and Co-Attentive Network by Kim et al.

Original: and Multiway Attention Network by Tan et al.
Revised: and Multiway Attention Network by Tan et al.

Original: Please refer to this papers for further implementation details.
Revised: Please refer to this papers for further implementation details.

Original: [Table]
Revised: [Table]

Original: UG representations contain enough information to non-trivially generalize the NLI task to unseen languages.
Revised: UG representations contain enough information to non-trivially generalize the NLI task to unseen languages.

Original: That being said, we do see a relatively large drop in performance moving across languages which hints that either our calculation of the Wasserstein distance may not be sufﬁciently accurate or the universal representations are biased toward speciﬁc languages or tasks.
Revised: That being said, we do see a relatively large drop in performance moving across languages which hints that either our calculation of the Wasserstein distance may not be sufﬁciently accurate or the universal representations are biased toward speciﬁc languages or tasks.

Original: One hypothesis might be that as we increase λ the cross lingual generalization gap (difference in test error on a task across languages) will vanish.
Revised: One hypothesis might be that as we increase λ the cross lingual generalization gap (difference in test error on a task across languages) will vanish.

Original: To test this hypothesis we conducted the same experiment where UG-WGAN was trained with a λ ranging from 0 to 10 .
Revised: To test this hypothesis we conducted the same experiment where UG-WGAN was trained with a λ ranging from 0 to 10 .

Original: From each of the experiments we picked the model epoch which showed the best perplexity.
Revised: From each of the experiments we picked the model epoch which showed the best perplexity.

Original: The NLI speciﬁc model was the Densely-Connected Recurrent and Co-Attentive Network.
Revised: The NLI speciﬁc model was the Densely-Connected Recurrent and Co-Attentive Network.

Original: Increasing λ doesn’t seem to have a signiﬁcant impact on the generalization gap but has a large impact on test error.
Revised: Increasing λ doesn’t seem to have a signiﬁcant impact on the generalization gap but has a large impact on test error.

Original: Our hypothesis is that a large λ doesn’t provide the model with enough freedom to learn useful representations since the optimizations focus would largely be on minimizing the
Revised: Our hypothesis is that a large λ doesn’t provide the model with enough freedom to learn useful representations since the optimizations focus would largely be on minimizing the

Original: Wasserstein distance, while a small λ permits this freedom.
Revised: Wasserstein distance, while a small λ permits this freedom.

Original: One reason we might be seeing this generalization gap might be due to the way we satisfy the Lipschitz constraint.
Revised: One reason we might be seeing this generalization gap might be due to the way we satisfy the Lipschitz constraint.

Original: It’s been shown that there are better constraints than clipping parameters to a compact space such as a gradient penalty (Gulrajani et al., 2017).
Revised: It’s been shown that there are better constraints than clipping parameters to a compact space such as a gradient penalty (Gulrajani et al., 2017).

Original: This is a future direction that can be explored.
Revised: This is a future direction that can be explored.

Original: =0.
Revised: =0.

Original: =0. =0.
Revised: =0. =0.

Original: =1.
Revised: =1.

Original: =5.
Revised: =5.

Original: =10.0 0% 10% 20% 30% 40% 50% 60%
Revised: =10.0 0% 10% 20% 30% 40% 50% 60%

Original: N L I ( r u ) t e s t
Revised: N L I ( r u ) t e s t

Original: e rr o r
Revised: e rr o r

Original: NLI (ru) test error by =0.
Revised: NLI (ru) test error by =0.

Original: =0. =0.
Revised: =0. =0.

Original: =1.
Revised: =1.

Original: =5.
Revised: =5.

Original: =10.0 0% 10% 20% 30% 40% 50% 60%
Revised: =10.0 0% 10% 20% 30% 40% 50% 60%

Original: C r o ss - L i n g u a l G e n e r a li z a t i o n G a p
Revised: C r o ss - L i n g u a l G e n e r a li z a t i o n G a p

Original: Cross-Lingual Generalization Gap by
Revised: Cross-Lingual Generalization Gap by

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Universal Grammar also comments on the learnability of grammar, stating that statistical information alone is not enough to learn grammar and some form of native language faculty must exist, sometimes titled the poverty of stimulus (POS) argument (Chomsky, 2010; Lewis & Elman, 2001).
Revised: Universal Grammar also comments on the learnability of grammar, stating that statistical information alone is not enough to learn grammar and some form of native language faculty must exist, sometimes titled the poverty of stimulus (POS) argument (Chomsky, 2010; Lewis & Elman, 2001) [Figure]

Original: The goal of our paper is not to make a statement on the Universal Grammar hypothesis.
Revised: The goal of our paper is not to make a statement on the Universal Grammar hypothesis.

Original: But from a machine learning perspective, we’re interested in extracting informative features.
Revised: But from a machine learning perspective [Figure] we’re interested in extracting informative features.

Original: That being said it is of interest to what extent language models capture grammar and furthermore the extent to which models trained toward the universal grammar objective learn grammar.
Revised: That being said it is of interest to what extent language models capture grammar and furthermore the extent to which models trained with our objective learn grammar

Original: [Figure]
Revised: [Figure]

Original: One way to measure universality is by studying perplexity of our multi-lingual language model as we increase the number of languages.
Revised: One way to measure universality is by studying perplexity of our multi-lingual language model as we increase the number of languages.

Original: To do so we trained 6 UG-WGAN models on the following languages: English, Russian, Arabic, Chinese, German, Spanish, French.
Revised: To do so we trained 6 UG-WGAN models on the following languages: English, Russian, Arabic, Chinese, German, Spanish, French.

Original: We maintain the same procedure as described above.
Revised: We maintain the same procedure as described above.

Original: The hidden size of the language model was increased to 1024 with 16K BPE tokens being used.
Revised: The hidden size of the language model was increased to 1024 with 16K BPE tokens being used.

Original: The ﬁrst model was trained on English Russian, second was trained on English Russian Arabic and so on.
Revised: The ﬁrst model was trained on English Russian, second was trained on English Russian Arabic and so on.

Original: For Arabic we still trained from left to right even though naturally the language is read from right to left.
Revised: For Arabic we still trained from left to right even though naturally the language is read from right to left.

Original: We report the results in Figure 5.
Revised: We report the results in Figure 5.

Original: As we increase the number of languages the perplexity gap between constrained and unconstrained UG-WGAN ( λ = 0 . 0 ) decreases which implies while controlling capacity, our constrained (universal λ = 0 . 1 ) language model, models language (almost) as well as jointly trained language models with no universal constraints ( λ = 0 . 0 ).
Revised: As we increase the number of languages the perplexity gap between constrained and unconstrained UG-WGAN ( λ = 0 . 0 ) decreases which implies while controlling capacity, our constrained (universal λ = 0 . 1 ) language model, models language (almost) as well as jointly trained language models with no universal constraints ( λ = 0 . 0 ).

Original: We see from Figure 2 that perplexity worsens proportional to λ . We explore the differences by sampling sentences from an unconstrained language model and λ = 0 . 1 language model trained towards English and Spanish in Table 3.
Revised: Furthermore, the heatmap in Figure 5 shows the perplexities of UG-WGAN trained on any combination of 2 languages from our set of 7.

Original: In general there is a very small difference between a language model trained with a Universal Grammar objective and one without. The Universal Grammar model tends to make more gender mistakes and mistakes due to Plural-Singular Form in Spanish. In English we saw virtually no fundamental differences between the language models. One explanation of this phenomena comes from the autonomy of syntax argument, which argues that semantics have λ = 0 .λ
Revised: We can treat the perplexities as a loose measure of distance λ = 0 .λ

Original: = 0 .en earth’s oxide is a monopoly that occurs towing of the carbon-booed trunks, resulting in a beam containing of oxygen through the soil, salt, warm waters, and the different proteins.
Revised: = 0 .en earth’s oxide is a monopoly that occurs towing of the carbon-booed trunks, resulting in a beam containing of oxygen through the soil, salt, warm waters, and the different proteins.

Original: the practice of epimatic behaviours may be required in many ways of all non-traditional entities.
Revised: the practice of epimatic behaviours may be required in many ways of all non-traditional entities.

Original: the groove and the products are numeric because they are called ”pressibility” (ms) nutrients containing speciﬁc different principles that are available from the root of their family, including a wide variety of molecular and biochemical elements.
Revised: the groove and the products are numeric because they are called ”pressibility” (ms) nutrients containing speciﬁc different principles that are available from the root of their family, including a wide variety of molecular and biochemical elements.

Original: a state line is a self-government environment for sta- tistical cooperation, which is affected by the monks of canada, the east midland of the united kingdom.
Revised: a state line is a self-government environment for sta- tistical cooperation, which is affected by the monks of canada, the east midland of the united kingdom.

Original: however, compared to the listing of special deﬁni- tions, it has evolved to be congruent with structural introductions, allowing to form the chemical form.
Revised: however, compared to the listing of special deﬁni- tions, it has evolved to be congruent with structural introductions, allowing to form the chemical form.

Original: the vernacular concept of physical law is not as an objection (the whis) but as a universal school.
Revised: the vernacular concept of physical law is not as an objection (the whis) but as a universal school.

Original: es la revista ms reciente vari el manuscrito original- mente por primera vez en la revista publicada en 1994.
Revised: es la revista ms reciente vari el manuscrito original- mente por primera vez en la revista publicada en 1994.

Original: en el municipio real se localiza al mar
Revised: en el municipio real se localiza al mar

Original: del norte y su entorno en escajros alto, con mayor variedad de cclica poblacin en forma de cerca de 1070 km2.
Revised: del norte y su entorno en escajros alto, con mayor variedad de cclica poblacin en forma de cerca de 1070 km2.

Original: de hecho la primera cancin de ”blebe cantas”, pahka zanjiwtryinvined cot de entre clases de fanticas, apareci en el ornitlogo sello triusion, jr., en la famosa publicacin playboy de john allen.
Revised: de hecho la primera cancin de ”blebe cantas”, pahka zanjiwtryinvined cot de entre clases de fanticas, apareci en el ornitlogo sello triusion, jr., en la famosa publicacin playboy de john allen.

Original: fue el ltimo habitantes de suecia, con tres hijos, atasaurus y aminkinano (nuestra).
Revised: fue el ltimo habitantes de suecia, con tres hijos, atasaurus y aminkinano (nuestra).

Original: The names of large predators in charlesosaurus in- clude bird turtles hibernated by aerial ﬁghters and ignored ﬁsh.
Revised: The names of large predators in charlesosaurus in- clude bird turtles hibernated by aerial ﬁghters and ignored ﬁsh.

Original: jaime en veracruz fue llamado papa del conde mayor de valdechio, hijo de diego de ziga.
Revised: jaime en veracruz fue llamado papa del conde mayor de valdechio, hijo de diego de ziga.

Revised: we see that English and Spanish are the most similar languages while Russian and Arabic are the most different, which aligns with intuition.

Revised: We see from Figure 2 that perplexity worsens proportional to λ .

Revised: We explore the differences by sampling sentences from an unconstrained language model and λ = 0 .

Revised: 1 language model trained towards English and Spanish in Table 3.

Original:  no weight on syntax (Higginbotham, 1987).
Revised: In general there is a very small difference between a language model trained with our objective and one without. The constrained model tends to make more gender mistakes and mistakes due to Plural-Singular Form in Spanish. In English we saw virtually no fundamental differences between the language models. One explanation of this phenomena comes from the autonomy of syntax argument, which argues that semantics have no weight on syntax (Higginbotham, 1987).

Original: Our hypothesis is that both models learn syntax well, but the models with better perplexity generate sentences with better or clearer semantic meaning.
Revised: Our hypothesis is that both models learn syntax well, but the models with better perplexity generate sentences with better or clearer semantic meaning.

Original: Although completely learning grammar from statistical signals might be improbable, we can still extract useful information.
Revised: Although completely learning grammar from statistical signals might be improbable, we can still extract useful information.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we introduced an unsupervised approach toward learning language agnostic universal representations by formalizing Universal Grammar as an optimization problem.
Revised: In this paper we introduced an unsupervised approach toward learning language agnostic universal representations by taking inspiration from the Universal Grammar hypothesis.

Original: We showed that we can use these representations to learn tasks in one language and automatically transfer them to others with no additional training.
Revised: We showed that we can use these representations to learn tasks in one language and automatically transfer them to others with no additional training.

Original: Furthermore we studied the importance of the Wasserstein constraint through the λ hyper-parameter.
Revised: Furthermore we studied the importance of the Wasserstein constraint through the λ hyper-parameter.

Original: And lastly we explored the difference between a standard multi-lingual language model and UG-WGAN by studying the generated outputs of the respective language models as well as the perplexity gap growth with respect to the number of languages.
Revised: And lastly we explored the difference between a standard multi-lingual language model and UG-WGAN by studying the generated outputs of the respective language models as well as the perplexity gap growth with respect to the number of languages.