Original: D E FINE : DE EP F ACTORIZED IN PUT W ORD E MBEDDINGS FOR N EURAL S EQUENCE M ODELING
Revised: D E FINE : DE EP F ACTORIZED IN PUT T OKEN E MBEDDINGS FOR N EURAL S EQUENCE M ODELING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: For sequence models with largeword-level vocabularies, a majority of network parameters lie in the input and output layers.
Revised: For sequence models with large vocabularies, a majority of network parameters lie in the input and output layers.

Original: In this work, we describe a new method, DeFINE , for learning deep word-level representations efﬁciently.
Revised: In this work, we describe a new method, DeFINE , for learning deep token representations efﬁciently.

Original: Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods.
Revised: Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods.

Original: DeFINE can be incorporated easily in new or existing sequence models.
Revised: DeFINE can be incorporated easily in new or existing sequence models.

Original: Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity.
Revised: Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity.

Original: On WikiText-103,
Revised: On WikiText-103,

Original: DeFINE reduces  total parameters of Transformer-XL by half with minimal impact on performance.
Revised: DeFINE reduces the total parameters of Transformer-XL by half with minimal impact on performance.

Original: On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters, achieving comparable performance to state-of-the-art methods with fewer parameters.
Revised: On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters, achieving comparable performance to state-of-the-art methods with fewer parameters.

Original: For machine translation, DeFINE improves the efﬁciency of the Transformer model by about 1 .times while delivering similar performance.
Revised: For machine translation, DeFINE improves the efﬁciency of the Transformer model by about 1 .times while delivering similar performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural models for NLP tasks, such as language modeling and machine translation, require large vocabularies for generality (Chelba et al., 2013; Bahdanau et al., 2015; Luong et al., 2015; Merity et al., 2017).
Revised: Neural models for NLP tasks, such as language modeling and machine translation, require large vocabularies for generality (Chelba et al., 2013; Bahdanau et al., 2015; Luong et al., 2015; Merity et al., 2017).

Original: These models often employ a similar architecture:  words,  represented as one-hot vectors, are mapped to a dense continuous space; they are then processed by a context model; ﬁnally, the contextualized representations are mapped back to a vocabulary-sized vector for computing next-token probabilities.
Revised: These models often employ a similar architecture: tokens (e g., words, sub-words, or characters), represented as one-hot vectors, are mapped to a dense continuous space; they are then processed by a context model; ﬁnally, the contextualized representations are mapped back to a vocabulary-sized vector for computing next-token probabilities.

Original: A language modeling example is shown in Figure 1a.
Revised: A language modeling example is shown in Figure 1a.

Original: The mapping in the ﬁrst and last steps often uses a shared learned look-up table, referred to as an embedding layer, which takes every word in the vocabulary to a ﬁxed m -dimensional vector.
Revised: The mapping in the ﬁrst and last steps often uses a shared learned lookup table, referred to as an embedding layer, which takes every token in the vocabulary to a ﬁxed m -dimensional vector.

Original: One drawback of this approach is that the number of parameters in the embedding layer increases as the vocabulary size grows, limiting us to small values of m over large vocabularies.
Revised: One drawback of this approach is that the number of parameters in the embedding layer increases as the vocabulary size grows, limiting us to small values of m over large vocabularies.

Original: Researchers have sought to improve the efﬁciency of the embedding layer by assigning lower frequency words smaller dimensional vectors, however, signiﬁcant parameter reductions come at the cost of performance (Morin & Bengio, 2005; Grave et al., 2017a; Baevski & Auli, 2019).
Revised: Researchers have sought to improve the efﬁciency of the embedding layer by assigning lower frequency tokens smaller dimensional vectors, however, signiﬁcant parameter reductions come at the cost of performance (Morin & Bengio, 2005; Grave et al., 2017a; Baevski & Auli, 2019).

Original: In all these approaches, word embedding is approximated with a linear function from words to vectors.
Revised: In all these approaches, token embedding is approximated with a linear function from tokens to vectors.

Original: In this work, we introduce DE ep F actorized IN put word E mbeddings ( DeFINE ) for neural sequence modeling.
Revised: In this work, we introduce De ep F actorized IN put token E mbeddings ( DeFINE ) for neural sequence modeling.

Original: DeFINE approximates the complicated word embedding function with far fewer parameters compared to standard methods.
Revised: DeFINE approximates the complicated token embedding function with far fewer parameters compared to standard methods.

Original: DeFINE allows for lower-dimensional input and output mappings in sequence models, reducing their computational burden without reducing performance.
Revised: DeFINE allows for lower-dimensional input and output mappings in sequence models, reducing their computational burden without reducing performance.

Original: The representations produced by DeFINE are more powerful than those of other factorization techniques and even standard embedding layers.
Revised: The representations produced by DeFINE are more powerful than those of other factorization techniques and even standard embedding layers.

Original: To accomplish this, DeFINE leverages a hierarchical group transformation ( HGT ) that learns deep representations efﬁciently and effectively.
Revised: To accomplish this, DeFINE leverages a hierarchical group transformation ( HGT ) that learns deep representations efﬁciently and effectively.

Original: HGT connects different subsets of the input using sparse and dense connections.
Revised: HGT connects different subsets of the input using sparse and dense connections.

Original: To improve the ﬂow of information, DeFINE introduces a new skip-connection that establishes a direct link with the input layer at every level of its hierarchy, allowing gradient to ﬂow back directly to the input via multiple paths.
Revised: To improve the ﬂow of information, DeFINE introduces a new skip-connection that establishes a direct link with the input layer at every level of its hierarchy, allowing gradients to ﬂow back directly to the input via multiple paths.

Original: DeFINE replaces standardword embedding layers, leaving the rest of the model untouched, and  so it can be used with a wide variety of sequence modeling architectures.
Revised: DeFINE replaces standard embedding layers, leaving the rest of the model untouched, and [Figure] so it can be used with a wide variety of sequence modeling architectures and token-types, including words and sub-words.

Original: Figure 1 shows how we incorporate DeFINE with Transformer-XL (Dai et al., 2019), a state-of-the-art Transformer-based language model and the resulting reduction in total parameters.[Figure]
Revised: Figure 1 shows how we incorporate DeFINE with Transformer-XL (Dai et al., 2019), a state-of-the-art Transformer-based language model, and the resulting reduction in total parameters.

Original: Our experiments show that both LSTM- and Transformer-based sequence models beneﬁt from the use of DeFINE .
Revised: Our experiments show that both LSTM- and Transformer-based sequence models beneﬁt from the use of DeFINE .

Revised: Furthermore, our experiments with word-level language modeling and sub-word level machine translation tasks show that DeFINE can be used with different token types.

Original: On the Wikitext-103 dataset, an LSTM-based language model with DeFINE provides a 9 point improvement over a full capacity model while using half as many parameters.
Revised: On the Wikitext-103 dataset, an LSTM-based language model with DeFINE provides a 9 point improvement over a full capacity model while using half as many parameters.

Original: When combined with adaptive input (Baevski & Auli, 2019) and output (Grave et al., 2017a) representations, DeFINE improves the performance by about 3 points across LSTM-based (see Table 1a) and 
Revised: When combined with adaptive input (Baevski & Auli, 2019) and output (Grave et al., 2017a) representations, DeFINE improves the performance by about 3 points across LSTM-based (see Table 1a) and Transformer-XL-based (see

Original: Transformer-XL-based (see Table 2) language models with a minimal increase in training parameters.
Revised: Table 2) language models with a minimal increase in training parameters.

Original: Computation time at inference is unaffected.
Revised: Computation time at inference is unaffected.

Original:  Incorporating DeFINE into the popular AWDLSTM language model (Merity et al., 2018b) without ﬁnetuning results in a test perplexity of 54. on the Penn Treebank dataset, outperforming both the original and ﬁne-tuned AWD-LSTM models as well as Transformer-XL and MoS (Yang et al., 2018).
Revised: 1 Incorporating DeFINE into the popular AWD-LSTM language model (Merity et al., 2018b) without ﬁnetuning results in a test perplexity of 54.2 on the Penn Treebank dataset, outperforming both the original and ﬁne-tuned AWD-LSTM models as well as Transformer-XL and MoS (Yang et al., 2018).

Original: For machine translation, DeFINE improves the efﬁciency of a Transformer model (Vaswani et al., 2017) by 26% while maintaining translation quality.
Revised: For machine translation, DeFINE improves the efﬁciency of a Transformer model (Vaswani et al., 2017) by 26% while maintaining translation quality.

Original: We provide substantive experiments which detail the impact of our architecture decisions and demonstrate the effectiveness of DeFINE across models of varying capacities.
Revised: We provide substantive experiments which detail the impact of our architecture decisions and demonstrate the effectiveness of DeFINE across models of varying capacities.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Many sequence modeling tasks – including language modeling and machine translation – have a large vocabulary.
Revised: Many sequence modeling tasks, including language modeling and machine translation, have a large vocabulary.

Original: As a consequence, the majority of a model’s parameters are located in the input (or embedding) and the output (or classiﬁcation) layers.
Revised: As a consequence, the majority of a model’s parameters are located in the input (or embedding) and the output (or classiﬁcation) layers.

Original: To reduce the computational load presented by these layers, Press & Wolf (2017) and Inan et al.
Revised: To reduce the computational load presented by these layers, Press & Wolf (2017) and Inan et al.

Original: (2017) introduce an effective mechanism called weight-tying that enables learning input and output representations jointly while signiﬁcantly reducing the number of network parameters.
Revised: (2017) introduce an effective mechanism called weight-tying that enables learning input and output representations jointly while signiﬁcantly reducing the number of network parameters.

Original: To further reduce the computational load from these layers, factorization-based methods, such as projective embeddings (Acharya et al., 2019; Shu & Nakayama, 2017; Dai et al., 2019), grouped embeddings (Chen et al.,2016; 2018; Grave et al., 2017a; Goodman, 2001; Mnih & Hinton, 2009; Morin & Bengio, 2005), and slim embeddings (Li et al., 2018), have been proposed.
Revised: To further reduce the computational load from these layers, factorization-based methods, such as projective embeddings (Dai et al., 2019), grouped embeddings (Chen et al., 2018; Grave et al., 2017a; Goodman, 2001; Mnih & Hinton, 2009; Morin & Bengio, 2005), and slim embeddings (Li et al., 2018), have been proposed.

Original: Projective embeddings approximate a large embedding matrix with two smaller matrices while grouped embeddings cluster input tokens by frequency and assign different capacities to different clusters using projective embedding methods.
Revised: Projective embeddings approximate a large embedding matrix with two smaller matrices while grouped embeddings cluster input tokens by frequency and assign different capacities to different clusters using projective embedding methods.

Original: We note that projective embeddings is a special case of grouped embeddings when the number of clusters is one.
Revised: We note that projective embeddings is a special case of grouped embeddings when the number of clusters is one.

Original: The adaptive input method of Baevski & Auli (2019) generalizes projective and grouped embedding methods and proposes a factorization method that allows for faster, memory-efﬁcient end-to-end training while providing similar or better beneﬁts compared to existing post-training methods which require a pretrained embedding matrix (Chen et al., 2018; Shu & Nakayama, 2017).
Revised: The adaptive input method of Baevski & Auli (2019) generalizes projective and grouped embedding methods and proposes a factorization method that allows for faster, memory-efﬁcient end-to-end training while providing similar or better beneﬁts compared to existing post-training methods which require a pretrained embedding matrix (Chen et al., 2018).

Original: Unlike projective and grouped embeddings, Li et al.
Revised: Unlike projective and grouped embeddings, Li et al.

Original: (2018) extends group transformation (Kuchaiev &Ginsburg,
Revised: (2018) extends group transformation (Kuchaiev &

Original:  2017; Mehta et al., 2018) with the shufﬂing algorithm of Fisher & Yates (1943) to factorize these layers.
Revised: Ginsburg, 2017; Mehta et al., 2018) with the shufﬂing algorithm of Fisher & Yates (1943) to factorize these layers.

Revised: Other techniques such as codebook learning (Shu & Nakayama, 2017; Chen et al., 2016; Acharya et al., 2019) and quantization (Rastegari et al., 2016; Hubara et al., 2017) can be used to further improve efﬁciency, especially in terms of storage requirements.

Original: DeFINE is orthogonal to thesefactorization methods; our empirical results in Sectionshow improved performance compared to these methods alone.
Revised: DeFINE is orthogonal to these methods; our empirical results in Section 4 show improved performance compared to these methods alone.

Original: Recent advances in sequence modeling, such as Transformers and multi-layer RNNs, demonstrate the power of deep architectures in NLP (Jozefowicz et al., 2016; Vaswani et al., 2017; Merity et al., 2018a).
Revised: Recent advances in sequence modeling, such as Transformers and multi-layer RNNs, demonstrate the power of deep architectures in NLP (Jozefowicz et al., 2016; Vaswani et al., 2017; Merity et al., 2018a).

Original: But while signiﬁcant attention has been given to modeling the interactions between words with deep architectures (e.g. ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019)), contextfree word representations are typically modeled with only corpus statistics (Pennington et al., 2014) or a single linear transformation (Mikolov et al., 2013; McCann et al., 2017).
Revised: But while signiﬁcant attention has been given to modeling the interactions between tokens with deep architectures (e.g., ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019)), contextfree token representations are typically modeled with only corpus statistics (Pennington et al., 2014) or a single linear transformation (Mikolov et al., 2013; McCann et al., 2017).

Original: Character-level models (Kim et al., 2016) also effect deep representations of words as a convolution over characters, however these models often require more capacity to deliver performance comparable to word-level models (Baevski & Auli, 2019).
Revised: Character-level models (Kim et al., 2016) also effect deep representations of words as a convolution over characters, however these models often require more capacity to deliver performance comparable to word- or sub-wordlevel models (Baevski & Auli, 2019).

Original: Still, DeFINE can be used to learn deep representations of a variety of token types, including words, characters, or byte-pair encodings (Sennrich et al., 2015).
Revised: Still, DeFINE can be used to learn deep representations of a variety of token types, including words, characters, or sub-words (byte-pair encodings) (Sennrich et al., 2015).

Original:  D E FINE
Revised:  D E FINE

Original: Word embedding is often treated as simple function of a one-hot vector to a dense continuous space.
Revised: Token embedding is often treated as simple function of a one-hot vector to a dense continuous space.

Original: The embedding layer can thus be thought of as a wide, shallow network consisting of a single linear transformation.
Revised: The embedding layer can thus be thought of as a wide, shallow network consisting of a single linear transformation.

Original: At its heart, the function that this network approximates (call it f ) takes a word from its orthographic form to a representation of those of its syntactic and semantic properties which are relevant for modeling an arbitrary number of contexts in which the word can occur.
Revised: At its heart, the function that this network approximates (call it f ) takes a token from its orthographic form to a representation of those of its morphosyntactic and semantic properties which are relevant for modeling an arbitrary number of contexts in which the token can occur.

Original: Most NLP research assumes a simple embedding layer can sufﬁciently approximate the intractable function f .
Revised: Most NLP research assumes a simple embedding layer can sufﬁciently approximate the intractable function f .

Original: We hypothesize that, due to the complexity of f , a shallow network would require exceptional capacity to learn a good approximation.
Revised: We hypothesize that, due to the complexity of f , a shallow network would require exceptional capacity to learn a good approximation.

Original: Time and data constraints prohibit learning such a high capacity shallow network.
Revised: Time and data constraints prohibit learning such a high capacity shallow network.

Original: We propose, based on recent theoretical results of Liang & Srikant (2017), 2 that a deeper network can approximate f with signiﬁcantly fewer parameters than a shallow network.
Revised: We propose, based on recent theoretical results of Liang & Srikant (2017), 2 that a deeper network can approximate f with signiﬁcantly fewer parameters than a shallow network.

Original: The validity of this assumption is evidenced by our experimental results in Section 4.
Revised: The validity of this assumption is evidenced by our experimental results in Section 4.

Original: In this work, we introduce DeFINE , an effective way of learning deep word-level representations in high-dimensional space with a minimum of additional parameters.
Revised: In this work, we introduce DeFINE , an effective way of learning deep token representations in high-dimensional space with a minimum of additional parameters.

Original: Our method is based on a Map-Expand-Reduce ( MER ) principle, described in Section 3.1, that ﬁrst maps an input word to a low dimensional embedding vector, then transforms it to a high-dimensional space using a computationally efﬁcient hierarchical group transformation ( HGT , Section 3.2), which is sketched in Figure 2c.
Revised: Our method is based on a MapExpand-Reduce ( MER ) principle, described in Section 3.1, that ﬁrst maps an input token to a low dimensional embedding vector, then transforms it to a high-dimensional space using a computationally efﬁcient hierarchical group transformation ( HGT , Section 3.2), which is sketched in Figure 2c.

Original: The resultant vector is then transformed to a low-dimensional space.
Revised: The resultant vector is then transformed to a low-dimensional space.

Original: Over the course of these transformations, we make use of a new connectivity pattern that establishes a direct link between the input and output layers (Figure 3), promoting feature reuse, and improving gradient ﬂow (Section 3.3).
Revised: Over the course of these transformations, we make use of a new connectivity pattern that establishes a direct link between the input and output layers (Figure 3), promoting feature reuse, and improving gradient ﬂow (Section 3.3).

Original: The output layer of DeFINE can then be used in place of a traditional embedding as an input to sequence modeling tasks.
Revised: The output layer of DeFINE can then be used in place of a traditional embedding as an input to sequence modeling tasks.

Original: We detail the various aspects of the architecture below.
Revised: We detail the various aspects of the architecture below.

Original: 3. T HE M AP -E XPAND -R EDUCE P RINCIPLE ( MER )
Revised: 3. T HE M AP -E XPAND -R EDUCE P RINCIPLE ( MER )

Original: The ﬁrst step in MER , M ap, is similar to standard sequence models.
Revised: The ﬁrst step in MER , M ap, is similar to standard sequence models.

Original: Every input word in the vocabulary V is mapped to a ﬁxed dimensional vector e
Revised: Every input token in the vocabulary V is mapped to a ﬁxed dimensional vector e

Original: i ∈ R n × 1 .
Revised: i ∈ R n × 1 .

Original: However, in our case, the value of n is small (say 64 or 128, compared to typical dimensions of 400 or more).
Revised: However, in our case, the value of n is small (say 64 or 128, compared to typical dimensions of 400 or more).

Original: The next step, E xpand,takes e i as an input and applies a hierarchical group transformation ( HGT ) to produce a very highdimensional vector ˆe i ∈ R k × 1 , where k >> n . Unlike a stack of fully connected layers, HGT learns deep representations efﬁciently from different subsets of the input using sparse and dense connections.
Revised: The next step, E xpand,

Original: The last step, R educe, projects the vector ˆe i to a lower dimensional space to produce the ﬁnal embedding vector e o ∈ R m × 1 for a given input word.

Original: The dimensions of e o can be matched to contextual representation models, such as LSTMs or Transformers, allowing DeFINE to serve as an input layer for these models.

Original: (cid:1) (cid:1)
Revised: (cid:1) (cid:1)

Original: Expand
Revised: Expand

Original: Map (a) LT (cid:1) (cid:1) (b) GLT (cid:1) (cid:1) (c) HGT
Revised: Map (a) LT (cid:1) (cid:1) (b) GLT (cid:1) (cid:1) (c) HGT

Original: Layer # parameters
Revised: Layer # parameters

Original: LT
Revised: LT

Original: N − 1 (cid:80) l =0 n l k l
Revised: N − 1 (cid:80) l =0 n l k l

Original: GLT
Revised: GLT

Original: N − 1 (cid:80) l =0 n l k l /g
Revised: N − 1 (cid:80) l =0 n l k l /g

Original: HGT
Revised: HGT

Original: N − 1 (cid:80) l =0 n l k l /g l (d) 
Revised: N − 1 (cid:80) l =0 n l k l /g l (d) takes e i as an input and applies a hierarchical group transformation ( HGT ) to produce a very highdimensional vector ˆe i ∈ R k × 1 , where k >> n . Unlike a stack of fully connected layers, HGT learns deep representations efﬁciently from different subsets of the input using sparse and dense connections.

Revised: The last step, R educe, projects the vector ˆe i to a lower dimensional space to produce the ﬁnal embedding vector e o ∈ R m × 1 for a given input token.

Revised: The dimensions of e o can be matched to contextual representation models, such as LSTMs or Transformers, allowing DeFINE to serve as an input layer for these models.

Original: 3. H IERARCHICAL GROUP TRANSFORMATION ( HGT )
Revised: 3. H IERARCHICAL GROUP TRANSFORMATION ( HGT )

Original: We introduce a hierarchical group transformation ( HGT ), sketched in Figure 2c, to learn deep wordlevel representations efﬁciently.
Revised: We introduce a hierarchical group transformation ( HGT ), sketched in Figure 2c, to learn deep token representations efﬁciently.

Original: HGT comprises of a stack of N layers.
Revised: HGT comprises of a stack of N layers.

Original: At each layer, HGT uses a different number of groups that allow it learn representations from different subsets of input.
Revised: At each layer, HGT uses a different number of groups that allows it learn representations from different subsets of input.

Original: HGT starts with g max groups at the ﬁrst layer and then subsequently decreases the number of groups by a factor of 2 at each level.
Revised: HGT starts with g max groups at the ﬁrst layer and then subsequently decreases the number of groups by a factor of 2 at each level.

Original: This hierarchical grouping mechanism sparsiﬁes the connections in fully connected (or linear) layers and allows us to learn representations efﬁciently with fewer parameters.
Revised: This hierarchical grouping mechanism sparsiﬁes the connections in fully connected (or linear) layers and allows us to learn representations efﬁciently with fewer parameters.

Original: Similar to a stack of fully connected layers, the N -th layer in HGT has access to every input element of the ﬁrst layer through multiple paths, thereby, allowing it to learn effective representations.
Revised: Similar to a stack of fully connected layers, the N -th layer in HGT has access to every input element of the ﬁrst layer through multiple paths, thereby, allowing it to learn effective representations.

Original: Group linear transformations (GLT), originally introduced to improve the efﬁciency of the LSTM, also sparsify the connections in fully connected layers and signiﬁcantly reduce computational costs (Kuchaiev & Ginsburg, 2017; Mehta et al., 2018).
Revised: Group linear transformations (GLT), originally introduced to improve the efﬁciency of the LSTM, also sparsify the connections in fully connected layers and signiﬁcantly reduce computational costs (Kuchaiev & Ginsburg, 2017; Mehta et al., 2018).

Original: However, if we stack multiple GLT layers, the outputs of  certain group are only derived from a small fraction of the input, thus learning weak representations.
Revised: However, if we stack multiple GLT layers, the outputs of a certain group are only derived from a small fraction of the input, thus learning weak representations.

Original: The hierarchical grouping mechanism in HGT allows the N -th layer to obtain input data from multiple paths, enabling HGT to learn stronger representations.
Revised: The hierarchical grouping mechanism in HGT allows the N -th layer to obtain input data from multiple paths, enabling HGT to learn stronger representations.

Original: A comparison of different transformations is given in Figure 2.
Revised: A comparison of different transformations is given in Figure 2.

Original: We can see that HGT is both efﬁcient and has better access to the input.
Revised: We can see that HGT is both efﬁcient and has better access to the input.

Original: Note that linear and group linear transforms are special cases of HGT when g l = 1 and g l = g (ﬁxed), respectively.
Revised: Note that linear and group linear transforms are special cases of HGT when g l = 1 and g l = g (ﬁxed), respectively.

Original: To transform e i ∈ R n × 1 to ˆe i ∈
Revised: To transform e i ∈ R n × 1 to ˆe i ∈

Original: R k × 1 , HGT
Revised: R k × 1 , HGT

Original: ﬁrst samples the space between n and k linearly to construct N intermediate layers of increasing dimensionality.
Revised: ﬁrst samples the space between n and k linearly to construct N intermediate layers of increasing dimensionality.

Original: Therefore, the output vector produced by l + 1 -th layer will have higher dimensionality than the l -th layer.
Revised: Therefore, the output vector produced by l + 1 -th layer will have higher dimensionality than the l -th layer.

Original: Assume that the linearly spaced vector dimensions are divisible by g max , we transform e i to ˆe i as follows:
Revised: Assume that the linearly spaced vector dimensions are divisible by g max , we transform e i to ˆe i as follows:

Original: [Equation]
Revised: [Equation]

Original: where g l = max (cid:0) (cid:98) g max 2 l − 1 (cid:99) , 1 (cid:1) , W l are the weights learned at l -th layer, and F G is a group transformation function deﬁned in Mehta et al.
Revised: where g l = max (cid:0) (cid:98) g max 2 l − 1 (cid:99) , 1 (cid:1) , W l are the weights learned at l -th layer, and F G is a group transformation function deﬁned in Mehta et al.

Original: Group transformation splits the input into g groups, each of which is processed independently using a linear transformation.
Revised: Group transformation splits the input into g groups, each of which is processed independently using a linear transformation.

Original: The output of these groups are then concatenated to produce ﬁnal output.
Revised: The output of these groups are then concatenated to produce ﬁnal output.

Original: See Section A.1 for details.
Revised: See Section A.1 for details.

Original: 3. D E FINE UNIT
Revised: 3. D E FINE UNIT

Original: The DeFINE unit is composed of HGT transformations that are designed using the MER principle.
Revised: The DeFINE unit is composed of HGT transformations that are designed using the MER principle.

Original: Though HGT layers are an efﬁcient approximation to computationally expensive fully connected  layers, they might impede training as the depth N of the DeFINE unit grows.
Revised: Though HGT layers are an efﬁcient approximation to computationally expensive fully connected [Figure] layers, they might impede training as the depth N of the DeFINE unit grows.

Original: Residual connections (He et al., 2016) have proved to be very effective at mitigating this issue, however, such connections are difﬁcult to implement in HGT because the input and output dimensions of each layer are different.
Revised: Residual connections (He et al., 2016) have proved to be very effective at mitigating this issue, however, such connections are difﬁcult to implement in HGT because the input and output dimensions of each layer are different.

Original: To maximize the ﬂow of information and facilitate training with deeper DeFINE units, we introduce a simple new skip-connection that establishes a direct link between any layer in HGT with the input e i .
Revised: To maximize the ﬂow of information and facilitate training with deeper DeFINE units, we introduce a simple new skip-connection that establishes a direct link between any layer in HGT with the input e i .

Original: Figure 3 visualizes the DeFINE unit with a depth of two ( N = 2 ).
Revised: Figure 3 visualizes the DeFINE unit with a depth of two ( N = 2 ).

Original: To enable the sparse connections in HGT to have access to the input e i and the output of the previous layer ( ˆe l − 1 i ), we chunk[Figure] the input and the output into g l groups using a split layer .
Revised: To enable the sparse connections in HGT to have access to the input e i and the output of the previous layer ( ˆe l − 1 i ), we chunk the input and the output into g l groups using a split layer .

Original: The chunked input and output vectors are then mixed such that the ﬁrst chunk of the input and the ﬁrst chunk of the l − 1 -th layer’s output are put together as the input for the ﬁrst group transformation in the l -th layer, and so on until g l inputs have been constructed.
Revised: The chunked input and output vectors are then mixed such that the ﬁrst chunk of the input and the ﬁrst chunk of the l − 1 -th layer’s output are put together as the input for the ﬁrst group transformation in the l -th layer and so on until g l inputs have been constructed.

Original: The resultant vector is then fed to  l -th layer.
Revised: The resultant vector is then fed to the l -th layer.

Original: This mechanism promotes input feature reuse efﬁciently.
Revised: This mechanism promotes input feature reuse efﬁciently.

Original: Additionally, it establishes a direct link with the input e i , allowing gradient to ﬂow back to the input via multiple paths and resulting in improved performance.
Revised: Additionally, it establishes a direct link with the input e i , allowing gradients to ﬂow back to the input via multiple paths and resulting in improved performance.

Original: 3. D E FINE FOR S EQUENCE M ODELING
Revised: 3. D E FINE FOR S EQUENCE M ODELING

Original: The DeFINE unit can be easily integrated with any new or existing sequence models.
Revised: The DeFINE unit can be easily integrated with any new or existing sequence models.

Original: Sequence models typically consist of a stack of an input layer (embedding or adaptive input layer), a contextual model (e.g. LSTM or Transformer), and a classiﬁcation layer (a fully-connected or adaptive softmax).
Revised: Sequence models typically consist of a stack of an input layer (embedding or adaptive input layer), a contextual model (e.g., LSTM or Transformer), and a classiﬁcation layer (a fully-connected or adaptive softmax).

Original: Since DeFINE learns deep word-level representations, we can easily stack it immediately after the input.
Revised: Since DeFINE learns deep token representations, we can easily stack it immediately after the input.

Original: An example is shown in Figure 1, where DeFINE is integrated with TransformerXL, a state-of-the-art language model.
Revised: An example is shown in Figure 1, where DeFINE is integrated with Transformer-XL, a state-of-the-art language model.

Original: DeFINE enables the use of relatively lower dimensions in the input layer, thus reducing network parameters.
Revised: DeFINE enables the use of relatively lower dimensions in the input layer, thus reducing network parameters.

Original: The input word-level representations, e i , ˆe i , and e o , that a neural model learns for each word are independent of other words.
Revised: The input token representations, e i , ˆe i , and e o , that a neural model learns for each token are independent of other tokens.

Original: This allows us to create another independent look-up table (after training a model) that caches the mapping between the input word and the output of the DeFINE unit ( e o ), resulting in a mechanism that allows to skip the computations of the DeFINE unit at inference time.
Revised: This allows us to create another independent look-up table (after training a model) that caches the mapping between the input token and the output of the DeFINE unit ( e o ), resulting in a mechanism that allows to skip the computations of the DeFINE unit at inference time.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We demonstrate the performance of DeFINE on two sequence modeling tasks: language modeling (Section 4.1) and machine translation (Section 4.2).
Revised: We demonstrate the performance of DeFINE on two sequence modeling tasks: language modeling (Section 4.1) and machine translation (Section 4.2).

Original: We compare the performance of DeFINE with existing factorization  methods in Section 4.3.
Revised: We compare the performance of DeFINE with existing factorization and compression-based methods in Section 4.3.

Original: We also provide ablations in Section 4.4 to show the effectiveness of our design decisions. 
Revised: We also provide ablations in Section 4.4 to show the effectiveness of our design decisions. Throughout this section, we use the [Table]

Revised: based language model (ours) on WT

Revised: [Table]

Original: Throughout this section, we use the following notation: n , k , and m are dimensions of e i , ˆe i , and e o respectively, and N represents depth of DeFINE .
Revised: For this experiment, we use two GPUs [Table] [Table] following notation: n , k , and m are dimensions of e i , ˆe i , and e o respectively, and N represents depth of DeFINE .

Original: 4. L ANGUAGE M ODELING
Revised: 4. L ANGUAGE M ODELING

Original: In this section, we study the performance of our models with LSTM- and Transformer-based language models on two datasets: WikiText-

Revised: In this section, we study the performance of our models with LSTM- and Transformer-based language models on two datasets: WikiText-

Original: (Merity et al., 2017) and the Penn Treebank (Marcus [Table]
Revised: (Merity et al., 2017) and the Penn Treebank (Marcus et al., 1994).

Original: LSTM-based language model (ours) on WT103.

Original: [Table]

Original: For this experiment, we use two GPUs.

Original: [Table]

Original: [Table]

Original: et al., 1994).

Original: On both datasets, we show that DeFINE is parameter efﬁcient and improves the performance of existing language models.
Revised: On both datasets, we show that DeFINE is parameter efﬁcient and improves the performance of existing language models.

Original: [Equation]
Revised: [Equation]

Original: Data and models: The WikiText-103 dataset (Merity et al., 2017) consists of 103M/217K/245K tokens for training, validation, and test  respectively and has a vocabulary size of about 260K.
Revised: Data and models: The WikiText-103 dataset (Merity et al., 2017) consists of 103M/217K/245K tokens for training, validation, and test sets respectively and has a vocabulary size of about 260K.

Original: This dataset is composed of Wikipedia articles and retains punctuation, numbers, and case.
Revised: This dataset is composed of Wikipedia articles and retains punctuation, numbers, and case.

Original: To evaluate the effectiveness of DeFINE , we study two different kinds of contextual models: LSTM, and Transformer (Transformer-XL (Dai et al., 2019)).
Revised: To evaluate the effectiveness of DeFINE , we study two different kinds of contextual models: LSTM, and Transformer (Transformer-XL (Dai et al., 2019)).

Original: We measure the performance of these models in terms of perplexity, a standard metric for language modeling.
Revised: We measure the performance of these models in terms of perplexity, a standard metric for language modeling.

Original: Lower values of perplexity indicate better performance.
Revised: Lower values of perplexity indicate better performance.

Original: Following recent works, including Merity et al.
Revised: Following recent works, including Merity et al.

Original: (2018a), Baevski & Auli (2019), and Dai et al.
Revised: (2018a), Baevski & Auli (2019), and Dai et al.

Original: (2019), we use adaptive inputs as a mapping function in DeFINE and adaptive softmax for classiﬁcation with tied weights.
Revised: (2019), we use adaptive inputs as a mapping function in DeFINE and adaptive softmax for classiﬁcation with tied weights.

Original: See A.3 for more details.
Revised: See A.3 for more details.

Original: Results of LSTM -based language models: Table 1 summarizes the results of LSTM-based language models.
Revised: Results of LSTM-based language models: Table 1 summarizes the results of LSTM-based language models.

Original: Though the adaptive input (Baevski & Auli, 2019) and output (Grave et al., 2017a) methods are effective and reduce the number of parameters signiﬁcantly, our method further improves performance by about 3 points while learning only 1.25% (or 0.4 million) more parameters.
Revised: Though the adaptive input (Baevski & Auli, 2019) and output (Grave et al., 2017a) methods are effective and reduce the number of parameters signiﬁcantly, our method further improves performance by about 3 points while learning only 1.25% (or 0.4 million) more parameters.

Original: It is important to note that the computational complexity of models in R2 and R3 is the same because our method allows caching outputs of DeFINE for use at inference (see Section 3.4).
Revised: It is important to note that the computational complexity of models in R2 and R3 is the same because our method allows caching outputs of DeFINE for use at inference (see Section 3.4).

Original: When we scale the depth of DeFINE from 3 to 11 layers (Table 1b) 3 , the performance improves by a further 6 points, delivering competitive performance to existing RNN-based methods with fewer parameters (e.g. 1 / 3 as many parameters as Merity et al.
Revised: When we scale the depth of DeFINE from 3 to 11 layers (Table 1b) 3 , the performance improves by a further 6 points, delivering competitive performance to existing RNN-based methods with fewer parameters (e.g., 1 / 3 as many parameters as Merity et al.

Original: (2018a)).
Revised: (2018a)).

Original: The performance of our model is better than existing methods such as Dauphin et al.
Revised: The performance of our model is better than existing methods such as Dauphin et al.

Original: (2017) and Bai et al.
Revised: (2017) and Bai et al.

Original: Results of Transformer-based model:
Revised: Results of Transformer-based model:

Original: Table 2 compares the performance of Transformer-XL, a state-of-the-art Transformer-based model, with and without DeFINE .
Revised: Table 2 compares the performance of Transformer-XL, a state-of-the-art Transformer-based model, with and without DeFINE .

Original: Table 2a shows our method is able to attain similar performance to Dai et al.
Revised: Table 2a shows our method is able to attain similar performance to Dai et al.

Original: (2019) while learning 10M fewer parameters. 
Revised: (2019) while learning 10M fewer parameters. It

Revised: [Table]

Revised: 4096 , and m = 384 .

Revised: For models without DeFINE , we use projective embeddings (Dai et al., 2019) that linearly projects the vector e i to a dimension of m = 384 .

Revised: Except the row marked with (cid:63) that uses inner model dimension of 2100, all other rows uses an inner model dimension of 1920.

Original: It is interesting to note that DeFINE enables us to reduce the computational burden from the input and output layers by a large amount with minimal impact on performance.
Revised: Best number in each group in Table 2a is highlighted in red while overall best numbers are marked in bold . Table 2a shows that adding DeFINE signiﬁcantly improves results with low overhead; Table 2b shows the parameter reduction using DeFINE for similar performance. is interesting to note that DeFINE enables us to reduce the computational burden from the input and output layers by a large amount with minimal impact on performance.

Original: With DeFINE , the performance of Transformer-XL drops only by about 2 points while the number of parameters are reduced by 50%.
Revised: With DeFINE , the performance of Transformer-XL drops only by about 2 points while the number of parameters are reduced by 50%.

Original: For similar reduction in the number of parameters, the performance of original
Revised: For similar reduction in the number of parameters, the performance of original

Original: Transformer-XL drops by 5 points, suggesting the proposed method for learning word-level representations is effective.
Revised: Transformer-XL drops by 5 points, suggesting the proposed method for learning word-level representations is effective.

Original: Table 2b highlights the fact that Transformer-XL with DeFINE is able to achieve comparable perplexity to a standard Transformer-XL with projective embeddings while using signiﬁcantly fewer parameters.[Table] [Table]
Revised: Table 2b highlights the fact that Transformer-XL with DeFINE is able to achieve comparable perplexity to a standard Transformer-XL with projective embeddings while using signiﬁcantly fewer parameters.

Original: = 4096 , and m = models without DeFINE we projective embeddings (Dai et al., 2019) that linearly vector

Original: Table 2a shows that adding DeFINE signiﬁcantly improves results with low overhead; Table 2b shows the parameter reduction using DeFINE for similar performance.

Original: 4.1. P ENN T REEBANK (PTB)
Revised: 4.1. P ENN T REEBANK (PTB)

Original: Data and models: The Penn Treebank dataset (Marcus et al., 1994) contains about 929K/74K/82K tokens in its train, validation, and test sets respectively.
Revised: Data and models: The Penn Treebank dataset (Marcus et al., 1994) contains about 929K/74K/82K tokens in its train, validation, and test sets respectively.

Original: It has a vocabulary size of about 10K.
Revised: It has a vocabulary size of about 10K.

Original: Following recent works, we use the processed version provided by Mikolov et al. (2010).
Revised: Following recent works, we use the processed version provided by Mikolov et al. (2010).

Original: To evaluate the effectiveness of our model, we compare to AWD-LSTM (Merity et al., 2018b).
Revised: To evaluate the effectiveness of our model, we compare to AWD-LSTM (Merity et al., 2018b).

Original: Our model replaces the embedding layer in AWD-LSTM with DeFINE unit with the following settings: n = 128 , k = 1024 , N = 7 , and m = 400 .
Revised: Our model replaces the embedding layer in AWD-LSTM with DeFINE unit with the following settings: n = 128 , k = 1024 , N = 7 , and m = 400 .

Original: We use the same hyper-parameters and PyTorch version as the original AWD-LSTM.
Revised: We use the same hyper-parameters and PyTorch version as the original AWD-LSTM.

Original: Results: Results are summarized in Table 1c.
Revised: Results: Results are summarized in Table 1c.

Original: The proposed method improves the performance of AWD-LSTM by 4 points while simultaneously reducing the number of parameters by 4 million.
Revised: The proposed method improves the performance of AWD-LSTM by 4 points while simultaneously reducing the number of parameters by 4 million.

Original: Without any ﬁnetuning, AWD-LSTM + DeFINE achieves comparable performance to state-of-theart methods, including Transformer-XL, with fewer parameters.
Revised: Without any ﬁnetuning, AWD-LSTM + DeFINE achieves comparable performance to state-of-theart methods, including Transformer-XL, with fewer parameters.

Original: 4. M ACHINE T RANSLATION
Revised: 4. M ACHINE T RANSLATION

Original: Data and models: We use the WMT 2014 English-German (EN-DE) dataset (Luong et al., 2015) for training.
Revised: Data and models: We use the WMT 2014 English-German (EN-DE) dataset (Luong et al., 2015) for training.

Original: Following Vaswani et al.
Revised: Following Vaswani et al.

Original: (2017), we encodethe sentences using byte-pair encoding (Britz et al., 2017) and use newstest2014 and newstest2017 as validation and test sets, respectively.
Revised: (2017), we encode sentences using byte-pair encoding (Britz et al., 2017) and use newstest2014 and newstest2017 as validation and test sets, respectively.

Original: We integrate DeFINE with the state-of-the-art Transformer model (Vaswani et al., 2017) with following parameters: n = 128 , k = 1024 , m = 512 , and N = 3 .
Revised: We integrate DeFINE with the state-of-the-art Transformer model (Vaswani et al., 2017) with following parameters: n = 128 , k = 1024 , m = 512 , and N = 3 .

Original: We use the implementation in OpenNMT-py (Klein et al., 2017) for training and evaluation with the recommended hyper-parameters. 
Revised: We use the implementation in OpenNMTpy (Klein et al., 2017) for training and evaluation with the recommended hyper-parameters. [Table] [Table]

Original: Results: Table 3 summarizes the results.
Revised: Results: Table 3 summarizes the results.

Original: DeFINE improves the performance of the Transformer model without ensembling by 2% while simultaneously reducing the total number of parameters by 26%, suggesting that DeFINE is effective.[Table] [Table] [Figure]
Revised: DeFINE improves the performance of the Transformer model without checkpoint averaging by 2% while simultaneously reducing the total number of parameters by 26%, suggesting that DeFINE is effective.

Original: DeFINE is able to approximate the standard embedding matrix efﬁciently.

Original: More visualizations are included in Appendix B.

Original: 4. C OMPARISON WITH DIFFERENTFACTORIZATION METHODS
Revised: 4. C OMPARISON WITH DIFFERENT METHODS

Revised: Factorization-based methods

Original:  Table 4 compares the performance of different factorization methods for different sequence models.
Revised: : Table 4 compares the performance of different factorization methods for different sequence models.

Original: With DeFINE , the performance and efﬁciency of sequence models improves across different tasks.
Revised: With DeFINE , the performance and efﬁciency of sequence models improves across different tasks.

Original: This is likely because the output of DeFINE more closely approximates the correlation pattern of a standard embedding layer compared to other embeddings (see Figure 4 and Appendix B).
Revised: This is likely because the output of DeFINE more closely approximates the correlation pattern of a standard embedding layer compared to other embeddings (see Figure 4 and Appendix B).

Original: Furthermore, we see that strong correlations between dimensions in the mapping layer of DeFINE are reduced over the course of the expansion layers (see Figures 8, 9, and 10 in Appendix).
Revised: Furthermore, we see that strong correlations between dimensions in the mapping layer of DeFINE are reduced over the course of the expansion layers (see Figures 8, 9, and 10 in Appendix).

Original: Figurein Appendix shows that groups within an expansion layer of DeFINE are not correlated, suggesting these matrices are learning different representations of their input.
Revised: Figure 11 in Appendix shows that groups within an expansion layer of DeFINE are not correlated, suggesting these matrices are learning different representations of their input.

Original: [Table]
Revised: Impact of compression-based methods: Compression-based methods allow for efﬁciently discretizing the continuous 32-bit full-precision embedding vectors, thus reducing the memory footprint of the input layer. With DeFINE , we also learn a continuous full precision 32-bit ﬂoating-point embedding vector (similar to Baevski & Auli (2019) and Dai et al. Therefore, compressionbased methods, such as (Shu & Nakayama, 2017), can be applied to sequence models with DeFINE and other factorization methods.

Original: [Table] [Table]
Revised: Table 5 shows that DeFINE embeddings can be compressed similarly to standard embeddings without loss of performance.

Original: 4. A BLATION STUDIES ON W IKI T EXT -103 DATASET
Revised: 4. A BLATION STUDIES ON W IKI T EXT -103 DATASET

Original: In this section, we provide an analysis of our design choices using an LSTM-based language model.
Revised: In this section, we provide an analysis of our design choices using an LSTM-based language model.

Original: In our ablations, we choose LSTM- over Transformer-based language models because they are less sensitive to hyper-parameters and can be trained on a single GPU.
Revised: In our ablations, we choose LSTM- over Transformer-based language models because they are less [Figure] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Original: We use the same hyper-parameters for training as described in Section 4.1.1, speciﬁcally N = 7 , n = 384 , k = 1024 , and m = 384 .
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: sensitive to hyper-parameters and can be trained on a single GPU.

Revised: We use the same hyper-parameters for training as described in Section 4.1.1, speciﬁcally N = 7 , n = 384 , k = 1024 , and m = 384 .

Original: Impact of different transformations: Table 5 summarizes our results.
Revised: Impact of different transformations: Table 6 summarizes our results.

Original: HGT is as effective as linear transformation while learning two million fewer parameters.
Revised: HGT is as effective as linear transformation while learning two million fewer parameters.

Original: Compared to group linear transform (GLT), HGT improves perplexity by about 5 points while learning a similar number of parameters.
Revised: Compared to group linear transform (GLT), HGT improves perplexity by about 5 points while learning a similar number of parameters.

Original: Furthermore, when we establish a direct connection with the input (see Section 3.2 for details), the performance further improves by 2.9 points with a minimal impact on number of parameters, suggesting that DeFINE learns good representations.
Revised: Furthermore, when we establish a direct connection with the input (see Section 3.2 for details), the performance further improves by 2.9 points with a minimal impact on number of parameters, suggesting that DeFINE learns good representations.

Original: Impact of scaling depth ( N ) and width ( k ): Table 6 summarizes the results of our scaling experiments.
Revised: Impact of scaling depth ( N ) and width ( k ): Table 7 summarizes the results of our scaling experiments.

Original: For the same value of k , the performance of the language model improves with the increase in the depth N .
Revised: For the same value of k , the performance of the language model improves with the increase in the depth N .

Original: However, when we scale the width k for a ﬁxed value of depth N , the performance does not improve.
Revised: However, when we scale the width k for a ﬁxed value of depth N , the performance does not improve.

Original: This is likely because, as we increase the size of k , more neurons are receiving their input from the same subset of dimensions and thus learning many redundant parameters.
Revised: This is likely because, as we increase the size of k , more neurons are receiving their input from the same subset of dimensions and thus learning many redundant parameters.

Original: DeFINE with different connections: Table 7a demonstrates the impact of residual connections in DeFINE .
Revised: DeFINE with different connections: Table 8a demonstrates the impact of residual connections in DeFINE .

Original: In order to facilitate residual connections inside DeFINE , we ﬁx the dimension of each layer ˆe li in DeFINE to be k 2 instead of linearly spanning from n to k .
Revised: In order to facilitate residual connections inside DeFINE , we ﬁx the dimension of each layer ˆe li in DeFINE to be k 2 instead of linearly spanning from n to k .

Original: We can clearly see that the proposed skip-connections are more effective.
Revised: We can clearly see that the proposed skip-connections are more effective.

Original: Impact of reduce operation in MER :
Revised: Impact of reduce operation in MER :

Original: In the MER strategy (Section 3.1), we project the highdimensional vector to a low-dimensional space before feeding it to a contextual model, such as an
Revised: In the MER strategy (Section 3.1), we project the highdimensional vector to a low-dimensional space before feeding it to a contextual model, such as an

Original: LSTM.
Revised: LSTM.

Original: We empirically found that the performance with and without this reduction step is similar, however, a model without the reduction step learns more parameters (Table 7b).
Revised: We empirically found that the performance with and without this reduction step is similar, however, a model without the reduction step learns more parameters (Table 8b).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: DeFINE uses a deep, hierarchical, sparse network with new skip connections to learn better word embeddings efﬁciently.
Revised: DeFINE uses a deep, hierarchical, sparse network with new skip connections to learn better token embeddings efﬁciently.

Original: Sequence models with DeFINE (e.g. Transformer and LSTM) perform comparably or better with state-of-the-art methods with fewer parameters. 
Revised: Sequence models with DeFINE (e.g., Transformer and LSTM) perform comparably or better with state-of-the-art methods with fewer parameters. Our experiments show [Table] [Figure]

Original: Our experiments show that the proposed architectural decisions each contribute to the effectiveness of the DeFINE unit.
Revised: [Table] [Table] [Table] that the proposed architectural decisions each contribute to the effectiveness of the DeFINE unit.

Original: We believe neural sequence models with DeFINE can be further improved with extended hyperparameter search, similar to Melis et al.
Revised: We believe neural sequence models with DeFINE can be further improved with extended hyperparameter search, similar to Melis et al.

Original: In future work, we will apply DeFINE to other sequence modeling tasks.
Revised: In future work, we will apply DeFINE to other sequence modeling tasks.

Original: For instance, we believe that pretrained language model architectures such as ELMo and BERT can beneﬁt from incorporating DeFINE to improve efﬁciency and performance.
Revised: For instance, we believe that pretrained language model architectures such as ELMo and BERT can beneﬁt from incorporating DeFINE to improve efﬁciency and performance.

Original: Another direction is to use the components of DeFINE – speciﬁcally MER , HGT , and mixing layers – in neural architecture search processes.
Revised: Another direction is to use the components of DeFINE – speciﬁcally MER , HGT , and mixing layers – in neural architecture search processes.

Original: We have shown the promise of these components here, but a thorough architecture search may discover more optimal conﬁgurations in the large search space deﬁned by the depth, grouping, and connectivity parameters.  
Revised: We have shown the promise of these components here, but a thorough architecture search may discover more optimal conﬁgurations in the large search space deﬁned by the depth, grouping, and connectivity parameters.  A CKNOWLEDGEMENTS

Revised: This research was supported by ONR N00014-18-1-2826, DARPA N66001-19-2-403, NSF (IIS1616112, IIS1252835), an Allen Distinguished Investigator Award, Samsung GRO and gifts from

Revised: Allen Institute for AI, Google, and Amazon.

Revised: Authors would also like to thank members of the UWNLP and the H2Lab at The University of Washington for their valuable feedback and comments.

Original: A A PPENDIX A. T RANSFORMATION FUNCTION F G
Revised: A A PPENDIX A. T RANSFORMATION FUNCTION F G

Original: To produce an output y ∈ R m × 1 from an input x ∈ R n × 1 and weight matrix W ∈ R
Revised: To produce an output y ∈ R m × 1 from an input x ∈ R n × 1 and weight matrix W ∈ R

Original: ng × mg , F G ﬁrst chunks the input x into g groups and then concatenates the chunked parts to produce ˆ x ∈ R g × ng .
Revised: ng × mg , F G ﬁrst chunks the input x into g groups and then concatenates the chunked parts to produce ˆ x ∈ R g × ng .

Original: ˆ x is then multiplied with weight matrix W to produce ˆ y
Revised: ˆ x is then multiplied with weight matrix W to produce ˆ y

Original: = ˆ x · W ∈ R
Revised: = ˆ x · W ∈ R

Original: g ×
Revised: g ×

Original: mg .
Revised: mg .

Original: The resultant vector ˆ y is then ﬂattened to produce y .
Revised: The resultant vector ˆ y is then ﬂattened to produce y .

Original: When g = 1 , we obtain the linear transform.
Revised: When g = 1 , we obtain the linear transform.

Original: A. B LOCK LEVEL DIAGRAMS OF DIFFERENT SKIP - CONNECTIONS IN D E FINE
Revised: A. B LOCK LEVEL DIAGRAMS OF DIFFERENT SKIP - CONNECTIONS IN D E FINE

Original: Block level diagrams of different variants of DeFINE are given in Figure 5.
Revised: Block level diagrams of different variants of DeFINE are given in Figure 5.

Original: Figure 5a stacks transformation layer F G (Eq. 1) and is the same as HGT in Figure 2c.
Revised: Figure 5a stacks transformation layer F G (Eq. 1) and is the same as HGT in Figure 2c.

Original: Figure 5b adds a residual connection to Figure 5a.
Revised: Figure 5b adds a residual connection to Figure 5a.

Original: Figure 5c is the same as Figure 3 while Figure 5d is the same as Figure 5c, but without split and mixer functionality.
Revised: Figure 5c is the same as Figure 3 while Figure 5d is the same as Figure 5c [Figure] but without split and mixer functionality

Original: [Figure]
Revised: [Figure]

Original: A. H YPER - PARAMETERS FOR TRAINING LANGUAGE MODELS
Revised: A. H YPER - PARAMETERS FOR TRAINING LANGUAGE MODELS

Original: For training LSTM-based language models, we use a single NVIDIA GTX 1080 Ti GPU with
Revised: For training LSTM-based language models, we use a single NVIDIA GTX 1080 Ti GPU with

Original: GB GPU memory while for training Transformer-XL, we used four GeForce RTX 2080 Ti GPUs, each with 11 GB of GPU memory (as recommended by authors).
Revised: GB GPU memory while for training Transformer-XL, we used four GeForce RTX 2080 Ti GPUs, each with 11 GB of GPU memory (as recommended by authors).

Original: Following recent works, including
Revised: Following recent works, including

Original: Merity et al.
Revised: Merity et al.

Original: (2018a)
Revised: (2018a)

Original: and Baevski & Auli (2019), we use adaptive inputs as a mapping function in DeFINE and adaptive softmax for classiﬁcation for our experiments with RNN-based sequence models.
Revised: and Baevski & Auli (2019), we use adaptive inputs as a mapping function in DeFINE and adaptive softmax for classiﬁcation for our experiments with RNN-based sequence models.

Original: We also tie weights between the adaptive inputs and outputs.
Revised: We also tie weights between the adaptive inputs and outputs.

Original: For Transformer-XL Dai et al.
Revised: For Transformer-XL Dai et al.

Original: (2019), we use projective embeddings (as done by authors).
Revised: (2019), we use projective embeddings (as done by authors).

Original: We train our models using PyTorch (v1.2).
Revised: We train our models using PyTorch (v1.2).

Original: For LSTM-based language models, we use similar hyper-parameters as Merity et al.
Revised: For LSTM-based language models, we use similar hyper-parameters as Merity et al.

Original: (2018a) which are summarized in Section 8.
Revised: (2018a) which are summarized in Section 9.

Original: A. P ERFORMANCE OF T RANSFORMER -XL ON W IKI T EXT - 
Revised: A. P ERFORMANCE OF T RANSFORMER -XL ON W IKI T EXT -

Original: Figure 6 plots the validation perplexity of Transformer-XL on the WikiText-103 as a function of training steps.
Revised: Figure 6 plots the validation perplexity of Transformer-XL on the WikiText-103 as a function of training steps.

Original: We can see that DeFINE enables Transformer-XL to deliver similar performance with fewer parameters. 
Revised: We can see that DeFINE enables Transformer-XL to deliver similar performance with fewer parameters. WikiText

Original: [Table]
Revised: [Table]

Revised: # of GPUs

Revised: Weight decay

Revised: Optimizer

Revised: SGD

Revised: LR

Revised: BPTT Length

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: as Merity et al.

Revised: (2018a)

Original: B C ORRELATION MAP VISUALIZATION FOR T RANSFORMER -XL ON W IKI T EXT - 
Revised: B C ORRELATION MAP VISUALIZATION FOR T RANSFORMER -XL ON W IKI T EXT -

Original: Computing correlation map: Let us say that we have an arbitrary look-up table E ∈ R V× m that maps every word in vocabulary V to a m -dimensional vector space.
Revised: Computing correlation map: Let us say that we have an arbitrary look-up table E ∈ R V× m that maps every token in vocabulary V to a m -dimensional vector space.

Original: We compute the correlation map M as: M = E T · E ∈ R m × m .
Revised: We compute the correlation map M as: M = E T · E ∈ R m × m .

Original: 4 If the correlation map is identity, then it suggests that the mdimensions in E are independent.
Revised: 4 If the correlation map is identity, then it suggests that the mdimensions in E are independent.

Original: To encode better contextual representations among words using context models such as LSTMs and Transformers, embedding dimensions should be independent.
Revised: To encode better contextual representations among tokens using context models such as LSTMs and Transformers, embedding dimensions should be independent.

Original: Can DeFINE approximate the standard embedding layer?
Revised: Can DeFINE approximate the standard embedding layer?

Original: Figure 7 visualizes the correlation maps of embeddings learned using a standard embedding layer (top row), projective embeddings (Acharya et al., 2019; Dai et al., 2019) (middle row), and DeFINE embeddings (bottom row) at different values of n , where n is the dimension of mapping layer in DeFINE .
Revised: Figure 7 visualizes the correlation maps of embeddings learned using a standard embedding layer (top row), projective embeddings (Acharya et al., 2019; Dai et al., 2019) (middle row), and DeFINE embeddings (bottom row) at different values of n , where n is the dimension of mapping layer in DeFINE .

Original: Compared to projective embeddings, DeFINE is able to approximate the standard embedding layer efﬁciently and effectively (see Table 2 for efﬁciency and performance comparison).
Revised: Compared to projective embeddings, DeFINE is able to approximate the standard embedding layer efﬁciently and effectively (see Table 2 for efﬁciency and performance comparison).

Original: Furthermore, we provide layer-wise comparison for DeFINE at different values of n in Figures 8, 9, and 10.
Revised: Furthermore, we provide layer-wise comparison for DeFINE at different values of n in Figures 8, 9, and 10.

Original: The mapping layer in DeFINE is in low-dimensional space and has correlations.
Revised: The mapping layer in DeFINE is in low-dimensional space and has correlations.

Original: As we learn deeper representations using DeFINE , these correlations are reduced and we obtain a correlation matrix similar to a standard embedding layer.
Revised: As we learn deeper representations using DeFINE , these correlations are reduced and we obtain a correlation matrix similar to a standard embedding layer.

Original: This suggests that DeFINE is effective in approximating the standard embedding layer.
Revised: This suggests that DeFINE is effective in approximating the standard embedding layer.

Original: Importantly, the groups at different expansion layers in DeFINE are independent (see Figure 11), suggesting these matrices are learning different representations of their input.
Revised: Importantly, the groups at different expansion layers in DeFINE are independent (see Figure 11), suggesting these matrices are learning different representations of their input.

Original: [Figure]
Revised: [Figure]

Original: [Equation] [Figure] [Equation] Map Layer in DeFINE
Revised: DeFINE

Original: Expansion Layer 1 in DeFINE Expansion Layer 2 in DeFINE Reduction Layer in DeFINE
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: DeFINE

Original: [Figure]
Revised: [Figure]

Original: DeFINE

Original: [Figure]
Revised: [Figure]

Original: DeFINE

Original: [Figure]
Revised: [Equation]

Original:  DeFINE [Figure] DeFINE n =n =n =
Revised: Expansion layer 2 in DeFINE with 2 groups

Original: [Figure]
Revised: [Equation]