Original: B AG - OF -V ECTORS A UTOENCODERS FOR U NSUPERVISED C ONDITIONAL T EXT G ENERATION
Revised: B AG - OF -V ECTORS A UTOENCODERS FOR U NSUPERVISED C ONDITIONAL T EXT G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Text autoencoders are often used for unsupervised conditional text generation by applying mappings in the latent space to change attributes to the desired values.
Revised: Text autoencoders are often used for unsupervised conditional text generation by applying mappings in the latent space to change attributes to the desired values.

Original: Recently, Mai et al.
Revised: Recently, Mai et al.

Original: (2020) proposed Emb2Emb , a method to learn these mappings in the embedding space of an autoencoder.
Revised: (2020) proposed Emb2Emb , a method to learn these mappings in the embedding space of an autoencoder.

Original: However, their method is restricted to autoencoders with a single-vector embedding, which limits how much information can be retained.
Revised: However, their method is restricted to autoencoders with a single-vector embedding, which limits how much information can be retained.

Original: We address this issue by extending their method to
Revised: We address this issue by extending their method to

Original: Bag-of-Vectors Autoencoders (BoV-AEs), which encode the text into a variablesize bag of vectors that grows with the size of the text, as in attention-based models.
Revised: Bag-of-Vectors Autoencoders (BoV-AEs), which encode the text into a variablesize bag of vectors that grows with the size of the text, as in attention-based models.

Original: This allows to encode and reconstruct much longer texts than standard autoencoders.
Revised: This allows to encode and reconstruct much longer texts than standard autoencoders.

Original: Analogous to conventional autoencoders, we propose regularization techniques that facilitate learning meaningful operations in the latent space.
Revised: Analogous to conventional autoencoders, we propose regularization techniques that facilitate learning meaningful operations in the latent space.

Original: Finally, we adapt Emb2Emb for a training scheme that learns to map an input bag to an output bag, including a novel loss function and neural architecture.
Revised: Finally, we adapt Emb2Emb for a training scheme that learns to map an input bag to an output bag, including a novel loss function and neural architecture.

Original: Our empirical evaluations on unsupervised sentiment transfer and sentence summarization show that our method performs substantially better than a standard autoencoder.
Revised: Our empirical evaluations on unsupervised sentiment transfer and sentence summarization show that our method performs substantially better than a standard autoencoder.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In conditional text generation, we would like to produce an output text given an input text.
Revised: In conditional text generation, we would like to produce an output text given an input text.

Original: Hence, parallel input-output pairs are required to train a good supervised machine learning model on this type of task.
Revised: Hence, parallel input-output pairs are required to train a good supervised machine learning model on this type of task.

Original: Large-scale pretraining (Peters et al., 2018; Devlin et al., 2019; Lewis et al., 2020) can alleviate the necessity for training examples to some extent, but even this requires a substantial number of annotations (Yogatama et al., 2019).
Revised: Large-scale pretraining (Peters et al., 2018; Devlin et al., 2019; Lewis et al., 2020) can alleviate the necessity for training examples to some extent, but even this requires a substantial number of annotations (Yogatama et al., 2019).

Original: This is an expensive process and can introduce unwanted artifacts itself, which are henceforth learned by the model (Gururangan et al., 2018).
Revised: This is an expensive process and can introduce unwanted artifacts itself, which are henceforth learned by the model (Gururangan et al., 2018).

Original: For these reasons, there is substantial interest in unsupervised solutions.
Revised: For these reasons, there is substantial interest in unsupervised solutions.

Original: Text autoencoders (AEs) don’t require labeled data for training, and are therefore a popular model for unsupervised approaches to many tasks, such as machine translation (Artetxe et al., 2018), sentence compression (F ´ evry &
Revised: Text autoencoders (AEs) don’t require labeled data for training, and are therefore a popular model for unsupervised approaches to many tasks, such as machine translation (Artetxe et al., 2018), sentence compression (F ´ evry &

Original: Phang, 2018) and sentiment transfer (Shen et al., 2017).
Revised: Phang, 2018) and sentiment transfer (Shen et al., 2017).

Original: The classical text AE (Bowman et al., 2016) embeds the input text into a single ﬁxed-size vector via the encoder, and then tries to reconstruct the input text from the single vector via the decoder.
Revised: The classical text AE (Bowman et al., 2016) embeds the input text into a single ﬁxed-size vector via the encoder, and then tries to reconstruct the input text from the single vector via the decoder.

Original: Single-vector embeddings are very useful, because they allow to perform conditional text generation through simple mappings in the embedding space, e.g. by adding a constant offset vector to change attributes such as sentiment (Shen et al., 2020).
Revised: Single-vector embeddings are very useful, because they allow to perform conditional text generation through simple mappings in the embedding space, e.g. by adding a constant offset vector to change attributes such as sentiment (Shen et al., 2020).

Original: Recently, Mai et al.
Revised: Recently, Mai et al.

Original: (2020) proposed Emb2Emb , a method that can learn these mappings directly in the embedding space of any pretrained single-vector AE.
Revised: (2020) proposed Emb2Emb , a method that can learn these mappings directly in the embedding space of any pretrained single-vector AE.

Original: This is a powerful framework, because the AE can then be pretrained on virtually inﬁnite amounts of unlabeled data before applying it to any downstream application.
Revised: This is a powerful framework, because the AE can then be pretrained on virtually inﬁnite amounts of unlabeled data before applying it to any downstream application.

Original: This concept, transfer learning , is arguably one of the most important drivers of progress in machine learning in the recent decade:
Revised: This concept, transfer learning , is arguably one of the most important drivers of progress in machine learning in the recent decade:

Original: These so-called Foundation Models (Bommasani et al., 2021) have revolutionized natural language understanding (e.g, BERT (Devlin et al., 2019)) and computer vision (e.g, DALL-E (Ramesh et al., 2021)), among others.
Revised: These so-called Foundation Models (Bommasani et al., 2021) have revolutionized natural language understanding (e.g, BERT (Devlin et al., 2019)) and computer vision (e.g, DALL-E (Ramesh et al., 2021)), among others.

Original: Since Emb2Emb was designed to work with any pretrained AE, it was an important step towards their scalability .
Revised: Since Emb2Emb was designed to work with any pretrained AE, it was an important step towards their scalability .

Original: However, as Bommasani et al.
Revised: However, as Bommasani et al.

Original: (2021) point out, another crucial model property is expressivity , the ability to represent the data distribution it is trained on.
Revised: (2021) point out, another crucial model property is expressivity , the ability to represent the data distribution it is trained on.

Original: In this regard, single-vector representations are fundamentally limited; they act as a bottleneck, causing the model to increasingly struggle to encode longer text (Bahdanau et al., 2015).
Revised: In this regard, single-vector representations are fundamentally limited; they act as a bottleneck, causing the model to increasingly struggle to encode longer text (Bahdanau et al., 2015).

Original: In this paper, we extent Emb2Emb from single-vector bottleneck AEs to Bag-of-Vector Autoencoders (BoV-AEs), which encode text into a variable-size representation where the number of vectors grows with the length of the text.
Revised: In this paper, we extent Emb2Emb from single-vector bottleneck AEs to Bag-of-Vector Autoencoders (BoV-AEs), which encode text into a variable-size representation where the number of vectors grows with the length of the text.

Original: This gives BoV-
Revised: This gives BoV-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: An autoencoder is trained on an unlabeled corpus, i.e., the encoder enc transforms an input text x into a continuous embedding z x , which is in turn used by the decoder dec to predict a reconstruction ˆ x of the input sentence.
Revised: An autoencoder is trained on an unlabeled corpus, i.e., the encoder enc transforms an input text x into a continuous embedding z x , which is in turn used by the decoder dec to predict a reconstruction ˆ x of the input sentence.

Original: Task Training :
Revised: Task Training :

Original: The encoder is frozen (grey), and a mapping Φ is trained (green) on input embeddings
Revised: The encoder is frozen (grey), and a mapping Φ is trained (green) on input embeddings

Original: z x to output predictions ˆz x such that it minimize some loss L ( ˆz y ) .
Revised: z x to output predictions ˆz y such that it minimize some loss L ( ˆz y ) .

Original: Inference : To obtain textual predictions ˆ y , the encoder is composed with Φ and the decoder.
Revised: Inference : To obtain textual predictions ˆ y , the encoder is composed with Φ and the decoder.

Original: AEs the same kind of representations as attention-based models.
Revised: AEs the same kind of representations as attention-based models.

Original: But this added expressivity comes with additional challenges, as illustrated in Figure 1.
Revised: But this added expressivity comes with additional challenges, as illustrated in Figure 1.

Original: In the single-vector case, an operation Φ in the vector space consists of a simple vector-to-vector mapping.
Revised: In the single-vector case, an operation Φ in the vector space consists of a simple vector-to-vector mapping.

Original: But with BoV-AEs, Φ needs to map a bag of vectors onto another bag of vectors, which requires more complicated operations.
Revised: But with BoV-AEs, Φ needs to map a bag of vectors onto another bag of vectors, which requires more complicated operations.

Original: In this paper, we demonstrate how such a mapping can be learned in the context of the Emb2Emb framework by making the following novel contributions: (i)
Revised: In this paper, we demonstrate how such a mapping can be learned in the context of the Emb2Emb framework by making the following novel contributions: (i)

Original: We propose a regularization scheme for
Revised: We propose a regularization scheme for

Original: BoV-AEs, (ii) a neural mapping architecture Φ for Emb2Emb , and (iii) a suitable training loss.
Revised: BoV-AEs, (ii) a neural mapping architecture Φ for Emb2Emb , and (iii) a suitable training loss.

Original: Empirically, we show on two unsupervised conditional text generation tasks, sentiment transfer (Shen et al., 2017) and sentence summarization (Rush et al., 2015), that BoV-AEs perform substantially better than standard AEs if the text is too long to be captured by one vector alone.
Revised: Empirically, we show on two unsupervised conditional text generation tasks, sentiment transfer (Shen et al., 2017) and sentence summarization (Rush et al., 2015), that BoV-AEs perform substantially better than standard AEs if the text is too long to be captured by one vector alone.

Original: Our ablation studies conﬁrm that our technical contributions are crucial for this success.
Revised: Our ablation studies conﬁrm that our technical contributions are crucial for this success.

Original: In the following section, we review the Emb2Emb framework, before we introduce BoV-AE (Section 3) and its integration within Emb2Emb (Section 4).
Revised: In the following section, we review the Emb2Emb framework, before we introduce BoV-AE (Section 3) and its integration within Emb2Emb (Section 4).

Original:  B ACKGROUND : Emb2Emb
Revised:  B ACKGROUND : Emb2Emb

Original: Embedding-to-Embedding ( Emb2Emb ) was introduced by Mai et al.
Revised: Embedding-to-Embedding ( Emb2Emb ) was introduced by Mai et al.

Original: (2020) as both a supervised and an unsupervised framework for conditional text generation.
Revised: (2020) as both a supervised and an unsupervised framework for conditional text generation.

Original: The core idea is to disentangle the transition from the discrete text space to a continuous latent space from the speciﬁc task, allowing for larger-scale pretraining with unlabeled data.
Revised: The core idea is to disentangle the transition from the discrete text space to a continuous latent space from the speciﬁc task, allowing for larger-scale pretraining with unlabeled data.

Original: The workﬂow of the framework is depicted in Figure 2.
Revised: The workﬂow of the framework is depicted in Figure 2.

Original: First, a text AE A =
Revised: First, a text AE A =

Original: dec ◦ enc is trained to map an input sentence from the discrete text space X to an embedding space Z via the encoder enc : X → Z , and back to X via a decoder dec : Z → X , such that A ( x ) = x , typically trained via negative log-likelihood, L rec =
Revised: dec ◦ enc is trained to map an input sentence from the discrete text space X to an embedding space Z via the encoder enc : X → Z , and back to X via a decoder dec : Z → X , such that A ( x ) = x , typically trained via negative log-likelihood, L rec =

Original: NLL( A ( x ) , x ) .
Revised: NLL( A ( x ) , x ) .

Original: In contrast to other methods, A can in principle be any AE, opening the possibility for large-scale AE pretraining with unlabeled data.
Revised: In contrast to other methods, A can in principle be any AE, opening the possibility for large-scale AE pretraining with unlabeled data.

Original: Second, taskspeciﬁc training is performed only in the embedding space Z of the AE.
Revised: Second, taskspeciﬁc training is performed only in the embedding space Z of the AE.

Original: To this end, the encoder is frozen, and a new mapping layer Φ : Z → Z is introduced, which is trained to transform the embedding of the input z x into the embedding of the predicted output ˆ z y .
Revised: To this end, the encoder is frozen, and a new mapping layer Φ : Z → Z is introduced, which is trained to transform the embedding of the input z x into the embedding of the predicted output ˆ z y .

Original: The concrete loss L (ˆ z y ) depends on the type of task.
Revised: The concrete loss L (ˆ z y ) depends on the type of task.

Original: In the supervised case, the true output is also encoded into space
Revised: In the supervised case, the true output is also encoded into space

Original: Z , and the distance between the true embedding and the predicted embedding is minimized.
Revised: Z , and the distance between the true embedding and the predicted embedding is minimized.

Original: In the unsupervised case, the loss needs to be deﬁned for the speciﬁc task at hand.
Revised: In the unsupervised case, the loss needs to be deﬁned for the speciﬁc task at hand.

Original: For example, for sentiment transfer, where the goal is to transform a negative review into a positive review while retaining as much of the input as possible, Mai et al.
Revised: For example, for sentiment transfer, where the goal is to transform a negative review into a positive review while retaining as much of the input as possible, Mai et al.

Original: (2020) compose the loss as a combination of two loss terms 1 , L ( ˆz y )
Revised: (2020) compose the loss as a combination of two loss terms 1 , L ( ˆz y )

Original: =
Revised: =

Original: L sim ( z x , ˆz y )
Revised: L sim ( z x , ˆz y )

Original: + λ sty L sty ( ˆz y ) .
Revised: + λ sty L sty ( ˆz y ) .

Original: L sty encourages ˆz y to be classiﬁed as a positive review according to a separately trained sentiment classiﬁer.
Revised: L sty encourages ˆz y to be classiﬁed as a positive review according to a separately trained sentiment classiﬁer.

Original: L sim encourages the output to be close to the input in embedding space, e.g. via euclidean distance.
Revised: L sim encourages the output to be close to the input in embedding space, e.g. via euclidean distance.

Original: λ sty is a hyperparameter that controls the importance of changing the sentiment of the predicted output.
Revised: λ sty is a hyperparameter that controls the importance of changing the sentiment of the predicted output.

Original: A main question in Emb2Emb is how to choose the embedding space Z .
Revised: A main question in Emb2Emb is how to choose the embedding space Z .

Original: Mai et al.
Revised: Mai et al.

Original: (2020) use a single continuous vector to encode all the information of the input, i.e. Z = R d .
Revised: (2020) use a single continuous vector to encode all the information of the input, i.e. Z = R d .

Original: This choice simpliﬁes the mapping Φ to an MLP and the training loss to vector space distances, which is relatively easy to train.
Revised: This choice simpliﬁes the mapping Φ to an MLP and the training loss to vector space distances, which is relatively easy to train.

Original: On the other hand, it limits the model in fundamental ways: The representation is ﬁxed-sized , i.e., the representation cannot grow in size.
Revised: On the other hand, it limits the model in fundamental ways: The representation is ﬁxed-sized , i.e., the representation cannot grow in size.

Original: Sequence-to-sequence models with a ﬁxed-size bottleneck struggle to encode long text sequences (Bahdanau et al., 2015), which is a key reason why attention-based models are now standard practice in sequence-to-sequence models.
Revised: Sequence-to-sequence models with a ﬁxed-size bottleneck struggle to encode long text sequences (Bahdanau et al., 2015), which is a key reason why attention-based models are now standard practice in sequence-to-sequence models.

Original: Hence, it would be desirable to adapt Emb2Emb in such a way that Z contains variable-sized embeddings instead.
Revised: Hence, it would be desirable to adapt Emb2Emb in such a way that Z contains variable-sized embeddings instead.

Original:  B AG - OF -V ECTORS A UTOENCODER
Revised:  B AG - OF -V ECTORS A UTOENCODER

Original: We propose Bag-of-Vectors Autoencoders (BoV-AEs) to be used with Emb2Emb .
Revised: We propose Bag-of-Vectors Autoencoders (BoV-AEs) to be used with Emb2Emb .

Original: Following the naming convention by Henderson (2020), we refer to a bag of vectors as a (multi)-set of vectors that (i) can grow arbitrarily large, and (ii) where the elements are not ordered (a basic property of sets).
Revised: Following the naming convention by Henderson (2020), we refer to a bag of vectors as a (multi)-set of vectors that (i) can grow arbitrarily large, and (ii) where the elements are not ordered (a basic property of sets).

Original: A type of BoV representation that is used very commonly is found in Transformer (Vaswani et al., 2017) encoder-decoder models, where there is one vector to represent each token of the input text, and the order of the vectors does not matter when the decoder accesses the output of the encoder.
Revised: A type of BoV representation that is used very commonly is found in Transformer (Vaswani et al., 2017) encoder-decoder models, where there is one vector to represent each token of the input text, and the order of the vectors does not matter when the decoder accesses the output of the encoder.

Original: In this work, we also rely on Transformer models as the backbone of our encoders and decoders.
Revised: In this work, we also rely on Transformer models as the backbone of our encoders and decoders.

Original: However, in principle, any encoder and decoder can be used, as long as the encoder produces a bag as output and the decoder takes a bag as input.
Revised: However, in principle, any encoder and decoder can be used, as long as the encoder produces a bag as output and the decoder takes a bag as input.

Original: Formally, Z = ( R d ) + , so the encoder produces a bag-of-vectors X
Revised: Formally, Z = ( R d ) + , so the encoder produces a bag-of-vectors X

Original: = { z 1 , ..., z n } := enc( x ) .
Revised: = { z 1 , ..., z n } := enc( x ) , where n is the number of vectors in the input bag.

Original: 3. R EGULARIZATION
Revised: 3. R EGULARIZATION

Original: The fact that we use a BoV-based AE presents a major challenge: AEs have to be regularized to prevent them from learning a simple identity mapping where the input is merely copied to the output, which does not result in a meaningful embedding space.
Revised: The fact that we use a BoV-based AE presents a major challenge: AEs have to be regularized to prevent them from learning a simple identity mapping where the input is merely copied to the output, which does not result in a meaningful embedding space.

Original: In ﬁxed-size embeddings, this is for example achieved through under-completeness (choosing a latent dimension that is smaller than the input dimension) or through injection of noise, either at the input or in the embedding space.
Revised: In ﬁxed-size embeddings, this is for example achieved through under-completeness (choosing a latent dimension that is smaller than the input dimension) or through injection of noise, either at the input or in the embedding space.

Original: While there exists a lot of research on regularizing ﬁxed-sized AEs, it is not clear how to achieve the same goal in a BoV-AE.
Revised: While there exists a lot of research on regularizing ﬁxed-sized AEs, it is not clear how to achieve the same goal in a BoV-AE.

Original: Here, regularizing the capacity of each vector is not enough.
Revised: Here, regularizing the capacity of each vector is not enough.

Original: As long as each vector can store a (constant) positive amount of information, a bag of unlimited size can still store inﬁnite information.
Revised: As long as each vector can store a (constant) positive amount of information, a bag of unlimited size can still store inﬁnite information.

Original: However, it is not clear to what extent the size of the bag needs to be restricted.
Revised: However, it is not clear to what extent the size of the bag needs to be restricted.

Original: By default, a standard Transformer model produces as many vectors as there are input tokens, but this is likely too many, as it makes copying from the input to the output trivial.
Revised: By default, a standard Transformer model produces as many vectors as there are input tokens, but this is likely too many, as it makes copying from the input to the output trivial.

Original: Hence, we want the encoder to output fewer vectors.
Revised: Hence, we want the encoder to output fewer vectors.

Original: In the following we explain how this is achieved in BoV-AEs.
Revised: In the following we explain how this is achieved in BoV-AEs.

Original: L0Drop Ideally, we want the model to decide for itself on a per- example basis which vectors it needs to retain for reconstruction.
Revised: L0Drop Ideally, we want the model to decide for itself on a per-example basis which vectors it needs to retain for reconstruction.

Original: To this end, we adopt L0Drop , a differentiable approximation to
Revised: To this end, we adopt L0Drop , a differentiable approximation to

Original: L0 regularization, which was originally developed by Zhang et al.
Revised: L0 regularization, which was originally developed by Zhang et al.

Original: (2021) for the purpose of speeding up a model through sparsiﬁcation.
Revised: (2021) for the purpose of speeding up a model through sparsiﬁcation.

Original: The model computes scalar gates g i
Revised: The model computes scalar gates g i

Original: = g ( z i ) ∈
Revised: = g ( z i ) ∈

Original: [0 , 1] (which can be exactly zero or one) for each encoder output.
Revised: [0 , 1] (which can be exactly zero or one) for each encoder output.

Original: After the gates are computed, we multiply them with their corresponding vector.
Revised: After the gates are computed, we multiply them with their corresponding vector.

Original: Vectors whose gates are near zero (i.e., smaller than some (cid:15) > 0 ) are removed from the bag entirely.
Revised: Vectors whose gates are near zero (i.e., smaller than some (cid:15) > 0 ) are removed from the bag entirely.

Original: An additional loss term, L L 0 ( X ) =
Revised: An additional loss term, L L 0 ( X ) =

Original: λ L 0 (cid:80) ni g
Revised: λ L 0 (cid:80) ni g

Original: i encourages the model to close as many gates as possible, where the hyperparameter λ L 0 controls the sparsity rate implicitly .
Revised: i encourages the model to close as many gates as possible, where the hyperparameter λ L 0 controls the sparsity rate implicitly .

Original: However, in initial experiments, we found λ L 0 difﬁcult to tune, as it is very sensitive with respect to other hyperparameters.
Revised: However, in initial experiments, we found λ L 0 difﬁcult to tune, as it is very sensitive with respect to other hyperparameters.

Original: We instead employ a modiﬁed loss that seeks to explicitly match a certain target ratio r of open gates.
Revised: We instead employ a modiﬁed loss that seeks to explicitly match a certain target ratio r of open gates.

Original: Similar to the free-bits objective that is used to prevent the posterior collapse problem in VAEs (Kingma et al., 2016), the objective becomes
Revised: Similar to the free-bits objective that is used to prevent the posterior collapse problem in VAEs (Kingma et al., 2016), the objective becomes

Original: [Equation]
Revised: [Equation]

Original: By setting λ L 0 to a large enough value (empirically, λ L 0 = 10 ), we ﬁnd that this objective reaches the target ratio r reliably for different r while at the same time reducing the reconstruction loss.
Revised: By setting λ L 0 to a large enough value (empirically, λ L 0 = 10 ), we ﬁnd that this objective reaches the target ratio r reliably for different r while at the same time reducing the reconstruction loss.

Original: This allows to compare different strengths of regularization while reducing the tuning effort substantially.
Revised: This allows to compare different strengths of regularization while reducing the tuning effort substantially.

Original:  E MB 2E MB WITH B O V-AE S
Revised:  E MB 2E MB WITH B O V-AE S

Original: In the following we describe how to adapt the Emb2Emb model to BoV-AEs, i.e., how to generate an output bag Y given an input bag X through the mapping Φ , and how to choose the loss function 
Revised: In the following we describe how to adapt the Emb2Emb model to BoV-AEs, i.e., how to generate an output bag ˆ X = { ˆ z 1 , . . , ˆ z n } given an input bag X through the mapping Φ( X ) , and how to choose the loss function L (ˆ X , X ) .

Original: L ( X, Y ) .
Revised: For example, in the case of style transfer, we want ˆ X to be similar to X .

Original: 4. M APPING Φ
Revised: 4. M APPING Φ

Original: In contrast to Mai et al.
Revised: In contrast to Mai et al.

Original: (2020), who use a single-vector embedding and hence Φ can be as simple as an MLP, in our work, Φ must be capable of producing a bag of vectors.
Revised: (2020), who use a single-vector embedding and hence Φ can be as simple as an MLP, in our work, Φ must be capable of producing a bag of vectors.

Original: The straight-forward choice for Φ is a Transformer decoder that uses cross-attention on the input BoV, and generates vectors autoregressively one at a time, formally ˆ zy t =
Revised: The straight-forward choice for Φ is a Transformer decoder that uses cross-attention on the input BoV, and generates vectors autoregressively one at a time, formally ˆ z

Original:  Transformer( z s , ˆ z 1 , . . 
Revised: = Transformer( z s , ˆ z 1 , . . .

Original: , ˆ z t − 1 , X ) , t ≥ 1 , where z s is the embedding of some starting symbol.
Revised: , ˆ z t − 1 , X ) , t ≥ 1 , where z s is the embedding of some starting symbol.

Original: Depending on the difﬁculty of the task, this mapping may be sufﬁciently powerful, in particular for short texts (i.e. bags with few vectors).
Revised: Depending on the difﬁculty of the task, this mapping may be sufﬁciently powerful, in particular for short texts (i.e. bags with few vectors).

Original: For longer texts, however, learning to map from one bag to another is difﬁcult, and may hence require the integration of inductive biases.
Revised: For longer texts, however, learning to map from one bag to another is difﬁcult, and may hence require the integration of inductive biases.

Original: Based on the assumption that the output should be close to the input in embedding space, Mai et al.
Revised: Based on the assumption that the output should be close to the input in embedding space, Mai et al.

Original: (2020) propose OffsetNet for the single vector case, which computes an offset vector to be added to the input.
Revised: (2020) propose OffsetNet for the single vector case, which computes an offset vector to be added to the input.

Original: With a similar motivation, we propose a variant of pointer-generator networks (See et al., 2017), which allows the model to choose between copying an input vector and generating a new one.
Revised: With a similar motivation, we propose a variant of pointer-generator networks (See et al., 2017), which allows the model to choose between copying an input vector and generating a new one.

Original: Instead of just copying, however, our model ( Transformer ++ ) allows to compute an offset vector to be added to the copied vector, analogous to Mai et al.
Revised: Instead of just copying, however, our model ( Transformer ++ ) allows to compute an offset vector to be added to the copied vector, analogous to Mai et al.

Original: Formally, at each timestep t ,
Revised: Formally, at each timestep t ,

Original: [Equation]
Revised: [Equation]

Original:  Intuitively, by controlling p gen ∈ (0 , 1) , the model makes the (soft) decision to either copy a vector from the input and add an offset, or to generate a completely new vector.
Revised: where z (cid:48) t = Transformer( z s , ..., ˆ z t − 1 , X ) . Intuitively, by controlling p gen ∈ (0 , 1) , the model makes the (soft) decision to either copy a vector from the input and add an offset, or to generate a completely new vector.

Original: Here, p gen is a function ofthe output z (cid:48) tof Transformer and the starting symbol which we treat as a context vector, p gen = σ ( W [ z s ; z (cid:48) t ]) .
Revised: Here, p gen is a function of z (cid:48) t and the starting symbol which we treat as a context vector, p gen = σ ( W [ z s ; z (cid:48) t ]) .

Original: Similarly, z offset is a one-layer MLP with [ z (cid:48) t ; z copy ] as input.
Revised: Similarly, z offset is a one-layer MLP with [ z (cid:48) t ; z copy ] as input.

Original: z copy is determined through an attention function:
Revised: z copy is determined through an attention function:

Original: [Equation]
Revised: [Equation]

Original: where W cpy is a learnable weight matrix.
Revised: where W cpy is a learnable weight matrix.

Original: We refer to this model as Transformer ++ .
Revised: We refer to this model as Transformer ++ .

Original: 4. G ENERATING V ARIABLE S IZED B AGS
Revised: 4. G ENERATING V ARIABLE S IZED B AGS

Original: The output bag is generated in an autoregressive manner.
Revised: The output bag is generated in an autoregressive manner.

Original: In the unsupervised case, it is not always clear how many vectors the bag should contain.
Revised: In the unsupervised case, it is not always clear how many vectors the bag should contain.

Original: However, due to the unsupervised nature, all information needed for computing the  training loss L ( X , Y ) are also available at inference time.
Revised: However, due to the unsupervised nature, all information needed for computing the (task-dependent) training loss L (ˆ X , X ) are also available at inference time.

Original: Hence, we can ﬁrst generate some ﬁxed maximum number N of vectors autoregressively, and then determine the optimal bag by computing the minimal (inference-time) loss value, ˆ
Revised: In this case, we can ﬁrst generate some ﬁxed maximum number N of vectors autoregressively, and then determine the optimal bag by computing the minimal (inference-time) loss value, X ∗ =

Original: X = min l =1 ,...,N L ( X 1: l , Y ) .
Revised: min l =1 ,...,N L (ˆ X 1: l , X ) .

Original: This can be valuable for tasks where we do not have a good prior on the size of the target bag.
Revised: This can be valuable for tasks where we do not have a good prior on the size of the target bag.

Original: For training, we minimize the loss locally at every step.
Revised: During training, we minimize the loss locally at every step.

Original: But we don’t necessarily care about the loss at very small or big bags, so we might want to weight the steps as L total ( X , Y ) =
Revised: But we don’t necessarily care about the loss at very small or big bags, so we might want to weight the steps as L total (ˆ X , X ) =

Original: (cid:80) 
Revised: (cid:80) Nl

Original: Nl =1 w l L ( X 1: l , Y ) .
Revised: =1 w l L (ˆ X 1: l , X ) .

Original: Here, w ∈ R N + could be any weighting, but it is more beneﬁcial for training to only backpropagate from bag sizes that we expect to be close to the optimal output bag size.
Revised: Here, w ∈ R N + could be any weighting, but it is more beneﬁcial for training to only backpropagate from bag sizes that we expect to be close to the optimal output bag size.

Original: For instance, in style transfer, the output typically has about the same length as the input.
Revised: For instance, in style transfer, the output typically has about the same length as the input.

Original: Hence, for an input size of length n , a useful weighting could be
Revised: Hence, for an input size of length n , a useful weighting could be

Original: [Equation]
Revised: [Equation]

Original: in which k denotes the size of a window around the input bag size.
Revised: in which k denotes the size of a window around the input bag size.

Original: 4. A LIGNING T WO B AGS OF V ECTORS
Revised: 4. A LIGNING T WO B AGS OF V ECTORS

Original: As described in Section 2, unsupervised sentiment transfer involves two loss terms, L sty and L sim .
Revised: As described in Section 2, unsupervised sentiment transfer involves two loss terms, L sty and L sim .

Original: In order to adapt L sty from the single vector case to the BoV case, we can simply switch from an
Revised: In order to adapt L sty from the single vector case to the BoV case, we can simply switch from an

Original: MLP classiﬁer to a Transformer-based classiﬁer.
Revised: MLP classiﬁer to a Transformer-based classiﬁer.

Original: For L sim , however, we need to switch to a loss function that is deﬁned on sets.
Revised: For L sim , however, we need to switch to a loss function that is deﬁned on sets.

Original: While there are well-known losses for the single-vector case, in NLP set-level loss functions are not well-studied.
Revised: While there are well-known losses for the single-vector case, in NLP set-level loss functions are not well-studied.

Original: Here, we propose a novel variant of the Hausdorff distance.
Revised: Here, we propose a novel variant of the Hausdorff distance.

Original: This distance is commonly used in vision applications: as a performance evaluation metric in e.g. medical image segmentation (Taha &
Revised: This distance is commonly used in vision applications: as a performance evaluation metric in e.g. medical image segmentation (Taha &

Original: Hanbury, 2015; Aydin et al., 2020), or in vision systems as a way to compare images (Huttenlocher et al., 1992; Tak ´ acs, 1998; Lin et al., 2003; Lu et al., 2001).
Revised: Hanbury, 2015; Aydin et al., 2020), or in vision systems as a way to compare images (Huttenlocher et al., 1992; Tak´acs, 1998; Lin et al., 2003; Lu et al., 2001).

Original: More recently, variants (different from ours) of the Hausdorff distance have also been used as loss functions to train neural networks (Fan et al., 2017; Ribera et al., 2019;
Revised: More recently, variants (different from ours) of the Hausdorff distance have also been used as loss functions to train neural networks (Fan et al., 2017; Ribera et al., 2019;

Original: Zhao et al., 2021).
Revised: Zhao et al., 2021).

Original: In NLP, its use is very rare (Nutanong et al., 2016; Chen, 2019; Kuo et al., 2020).
Revised: In NLP, its use is very rare (Nutanong et al., 2016; Chen, 2019; Kuo et al., 2020).

Original: To the best of our knowledge, our paper is the ﬁrst to present a novel, fully differentiable variant of the Hausdorff distance as a loss for language learning.
Revised: To the best of our knowledge, our paper is the ﬁrst to present a novel, fully differentiable variant of the Hausdorff distance as a loss for language learning.

Original: The Hausdorff distance is a method for aligning two sets.
Revised: The Hausdorff distance is a method for aligning two sets.

Original: Given two sets X and Y , their Hausdorff distance H is deﬁned as
Revised: Given two sets X and ˆ X , their Hausdorff distance H is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Intuitively, two sets are close if each point in either set has a counterpart in the other set that is close to it according to some distance metric d .
Revised: Intuitively, two sets are close if each point in either set has a counterpart in the other set that is close to it according to some distance metric d .

Original: We choose d to be the euclidean distance, but in principle any differentiable distance metric could be used (e.g. cosine distance).
Revised: We choose d to be the euclidean distance, but in principle any differentiable distance metric could be used (e.g. cosine distance).

Original: However, the vanilla Hausdorff distance is very prone to outliers, and therefore often reduced to the average
Revised: However, the vanilla Hausdorff distance is very prone to outliers, and therefore often reduced to the average

Original: Hausdorff distance (Dubuisson & Jain, 1994), where
Revised: Hausdorff distance (Dubuisson & Jain, 1994), where

Original: [Equation]
Revised: [Equation]

Original: The average Hausdorff function is step-wise smooth and differentiable.
Revised: The average Hausdorff function is step-wise smooth and differentiable.

Original: Empirically, however, we ﬁnd step-wise smoothness to be insufﬁcient for the best training outcome.
Revised: Empirically, however, we ﬁnd step-wise smoothness to be insufﬁcient for the best training outcome.

Original: Therefore, we propose a fully differentiable version of the Hausdorff distance by replacing the min operation with softmin like follows:
Revised: Therefore, we propose a fully differentiable version of the Hausdorff distance by replacing the min operation with softmin like follows:

Original: [Equation]
Revised: [Equation]

Original: This variant is reminiscent of the attention mechanism Bahdanau et al.
Revised: This variant is reminiscent of the attention mechanism Bahdanau et al.

Original: (2015) in the sense that a weighted average is computed, which has been very successful at smoothly approximating discrete decisions, e.g., read and write operations in the Differentiable Neural Computer (Graves et al., 2016) among many others.
Revised: (2015) in the sense that a weighted average is computed, which has been very successful at smoothly approximating discrete decisions, e.g., read and write operations in the Differentiable Neural Computer (Graves et al., 2016) among many others.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments are designed to test the following two hypotheses.
Revised: Our experiments are designed to test the following two hypotheses.

Original: H1 :
Revised: H1 :

Original: If the input text is too long to be encoded into a ﬁxed-size single vector representation, BoV-AE-based Emb2Emb provides a substantial advantage over the ﬁxed-sized model.
Revised: If the input text is too long to be encoded into a ﬁxed-size single vector representation, BoV-AE-based Emb2Emb provides a substantial advantage over the ﬁxed-sized model.

Original: H2 : Our technical contributions, namely L0Drop regularization, the training loss, and the mapping architecture, are necessary for BoV-AE’s success.
Revised: H2 : Our technical contributions, namely L0Drop regularization, the training loss, and the mapping architecture, are necessary for BoV-AE’s success.

Original: We evaluate our model on two unsupervised conditional text generation tasks:
Revised: We evaluate our model on two unsupervised conditional text generation tasks:

Original: In Section 5.1, we show that H1 holds even when the single-vector dimensionality is large ( d =512 ).
Revised: In Section 5.1, we show that H1 holds even when the single-vector dimensionality is large ( d =512 ).

Original: To this end, we create a new sentiment transfer dataset, Yelp-Reviews, whose inputs are relatively long.
Revised: To this end, we create a new sentiment transfer dataset, Yelp-Reviews, whose inputs are relatively long.

Original: However, training on this dataset is computationally very demanding 2 .
Revised: However, training on this dataset is computationally very demanding 2 .

Original: Therefore, we turn to shorter text datasets to test hypothesis H2 .
Revised: Therefore, we turn to shorter text datasets to test hypothesis H2 .

Original: Concretely, we test on a sentiment transfer dataset of short texts (Section 5.2), and a sentence summarization dataset of medium length texts (Section 5.3).
Revised: Concretely, we test on a sentiment transfer dataset of short texts (Section 5.2), and a sentence summarization dataset of medium length texts (Section 5.3).

Original: Evaluation metrics:
Revised: Evaluation metrics:

Original: In sentiment transfer, the goal is to rewrite a negative review as a positive review while keeping as much of the content as possible.
Revised: In sentiment transfer, the goal is to rewrite a negative review as a positive review while keeping as much of the content as possible.

Original: Hence, two metrics are important, sentiment transfer ability and content retention.
Revised: Hence, two metrics are important, sentiment transfer ability and content retention.

Original: Following common practice (Hu et al., 2017; Shen et al., 2017;
Revised: Following common practice (Hu et al., 2017; Shen et al., 2017;

Original: Lample et al., 2019), we measure the former with a separately trained style classiﬁer based on DistilBERT (Sanh et al., 2019), and content retention in terms of self-BLEU (Papineni et al., 2002) between the input and the predicted output.
Revised: Lample et al., 2019), we measure the former with a separately trained style classiﬁer based on DistilBERT (Sanh et al., 2019), and content retention in terms of self-BLEU (Papineni et al., 2002) between the input and the predicted output.

Original: Sometimes, comparing models is easiest via a single value.
Revised: Sometimes, comparing models is easiest via a single value.

Original: Therefore, others have argued to aggregate content retention and transfer accuracy metrics (Xu et al., 2018; Krishna et al., 2020).
Revised: Therefore, others have argued to aggregate content retention and transfer accuracy metrics (Xu et al., 2018; Krishna et al., 2020).

Original: We follow the argumentation of Krishna et al.
Revised: We follow the argumentation of Krishna et al.

Original: (2020) that this aggregation should be per sentence and compute a single score = 1 M (cid:80) Mi
Revised: (2020) that this aggregation should be per sentence and compute a single score = 1 M (cid:80) Mi

Original: =1 ACC(ˆ y ) · BLEU(ˆ y, x ) where x is the input sentence, ˆ y is the predicted sentence, and M is the number of data points.
Revised: =1 ACC(ˆ y ) · BLEU(ˆ y, x ) where x is the input sentence, ˆ y is the predicted sentence, and M is the number of data points.

Original: For readability, we multiply all metrics by 100 before reporting.
Revised: For readability, we multiply all metrics by 100 before reporting.

Original: As is standard practice in summarization, we evaluate performance on this task with ROUGE-L (Lin, 2004).
Revised: As is standard practice in summarization, we evaluate performance on this task with ROUGE-L (Lin, 2004).

Original: Note, however, that ROUGE scores can be misleading if summaries are longer than the gold summary (Napoles et al., 2011).
Revised: Note, however, that ROUGE scores can be misleading if summaries are longer than the gold summary (Napoles et al., 2011).

Original: For this reason, we also report the average output length.
Revised: For this reason, we also report the average output length.

Original: 5. Y ELP -R EVIEWS
Revised: 5. Y ELP -R EVIEWS

Original: Our hypothesis is that AEs with a single vector bottleneck are unable to reliably compress the text when it is too long.
Revised: Our hypothesis is that AEs with a single vector bottleneck are unable to reliably compress the text when it is too long.

Original: Here, we test if this holds true even for a large single-vector model with d =512 .
Revised: Here, we test if this holds true even for a large single-vector model with d =512 .

Original: To this end, we create the dataset Yelp-Reviews , which consists of strongly positive and strongly negative English restaurant reviews on Yelp (see appendix A.3.1 for a detailed description).
Revised: To this end, we create the dataset Yelp-Reviews , which consists of strongly positive and strongly negative English restaurant reviews on Yelp (see appendix A.3.1 for a detailed description).

Original: This dataset is very similar to Yelp-Sentences introduced by Shen et al.
Revised: This dataset is very similar to Yelp-Sentences introduced by Shen et al.

Original: However, while YelpSentences consists of single sentences of about 10 words on average, Yelp-Reviews consists of entire reviews of 52 words on average.
Revised: However, while YelpSentences consists of single sentences of about 10 words on average, Yelp-Reviews consists of entire reviews of 52 words on average.

Original: To assess the reconstruction ability of BoV-AEs and ﬁxed-sized
Revised: To assess the reconstruction ability of BoV-AEs and ﬁxed-sized

Original: AEs, we train 5 models that only differ in the target ratio.
Revised: AEs, we train 5 models that only differ in the target ratio.

Original: Models named L0-r denote L0Drop-based
Revised: Models named L0-r denote L0Drop-based

Original: BoV-AE models with a target ratio r .
Revised: BoV-AE models with a target ratio r .

Original: The single vector ﬁxed-size AE is obtained by averaging the representations at the last layer of the encoder.
Revised: The single vector ﬁxed-size AE is obtained by averaging the representations at the last layer of the encoder.

Original: For style transfer, we train a Transformer ++ mapping using the loss described in Section 2.
Revised: For style transfer, we train a Transformer ++ mapping using the loss described in Section 2.

Original: To obtain results at varying transfer levels, we train multiple times with varying λ sty .
Revised: To obtain results at varying transfer levels, we train multiple times with varying λ sty , resulting in multiple points for each model in Figure 3 and 5.

Original: Further details can be found in the appendix A.3.2.
Revised: Further details can be found in the appendix A.3.2.

Original: Results: The results (full graph shown in Figure 8 in the appendix) indicate that even large single vector models ( d =512 ) are unable to compress the text well; the NLL loss on the validation set of the ﬁxed-size model is ≈ 3 . 9 .
Revised: Results: The results (full graph shown in Figure 7 in the appendix) indicate that even large single vector models ( d =512 ) are unable to compress the text well; the NLL loss on the validation set of the ﬁxed-size model is ≈ 3 . 9 .

Original: L0-0.05 is only slightly better than the ﬁxed-size model, whereasL0-0.
Revised: L0-0.05 is only slightly better than the ﬁxed-size model, whereas

Original:  already reaches a substantially lower reconstruction loss ( ≈ 2 . 1 ).
Revised: L0-0.1 already reaches a substantially lower reconstruction loss ( ≈ 2 . 1 ).

Original: We evaluated the downstream sentiment transfer performance of Transformer ++ with L0-0.1 3 and the ﬁxed-size model, respectively.
Revised: We evaluated the downstream sentiment transfer performance of Transformer ++ with L0-0.1 3 and the ﬁxed-size model, respectively.

Original: Figure 3 shows a scatter plot of the results, where results that are further to the top-right corner are better.
Revised: Figure 3 shows a scatter plot of the results, where results that are further to the top-right corner are better.

Original: We see that at a comparable transfer level, the BoV is substantially better at retaining the input content.
Revised: We see that at a comparable transfer level, the BoV is substantially better at retaining the input content.

Original: This supports hypothesis H1 that variable-size BoV models are particularly beneﬁcial in cases where the text length is too long to be encoded in a single-vector.
Revised: This supports hypothesis H1 that variable-size BoV models are particularly beneﬁcial in cases where the text length is too long to be encoded in a single-vector.

Original: 5. Y ELP -S ENTENCES
Revised: 5. Y ELP -S ENTENCES

Original: In order to answer research question H2 , we perform a large set of controlled experiments over our model’s components.
Revised: In order to answer research question H2 , we perform a large set of controlled experiments over our model’s components.

Original: Due to the high computational demand, we turn to the popular YelpSentences sentiment transfer dataset by Shen et al.
Revised: Due to the high computational demand, we turn to the popular YelpSentences sentiment transfer dataset by Shen et al.

Original: Texts in this dataset are ≈ 10 words on average.
Revised: Texts in this dataset are ≈ 10 words on average.

Original: As these sentences are much easier to reconstruct, we set the embedding size to d =32 so that the condition for hypothesis H1 is still valid.
Revised: As these sentences are much easier to reconstruct, we set the embedding size to d =32 so that the condition for hypothesis H1 is still valid.

Original: Here, we again train BoV-AEs for target rates r = 0 .
Revised: Here, we again train BoV-AEs for target rates r = 0 .

Original: and then evaluate their reconstruction and style transfer ability in the same fashion as for Yelp-Reviews.
Revised: and then evaluate their reconstruction and style transfer ability in the same fashion as for Yelp-Reviews.

Original: Finally, we investigate the impact of the window size and differentiable
Revised: Finally, we investigate the impact of the window size and differentiable

Original: Hausdorff loss.
Revised: Hausdorff loss.

Original: Experimental details are given in appendix A.4.2.
Revised: Experimental details are given in appendix A.4.2.

Revised: For completeness, we provide an

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure][Table]
Revised: [Table] [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] [Figure]
Revised: sty .

Original:  content retention and style transfer are important for style transfer, the further a graph is to the top right, the better the model
Revised: As both content retention and style transfer are important for style transfer, the further a graph is to the top right, the better the model.

Revised: analysis of the computational complexity of BoV-AE in Appendix B.2.2, and a qualitative analysis in Appendix B.2.3.

Original: Reconstruction ability: Figure 4 shows the reconstruction loss on the validation set for the ﬁxedsize model compared to BoV models.
Revised: Reconstruction ability: Figure 4 shows the reconstruction loss on the validation set for the ﬁxedsize model compared to BoV models.

Original: The ﬁxed-size AE does not reach satisfactory reconstruction ability, converging at an NLL loss value of about 3.
Revised: The ﬁxed-size AE does not reach satisfactory reconstruction ability, converging at an NLL loss value of about 3.

Original: In contrast, BoV models are able to outperform the ﬁxed-size model considerably.
Revised: In contrast, BoV models are able to outperform the ﬁxed-size model considerably.

Original: As expected, higher target ratios lead to better reconstruction, because the model can use more vectors to store the information.
Revised: As expected, higher target ratios lead to better reconstruction, because the model can use more vectors to store the information.

Original: Models with a higher target ratio also reach their optimal loss value more quickly.
Revised: Models with a higher target ratio also reach their optimal loss value more quickly.

Original: While L0-0.6 approaches the best reconstruction value ( ≈ 1 . 0 ) eventually, the model needs more than 1 million training steps to reach it.
Revised: While L0-0.6 approaches the best reconstruction value ( ≈ 1 . 0 ) eventually, the model needs more than 1 million training steps to reach it.

Original: In contrast,
Revised: In contrast,

Original: L0-0.8 needs less than 100k steps to converge, which could indicate that L0-0.8 learns to copy rather then compress the input, resulting in a bad latent space.
Revised: L0-0.8 needs less than 100k steps to converge, which could indicate that L0-0.8 learns to copy rather then compress the input, resulting in a bad latent space.

Original: L0-0.4 yields to a higher loss, but is still drastically better than the ﬁxed size model.
Revised: L0-0.4 yields to a higher loss, but is still drastically better than the ﬁxed size model.

Original: L0-0.2 is not enough to outperform the ﬁxed-size model.
Revised: L0-0.2 is not enough to outperform the ﬁxed-size model.

Original: Overall, these results show we have the right settings for evaluating H1 and H2 , as 10 words is too long to be encoded well into a single vector of d =32 , whereas a BoV-AE with a high enough target ratio r can ﬁt it well.
Revised: Overall, these results show we have the right settings for evaluating H1 and H2 , as 10 words is too long to be encoded well into a single vector of d =32 , whereas a BoV-AE with a high enough target ratio r can ﬁt it well.

Original: Style transfer ability: Results are shown in Figure 5.
Revised: Style transfer ability: Results are shown in Figure 5.

Original: Up to r =0 .
Revised: Up to r =0 .

Original: 6 , they correspond well to the reconstruction ability, in that BoV models with higher target ratios yield higher self-BLEU scores at comparable transfer abilities, outperforming the ﬁxed-size model ( H1 ).
Revised: 6 , they correspond well to the reconstruction ability, in that BoV models with higher target ratios yield higher self-BLEU scores at comparable transfer abilities, outperforming the ﬁxed-size model ( H1 ).

Original: However, at r =0 . 8 , the performance suddenly deteriorates at medium to high transfer levels.
Revised: However, at r =0 . 8 , the performance suddenly deteriorates at medium to high transfer levels.

Original: This supports the hypothesis that L0-0.8 lacks smoothness in the embedding space due to insufﬁcient regularization, which in turn complicates downstream training.
Revised: This supports the hypothesis that L0-0.8 lacks smoothness in the embedding space due to insufﬁcient regularization, which in turn complicates downstream training.

Original: This is the ﬁrst piece of evidence that L0Drop is necessary for the success of our model ( H2 ).
Revised: This is the ﬁrst piece of evidence that L0Drop is necessary for the success of our model ( H2 ).

Original: Window size: The window size k determines which bag sizes around the input bag size we backpropagate from.
Revised: Window size: The window size k determines which bag sizes around the input bag size we backpropagate from.

Original: Here, we investigates its inﬂuence on the model’s performance.
Revised: Here, we investigates its inﬂuence on the model’s performance.

Original: Since the λ sty hyperparameter is very sensitive to other model hyperparameters, we train with varying λ sty for each ﬁxed window size and report the best style transfer score for each window size.
Revised: Since the λ sty hyperparameter is very sensitive to other model hyperparameters, we train with varying λ sty for each ﬁxed window size and report the best style transfer score for each window size.

Original: Further exper- imental details and full results can be found in appendix A.4.3.
Revised: Further experimental details and full results can be found in appendix A.4.3.

Original: Our results indicate that increasing the window size from zero (score 22.7) is beneﬁcial up to some point ( k =5 , score 32.6), whereas increasing by too much ( k =20 , score 15.6) is detrimental to model performance even compared to a window size of zero.
Revised: Our results indicate that increasing the window size from zero (score 22.7) is beneﬁcial up to some point ( k =5 , score 32.6), whereas increasing by too much ( k =20 , score 15.6) is detrimental to model performance even compared to a window size of zero.

Original: We hypothesize that backpropagating bags that are either very small or very large is detrimental because it forces the model to adjust its parameters to optimize unrealistic bags, taking away capacity for ﬁtting realistic bags.
Revised: We hypothesize that backpropagating bags that are either very small or very large is detrimental because it forces the model to adjust its parameters to optimize unrealistic bags, taking away capacity for ﬁtting realistic bags.

Original: Differentiable Hausdorff:
Revised: Differentiable Hausdorff:

Original: In Section 4.3, we argue that the min operation should be replaced by softmin in order to facilitate backpropagation.
Revised: In Section 4.3, we argue that the min operation should be replaced by softmin in order to facilitate backpropagation.

Original: Here, we test if the differentiable version is really necessary, that is, we compare Eq.
Revised: Here, we test if the differentiable version is really necessary, that is, we compare Eq.

Original: 6 to Eq.
Revised: 6 to Eq.

Original: Like before, we train the two variants with different λ sty and select the best style transfer score on the validation set.
Revised: Like before, we train the two variants with different λ sty and select the best style transfer score on the validation set.

Original: Further experimental details can be found in appendix A.4.3.
Revised: Further experimental details can be found in appendix A.4.3.

Original: The difference is substantial: Average Hausdorff reaches 14.6, whereas differentiable Hausdorff reaches 24.2.
Revised: The difference is substantial: Average Hausdorff reaches 14.6, whereas differentiable Hausdorff reaches 24.2.

Original: We hypothesize that this discrepancy is due to the difﬁcult nature of the style transfer problem, which requires carefully balancing the two objectives, content retention (via Hausdorff) and style transfer (via the classiﬁer).
Revised: We hypothesize that this discrepancy is due to the difﬁcult nature of the style transfer problem, which requires carefully balancing the two objectives, content retention (via Hausdorff) and style transfer (via the classiﬁer).

Original: This is easier when the objective functions are smooth, which is the key advantage of differentiable Hausdorff.
Revised: This is easier when the objective functions are smooth, which is the key advantage of differentiable Hausdorff.

Original: 5. S ENTENCE S UMMARIZATION
Revised: 5. S ENTENCE S UMMARIZATION

Original: In sentence summarization (Rush et al., 2015), the goal is to capture the essence of a sentence in fewer words.
Revised: In sentence summarization (Rush et al., 2015), the goal is to capture the essence of a sentence in fewer words.

Original: We evaluate on the Gigaword corpus (Graff et al., 2003) similar to Rush et al.
Revised: We evaluate on the Gigaword corpus (Graff et al., 2003) similar to Rush et al.

Original: This corpus consists of more than 8.5 million training samples, but we use a random subset of 500k to limit the computational cost.
Revised: This corpus consists of more than 8.5 million training samples, but we use a random subset of 500k to limit the computational cost.

Original: Inputs are on average 27 words long, which is medium length compared to the other two datasets in this study.
Revised: Inputs are on average 27 words long, which is medium length compared to the other two datasets in this study.

Original: We use moderately sized vectors of d =128 and again train different BoV-AEs with target ratios r = 0 .
Revised: We use moderately sized vectors of d =128 and again train different BoV-AEs with target ratios r = 0 .

Original: When applying the model to the sentence summarization downstream task, we train using the loss term L ( ˆz y )
Revised: When applying the model to the sentence summarization downstream task, we train using the loss term L ( ˆz y )

Original: =
Revised: =

Original: L sim ( z x , ˆz y )
Revised: L sim ( z x , ˆz y )

Original: + λ len L len ( ˆz y ) .
Revised: + λ len L len ( ˆz y ) .

Original: This loss term is conceptually similar to the loss term used for style transfer, except that L len denotes the prediction of a  length regressor.
Revised: This loss term is conceptually similar to the loss term used for style transfer, except that L len denotes the prediction of a model trained to predict the length of the input text from the text’s latent representation (the shorter the better).

Original: We train with varying values of λ len = 0 .
Revised: We train with varying values of λ len = 0 .

Original: 5 , 1 , 2 , 5 , 10 and select the best model (ROUGE-L) on the development set.
Revised: 5 , 1 , 2 , 5 , 10 and select the best model (ROUGE-L) on the development set.

Original: Intuitively, this model learns to retain as much from the input as possible while minimizing the output length.
Revised: Intuitively, this model learns to retain as much from the input as possible while minimizing the output length.

Original: Note that this model of summarization could certainly be improved further, e.g. by accounting for relevancy and informativeness of the output (Peyrard, 2019).
Revised: Note that this model of summarization could certainly be improved further, e.g. by accounting for relevancy and informativeness of the output (Peyrard, 2019).

Original: However, our goal is not to create the best task-speciﬁc model possible, so these considerations are out of scope for this paper.
Revised: However, our goal is not to create the best task-speciﬁc model possible, so these considerations are out of scope for this paper.

Original: The input texts in this task are relatively long.
Revised: The input texts in this task are relatively long.

Original: Due to the higher number of vectors in a BoV, it may be difﬁcult to learn the mapping, especially for large target ratios r .
Revised: Due to the higher number of vectors in a BoV, it may be difﬁcult to learn the mapping, especially for large target ratios r .

Original: We experiment with
Revised: We experiment with

Original: Transformer ++ to observe to what extent this can facilitate learning.
Revised: Transformer ++ to observe to what extent this can facilitate learning.

Original: Results: Despite the moderately large vector dimensionality, the single-vector bottleneck model achieves only considerably lower reconstruction performance than the BoV models (for details, please refer to Figure 8 in the appendix).
Revised: Results: Despite the moderately large vector dimensionality, the single-vector bottleneck model achieves only considerably lower reconstruction performance than the BoV models (for details, please refer to Figure 8 in the appendix).

Original: Again, larger target rates r lead to faster convergence, and all BoV models converge to approximately the same validation loss value (0.9).
Revised: Again, larger target rates r lead to faster convergence, and all BoV models converge to approximately the same validation loss value (0.9).

Original: The only exception is L0-0.2 , which converges to a higher loss value (1.25), but is still vastly stronger than the ﬁxed size model (3.01).
Revised: The only exception is L0-0.2 , which converges to a higher loss value (1.25), but is still vastly stronger than the ﬁxed size model (3.01).

Original: However, as shown in Table 1, L0-0.2 performs the best on the downstream task, outperforming the single-vector model by more than 5 ROUGE-L points while simultaneously requiring much fewer output words.
Revised: However, as shown in Table 1, L0-0.2 performs the best on the downstream task, outperforming the single-vector model by more than 5 ROUGE-L points while simultaneously requiring much fewer output words.

Original: BoV models with higher target ratios than r =0 .
Revised: BoV models with higher target ratios than r =0 .

Original: 2 perform worse.
Revised: 2 perform worse.

Original: Moreover, the
Revised: Moreover, the

Original: Transformer ++ architecture tends to improve results, particularly with target rates r > 0 .
Revised: Transformer ++ architecture tends to improve results, particularly with target rates r > 0 .

Original: The
Revised: The

Original: ROUGE-L score itself does not improve for r =0 . 2 , but note that this comes at the expense of more than doubling the output length.
Revised: ROUGE-L score itself does not improve for r =0 . 2 , but note that this comes at the expense of more than doubling the output length.

Original: Also note that L0-0.6 and L0-0.8 only obtain relatively high scores because they produce long outputs that even exceed the length of the input.
Revised: Also note that L0-0.6 and L0-0.8 only obtain relatively high scores because they produce long outputs that even exceed the length of the input.

Original: In fact, for r = 0 . 6 , 0 .no value of λ len produces outputs that are reasonably good ( > 10 ROUGE-L) and short ( <
Revised: In fact, for r = 0 . 6 , 0 .no value of λ len produces outputs that are reasonably good ( > 10 ROUGE-L) and short ( <

Original: BLEU)
Revised: BLEU)

Original: at the same time.
Revised: at the same time.

Original: The above results conﬁrm both our hypotheses: First ( H1 ), it is beneﬁcial to use a BoV model over a single-vector model to reduce the compression issues induced by the ﬁxed-size bottleneck.
Revised: The above results conﬁrm both our hypotheses: First ( H1 ), it is beneﬁcial to use a BoV model over a single-vector model to reduce the compression issues induced by the ﬁxed-size bottleneck.

Original: Secondly ( H2 ), when using a BoV model, it is imperative to regularize the number of vectors in the bag as a way of smoothing the embedding space, making it easier to learn the mapping for unsupervised text generation tasks.
Revised: Secondly ( H2 ), when using a BoV model, it is imperative to regularize the number of vectors in the bag as a way of smoothing the embedding space, making it easier to learn the mapping for unsupervised text generation tasks.

Original: Moreover, if the number of vectors in the bag is large, our Transformer ++ architecture can substantially facilitate learning the mapping.
Revised: Moreover, if the number of vectors in the bag is large, our Transformer ++ architecture can substantially facilitate learning the mapping.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Manipulations in latent space: Besides Emb2Emb , latent space manipulations for textual style transfer are performed either via gradient descent (Wang et al., 2019; Liu et al., 2020) or by adding constant style vectors to the input (Shen et al., 2020; Montero et al., 2021).
Revised: Manipulations in latent space: Besides Emb2Emb , latent space manipulations for textual style transfer are performed either via gradient descent (Wang et al., 2019; Liu et al., 2020) or by adding constant style vectors to the input (Shen et al., 2020; Montero et al., 2021).

Original: In computer vision, discovering latent space manipulations for image style transfer has recently become a topic of increased interest, in both supervised (Jahanian et al., 2020; Zhuang et al., 2021) and unsupervised ways (H¨ark¨onen et al., 2020; Voynov & Babenko, 2020).
Revised: In computer vision, discovering latent space manipulations for image style transfer has recently become a topic of increased interest, in both supervised (Jahanian et al., 2020; Zhuang et al., 2021) and unsupervised ways (H¨ark¨onen et al., 2020; Voynov & Babenko, 2020).

Original: While these vision methods are similar to
Revised: While these vision methods are similar to

Original: Emb2Emb conceptually, they differ from our work in important ways.
Revised: Emb2Emb conceptually, they differ from our work in important ways.

Original: First, they focus on the latent space of GANs (Goodfellow et al., 2014), which work well for image generation but are known to struggle with text (Caccia et al., 2020).
Revised: First, they focus on the latent space of GANs (Goodfellow et al., 2014), which work well for image generation but are known to struggle with text (Caccia et al., 2020).

Original: Secondly, images typically have a ﬁxed size, and consequently their latent representations consist of single vectors.
Revised: Secondly, images typically have a ﬁxed size, and consequently their latent representations consist of single vectors.

Original: Our work focuses on data of variable size, which may have important insights for modalities other than text, e.g. videos and speech.
Revised: Our work focuses on data of variable size, which may have important insights for modalities other than text, e.g. videos and speech.

Original: Unsupervised conditional text generation: Modern unsupervised conditional text generation approaches are based on either (a) language models (LMs) or (b) autoencoders (AEs).
Revised: Unsupervised conditional text generation: Modern unsupervised conditional text generation approaches are based on either (a) language models (LMs) or (b) autoencoders (AEs).

Original: (a) One type of
Revised: (a) One type of

Original: LM approach explicitly conditions on attributes during pretraining (Keskar et al., 2019), which puts restrictions on the training data that can be used for training.
Revised: LM approach explicitly conditions on attributes during pretraining (Keskar et al., 2019), which puts restrictions on the training data that can be used for training.

Original: Another type adapts pretrained LMs for conditional text generation by learning modiﬁcations in the embedding space (Dathathri et al., 2020).
Revised: Another type adapts pretrained LMs for conditional text generation by learning modiﬁcations in the embedding space (Dathathri et al., 2020).

Original: These approaches work well because LMs are pretrained with very large amounts of data and compute power, which results in exceptional generative ability (Radford et al., 2019; Brown et al., 2020) that even enables impressive zero-shot style transfer results (Reif et al., 2021).
Revised: These approaches work well because LMs are pretrained with very large amounts of data and compute power, which results in exceptional generative ability (Radford et al., 2019; Brown et al., 2020) that even enables impressive zero-shot style transfer results (Reif et al., 2021).

Original: However, in contrast to AEs, LMs are not designed to have a latent space that facilitates learning in it.
Revised: However, in contrast to AEs, LMs are not designed to have a latent space that facilitates learning in it.

Original: We therefore argue that AE approaches could perform even better than LMs if they were given equal resources.
Revised: We therefore argue that AE approaches could perform even better than LMs if they were given equal resources.

Original: This motivates our research.
Revised: This motivates our research.

Original: (b) A very common approach to AE-based unsupervised conditional text generation is to learn a shared latent space for input and output corpora that is agnostic to the attribute of interest (e.g., sentiment transfer (Shen et al., 2017), style transfer (Lample et al., 2019), summarization (Liu et al., 2019), machine translation (Artetxe et al., 2018)).
Revised: (b) A very common approach to AE-based unsupervised conditional text generation is to learn a shared latent space for input and output corpora that is agnostic to the attribute of interest (e.g., sentiment transfer (Shen et al., 2017), style transfer (Lample et al., 2019), summarization (Liu et al., 2019), machine translation (Artetxe et al., 2018)).

Original: However, in these approaches, the decoder is explicitly conditioned on the desired attribute that must be available for all data points, complicating pretraining on unlabeled data.
Revised: However, in these approaches, the decoder is explicitly conditioned on the desired attribute that must be available for all data points, complicating pretraining on unlabeled data.

Original: To overcome this, Mai et al.
Revised: To overcome this, Mai et al.

Original: (2020) recently proposed Emb2Emb , which disentangles AE pretraining from learning to change the attributes via a simple mapping.
Revised: (2020) recently proposed Emb2Emb , which disentangles AE pretraining from learning to change the attributes via a simple mapping.

Original: Our paper makes an important contribution by improving the expressivity of Emb2Emb through variable-size representations.
Revised: Our paper makes an important contribution by improving the expressivity of Emb2Emb through variable-size representations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our paper addresses a fundamental research question from representation learning: How do we learn text representations in such a way that NLP tasks, speciﬁcally conditional text generation, can be learned in the latent space?
Revised: Our paper addresses a fundamental research question from representation learning: How do we learn text representations in such a way that NLP tasks, speciﬁcally conditional text generation, can be learned in the latent space?

Original: Previous work ( Emb2Emb ) uses single-vector bottleneck autoencoders, which we argued to be fundamentally limited in how much text they can capture.
Revised: Previous work ( Emb2Emb ) uses single-vector bottleneck autoencoders, which we argued to be fundamentally limited in how much text they can capture.

Original: We presented Bagof-Vectors Autoencoders whose latent representation grows in size with the input as in attentionbased models.
Revised: We presented Bagof-Vectors Autoencoders whose latent representation grows in size with the input as in attentionbased models.

Original: We proposed L0Drop regularization, Transformer ++ , and differentiable Hausdorff to facilitate training in its embedding space.
Revised: We proposed L0Drop regularization, Transformer ++ , and differentiable Hausdorff to facilitate training in its embedding space.

Original: Controlled experiments revealed that BoV-AEs perform substantially better when the text is too long to be encoded into a single vector, even of size d = 512 .
Revised: Controlled experiments revealed that BoV-AEs perform substantially better when the text is too long to be encoded into a single vector, even of size d = 512 .

Original: Our study is fundamental in nature; we do not focus on any particular task.
Revised: Our study is fundamental in nature; we do not focus on any particular task.

Original: Instead, we systematically demonstrate the beneﬁt of Emb2Emb with variable-size representations rather than ﬁxed-sized representations via controlled experiments.
Revised: Instead, we systematically demonstrate the beneﬁt of Emb2Emb with variable-size representations rather than ﬁxed-sized representations via controlled experiments.

Original: However, our model is in principle ﬁt for the future.
Revised: However, our model is in principle ﬁt for the future.

Original: Mai et al.
Revised: Mai et al.

Original: (2020) showed that the Emb2Emb framework beneﬁts immensely from unlabeled data.
Revised: (2020) showed that the Emb2Emb framework beneﬁts immensely from unlabeled data.

Original: As such, given enough compute and data for large-scale pretraining, Bag-of-Vectors Autoencoders could have the potential to become a Foundation Model (Bommasani et al., 2021) like BERT and
Revised: As such, given enough compute and data for large-scale pretraining, Bag-of-Vectors Autoencoders could have the potential to become a Foundation Model (Bommasani et al., 2021) like BERT and

Original: GPT-3.
Revised: GPT-3.

Original: Our study paves the way for the application of BoV-AEs for unsupervised tasks by demonstrating how to learn in their latent space.
Revised: Our study paves the way for the application of BoV-AEs for unsupervised tasks by demonstrating how to learn in their latent space.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Applications The focus of our study is not any particular application, but concerns fundamental questions in unsupervised conditional text generation in general.
Revised: Applications The focus of our study is not any particular application, but concerns fundamental questions in unsupervised conditional text generation in general.

Original: Unsupervised applications are useful in scenarios where few annotations exists, which is particularly common in understudied low-resource languages (e.g. unsupervised neural machine translation (Kuwanto et al., 2021)).
Revised: Unsupervised applications are useful in scenarios where few annotations exists, which is particularly common in understudied low-resource languages (e.g. unsupervised neural machine translation (Kuwanto et al., 2021)).

Original: Of course, oftentimes unsupervised solutions perform worse than supervised ones, requiring extra care during deployment to avoid harm from potential mistakes.
Revised: Of course, oftentimes unsupervised solutions perform worse than supervised ones, requiring extra care during deployment to avoid harm from potential mistakes.

Original: Despite the fundamental nature of our study, we test our model on two concrete problems, a) text style transfer and b) sentence summarization.
Revised: Despite the fundamental nature of our study, we test our model on two concrete problems, a) text style transfer and b) sentence summarization.

Original: a) Style transfer has applications that are beneﬁcial to society, such as expressing ”complicated” text in simpler terms ( text simpliﬁcation ) or avoiding potentially offensive language ( detoxiﬁcation ), both of which are particularly beneﬁcial for traditionally underprivileged groups such as non-native English speakers.
Revised: a) Style transfer has applications that are beneﬁcial to society, such as expressing ”complicated” text in simpler terms ( text simpliﬁcation ) or avoiding potentially offensive language ( detoxiﬁcation ), both of which are particularly beneﬁcial for traditionally underprivileged groups such as non-native English speakers.

Original: However, the same technology can also be used maliciously by simply inverting the style transfer direction.
Revised: However, the same technology can also be used maliciously by simply inverting the style transfer direction.

Original: In this paper, we decided to study sentiment transfer of restaurant reviews as a style transfer task.
Revised: In this paper, we decided to study sentiment transfer of restaurant reviews as a style transfer task.

Original: The reasons are primarily practical; deriving both from the Yelp dataset, we can study the effectiveness of our model on two datasets (sentences and full reviews) that are very similar in content but considerably different in length.
Revised: The reasons are primarily practical; deriving both from the Yelp dataset, we can study the effectiveness of our model on two datasets (sentences and full reviews) that are very similar in content but considerably different in length.

Original: On one hand, this allows us to demonstrate the effectiveness of our model in a realistic, but computationally demanding setting.
Revised: On one hand, this allows us to demonstrate the effectiveness of our model in a realistic, but computationally demanding setting.

Original: On the other hand, we can perform ablations in a less expensive setting.
Revised: On the other hand, we can perform ablations in a less expensive setting.

Original: Apart from serving as a test bed for scientiﬁc research, sentiment transfer itself has no obvious real-world application.
Revised: Apart from serving as a test bed for scientiﬁc research, sentiment transfer itself has no obvious real-world application.

Original: With enough imagination one can construe a scenario where a bad actor hacks into the database of a review platform like Yelp to e.g. manipulate the content of existing reviews.
Revised: With enough imagination one can construe a scenario where a bad actor hacks into the database of a review platform like Yelp to e.g. manipulate the content of existing reviews.

Original: However, we rate this as highly unrealistic due to high opportunity cost, as it is much easier to generate fake reviews with large language models rather than hack into a system and alter existing reviews.
Revised: However, we rate this as highly unrealistic due to high opportunity cost, as it is much easier to generate fake reviews with large language models rather than hack into a system and alter existing reviews.

Original: b) Summarization systems can be very valuable for society by enabling people to process information faster.
Revised: b) Summarization systems can be very valuable for society by enabling people to process information faster.

Original: But this depends on the system’s output to be mostly factual, which neural summarization systems struggle with (Maynez et al., 2020).
Revised: But this depends on the system’s output to be mostly factual, which neural summarization systems struggle with (Maynez et al., 2020).

Original: Unfaithful outputs may convey misinformation, which can potentially harm users.
Revised: Unfaithful outputs may convey misinformation, which can potentially harm users.

Original: Deployment
Revised: Deployment

Original: While we argue above that sentiment transfer has no useful real-world application, the model can still be deployed for demonstration purposes, or be trained and deployed for other tasks, e.g., sentence simpliﬁcation.
Revised: While we argue above that sentiment transfer has no useful real-world application, the model can still be deployed for demonstration purposes, or be trained and deployed for other tasks, e.g., sentence simpliﬁcation.

Original: However, we urge not to deploy the models developed in this paper directly without adaptation for several reasons.
Revised: However, we urge not to deploy the models developed in this paper directly without adaptation for several reasons.

Original: i)
Revised: i)

Original: The absolute performance is suboptimal (e.g., no large-scale pretraining) and hence makes many mistakes that a real-world application should avoid to prevent harm.
Revised: The absolute performance is suboptimal (e.g., no large-scale pretraining) and hence makes many mistakes that a real-world application should avoid to prevent harm.

Original: ii)
Revised: ii)

Original: The model can occasionally produce toxic output.
Revised: The model can occasionally produce toxic output.

Original: Of course, the extent to which this happens strongly depends on the training data.
Revised: Of course, the extent to which this happens strongly depends on the training data.

Original: E.g., Yelp restaurant reviews can sometimes contain vulgar language.
Revised: E.g., Yelp restaurant reviews can sometimes contain vulgar language.

Original: Any real-world application should hence consider pre- and post-ﬁltering methods.
Revised: Any real-world application should hence consider pre- and post-ﬁltering methods.

Original: iii)
Revised: iii)

Original: The model might be biased towards certain populations, the extent of which is not the subject of this study.
Revised: The model might be biased towards certain populations, the extent of which is not the subject of this study.

Original: For example, the sentiment transfer models would likely work better for fast food restaurants than restaurants of African cuisine, because the former is more common in the mostly US-centric data that the model is trained on.
Revised: For example, the sentiment transfer models would likely work better for fast food restaurants than restaurants of African cuisine, because the former is more common in the mostly US-centric data that the model is trained on.

Original: A real-world application needs to consider the requirements of the target audience.
Revised: A real-world application needs to consider the requirements of the target audience.

Original: Similarly, we argue that the sentence summarization model studied in this paper needs further improvements before deployment, some of which we mentioned in the main paper.
Revised: Similarly, we argue that the sentence summarization model studied in this paper needs further improvements before deployment, some of which we mentioned in the main paper.

Original: Large-scale pretraining could also help to mitigate hallucinated facts (Maynez et al., 2020).
Revised: Large-scale pretraining could also help to mitigate hallucinated facts (Maynez et al., 2020).

Original: Dataset The Yelp-Reviews dataset is a direct derivative of the Yelp Open Dataset 4 .
Revised: Dataset The Yelp-Reviews dataset is a direct derivative of the Yelp Open Dataset 4 .

Original: Their license agreement states that any derivative remains the property of Yelp, hence we can not directly release the dataset.
Revised: Their license agreement states that any derivative remains the property of Yelp, hence we can not directly release the dataset.

Original: However, academic researchers can easily obtain their own license for non-commercial use and recreate the dataset used in this study via the script we provide in the supplementary material.
Revised: However, academic researchers can easily obtain their own license for non-commercial use and recreate the dataset used in this study via the script we provide in the supplementary material.

Original: No further data collection was conducted.
Revised: No further data collection was conducted.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We took several precautions to ensure that our work is reproducible.
Revised: We took several precautions to ensure that our work is reproducible.

Original: Datasets Our study is based on two existing datasets, Gigaword sentence summarization, and
Revised: Datasets Our study is based on two existing datasets, Gigaword sentence summarization, and

Original: Yelp-Sentences style transfer.
Revised: Yelp-Sentences style transfer.

Original: For these two datasets, we provide scripts that preprocess them as in our study.
Revised: For these two datasets, we provide scripts that preprocess them as in our study.

Original: For Yelp-Reviews dataset, we provide a detailed description in appendix A.3.1.
Revised: For Yelp-Reviews dataset, we provide a detailed description in appendix A.3.1.

Original: Moreover, we provide a script that allows to construct the dataset as a derivative from Yelp data.
Revised: Moreover, we provide a script that allows to construct the dataset as a derivative from Yelp data.

Original: In order to get access to Yelp data, practitioners have to obtain a license from Yelp that is free of charge.
Revised: In order to get access to Yelp data, practitioners have to obtain a license from Yelp that is free of charge.

Original: The data may only be used for non-commercial or academic purposes, but this sufﬁces to reproduce our study.
Revised: The data may only be used for non-commercial or academic purposes, but this sufﬁces to reproduce our study.

Original: The Gigaword corpus is commonly used, and can be downloaded from the
Revised: The Gigaword corpus is commonly used, and can be downloaded from the

Original: Linguistic Dataset Consortium at https://catalog.ldc.upenn.edu/LDC2012T21 .
Revised: Linguistic Dataset Consortium at https://catalog.ldc.upenn.edu/LDC2012T21 .

Original: For downloading, a membership is mandatory, or otherwise fees apply.
Revised: For downloading, a membership is mandatory, or otherwise fees apply.

Original: However, this commonplace in
Revised: However, this commonplace in

Original: NLP research institutes.
Revised: NLP research institutes.

Original: Code We provide anonymized code to reproduce all our experiment in the supplementary materials.
Revised: Code We provide anonymized code to reproduce all our experiment in the supplementary materials.

Original: Upon acceptance, the code will be made available to the public.
Revised: Upon acceptance, the code will be made available to the public.

Original: Experiments We provide details on each experiment’s setup in the appendix.
Revised: Experiments We provide details on each experiment’s setup in the appendix.

Original: However, it’s impractical to report all details that may impact the outcome.
Revised: However, it’s impractical to report all details that may impact the outcome.

Original: Therefore, for each experiment we additionally provide a csv ﬁle in the supplementary material.
Revised: Therefore, for each experiment we additionally provide a csv ﬁle in the supplementary material.

Original: The ﬁle contains information on all training parameters, model hyperparameters and results.
Revised: The ﬁle contains information on all training parameters, model hyperparameters and results.

Original: In combination with the code, this allows to reconstruct almost the exact experimental setup used in our study apart from parameters that are beyond our control, such as the computation environment.
Revised: In combination with the code, this allows to reconstruct almost the exact experimental setup used in our study apart from parameters that are beyond our control, such as the computation environment.

Original: Appendix A E XPERIMENTAL D ETAILS
Revised: Appendix A E XPERIMENTAL D ETAILS

Original: Here, we describe the experimental setup used in our experiments.
Revised: Here, we describe the experimental setup used in our experiments.

Original: We try to be exhaustive, but the exact training conﬁgurations and code will also be given as downloadable source code for reference.
Revised: We try to be exhaustive, but the exact training conﬁgurations and code will also be given as downloadable source code for reference.

Original: [Table]
Revised: [Table]

Original: A. A UTOENCODER P RETRAINING
Revised: A. A UTOENCODER P RETRAINING

Original: All autoencoders consist of standard Transformer encoders and decoders (Vaswani et al., 2017), with 3 encoder and decoder layers, respectively.
Revised: All autoencoders consist of standard Transformer encoders and decoders (Vaswani et al., 2017), with 3 encoder and decoder layers, respectively.

Original: The Transformers have 2 heads and the dimensionality is set to the same as the latent vectors (Yelp-Reviews: 512, Yelp-Sentences: 32, Gigaword: 128).
Revised: The Transformers have 2 heads and the dimensionality is set to the same as the latent vectors (Yelp-Reviews: 512, Yelp-Sentences: 32, Gigaword: 128).

Original: In case of the ﬁxed sized model, the representations at the last layer are averaged.
Revised: In case of the ﬁxed sized model, the representations at the last layer are averaged.

Original: Otherwise we perform L0Drop as described in Section 3.
Revised: Otherwise we perform L0Drop as described in Section 3.

Original: We set λ L 0 = 10 for all BoV models and only vary the target ratio.
Revised: We set λ L 0 = 10 for all BoV models and only vary the target ratio.

Original: All models are trained with a dropout (Srivastava et al., 2014) probability of 0 .and
Revised: All models are trained with a dropout (Srivastava et al., 2014) probability of 0 .and

Original: a denoising objective, i.e, tokens have a chance of 10% to be dropped from the sentence.
Revised: a denoising objective, i.e, tokens have a chance of 10% to be dropped from the sentence.

Original: We train the model with the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of lr = 0 .
Revised: We train the model with the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of lr = 0 .

Original: (Yelp-Reviews and Gigaword) or lr = 0 .
Revised: (Yelp-Reviews and Gigaword) or lr = 0 .

Original: (Yelp-Sentences) and a batch size of 64.
Revised: (Yelp-Sentences) and a batch size of 64.

Original: We experimented with other learning rates ( 0 . 00005 , 0 . 0005 ) for the ﬁxed-size model on YelpReviews, but the results did not improve.
Revised: We experimented with other learning rates ( 0 . 00005 , 0 . 0005 ) for the ﬁxed-size model on YelpReviews, but the results did not improve.

Original: Models are trained for 2 million steps on Gigaword and
Revised: Models are trained for 2 million steps on Gigaword and

Original: Yelp-Reviews and for 1.5 million steps on Yelp-Sentences.
Revised: Yelp-Reviews and for 1.5 million steps on Yelp-Sentences.

Original: We check the validation set performance every 20,000 steps and select the best model according to validation reconstruction performance.
Revised: We check the validation set performance every 20,000 steps and select the best model according to validation reconstruction performance.

Original: A. D OWNSTREAM T ASK T RAINING
Revised: A. D OWNSTREAM T ASK T RAINING

Original: After the autoencoder pretraining, we train downstream by freezing the parameters of the encoder and decoder.
Revised: After the autoencoder pretraining, we train downstream by freezing the parameters of the encoder and decoder.

Original: The dimensionality of the one-layer mapping Φ (a Transformer decoder with 4 heads) is set to the same as the latent representation (Yelp-Reviews: 512, Yelp-Sentences: 32, Gigaword: 128).
Revised: The dimensionality of the one-layer mapping Φ (a Transformer decoder with 4 heads) is set to the same as the latent representation (Yelp-Reviews: 512, Yelp-Sentences: 32, Gigaword: 128).

Original: We set the maximum number of output vectors to N = 250 on Yelp-Reviews and Gigaword, and N = 30 on Yelp-Sentences.
Revised: We set the maximum number of output vectors to N = 250 on Yelp-Reviews and Gigaword, and N = 30 on Yelp-Sentences.

Original: The batch size is 64 for Yelp-Sentences and Gigaword andon Yelp-Reviews.
Revised: The batch size is 64 for Yelp-Sentences and Gigaword andon Yelp-Reviews.

Original: We train for 10 epochs on Yelp-Sentences and Gigaword, and for 3 epochs on
Revised: We train for 10 epochs on Yelp-Sentences and Gigaword, and for 3 epochs on

Original: Yelp-Reviews.
Revised: Yelp-Reviews.

Original: The validation performance is evaluated after each epoch.
Revised: The validation performance is evaluated after each epoch.

Original: Losses: In all tasks we have two loss components.
Revised: Losses: In all tasks we have two loss components.

Original: For L sim , we use differentiable Hausdorff unless speciﬁed otherwise (in the ablation).
Revised: For L sim , we use differentiable Hausdorff unless speciﬁed otherwise (in the ablation).

Original: L sty and L len depend on classiﬁers / regressors, which we train separately after the autoencoder pretraining as a one-layer Transformer encoder.
Revised: L sty and L len depend on classiﬁers / regressors, which we train separately after the autoencoder pretraining as a one-layer Transformer encoder.

Original: The embeddings are then averaged and plugged into a one-layer MLP whose hidden size is half of the input size and uses the tanh activation function.
Revised: The embeddings are then averaged and plugged into a one-layer MLP whose hidden size is half of the input size and uses the tanh activation function.

Original: These classiﬁers are trained via Adam ( lr = 0 . 0001 ) for 10 epochs and we evaluate the validation set performance after each.
Revised: These classiﬁers are trained via Adam ( lr = 0 . 0001 ) for 10 epochs and we evaluate the validation set performance after each.

Original: The total loss depends on a window size as described in Equation 4.
Revised: The total loss depends on a window size as described in Equation 4.

Original: For performance reasons (multiple computations of the loss), we set k = 0 unless speciﬁed differently.
Revised: For performance reasons (multiple computations of the loss), we set k = 0 unless speciﬁed differently.

Original: A. Y ELP -R EVIEWS A.3. D ATASET
Revised: A. Y ELP -R EVIEWS A.3. D ATASET

Original: The dataset was obtained from https://www.yelp.com/dataset in May 2021.
Revised: The dataset was obtained from https://www.yelp.com/dataset in May 2021.

Original: Our goal is to obtain texts long enough such they cannot be reconstructed by a reasonably sized autoencoder with a single-vector bottleneck.
Revised: Our goal is to obtain texts long enough such they cannot be reconstructed by a reasonably sized autoencoder with a single-vector bottleneck.

Original: We ﬁnd that to be the case when limiting ourselves to reviews of maximum 100 words.
Revised: We ﬁnd that to be the case when limiting ourselves to reviews of maximum 100 words.

Original: We apply this limit due to the computational complexity of Transformers on long texts.
Revised: We apply this limit due to the computational complexity of Transformers on long texts.

Original: Otherwise, we stick with similar ﬁltering criteria as Shen et al.
Revised: Otherwise, we stick with similar ﬁltering criteria as Shen et al.

Original: (2017): We only consider restaurant businesses.
Revised: (2017): We only consider restaurant businesses.

Original: We consider reviews with 1 or 2 stars as negative, and reviews with 5 stars as positive.
Revised: We consider reviews with 1 or 2 stars as negative, and reviews with 5 stars as positive.

Original: We don’t consider reviews with 3 or 4 stars to avoid including neutral reviews.
Revised: We don’t consider reviews with 3 or 4 stars to avoid including neutral reviews.

Original: We subsample 400,000 positive and negative reviews for training, respectively, and use 50,000 for validation and test set each.
Revised: We subsample 400,000 positive and negative reviews for training, respectively, and use 50,000 for validation and test set each.

Original: In order to demonstrate the usefulness of our model on long texts, we turn to the original Yelp dataset 5 .
Revised: In order to demonstrate the usefulness of our model on long texts, we turn to the original Yelp dataset 5 .

Original: Our goal is to obtain texts long enough such they cannot be reconstructed by a reasonably sized autoencoder with a single-vector bottleneck.
Revised: Our goal is to obtain texts long enough such they cannot be reconstructed by a reasonably sized autoencoder with a single-vector bottleneck.

Original: We ﬁnd that to be the case when limiting ourselves to reviews of maximum 100 words 6 .
Revised: We ﬁnd that to be the case when limiting ourselves to reviews of maximum 100 words 6 .

Original: Otherwise, we stick with similar ﬁltering criteria as
Revised: Otherwise, we stick with similar ﬁltering criteria as

Original: Shen et al.
Revised: Shen et al.

Original: (2017): We only consider restaurant businesses.
Revised: (2017): We only consider restaurant businesses.

Original: We consider reviews with 1 or 2 stars as negative, and reviews with 5 stars as positive.
Revised: We consider reviews with 1 or 2 stars as negative, and reviews with 5 stars as positive.

Original: We don’t consider reviews with 3 or 4 stars to avoid including neutral reviews.
Revised: We don’t consider reviews with 3 or 4 stars to avoid including neutral reviews.

Original: We subsample 400,000 positive and negative reviews for training, respectively, and use 50,000 for validation and test set each.
Revised: We subsample 400,000 positive and negative reviews for training, respectively, and use 50,000 for validation and test set each.

Original: A.3. D OWNSTREAM T RAINING
Revised: A.3. D OWNSTREAM T RAINING

Original: For both the ﬁxed-size model and the BoV model ( L0-0.1 ), we choose the best learning rate among lr = 0 .
Revised: For both the ﬁxed-size model and the BoV model ( L0-0.1 ), we choose the best learning rate among lr = 0 .

Original: 0001 and lr = 0 .
Revised: 0001 and lr = 0 .

Original: 0005 on the validation set and report test set results.
Revised: 0005 on the validation set and report test set results.

Original: We train with
Revised: We train with

Original: L sty ∈ { 0 . 1 , 0 . 2 , 0 .
Revised: L sty ∈ { 0 . 1 , 0 . 2 , 0 .

Original: 5 , 1 , 2 , 5 , 10 } , resulting in the scatter plot in Figure 3.
Revised: 5 , 1 , 2 , 5 , 10 } , resulting in the scatter plot in Figure 3.

Original: A. Y ELP -S ENTENCES A.4. D ATASET
Revised: A. Y ELP -S ENTENCES A.4. D ATASET

Original: Yelp-Sentences consists of the sentiment transfer dataset created by
Revised: Yelp-Sentences consists of the sentiment transfer dataset created by

Original: Shen et al.
Revised: Shen et al.

Original: (2017), who made their data available at https://github.com/shentianxiao/ language-style-transfer/tree/master/data/yelp .
Revised: (2017), who made their data available at https://github.com/shentianxiao/ language-style-transfer/tree/master/data/yelp .

Original: We use their data as is without further preprocessing.
Revised: We use their data as is without further preprocessing.

Original: Table 2 presents some basic statistics about this dataset.
Revised: Table 2 presents some basic statistics about this dataset.

Original: A.4. D OWNSTREAM T RAINING
Revised: A.4. D OWNSTREAM T RAINING

Original: We train BoV models with λ sty ∈ { 1 , 2 , 5 , 10 , 20 , 50 , 100 } .
Revised: We train BoV models with λ sty ∈ { 1 , 2 , 5 , 10 , 20 , 50 , 100 } .

Original: To make sure that our results are not due to insufﬁcient tuning, for the ﬁxed-sized model, we use the following larger range: { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 1 , 2 , 5 , 10 , 20 , 50 , 100 } .
Revised: To make sure that our results are not due to insufﬁcient tuning, for the ﬁxed-sized model, we use the following larger range: { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 1 , 2 , 5 , 10 , 20 , 50 , 100 } .

Original: All conﬁgurations are trained with lr = 0 . 0005 .
Revised: All conﬁgurations are trained with lr = 0 . 0005 .

Original: These results produce the scatter plot in Figure 5.
Revised: These results produce the scatter plot in Figure 5.

Original: A.4. A BLATIONS
Revised: A.4. A BLATIONS

Original: For the ablations on differentiable Hausdorff distance and the window size, we use the L0-0.6 model.
Revised: For the ablations on differentiable Hausdorff distance and the window size, we use the L0-0.6 model.

Original: For each option, we train with L sty ∈ { 0 . 1 , 0 . 2 , 0 .
Revised: For each option, we train with L sty ∈ { 0 . 1 , 0 . 2 , 0 .

Original: 5 , 1 , 2 , 5 , 10 , 20 , 40 , 60 , 80 , 100 } and report the best value in terms of style transfer score on the validation set.
Revised: 5 , 1 , 2 , 5 , 10 , 20 , 40 , 60 , 80 , 100 } and report the best value in terms of style transfer score on the validation set.

Original: A. S ENTENCE S UMMARIZATION A.5. D ATASET
Revised: A. S ENTENCE S UMMARIZATION A.5. D ATASET

Original: The dataset is based on the Gigaword corpus (Graff et al., 2003).
Revised: The dataset is based on the Gigaword corpus (Graff et al., 2003).

Original: We largely follow the preprocessing in Rush et al.
Revised: We largely follow the preprocessing in Rush et al.

Original: (2015), which we obtained from the paper’s
Revised: (2015), which we obtained from the paper’s

Original: GitHub repository at https://github.com/facebookarchive/NAMAS .
Revised: GitHub repository at https://github.com/facebookarchive/NAMAS .

Original: Different from them, we convert all inputs and outputs to lower case and use a smaller split (1 million examples).
Revised: Different from them, we convert all inputs and outputs to lower case and use a smaller split (1 million examples).

Original: We provide the scripts for constructing the dataset from a copy of the Gigaword corpus (which can be obtained from the
Revised: We provide the scripts for constructing the dataset from a copy of the Gigaword corpus (which can be obtained from the

Original: Linguistic Dataset Consortium) together with the rest of our code.
Revised: Linguistic Dataset Consortium) together with the rest of our code.

Original: A.5. D OWNSTREAM T RAINING
Revised: A.5. D OWNSTREAM T RAINING

Original: We train all models with lr = 0 .
Revised: We train all models with lr = 0 .

Original: For each target ratio r and each of Transformer and
Revised: For each target ratio r and each of Transformer and

Original: Transformer ++ , we select the best λ len ∈ { 0 . 1 , 0 . 2 , 0 .
Revised: Transformer ++ , we select the best λ len ∈ { 0 . 1 , 0 . 2 , 0 .

Original: 5 , 1 , 2 , 5 , 10 } in terms of ROUGE-L on the validation set and report test set results in Table 1.
Revised: 5 , 1 , 2 , 5 , 10 } in terms of ROUGE-L on the validation set and report test set results in Table 1.

Original: B A DDITIONAL R ESULTS B. Y ELP -R EVIEWS
Revised: B A DDITIONAL R ESULTS B. Y ELP -R EVIEWS

Original: In Figure 7, we plot the reconstruction ability of the ﬁxed-size model compared to the BoV-AEs on the validation set.
Revised: In Figure 7, we plot the reconstruction ability of the ﬁxed-size model compared to the BoV-AEs on the validation set.

Original: Again, despite a large dimensionality ( d = 512 ), the single-vector model achieves substantially lower reconstruction ability than BoV-AE.
Revised: Again, despite a large dimensionality ( d = 512 ), the single-vector model achieves substantially lower reconstruction ability than BoV-AE.

Original: With respect to the target sparsity rate, we ﬁnd that r = 0 .
Revised: With respect to the target sparsity rate, we ﬁnd that r = 0 .

Original: 1 is enough to reach dramatically better results than the ﬁxed-size model, whereas r = 0 .
Revised: 1 is enough to reach dramatically better results than the ﬁxed-size model, whereas r = 0 .

Original: 05 only reaches slightly better results after two million training steps.
Revised: 05 only reaches slightly better results after two million training steps.

Original: However, the plot shows clearly that
Revised: However, the plot shows clearly that

Original: L0-0.05 has not converged, suggesting that L0-0.05 could reach much better performance if trained for even longer.
Revised: L0-0.05 has not converged, suggesting that L0-0.05 could reach much better performance if trained for even longer.

Original: B. Y ELP -S ENTENCES 
Revised: B. Y ELP -S ENTENCES B.2. W INDOW SIZE

Original: Window size In Figure 6, we plot the style transfer score as a function of the window size.
Revised: In Figure 6, we plot the style transfer score as a function of the window size.

Original: As already described in Section 5.2, the performance improves almost monotonically up to a certain point, after which the performance decreases again.
Revised: As already described in Section 5.2, the performance improves almost monotonically up to a certain point, after which the performance decreases again.

Original: This demonstrates the importance of choosing the right window size
Revised: This demonstrates the importance of choosing the right window size.

Original: . B. S ENTENCE S UMMARIZATION
Revised: B.2. C OMPUTATION TIME

Revised: Our experiments have shown that bag-of-vector representations are more powerful than single-vector representations.

Revised: However, the increased capacity of BoV-AE comes at the expense of higher computation time.

Revised: The size of the latent representation impacts the computation time in two places: During cross-attention in the decoder and when computing the mapping.

Revised: Asymptotically, the decoder’s cross-attention mechanism computes O ( n · | s | ) dot-products, where n is the number of vectors in the latent representation and | s | is the length of the text sequence s .

Revised: When computing the mapping,

Revised: [Table]

Revised: [Table]

Revised: both at training and inference time, we produce a ﬁxed number N of vectors autoregressively, but in most applications, N can reasonably be bound by a linear function of n (e.g., 2 n in style transfer or 0 .

Revised: 5 n in summarization).

Revised: The mapping is essentially a Transformer decoder, so both the cross attention and self attention parts compute O ( n 2 ) dot-products.

Revised: Given that n = 1 for single-vector

Revised: AEs and n =

Revised: O ( | s | ) for BoV-AEs with L0Drop, we obtain the asymptotic complexities as shown in

Revised: Table 3.

Revised: To assess the empirical impact, we measure the wallclock time of Emb2Emb ’s ”Inference” stage (cf.

Revised: Figure 2).

Revised: We take separate measurements for encoding, mapping, and decoding, respectively.

Revised: Since decoding speed depends on the quality of generation (e.g., when the end-of-sequence symbol is generated late due to repetitions), we do the following to enable fairer comparisons.

Revised: We enforce the same ﬁxed number of decoding steps (10) in all models.

Revised: The mapping is set to produce as many output vectors as input vectors.

Revised: We use a batch size of 1, but note that the results would largely extend to larger batch sizes when binned batching is used.

Revised: The results are shown in Table 4.

Revised: Both the encoding and the mapping stages of Emb2Emb are more expensive in BoV models than in the ﬁxed-size model.

Revised: The difference in the encoding stage can be explained by the overhead through the L0Drop layer, which includes identifying near-zero gates and discarding their respective vectors.

Revised: The difference in the mapping grows with higher L0Drop target ratios.

Revised: This is expected since the number of autoregressive steps decreases with the target ratio.

Revised: Finally, we do not observe any meaningful speed differences between the models at decoding time.

Revised: This is somewhat surprising, but could be explained by two factors.

Revised: First, the self-attention part of the decoder already has a complexity of O ( | s | 2 ) , which probably dominates the total computation time.

Revised: Secondly, the computation of the dot-product is easy to parallelize.

Revised: In summary, we ﬁnd that BoV models are slower overall, especially in the mapping.

Revised: However, since our L0Drop implementation prunes near-zero vectors, lower target rates mitigated the additional computation overhead.

Revised: This is especially evident when comparing training speeds.

Revised: While L0-0.8 processes 15 sentences per second, L0-0.4 processes can process 21 (ﬁxed-size: 42).

Revised: B.2. Q UALITATIVE A NALYSIS

Revised: Standard autoencoders suffer from poor performance with Emb2Emb if the text is too long to be encoded into a single vector.

Revised: BoV-AEs were designed to alleviate this issue.

Revised: Here, we conduct a qualitative analysis of 10 randomly selected model outputs on Yelp-Sentences.

Revised: For comparability, we select models with similar levels of style transfer accuracy, namely the ﬁxed size model with a performance of 59% accuracy and 17 points self-BLEU to L0-0.4 with a performance of 55% accuracy and 38 points self-BLEU.

Revised: We randomly sample 10 examples and show them in Table 5.

Revised: By design of the Yelp-Sentences dataset (Shen et al., 2017), the input sentences are sentences drawn from negative reviews, whose sentiment are supposed to be changed to positive.

Revised: Note that due to how the dataset was constructed, some of the input sentences are already positive (#7) or just neutral (#2).

Revised: We observe several trends:

Revised: [Table]

Revised: generally speaking it was nothing worth coming back to .

Revised: but there here here and it will enjoy it

Revised: generally remain it was it worth it and always happy !

Revised: then why did n’t they put some in ?

Revised: then she , you ta are the in the

Revised: then

Revised: why n

Revised: t they put

Revised: some delicious !

Revised: horrible

Revised: experience !

Revised: horrible !

Revised: horrible experience !

Revised: it was a shame because we were really looking forward to

Revised: dining there .

Revised: it was a a fun , there and we have been to .

Revised: it really nice shame because we were really looking forward forward and fantastic !

Revised: sufﬁce to stay , this is not a great place to stay

Revised: sufﬁce to to not stay to this place is a stay .

Revised: sufﬁce is not stay , this is a great place and always great !

Revised: the chicken was weird .

Revised: the chicken was weird .

Revised: the chicken was weird .

Revised: my mom ordered the margarita panini which was pretty good .

Revised: my my margarita was ordered which was very good

Revised: my mom

Revised: ordered the mar-

Revised: garita

Revised: panini which was

Revised: pretty good .

Revised: i ’m not willing to take the chance .

Revised: i will deﬁnitely recommend your

Revised: time or you

Revised: i ’ m not willing to take the great .

Revised: i would say for the price point that it was uninspired .

Revised: i had this place at the food

Revised: , it ’s super .

Revised: i would say for the price point that it was delicious .

Revised: the only pool complaint i have

Revised: was from the last

Revised: day

Revised: of our

Revised: stay

Revised: the waitress was the the the the time here a last time the only pool complaint i have was from the day was wonderful !

Revised: The ﬁxed-sized model has a difﬁcult time retaining the aspect discussed in the input sentence (#10: staff instead of location, #9: food instead of price), whereas the BoV-AE stays on topic.

Revised: This is likely a consequence of the ﬁxed-sized model’s inability to encode the input well into a single vector.

Revised: The outputs of the ﬁxed-sized models are often completely unusable (#1, #2) or nonsensical (#5, #9, #10), whereas the outputs of the BoV-AE are at least intelligible.

Revised: In

Revised: absolute terms, the outputs of neither model are reliably

Revised: grammatical

Revised: or able to ﬂip

Revised: the sentiment.

Revised: This is understandable since no large pretrained language model is used.

Revised: We would like to stress that therefore our models should not be used in production as they are (cf.

Revised: Ethics Statement).

Revised: Large-scale pretraining is needed to produce coherent outputs (Brown et al., 2020), which then produces impressive outputs on style transfer (Reif et al., 2021).

Revised: As we argue in Section 1 and 7, our paper contributes to the foundation for large scale pretraining of autoencoder models to be used in Emb2Emb .

Revised: B. S ENTENCE S UMMARIZATION

Original: Figure 8 shows the development of the reconstruction loss on the validation set over the course ofmillion training steps.
Revised: Figure 8 shows the development of the reconstruction loss on the validation set over the course ofmillion training steps.

Original: Despite a moderately sized dimensionality ( d = 128 ), the single-vector bottleneck model achieves only considerably lower reconstruction performance than the BoV models.
Revised: Despite a moderately sized dimensionality ( d = 128 ), the single-vector bottleneck model achieves only considerably lower reconstruction performance than the BoV models.

Original: Moreover, we can see again that larger target rates r lead to faster convergence, and all BoV models converge to approximately the same validation loss value.
Revised: Moreover, we can see again that larger target rates r lead to faster convergence, and all BoV models converge to approximately the same validation loss value.

Original: [Figure]
Revised: [Figure]

Original: Validation
Revised: Validation

Original: reconstruction loss run fixed
Revised: reconstruction loss run fixed

Original: L0-0. L0-0.
Revised: L0-0. L0-0.

Original: L0-0. L0-0.
Revised: L0-0. L0-0.

Original: [Figure]
Revised: [Figure]

Original: Validation reconstruction loss
Revised: Validation reconstruction loss

Original: run fixed
Revised: run fixed

Original: L0-0. L0-0.
Revised: L0-0. L0-0.

Original: L0-0. L0-0.
Revised: L0-0. L0-0.