Original: L EARNING L OCALITY AND I SOTROPY IN D IALOGUE M ODELING
Revised: L EARNING L OCALITY AND I SOTROPY IN D IALOGUE M ODELING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing dialogue modeling methods have achieved promising performance on various dialogue tasks with the aid of Transformer and the large-scale pre-trained language models.
Revised: Existing dialogue modeling methods have achieved promising performance on various dialogue tasks with the aid of Transformer and the large-scale pre-trained language models.

Original: However, some recent studies revealed that the context representations produced by these methods suffer the problem of anisotropy .
Revised: However, some recent studies revealed that the context representations produced by these methods suffer the problem of anisotropy .

Original: In this paper, we find that the generated representations are also not conversational , losing the conversation structure information during the context modeling stage.
Revised: In this paper, we ﬁnd that the generated representations are also not conversational , losing the conversation structure information during the context modeling stage.

Original: To this end, we identify two properties in dialogue modeling, i.e., locality and isotropy, and present a simple method for dialogue representation calibration, namely SimDRC, to build isotropic and conversational feature spaces.
Revised: To this end, we identify two properties in dialogue modeling, i.e., locality and isotropy, and present a simple method for dialogue representation calibration, namely SimDRC, to build isotropic and conversational feature spaces.

Original: Experimental results show that our approach significantly outperforms current state-of-the-art models on three open-domain dialogue tasks with eight benchmarks across both automatic and human evaluation metrics.
Revised: Experimental results show that our approach signiﬁcantly outperforms current state-of-the-art models on three open-domain dialogue tasks with eight benchmarks.

Original: More in-depth analyses further confirm the effectiveness of our proposed approach.
Revised: More in-depth analyses further conﬁrm the effectiveness of our proposed approach.

Revised: We release the code at https://github.com/hahahawu/SimDRC .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dialogue modeling (Serban et al., 2016; Mehri et al., 2019; Liu et al., 2021) is to encode the raw text of the input dialogue to the contextual representations.
Revised: Dialogue modeling (Serban et al., 2016; Mehri et al., 2019; Liu et al., 2021) is to encode the raw text of the input dialogue to the contextual representations.

Original: Although the Transformer-based dialogue modeling methods (Hosseini-Asl et al., 2020; Liu et al., 2021) have achieved great success on various dialogue tasks, there are still some impediments in these methods that are not well explored nowadays.
Revised: Although the Transformer-based dialogue modeling methods (Hosseini-Asl et al., 2020; Liu et al., 2021) have achieved great success on various dialogue tasks, there are still some impediments in these methods that are not well explored nowadays.

Original: Specifically, recent studies (Ethayarajh, 2019; Su et al., 2022) have revealed that on dialogue generation tasks, the representations produced by existing dialogue modeling methods are anisotropic , i.e. features occupy a narrow cone in the vector space, thus leading to the problem of degeneration.
Revised: Speciﬁcally, recent studies (Ethayarajh, 2019; Su et al., 2022) have revealed that on dialogue generation tasks, the representations produced by existing dialogue modeling methods are anisotropic , i.e. features occupy a narrow cone in the vector space, thus leading to the problem of degeneration.

Original: To alleviate this problem, previous solutions (e.g. SimCTG)
Revised: To alleviate this problem, previous solutions (e.g. SimCTG)

Original: (Su et al., 2021; 2022) encourage the model to learn isotropic token embeddings by pushing away the representations of distinct tokens.
Revised: (Su et al., 2021; 2022) encourage the model to learn isotropic token embeddings by pushing away the representations of distinct tokens.

Original: While building the more discriminative and isotropic feature space, these methods still ignore learning dialogue-specific features, such as inter-speaker correlations and conversational structure information, in the dialogue modeling stage.
Revised: While building the more discriminative and isotropic feature space, these methods still ignore learning dialogue-speciﬁc features, such as inter-speaker correlations and conversational structure information, in the dialogue modeling stage.

Original: Therefore, a question is naturally raised - are the representations produced by existing dialogue modeling methods really conversational?
Revised: Therefore, a question is naturally raised - are the representations produced by existing dialogue modeling methods really conversational?

Original: To answer this question, in Figure 1(a), we showcase the cosine similarity matrix of token representations produced by BART (Lewis et al., 2020) that is well trained on response generation task.
Revised: To answer this question, in Figure 1(a), we showcase the cosine similarity matrix of token representations produced by BART (Lewis et al., 2020) that is well trained on response generation task.

Original: First, we can easily observe the phenomenon of anisotropy from the heatmap where the similarities of distinct tokens are relatively high, over 0.5 for most token pairs.
Revised: First, we can easily observe the phenomenon of anisotropy from the heatmap where the similarities of distinct tokens are relatively high, over 0.5 for most token pairs.

Original: Then, Figure 1(b) illustrates the similarity heatmap of token representations produced by SimCTG where the color is faded on the whole, suggesting the problem of anisotropy is relaxed.
Revised: Then, Figure 1(b) illustrates the similarity heatmap of token representations produced by SimCTG where the color is faded on the whole, suggesting the problem of anisotropy is relaxed.

Original: However, another critical problem still remains, is that the representations of tokens in different utterances are nearby to each other, making the utterance indistinguishable on the token representations.
Revised: However, another critical problem still remains, is that the representations of tokens in different utterances are nearby to each other, making the utterance indistinguishable on the token representations.

Original: It is undesirable that no conversational features can be captured from the token similarity matrix while the matrix is produced by a “dialogue modeling” method trained on the dialogue task using dialogue data.
Revised: It is undesirable that no conversational features can be captured from the token similarity matrix while the matrix is produced by a “dialogue modeling” method trained on the dialogue task using dialogue data.

Original: Ideally, we expect that the representations of tokens within an utterance are close to voice a concentrated idea of the utterance, and the representations of different utterances are discriminative and isotropic to convey the maximal information of the dialogue.
Revised: Ideally, we expect that the representations of tokens within an utterance are close to voice a concentrated idea of the utterance, and the representations of different utterances are discriminative and isotropic to convey the maximal information of the dialogue.

Original: Accordingly, the ideal similarity matrix of token representations should be similar to Figure 1(c), where tokens within an utterance are intensive and different utterances are easily distinguishable on representations.
Revised: Accordingly, the ideal similarity matrix of token representations should be similar to Figure 1(c), where tokens within an utterance are intensive and different utterances are easily distinguishable on representations.

Original: Our motivation is that humans pay more attention to the central idea of the utterance rather than how the utterance is organized by words when humans utter, and humans also prefer to express more information with fewer utterances (Woolnough et al., 2021).
Revised: Our motivation is that humans pay more attention to the central idea of the utterance rather than how the utterance is organized by words when humans utter, and humans also prefer to express more information with fewer utterances (Woolnough et al., 2021).

Original: Based on the above observation and motivation, we identify two properties, i.e., locality and isotropy in dialogue modeling, and then present SimDRC , a sim ple d ialogue r epresentation c alibration method, which encourages the model to aggregate the representations of tokens within an utterance and push away the representations of distinct utterances.
Revised: Based on the above observation and motivation, we identify two properties, i.e., locality and isotropy in dialogue modeling, and then present SimDRC , a sim ple d ialogue r epresentation c alibration method, which encourages the model to aggregate the representations of tokens within an utterance and push away the representations of distinct utterances.

Original: We evaluate our approach on three open-domain dialogue tasks, including multi-turn dialogue response generation (Li et al., 2017), conversational response retrieval (Lowe et al., 2015) and conversational semantic role labeling (Xu et al., 2021).
Revised: We evaluate our approach on three open-domain dialogue tasks, including multi-turn dialogue response generation (Li et al., 2017), conversational response retrieval (Lowe et al., 2015) and conversational semantic role labeling (Xu et al., 2021).

Original: The experimental results show that our approach achieves comparable or better performance against the current state-of-the-art methods across the three dialogue tasks on both automatic and human evaluations.
Revised: The experimental results show that our approach achieves comparable or better performance against the current state-of-the-art methods across the three dialogue tasks on both automatic and human evaluations.

Original: In-depth analyses towards the effects of the hyper-parameters and the measurements of locality and isotropy further verify the effectiveness of our proposed approach.
Revised: In-depth analyses towards the effects of the hyper-parameters and the measurements of locality and isotropy further verify the effectiveness of our proposed approach.

Original:  R ELATED W ORK 2. D IALOGUE M ODELING
Revised:  R ELATED W ORK 2. D IALOGUE M ODELING

Original: Dialogue modeling is to transform the raw text of the dialogue to machine-readable representations, which is an indispensable step to most dialogue tasks (Li et al., 2017; Liu et al., 2021).
Revised: Dialogue modeling is to transform the raw text of the dialogue to machine-readable representations, which is an indispensable step to most dialogue tasks (Li et al., 2017; Liu et al., 2021).

Original: To achieve this goal, conventional approaches (Serban et al., 2016; 2017; Xing et al., 2018) with recurrent neural networks (RNN) (Hochreiter & Schmidhuber, 1997; Mikolov et al., 2010) prefer to hierarchically learn the representations due to the long distance dependency problems in RNNs.
Revised: To achieve this goal, conventional approaches (Serban et al., 2016; 2017; Xing et al., 2018) with recurrent neural networks (RNN) (Hochreiter & Schmidhuber, 1997; Mikolov et al., 2010) prefer to hierarchically learn the representations due to the long distance dependency problems in RNNs.

Original: With the remarkable success of Transformer (Vaswani et al., 2017) and pre-trained language models (Devlin et al., 2019;
Revised: With the remarkable success of Transformer (Vaswani et al., 2017) and pre-trained language models (Devlin et al., 2019;

Original: Raffel et al., 2020) on various NLP tasks, Transformer-based dialogue modeling methods (HosseiniAsl et al., 2020; Gu et al., 2020; Liu et al., 2021; Wu et al., 2021) are widely used, and significantly outperform the traditional methods on many dialogue tasks, such as response generation (Li et al., 2017) and response retrieval (Lowe et al., 2015).
Revised: Raffel et al., 2020) on various NLP tasks, Transformer-based dialogue modeling methods (HosseiniAsl et al., 2020; Gu et al., 2020; Liu et al., 2021; Wu et al., 2021) are widely used, and signiﬁcantly outperform the traditional methods on many dialogue tasks, such as response generation (Li et al., 2017) and response retrieval (Lowe et al., 2015).

Original: In this work, we concentrate on the standard
Revised: In this work, we concentrate on the standard

Original: Transformer-based dialogue modeling method, which directly encodes the flattened dialogue context with the pre-trained language models.
Revised: Transformer-based dialogue modeling method, which directly encodes the ﬂattened dialogue context with the pre-trained language models.

Original: By studying the geometry of the representation space of the model, we find that the contextual representations produced by existing dialogue modeling methods are not isotropic and conversational.
Revised: By studying the geometry of the representation space of the model, we ﬁnd that the contextual representations produced by existing dialogue modeling methods are not isotropic and conversational.

Original: 2. R EPRESENTATION C ALIBRATION
Revised: 2. R EPRESENTATION C ALIBRATION

Original: Outside dialogue modeling, many other representation learning approaches also attempt to normalize their feature distributions from different perspectives.
Revised: Outside dialogue modeling, many other representation learning approaches also attempt to normalize their feature distributions from different perspectives.

Original: A bunch of studies theoretically verify that uniformly distributing points on unit hypersphere can provide good representations (Bojanowski &
Revised: A bunch of studies theoretically verify that uniformly distributing points on unit hypersphere can provide good representations (Bojanowski &

Original: Joulin, 2017; Wang & Isola, 2020).
Revised: Joulin, 2017; Wang & Isola, 2020).

Original: Specifically, Wang & Isola (2020) directly uses the alignment and uniformity properties to guide the representation learning, finally achieving comparable or even better performance on downstream tasks than the original contrastive learning framework.
Revised: Speciﬁcally, Wang & Isola (2020) directly uses the alignment and uniformity properties to guide the representation learning, ﬁnally achieving comparable or even better performance on downstream tasks than the original contrastive learning framework.

Original: With the rapid spread of Transformer, a set of works (Dong et al., 2021; Su et al., 2022) also empirically explore the intrinsic properties of representations produced by the models, and find that the contextual words representations from most models are anisotropy , occupying a narrow cone in the feature space (Ethayarajh, 2019).
Revised: With the rapid spread of Transformer, a set of works (Dong et al., 2021; Su et al., 2022) also empirically explore the intrinsic properties of representations produced by the models, and ﬁnd that the contextual words representations from most models are anisotropy , occupying a narrow cone in the feature space (Ethayarajh, 2019).

Original: This observation strongly agrees with our intuition that refining dialogue feature space is a desirable step.
Revised: This observation strongly agrees with our intuition that reﬁning dialogue feature space is a desirable step.

Original: To this problem, SimCTG (Su et al., 2022) presents a token-level contrastive learning framework to calibrate the model’s representation space regarding the anisotropy problem.
Revised: To this problem, SimCTG (Su et al., 2022) presents a token-level contrastive learning framework to calibrate the model’s representation space regarding the anisotropy problem.

Original: However, the essential difference with our work is that we normalize the feature space on multi-granularity levels instead of just the token level, which matches better with the structural characteristics of the dialogue and produces more isotropic and conversational representations.
Revised: However, the essential difference with our work is that we normalize the feature space on multi-granularity levels instead of just the token level, which matches better with the structural characteristics of the dialogue and produces more isotropic and conversational representations.

Original:  M ETHODOLOGY 3. D IALOGUE M ODELING
Revised:  M ETHODOLOGY 3. D IALOGUE M ODELING

Original: Given a dialogue D = { u 1 , u 2 , ..., u N } consisting of N utterances, where u i = { w i, 1 , w i, 2 , ..., w i, | u i | } is a sequence of words, the goal of dialogue modeling is to learn the contextual representation H
Revised: Given a dialogue D = { u 1 , u 2 , ..., u N } consisting of N utterances, where u i = { w i, 1 , w i, 2 , ..., w i, | u i | } is a sequence of words, the goal of dialogue modeling is to learn the contextual representation H

Original: = { h 1 , 1 , ..., h i,j , ..., h N, | u N | } over the dialogue context.
Revised: = { h 1 , 1 , ..., h i,j , ..., h N, | u N | } over the dialogue context.

Original: Typically, the
Revised: Typically, the

Original: Transformer-based dialogue modeling method obtain the dialogue contextual representations by taking the utterances as a consecutive sequence with special separate tokens, which is presented as:
Revised: Transformer-based dialogue modeling method obtain the dialogue contextual representations by taking the utterances as a consecutive sequence with special separate tokens, which is presented as:

Original: [Equation]
Revised: [Equation]

Original: where [EOU] and [CONTEXT] are special tokens which would further be used to represent the whole utterance or dialogue.
Revised: where [EOU] and [CONTEXT] are special tokens which would further be used to represent the whole utterance or dialogue.

Original: [EOU] and [CONTEXT] are respectively inserted as the last token of each utterance and the last token of the whole dialogue.
Revised: [EOU] and [CONTEXT] are respectively inserted as the last token of each utterance and the last token of the whole dialogue.

Original: 3. L OCALITY AND I SOTROPY D ISTANCE
Revised: 3. L OCALITY AND I SOTROPY D ISTANCE

Original: Conventional wisdom says that a good dialogue contextual representation should be informative, discriminative and structurally sensitive (Gu et al., 2020;
Revised: Conventional wisdom says that a good dialogue contextual representation should be informative, discriminative and structurally sensitive (Gu et al., 2020;

Original: Wu et al., 2021;
Revised: Wu et al., 2021;

Original: Su et al., 2022).
Revised: Su et al., 2022).

Original: Therefore, we identify two properties for good dialogue representations as following:
Revised: Therefore, we identify two properties for good dialogue representations as following:

Original: • Locality : tokens within an utterance are mapped to nearby features, thus concentrating on expressing a core idea of the utterance.
Revised: • Locality : tokens within an utterance are mapped to nearby features, thus concentrating on expressing a core idea of the utterance.

Original: • Isotropy : utterance representations are roughly uniformly distributed in the feature space, thus preserving as much information of the whole dialogue as possible and enabling the utterance discriminative to each other.
Revised: • Isotropy : utterance representations are roughly uniformly distributed in the feature space, thus preserving as much information of the whole dialogue as possible and enabling the utterance discriminative to each other.

Original: For locality, it encourages the learned feature representations of the tokens within an utterance to be similar.
Revised: For locality, it encourages the learned feature representations of the tokens within an utterance to be similar.

Original: In practice, the special token (i.e., [EOU]) is often served as the representative token of the utterance to speak for the utterance (Zhu et al., 2020; Liu et al., 2021).
Revised: In practice, the special token (i.e., [EOU]) is often served as the representative token of the utterance to speak for the utterance (Zhu et al., 2020; Liu et al., 2021).

Original: Therefore, we model locality by bridging the gap between each token and the representative token of the utterance it belongs.
Revised: Therefore, we model locality by bridging the gap between each token and the representative token of the utterance it belongs.

Original: The benefits of this design are 1) the representations of the tokens within an utterance are pushed close to the representation of the utterance representative token for better expressing the utterance’s concentrated idea; 2) the representative token are encouraged to combine the information from all tokens within the utterance, thus becoming more informative and representative.
Revised: The beneﬁts of this design are 1) the representations of the tokens within an utterance are pushed close to the representation of the utterance representative token for better expressing the utterance’s concentrated idea; 2) the representative token are encouraged to combine the information from all tokens within the utterance, thus becoming more informative and representative.

Original: Formally, given a dialogue D = { w 1 , 1 , ..., w i,j , ..., w N, | u N | } , for each token w i,j ( j ̸ = | u i | ) in the context, we obtain the locality value by calculating the cosine similarity between the token and the corresponding utterance representative token w i, | u i | , 
Revised: Formally, given a dialogue D = { w 1 , 1 , ..., w i,j , ..., w N, | u N | } , for each token w i,j ( j (cid:54) = | u i | ) in the context, we obtain the locality value by calculating the cosine similarity between the token and the corresponding utterance representative token w i, | u i | , locality_value ( i,j ) = h Ti,j h i, | u i | (cid:107) h i,j (cid:107) · (cid:107) h

Revised: i, | u i | (cid:107) ,

Original: [Equation]
Revised: [Equation]

Original: where h i,j is the representation of token w i,j .
Revised: where h i,j is the representation of token w i,j .

Original: The value will be larger when the two vectors are closer.
Revised: The value will be larger when the two vectors are closer.

Original: Then, we compute the overall locality distance for the whole dialogue context as:
Revised: Then, we compute the overall locality distance for the whole dialogue context as:

Original: [Equation]
Revised: [Equation]

Original: For isotropy, it encourages the model to push away the features of distinct utterances, thus preserving the maximal information of the dialogue and enabling the utterance discriminative and isotropic.
Revised: For isotropy, it encourages the model to push away the features of distinct utterances, thus preserving the maximal information of the dialogue and enabling the utterance discriminative and isotropic.

Original: Similar to locality, we model isotropy by enlarging the gap between distinct representative tokens.
Revised: Similar to locality, we model isotropy by enlarging the gap between distinct representative tokens.

Original: Formally, given two distinct utterances u i and u j , we obtain the isotropy value by computing the cosine similarity between two utterance representative tokens,
Revised: Formally, given two distinct utterances u i and u j , we obtain the isotropy value by computing the cosine similarity between two utterance representative tokens,

Original: [Equation]
Revised: [Equation]

Original: Similarly, the overall isotropy distance of the whole dialogue context is formulated as:
Revised: Similarly, the overall isotropy distance of the whole dialogue context is formulated as:

Original: [Equation]
Revised: [Equation]

Original: 3. A DJUSTING F EATURE S PACE
Revised: 3. A DJUSTING F EATURE S PACE

Original: Based on the locality distance and isotropy distance, we present locality loss, isotropy loss and
Revised: Based on the locality distance and isotropy distance, we present locality loss, isotropy loss and

Original: SimDRC loss to adjust the feature space for modeling better representations.
Revised: SimDRC loss to adjust the feature space for modeling better representations.

Original: Locality loss attempts to close the representations of the tokens within an utterance to the representations of the corresponding utterance representative token.
Revised: Locality loss attempts to close the representations of the tokens within an utterance to the representations of the corresponding utterance representative token.

Original: Therefore, we introduce locality loss, aiming at maximizing the locality distance defined in Equation 3,
Revised: Therefore, we introduce locality loss, aiming at maximizing the locality distance deﬁned in Equation 3,

Original: [Equation]
Revised: [Equation]

Original: where δ 1 ∈ [0 , 1] is the margin value.
Revised: where δ 1 ∈ [0 , 1] is the margin value.

Original: When δ 1 equals to 0, the loss degrades to 0.
Revised: When δ 1 equals to 0, the loss degrades to 0.

Original: Isotropy loss tries to push away the representations of distinct utterances.
Revised: Isotropy loss tries to push away the representations of distinct utterances.

Original: To this end, we introduce isotropy loss to minimize the isotropy distance defined in Equation 5,
Revised: To this end, we introduce isotropy loss to minimize the isotropy distance deﬁned in Equation 5,

Original: [Equation]
Revised: [Equation]

Original: where δ 2 ∈ [ − 1 , 1] is the margin value.
Revised: where δ 2 ∈ [ − 1 , 1] is the margin value.

Original: When δ 2 equals to -1, the loss degrades to 0.
Revised: When δ 2 equals to -1, the loss degrades to 0.

Original: SimDRC loss combines the locality loss and isotropy loss to calibrate the representation space.
Revised: SimDRC loss combines the locality loss and isotropy loss to calibrate the representation space.

Original: Formally, we present the SimDRC loss as,
Revised: Formally, we present the SimDRC loss as,

Original: [Equation]
Revised: [Equation]

Original: where δ ∈ [ − 1 , 1] is the margin value, shared between the locality loss and isotropy loss.
Revised: where δ ∈ [ − 1 , 1] is the margin value, shared between the locality loss and isotropy loss.

Original: α is the hyper-parameter of loss weight.
Revised: α is the hyper-parameter of loss weight.

Original: The SimDRC loss degenerates to locality loss when α = 1 , and to isotropy loss when − 1 < δ ≤ 0 or α = 0 , and to 

Revised: The SimDRC loss degenerates to locality loss when α = 1 , and to isotropy loss when − 1 < δ ≤ 0 or α = 0 , and to

Original: when δ = − 1 .
Revised: when δ = − 1 .

Original: It is worth noting that, the isotropy loss is more sparse because it only works on a few utterance representative tokens while the locality loss computes on all tokens within the utterances.
Revised: It is worth noting that, the isotropy loss is more sparse because it only works on a few utterance representative tokens while the locality loss computes on all tokens within the utterances.

Original: Therefore, the model would be biased to modeling more local features if we give moderate values of δ and α (e.g. δ =0, α =0.5).
Revised: Therefore, the model would be biased to modeling [Table] contrastive 11. [Table]

Revised: [Table]

Revised: [Table]

Revised: SimDRC ( δ =0.7, α =0.3) greedy 12.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 474 beam 12.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: nucleus

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: To this problem, we empirically scale up the isotropy loss by applying the large value of δ and the small value of α .
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: more local features if we give moderate values of δ and α (e.g. δ =0, α =0.5).

Revised: To this problem, we empirically scale up the isotropy loss by applying the large value of δ and the small value of α .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method on three kinds of dialogue tasks, i.e., multi-turn dialogue response generation (Li et al., 2017), conversational response retrieval (Lowe et al., 2015) and conversational semantic role labeling (Xu et al., 2021).
Revised: We evaluate our method on three kinds of dialogue tasks, i.e., multi-turn dialogue response generation (Li et al., 2017), conversational response retrieval (Lowe et al., 2015) and conversational semantic role labeling (Xu et al., 2021).

Original: Note that we just focus on open-domain dialogue tasks in this work since previous studies (Jakobovits et al., 2022) revealed that most task-oriented datasets, like MultiWOZ (Budzianowski et al., 2018) are essentially not conversational and contextualized while our method aims to learn these features in dialogue modeling stage.
Revised: Note that we just focus on open-domain dialogue tasks in this work since previous studies (Jakobovits et al., 2022) revealed that most task-oriented datasets, like MultiWOZ (Budzianowski et al., 2018) are essentially not conversational and contextualized while our method aims to learn these features in dialogue modeling stage.

Original: 4. M ULTI - TURN D IALOGUE R ESPONSE G ENERATION
Revised: 4. M ULTI - TURN D IALOGUE R ESPONSE G ENERATION

Original: Task & Data Multi-turn dialogue response generation is to automatically produce the human-like response given the dialogue context.
Revised: Task & Data Multi-turn dialogue response generation is to automatically produce the human-like response given the dialogue context.

Original: Hopefully, we expect the generated responses are diverse, fluent, coherent and informative.
Revised: Hopefully, we expect the generated responses are diverse, ﬂuent, coherent and informative.

Original: We evaluate our approach on DailyDialog (Li et al., 2017) and LCCC (Wang et al., 2020), wherein DailyDialog is a multi-turn open-domain English dialogue dataset and
Revised: We evaluate our approach on DailyDialog (Li et al., 2017) and LCCC (Wang et al., 2020), wherein DailyDialog is a multi-turn open-domain English dialogue dataset and

Original: LCCC is a open-domain Chinese short-text conversation dataset.
Revised: LCCC is a open-domain Chinese short-text conversation dataset.

Original: Models & Decoding We compare the vanilla BART and DialoGPT models to their fine-tuned versions with SimCTG and SimDRC.
Revised: Models & Decoding We compare the vanilla BART and DialoGPT models to their ﬁne-tuned versions with SimCTG and SimDRC.

Original: For each model, we conduct the inference using four decoding strategies, including 1) greedy search; 2) beam search (beam size = 5); 3) nucleus sampling ( p = 0 .
Revised: For each model, we conduct the inference using four decoding strategies, including 1) greedy search; 2) beam search (beam size = 5); 3) nucleus sampling ( p = 0 .

Original: 95 ) (Holtzman et al., 2019); 4) contrastive decoding ( k = 5 , α = 0 . 6 )
Revised: 95 ) (Holtzman et al., 2019); 4) contrastive decoding ( k = 5 , α = 0 . 6 )

Original: (Su et al., 2022).
Revised: (Su et al., 2022).

Original: Evaluation It is a long-standing problem to accurately and comprehensively evaluate the quality of the generated texts (Celikyilmaz et al., 2020).
Revised: Evaluation It is a long-standing problem to accurately and comprehensively evaluate the quality of the generated texts (Celikyilmaz et al., 2020).

Original: The traditional n-gram overlap and text matching metrics, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), are not good choices to open-domain dialogue systems which permit significant diversity and allow multiple plausible outputs for a given input.
Revised: The traditional n-gram overlap and text matching metrics, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), are not good choices to open-domain dialogue systems which permit signiﬁcant diversity and allow multiple plausible outputs for a given input.

Original: Therefore, we choose to measure the generation quality over following automatic evaluation metrics, including BERTScore (Zhang et al., 2019), BARTScore (Yuan et al., 2021),
Revised: Therefore, we choose to measure the generation quality over following automatic evaluation metrics, including BERTScore (Zhang et al., 2019), BARTScore (Yuan et al., 2021),

Original: BLEURT (Sellam et al., 2020), Distinct2/4 (Li et al., 2016).
Revised: BLEURT (Sellam et al., 2020), Distinct2/4 (Li et al., 2016).

Original: We also conduct the human evaluation with the help of proficient English/Chinese speakers recruited from the crowdsourcing platform.
Revised: We also conduct the human evaluation with the help of proﬁcient English/Chinese speakers recruited from the crowdsourcing platform.

Original: We randomly sample 200 dialogue contexts from DailyDialog and LCCC test sets, respectively.
Revised: We randomly sample 200 dialogue contexts from DailyDialog and LCCC test sets, respectively.

Original: For each dialogue context, we generate responses using different models (vanilla BART, BART+SimCTG and BART+SimDRC) with different decoding strategies (greedy, beam search, nucleus sampling and contrastive decoding).
Revised: For each dialogue context, we generate responses using different models (vanilla BART, BART+SimCTG and BART+SimDRC) with different decoding strategies (greedy, beam search, nucleus sampling and contrastive decoding).

Original: Five annotators are employed to independently evaluate these 2,400 samples by giving a score ranging from 1 to 5 over following aspects, including fluency , informativeness , coherence , and semantic coverage 1 .
Revised: Five annotators are employed to independently evaluate these 2,400 samples by giving a score ranging from 1 to 5 over following aspects, including ﬂuency , informativeness , coherence , and semantic coverage 1 .

Original:  Results & Discussion Table 1 lists the automatic evaluation results of the different methods (i.e.
Revised: [Table] Results & Discussion Table 1 lists the automatic evaluation results of the different methods (i.e.

Original: BART, BART+SimCTG and BART+SimDRC) 2 with the different decoding strategies.
Revised: BART, BART+SimCTG and BART+SimDRC) 2 with the different decoding strategies.

Original: On both[Table] datasets, we can see that SimDRC, especially when equipped with nucleus sampling, achieves the best performance across the board (sign test p < 0.01 compared to SimCTG+contrastive), indicating that SimDRC can produce diverse (high distinct scores), semantically consistent (high BERTScores,
Revised: On both datasets, we can see that SimDRC, especially when equipped with nucleus sampling, achieves the best performance across the board (sign test p < 0.01 compared to SimCTG+contrastive), indicating that SimDRC can produce diverse (high distinct scores), semantically consistent (high BERTScores,

Original: BLEURT scores and BARTScores) and informative (high distinct and BARTscores) responses.
Revised: BLEURT scores and BARTScores) and informative (high distinct and BARTscores) responses.

Original: Inconsistent with the findings of SimCTG where their human evaluation showed that both SimCTG and contrastive decoding could improve the quality of the generated texts on the two datasets in
Revised: Inconsistent with the ﬁndings of SimCTG where their human evaluation showed that both SimCTG and contrastive decoding could improve the quality of the generated texts on the two datasets in

Original: English and Chinese, our automatic evaluation results find the difference that SimCTG+contrastive performs comparably with our method on the LCCC dataset, but performs poorly on the DailyDialog dataset.
Revised: English and Chinese, our automatic evaluation results ﬁnd the difference that SimCTG+contrastive performs comparably with our method on the LCCC dataset, but performs poorly on the DailyDialog dataset.

Original: In contrast, SimDRC constantly performs well regardless of the decoding methods and datasets used, suggesting SimDRC is a more robust solution.
Revised: In contrast, SimDRC constantly performs well regardless of the decoding methods and datasets used, suggesting SimDRC is a more robust solution.

Original: We also conduct ablation studies regarding the locality and isotropy training objectives.
Revised: We also conduct ablation studies regarding the locality and isotropy training objectives.

Original: We find that after removing either locality loss or isotropy loss, the quality of generated texts degrades to a certain extent (details in Section 5.2).
Revised: We ﬁnd that after removing either locality loss or isotropy loss, the quality of generated texts degrades to a certain extent (details in Section 5.2).

Original: Table 2 summaries the results of human evaluation on DailyDialog and LCCC datasets.
Revised: Table 2 summaries the results of human evaluation on DailyDialog and LCCC datasets.

Original: The first row indicates the inter-rater agreements calculated by Fleiss’ kappa coefficient (Fleiss, 1971).
Revised: The ﬁrst row indicates the inter-rater agreements calculated by Fleiss’ kappa coefﬁcient (Fleiss, 1971).

Original: In the rest part, we find that the results of human evaluation are highly consistent with automatic evaluation’s, showing that SimDRC+nucleus significantly outperforms baselines across the board.
Revised: In the rest part, we ﬁnd that the results of human evaluation are highly consistent with automatic evaluation’s, showing that SimDRC+nucleus signiﬁcantly outperforms baselines across the board.

Original: The exception is that SimDRC+contrastive obtains better informativeness score than SimDRC+nucleus on DailyDialog.
Revised: The exception is that SimDRC+contrastive obtains better informativeness score than SimDRC+nucleus on DailyDialog.

Original: We think this is reasonable since contrastive decoding encourages the model to generate the response with fewer repetitions and more novel terms.
Revised: We think this is reasonable since contrastive decoding encourages the model to generate the response with fewer repetitions and more novel terms.

Original: It is also in line with the results of automatic evaluation that SimDRC+contrastive achieves better distinct scores on DailyDialog.
Revised: It is also in line with the results of automatic evaluation that SimDRC

Original: 4. C ONVERSATIONAL R ESPONSE R ETRIEVAL
Revised: 4. C ONVERSATIONAL R ESPONSE R ETRIEVAL

Original: Task & Data Conversational response retrieval (Lowe et al., 2015) is to automatically select the appropriate response from the corpus given a prefix context.
Revised: Task & Data Conversational response retrieval (Lowe et al., 2015) is to automatically select the appropriate response from the corpus given a preﬁx context.

Original: Unlike response generation which produces the target sequence word by word, response retrieval finds the potential responses by matching the context representations with the candidate utterance representations.
Revised: Unlike response generation which produces the target sequence word by word, response retrieval ﬁnds the potential responses by matching the context representations with the candidate utterance representations.

Original: We evaluate our approach on three commonly used response retrieval datasets, including Douban (Wu et al., 2017),
Revised: We evaluate our approach on three commonly used response retrieval datasets, including Douban (Wu et al., 2017),

Original: E_commerce (Zhang et al., 2018) and Ubuntu (Lowe et al., 2015), wherein Douban is a daily chit-chat dataset while E_commerce and Ubuntu are domain-specific.
Revised: E_commerce (Zhang et al., 2018) and Ubuntu (Lowe et al., 2015), wherein Douban is a daily chit-chat dataset while E_commerce and Ubuntu are domain-speciﬁc.

Original: Model & Baselines As our approach is architecture-agnostic and can be applied to most existing models, we choose the current best-performing response retrieval model, i.e. BERT-FP (Han et al., 2021), as our backbone.
Revised: Model & Baselines As our approach is architecture-agnostic and can be applied to most existing models, we choose the current best-performing response retrieval model, i.e. BERT-FP (Han et al., 2021), as our backbone.

Original: BERT-FP is a simple BERT-based retrieval model but benefits from the fine-grained post-training that reflects the characteristics of the multi-turn dialogue.
Revised: BERT-FP is a simple BERT-based retrieval model but beneﬁts from the ﬁne-grained post-training that reﬂects the characteristics of the multi-turn dialogue.

Original: We directly apply
Revised: We directly apply

Original: SimDRC or SimCTG to the hidden states from the last layer of the original BERT-FP model.
Revised: SimDRC or SimCTG to the hidden states from the last layer of the original BERT-FP model.

Original: Results & Discussion Table 3 summarizes the evaluation results on three datasets.
Revised: Results & Discussion Table 3 summarizes the evaluation results on three datasets.

Original: We can see that our approach (BERT-FP + SimDRC) achieves comparable or better performance against the  baselines over all metrics, indicating that SimDRC is also an effective solution to retrieval-based tasks.
Revised: We can see that our approach (BERT-FP + SimDRC) achieves comparable or better performance against the [Table] [Table] baselines over all metrics, indicating that SimDRC is also an effective solution to retrieval-based tasks.

Original: Specifically, our approach reaches the new state-of-the-art performance on the Douban corpus (sign test p < 0.01), but we find a few performance drops on the Ubuntu and E-commerce corpus.
Revised: Speciﬁcally, our approach reaches the new state-of-the-art performance on the Douban corpus (sign test p < 0.01), but we ﬁnd a few performance drops on the Ubuntu and E-commerce corpus.

Original: We think this is because there is an extremely small room for performance improvements since the[Table] [Table] previous best-performing model has achieved over 99% accuracy on R 10 @5 .
Revised: We think this is because there is an extremely small room for performance improvements since the previous best-performing model has achieved over 99% accuracy on R 10 @5 .

Original: For ablation studies, we find separately employing locality or isotropy also outperforms the baselines.
Revised: For ablation studies, we ﬁnd separately employing locality or isotropy also outperforms the baselines.

Original: 4. C ONVERSATIONAL S EMANTIC R OLE L ABELING
Revised: 4. C ONVERSATIONAL S EMANTIC R OLE L ABELING

Original: Task & Data Conversational semantic role labeling (CSRL) (Xu et al., 2021) is a challenging dialogue understanding task, which extends the standard SRL task (Palmer et al., 2010) to dialogue scenario and aims to find the potential arguments over the entire conversation with the given predicate.
Revised: Task & Data Conversational semantic role labeling (CSRL) (Xu et al., 2021) is a challenging dialogue understanding task, which extends the standard SRL task (Palmer et al., 2010) to dialogue scenario and aims to ﬁnd the potential arguments over the entire conversation with the given predicate.

Original: Following previous work (Xu et al., 2021), we conduct experiments on three CSRL datasets, namely
Revised: Following previous work (Xu et al., 2021), we conduct experiments on three CSRL datasets, namely

Original: DuConv, NewsDialog and PersonalDialog.
Revised: DuConv, NewsDialog and PersonalDialog.

Original: DuConv annotations are split into 80%/10%/10% as train/dev/test sets, while NewsDialog and PersonalDialog are used as out-of-domain test sets.
Revised: DuConv annotations are split into 80%/10%/10% as train/dev/test sets, while NewsDialog and PersonalDialog are used as out-of-domain test sets.

Original: Model & Baselines We adopt CSAGN (Wu et al., 2021) model as our baseline method.
Revised: Model & Baselines We adopt CSAGN (Wu et al., 2021) model as our baseline method.

Original: CSAGN firstly obtains the dialogue representations from pre-trained BERT, and then employs a predicateaware module and a conversation structure-aware graph network to capture high-level features.
Revised: CSAGN ﬁrstly obtains the dialogue representations from pre-trained BERT, and then employs a predicateaware module and a conversation structure-aware graph network to capture high-level features.

Original: We apply SimDRC or SimCTG on the generated representations from the last layer of BERT.
Revised: We apply SimDRC or SimCTG on the generated representations from the last layer of BERT.

Original: Results & Discussion Table 4 shows the evaluation results on three datasets.
Revised: Results & Discussion Table 4 shows the evaluation results on three datasets.

Original: We can see that our approach (CSAGN+SimDRC) significantly outperforms all baseline models across the board (sign test with p-value < 0.01), suggesting that SimDRC helps the model to capture both local and conversational features.
Revised: We can see that our approach (CSAGN+SimDRC) signiﬁcantly outperforms all baseline models across the board (sign test with p-value < 0.01), suggesting that SimDRC helps the model to capture both local and conversational features.

Original: SimCTG helps the model find more cross-arguments on two out-of-domain test sets while also missing more intra-arguments.
Revised: SimCTG helps the model ﬁnd more cross-arguments on two out-of-domain test sets while also missing more intra-arguments.

Original: This is in line with the characteristics of SimCTG because pushing away all tokens makes the representations more isotropic but less concentrated on local information.
Revised: This is in line with the characteristics of SimCTG because pushing away all tokens makes the representations more isotropic but less concentrated on local information.

Original: Superior to SimCTG, SimDRC improves both locality and isotropy.
Revised: Superior to SimCTG, SimDRC improves both locality and isotropy.

Original: For ablation studies, we can find 1) individually adopting locality loss also outperforms the baselines; 2) locality consistently enhances the capacity of local feature modeling while isotropy tends to enhance the capacity of cross-turn feature modeling.
Revised: For ablation studies, we can ﬁnd 1) individually adopting locality loss also outperforms the baselines; 2) locality consistently enhances the capacity of local feature modeling while isotropy tends to enhance the capacity of cross-turn feature modeling.

Original:  I N - DEPTH A NALYSES 5. C ONVERSATIONAL C OHERENCE
Revised:  I N - DEPTH A NALYSES 5. C ONVERSATIONAL C OHERENCE

Original: A main concern of our approach is that SimDRC may weaken the intrinsic correlations among utterances since it tries to push away the utterance representations and make them discriminative.
Revised: A main concern of our approach is that SimDRC may weaken the intrinsic correlations among utterances since it tries to push away the utterance representations and make them discrimina- tive.

Original: Intuitively, the most straightforward idea of improving the quality of dialogue modeling should be strengthening the connections among utterances, thus making the dialogue more coherent.
Revised: Intuitively, the most straightforward idea of improving the quality of dialogue modeling should be strengthening the connections among utterances, thus making the dialogue more coherent.

Original: To this end, we study the conversational coherence in SimDRC-based models.
Revised: To this end, we study the conversational coherence in SimDRC-based models.

Original: As there are no available methods to automatically measure the conversational coherence based on the contex- tual representations, we take the average score of the similarities between the dialogue embedding and the utterance embeddings as the coherence score, which is formulated as: coherence = 1 N (cid:80) Ni =
Revised: As there are no available methods to automatically measure the conversational coherence based on the contextual representations, we take the average score of the similarities between the dialogue embedding and the utterance embeddings as the coherence score, which is formulated as: coherence = [Figure]

Original: ∥∥ v context ∥ , where v u i is the utterance representation of utterance u i and v context is the dialogue representation.
Revised: , where v u i is the utterance representation of utterance u i and v context is the dialogue representation.

Original: The motivation behind is that the utterances of a coherent dialogue should consistently follow a main topic, which is expressed as the dialogue embedding in our settings.
Revised: The motivation behind is that the utterances of a coherent dialogue should consistently follow a main topic, which is expressed as the dialogue embedding in our settings.

Original: [Table]
Revised: [Table]

Original: We calculate the coherence score on the
Revised: We calculate the coherence score on the

Original: DailyDialog dataset.
Revised: DailyDialog dataset.

Original: As shown in Table 5, SimDRC obtains the highest coherence score over other methods while the score of SimCTG is even lower than the vanilla BART model.
Revised: As shown in Table 5, SimDRC obtains the highest coherence score over other methods while the score of SimCTG is even lower than the vanilla BART model.

Original: We think this is because SimCTG constantly focuses on pushing away all token representations instead of modeling the conversational structure-aware representations, like SimDRC.
Revised: We think this is because SimCTG constantly focuses on pushing away all token representations instead of modeling the conversational structure-aware representations, like SimDRC.

Original: From another perspective, on the CSRL task, SimDRC greatly improves CSAGN’s capacity of identifying cross-arguments, further showing its effectiveness on enhancing or keeping the conversational coherence.
Revised: Moreover, on the CSRL task, SimDRC greatly improves CSAGN’s capacity of identifying cross-arguments, consistently showing its ability to enhance conversational coherence.

Original: 5. E FFECTS OF L OSS WEIGHT AND M ARGIN VALUE
Revised: 5. E FFECTS OF L OSS WEIGHT AND M ARGIN VALUE

Original: Here, we explore how the loss weight α and margin value δ affects the performance of the response generation task.
Revised: Here, we explore how the loss weight α and margin value δ affects the performance of the response generation task.

Original: On the DailyDialog dataset, we fine-tune the BART+SimDRC models with different loss weights α ∈ [0 . 1 , 0 . 9] and margin values δ ∈ [0 . 1 , 0 . 9] and measure the perplexity on the test set.
Revised: On the DailyDialog dataset, we ﬁne-tune the BART+SimDRC models with different loss weights α ∈ [0 . 1 , 0 . 9] and margin values δ ∈ [0 . 1 , 0 . 9] and measure the perplexity on the test set.

Original: We experimentally find that the large value of α (e.g. 0.6 ∼ 0.9) and the marginal values of δ (e.g. 0.1 ∼ 0.2, 0.8 ∼ 0.9) would hurt the performance, which is consistent with the discussion in the last paragraph of Section 3.3.
Revised: We experimentally ﬁnd that the large value of α (e.g. 0.6 ∼ 0.9) and the marginal values of δ (e.g. 0.1 ∼ 0.2, 0.8 ∼ 0.9) would hurt the performance, which is consistent with the discussion in the last paragraph of Section 3.3.

Original: To this end, we narrow the candidate sets of α and δ to α ∈ [0 . 1 , 0 . 5] and δ ∈ [0 . 3 , 0 . 7] .
Revised: To this end, we narrow the candidate sets of α and δ to α ∈ [0 . 1 , 0 . 5] and δ ∈ [0 . 3 , 0 . 7] .

Original: Figure 2(a) plots the results of the models trained with different hyper-parameters.
Revised: Figure 2(a) plots the results of the models trained with different hyper-parameters.

Original: We can see that SimDRC consistently achieves comparable or better performance than the baselines.
Revised: We can see that SimDRC consistently achieves comparable or better performance than the baselines.

Original: And it obtains the best performance when α = 0 .
Revised: And it obtains the best performance when α = 0 .

Original: 3 and δ = 0 .
Revised: 3 and δ = 0 .

Original: For the CSRL task, we find the variations of α do not lead to a big weave of the performance.
Revised: For the CSRL task, we ﬁnd the variations of α do not lead to a big weave of the performance.

Original: Based on this observation, we fix the value of α = 0 .
Revised: Based on this observation, we ﬁx the value of α = 0 .

Original: 5 to study the effects of margin value δ ∈ [0 . 1 , 0 . 9] .
Revised: 5 to study the effects of margin value δ ∈ [0 . 1 , 0 . 9] .

Original: Figure 2(b) illustrates the results of the models trained with different margin values.
Revised: Figure 2(b) illustrates the results of the models trained with different margin values.

Original: We see that
Revised: We see that

Original: SimDRC obtains better scores at the most time.
Revised: SimDRC obtains better scores at the most time.

Original: Still, the marginal values of δ would affect the performance.
Revised: Still, the marginal values of δ would affect the performance.

Original: Furthermore, individually introducing isotropy loss to CSAGN is not an effective solution while only using locality loss performs comparably.
Revised: Furthermore, individually introducing isotropy loss to CSAGN is not an effective solution while only using locality loss performs comparably.

Original: The optimal values of α and δ generally differ from task to task.
Revised: The optimal values of α and δ generally differ from task to task.

Original: It would take much effort to search the best hyper-parameters for different tasks.
Revised: It would take much effort to search the best hyper-parameters for different tasks.

Original: With the extensive experiments in this work, we find that the settings of α ∈ [0 . 2 , 0 . 4] and δ ∈ [0 . 4 , 0 . 6] work well for most tasks.
Revised: With the extensive experiments in this work, we ﬁnd that the settings of α ∈ [0 . 2 , 0 . 4] and δ ∈ [0 . 4 , 0 . 6] work well for most tasks.

Original: 5. M EASUREMENTS OF L OCALITY AND I SOTROPY D ISTANCE
Revised: 5. M EASUREMENTS OF L OCALITY AND I SOTROPY D ISTANCE

Original: To ensure that SimDRC exactly optimizes the model regarding the properties of locality and isotropy, we measure the locality distance and isotropy distance along with the training process.
Revised: To ensure that SimDRC exactly optimizes the model regarding the properties of locality and isotropy, we measure the locality distance and isotropy distance along with the training process.

Original: Specifically, we fine-tune the BART-based models on the DailyDialog training set and measure the locality and isotropy distances on the development set, following the Equation 3 and 5.
Revised: Speciﬁcally, we ﬁne-tune the BART-based models on the DailyDialog training set and measure the locality and isotropy distances on the development set, following the Equation 3 and 5.

Original: Figure 3 plots the values of locality, isotropy distances and the development loss at different training stages.
Revised: Figure 3 plots the values of locality, isotropy distances and the development loss at different training stages.

Original: We see that SimDRC (red line) obtains much better locality and isotropy distances than the vanilla BART  and SimCTG (green line), suggesting that SimDRC is essentially working on these two properties.
Revised: We see that SimDRC (red line) obtains much better locality and isotropy distances than the vanilla BART (blue line) and SimCTG (green line), suggesting that SimDRC is essentially working on these two properties.

Original: Additionally, larger fluctuations on these two distance values are spotted when training with BART and SimCTG, while the values are steady with SimDRC, especially in terms of isotropy distance.
Revised: Additionally, larger ﬂuctuations on these two distance values are spotted when training with BART and SimCTG, while the values are steady with SimDRC, especially in terms of isotropy distance.

Original: We think this is also the reason that SimDRC has a more smooth development loss curve than baselines.
Revised: We think this is also the reason that SimDRC has a more smooth development loss curve than baselines.

Original: 5. V ISUALIZATION OF T OKEN S IMILARITY M ATRIX
Revised: 5. V ISUALIZATION OF T OKEN S IMILARITY M ATRIX

Original: In this part, we visualize the token similarity matrices of different models, i.e. vanilla BART,
Revised: In this part, we visualize the token similarity matrices of different models, i.e. vanilla BART,

Original: BART+SimCTG and BART+SimDRC.
Revised: BART+SimCTG and BART+SimDRC.

Original: We first fine-tune the models on DailyDialog and LCCC datasets, and then employ the well-trained models to encode the dialogue context and generate the token similarity matrices.
Revised: We ﬁrst ﬁne-tune the models on DailyDialog and LCCC datasets, and then employ the well-trained models to encode the dialogue context and generate the token similarity matrices.

Original: Figure 4 showcases the resulted heatmaps on these two datasets.
Revised: Figure 4 showcases the resulted heatmaps on these two datasets.

Original: We can see that 1)
Revised: We can see that 1)

Original: The token similarities of vanilla BART are relatively high and uniform, i.e. the token is close to most of the other tokens on representations; 2) the token similarities are overall decreased after introducing SimCTG, indicating it alleviates the problem of anisotropy, especially on the
Revised: The token similarities of vanilla BART are relatively high and uniform, i.e. the token is close to most of the other tokens on representations; 2) the token similarities are overall decreased after introducing SimCTG, indicating it alleviates the problem of anisotropy, especially on the

Original: LCCC corpus.
Revised: LCCC corpus.

Original: However, the utterances are still indistinguishable on representations, suggesting that
Revised: However, the utterances are still indistinguishable on representations, suggesting that

Original: SimCTG also ignores modeling the conversational features; 3) the models with SimDRC generates isotropic and conversational representations, making all utterance discriminative on representations.
Revised: SimCTG also ignores modeling the conversational features; 3) the models with SimDRC generates isotropic and conversational representations, making all utterance discriminative on representations.

Original: The observations are in agreement with our motivations and discussions above.
Revised: The observations are in agreement with our motivations and discussions above.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this work, we present a simple dialogue representation calibration method to encourage the model to learn isotropic and conversational features in dialogue modeling stage.
Revised: In this work, we present a simple dialogue representation calibration method to encourage the model to learn isotropic and conversational features in dialogue modeling stage.

Original: Experimental results show that our method achieves impressive performance on three dialogue tasks with eight benchmarks.
Revised: Experimental results show that our method achieves impressive performance on three dialogue tasks with eight benchmarks.

Original: For future work, applying SimDRC into dialogue pre-training would be a promising research direction.
Revised: For future work, applying SimDRC into dialogue pre-training would be a promising research direction.

Original: E THICS S TATEMENT
Revised: A CKNOWLEDGEMENTS

Revised: We sincerely appreciate the valuable and constructive comments from the reviewers.

Revised: This work was supported in part by the Changsha Science and Technology Program International and Regional

Revised: Science and Technology Cooperation Project under Grants kh2201026, the Hong Kong RGC grant

Revised: ECS 21212419, the Technological Breakthrough Project of Science, Technology and Innovation

Revised: Commission of Shenzhen Municipality under Grants JSGG20201102162000001, InnoHK initiative, the Government of the HKSAR, Laboratory for AI-Powered Financial Technologies, and the Tencent

Revised: AI Lab Rhino-Bird Gift Fund with Hong Kong RGC RMGS.

Revised: E THICS S TATEMENT

Original: In this work we use the publicly released datasets to train/valid/test our models.
Revised: In this work, we use the publicly released datasets to train/valid/test our models.

Original: Generally, these previous works have considered the ethical issues when creating the datasets.
Revised: Generally, these previous works have considered the ethical issues when creating the datasets.

Original: For the datasets we used in this work, we have manually checked some samples, and do not find any obvious ethical concerns, such as violent or offensive content.
Revised: For the datasets we used in this work, we have manually checked some samples, and do not ﬁnd any obvious ethical concerns, such as violent or offensive content.

Original: We will also release the source code and the well-trained models along with friendly instructions to support its correct use.
Revised: We will also release the source code and the well-trained models along with friendly instructions to support its correct use.

Original: However, we still need to emphasize that text generation is not as controllable as we think.
Revised: However, we still need to emphasize that text generation is not as controllable as we think.

Original: It still would generate some novel or unexpected words occasionally.
Revised: It still would generate some novel or unexpected words occasionally.

Original: We may take the actions to decrease the generation diversity to alleviate this problem.
Revised: We may take the actions to decrease the generation diversity to alleviate this problem.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: In this work we present a simple method for dialogue representation calibration.
Revised: In this work, we present a simple method for dialogue representation calibration.

Original: We have provided detailed descriptions of our proposed algorithm in Section 3.
Revised: We have provided detailed descriptions of our proposed algorithm in Section 3.

Original: For different downstream evaluated tasks, we elaborate the experimental settings and hyper-parameters in Section 4 and Appendix B. All datasets used in this work have been publicly released.
Revised: For different downstream evaluated tasks, we elaborate the experimental settings and hyper-parameters in Section 4 and Appendix B. All datasets used in this work have been publicly released.

Original: Upon the acceptance, we will release our source code and the checkpoints to facilitate the further research.

Original: A H UMAN E VALUATION I NSTRUCTIONS
Revised: A H UMAN E VALUATION I NSTRUCTIONS

Original: Please rate the quality of the generated response based on the given dialogue context and the target response over following aspects: (1) Fluency; (2) Informativeness; (3) Coherence; (4)Semantic
Revised: Please rate the quality of the generated response based on the given dialogue context and the target response over following aspects: (1) Fluency; (2) Informativeness; (3) Coherence; (4)Semantic

Original: Coverage.
Revised: Coverage.

Original: We provide some instructions for your rating.
Revised: We provide some instructions for your rating.

Original: A. F LUENCY
Revised: A. F LUENCY

Original: This measures whether the generated text has no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read.
Revised: This measures whether the generated text has no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difﬁcult to read.

Original: The definitions of different scores are:
Revised: The deﬁnitions of different scores are:

Original: • 5 : The text is fluent, grammatically correct, and has no errors.
Revised: • 5 : The text is ﬂuent, grammatically correct, and has no errors.

Original: It is easy to read.
Revised: It is easy to read.

Original: • 4 : The text is grammatically correct, but has a few spelling or capitalization errors, which does not affect your understanding.
Revised: • 4 : The text is grammatically correct, but has a few spelling or capitalization errors, which does not affect your understanding.

Original: • 3 : The text has minor errors in both grammar and spelling.
Revised: • 3 : The text has minor errors in both grammar and spelling.

Original: The errors slightly affect your understanding.
Revised: The errors slightly affect your understanding.

Original: • 2 : The text has major errors in both grammar and spelling.
Revised: • 2 : The text has major errors in both grammar and spelling.

Original: The errors make the text hard to read.
Revised: The errors make the text hard to read.

Original: • 1 : The text does not make sense and it is unreadable.
Revised: • 1 : The text does not make sense and it is unreadable.

Original: A. I NFORMATIVENESS
Revised: A. I NFORMATIVENESS

Original: This measures whether the generated text has diverse, informative, novel or logically related contents.
Revised: This measures whether the generated text has diverse, informative, novel or logically related contents.

Original: The definitions of different scores are:
Revised: The deﬁnitions of different scores are:

Original: • 5 : The text contains very diverse, informative and novel contents.
Revised: • 5 : The text contains very diverse, informative and novel contents.

Original: It is enjoyable to read the text.
Revised: It is enjoyable to read the text.

Original: • 4 : The text contains many informative and novel contents.
Revised: • 4 : The text contains many informative and novel contents.

Original: (Choose this score when you hesitate between 3 and 5.)
Revised: (Choose this score when you hesitate between 3 and 5.)

Original: • 3 : The text contains some new information but also contains a few repetitions of the context.
Revised: • 3 : The text contains some new information but also contains a few repetitions of the context.

Original: • 2 : The text only contains a few informative and new terms.
Revised: • 2 : The text only contains a few informative and new terms.

Original: (Choose this score when you hesitate between 1 and 3.)
Revised: (Choose this score when you hesitate between 1 and 3.)

Original: • 1 : The text is dull, repetitive and has no new information.
Revised: • 1 : The text is dull, repetitive and has no new information.

Original: All contents are from the dialogue context.
Revised: All contents are from the dialogue context.

Original: A. C OHERENCE
Revised: A. C OHERENCE

Original: This measures whether the generated text is semantically and factually consistent with the dialogue context.
Revised: This measures whether the generated text is semantically and factually consistent with the dialogue context.

Original: The definitions of different scores are:
Revised: The deﬁnitions of different scores are:

Original: • 5 : The text is semantically, factually and topically consistent with the dialogue context.
Revised: • 5 : The text is semantically, factually and topically consistent with the dialogue context.

Original: All contents of the text are related to the source text or can be inferred from the source.
Revised: All contents of the text are related to the source text or can be inferred from the source.

Original: • 4 : The text is very related to the context but has minor inconsistency or contradictions that do not affect its overall relevance.
Revised: • 4 : The text is very related to the context but has minor inconsistency or contradictions that do not affect its overall relevance.

Original: • 3 : The text is related to the context but has some obvious inconsistency and contradictions.
Revised: • 3 : The text is related to the context but has some obvious inconsistency and contradictions.

Original: • 2 : The text is slightly consistent with the context.
Revised: • 2 : The text is slightly consistent with the context.

Original: Many inconsistency and contradictions to the context can be found.
Revised: Many inconsistency and contradictions to the context can be found.

Original: • 1 : The text is totally inconsistent with the context.
Revised: • 1 : The text is totally inconsistent with the context.

Original: It is semantically or factually contradicted to the context.
Revised: It is semantically or factually contradicted to the context.

Original: A. S EMANTIC C OVERAGE
Revised: A. S EMANTIC C OVERAGE

Original: This measures how many semantic content units from the target response are covered by the generated text.
Revised: This measures how many semantic content units from the target response are covered by the generated text.

Original: The definitions of different scores are:
Revised: The deﬁnitions of different scores are:

Original: • 5 : All semantic content units of the target text can be found from the generated text.
Revised: • 5 : All semantic content units of the target text can be found from the generated text.

Original: They are semantically consistent.
Revised: They are semantically consistent.

Original: • 4 : Most of the content units of the target text can be found from the generated text while a few missing units do not affect the overall coverage.
Revised: • 4 : Most of the content units of the target text can be found from the generated text while a few missing units do not affect the overall coverage.

Original: • 3 : Some semantic content units can be found from the generated text but also miss some important units.
Revised: • 3 : Some semantic content units can be found from the generated text but also miss some important units.

Original: • 2 : Most of semantic content units are not covered.
Revised: • 2 : Most of semantic content units are not covered.

Original: Only a few insignificant units can be found in the generated text.
Revised: Only a few insigniﬁcant units can be found in the generated text.

Original: • 1 : The text does not have any overlapping semantic content units with the target text.
Revised: • 1 : The text does not have any overlapping semantic content units with the target text.

Original: We recruit five human workers to annotate 24,000 samples.
Revised: We recruit ﬁve human workers to annotate 24,000 samples.

Original: To make sure the workers are fairly paid, we pay 0.1 dollars for each sample.
Revised: To make sure the workers are fairly paid, we pay 0.1 dollars for each sample.

Original: Therefore, the total amount spent on participant compensation is 2,400 dollars.
Revised: Therefore, the total amount spent on participant compensation is 2,400 dollars.

Original: The annotators take 48 hours to finish the task, suggesting the hourly wage for each worker is 10 dollars.
Revised: The annotators take 48 hours to ﬁnish the task, suggesting the hourly wage for each worker is 10 dollars.

Original: B M ORE D ETAILS OF THE T ASKS B. M ULTI - TURN DIALOGUE RESPONSE GENERATION
Revised: B M ORE D ETAILS OF THE T ASKS B. M ULTI - TURN DIALOGUE RESPONSE GENERATION

Original: Training We fine-tune the models on the DailyDialog and LCCC datasets for 8k and 40k steps, respectively.
Revised: Training We ﬁne-tune the models on the DailyDialog and LCCC datasets for 8k and 40k steps, respectively.

Original: We use a batch size of 128 and truncate the training samples to a maximum length of 256.
Revised: We use a batch size of 128 and truncate the training samples to a maximum length of 256.

Original: The parameters of the models are initialized from Huggingface Libraries (Wolf et al., 2019a) and updated by Adam optimizer (Kingma & Ba, 2015) with a learning rate of 3e-5.
Revised: The parameters of the models are initialized from Huggingface Libraries (Wolf et al., 2019a) and updated by Adam optimizer (Kingma & Ba, 2015) with a learning rate of 3e-5.

Original: The margin values of SimCTG and SimDRC are set to 0.5 and 0.7, respectively.
Revised: The margin values of SimCTG and SimDRC are set to 0.5 and 0.7, respectively.

Original: The loss weight α is 0.3.
Revised: The loss weight α is 0.3.

Original: All hyper-parameters are selected from the development set.
Revised: All hyper-parameters are selected from the development set.

Original: The training process on the DailyDialog and LCCC datasets takes 0.7 hours and 4 hours on four A100 GPUs, respectively.
Revised: The training process on the DailyDialog and LCCC datasets takes 0.7 hours and 4 hours on four A100 GPUs, respectively.

Original: Evaluation We conduct automatic evaluations for the response generation task on following metrics: 1) BERTScore (Zhang et al., 2019) which calculates the similarities of token representations between the generated response and the target response using the pre-trained BERT model; 2) BARTScore (Yuan et al., 2021) which estimates the difficulties of converting the generated text to the reference output by the text generation method; 3)
Revised: Evaluation We conduct automatic evaluations for the response generation task on following metrics: 1) BERTScore (Zhang et al., 2019) which calculates the similarities of token representations between the generated response and the target response using the pre-trained BERT model; 2) BARTScore (Yuan et al., 2021) which estimates the difﬁculties of converting the generated text to the reference output by the text generation method; 3)

Original: BLEURT (Sellam et al., 2020) which is a reference-based text generation metrics that is robust to both domain and quality drifts; and 4) Distinct2/4 (Li et al., 2016) which computes the generation repetition at different n-gram levels.
Revised: BLEURT (Sellam et al., 2020) which is a reference-based text generation metrics that is robust to both domain and quality drifts; and 4) Distinct2/4 (Li et al., 2016) which computes the generation repetition at different n-gram levels.

Original: B. C ONVERSATIONAL RESPONSE RETRIEVAL
Revised: B. C ONVERSATIONAL RESPONSE RETRIEVAL

Original: Training For each dataset, we fine-tune the model on the training set for 3 epochs, and save the best model according to the performance on the development set.
Revised: Training For each dataset, we ﬁne-tune the model on the training set for 3 epochs, and save the best model according to the performance on the development set.

Original: The model parameters are initialized from the post-training checkpoint released by Han et al.
Revised: The model parameters are initialized from the post-training checkpoint released by Han et al.

Original: (2021) and updated by Adam optimizer with a learning rate of 1 e − 5 .
Revised: (2021) and updated by Adam optimizer with a learning rate of 1 e − 5 .

Original: The margin values in SimCTG and SimDRC are set to 0.5 and 0.8, respectively.
Revised: The margin values in SimCTG and SimDRC are set to 0.5 and 0.8, respectively.

Original: The value of α in SimDRC is set to 0.6.
Revised: The value of α in SimDRC is set to 0.6.

Original: All hyper-parameters are selected on the development set.
Revised: All hyper-parameters are selected on the development set.

Original: The training process on each dataset takes around 10 hours on a single A100 GPU.
Revised: The training process on each dataset takes around 10 hours on a single A100 GPU.

Original: Evaluation Following previous work (Zhang et al., 2018; Han et al., 2021), we use Recall (i.e.
Revised: Evaluation Following previous work (Zhang et al., 2018; Han et al., 2021), we use Recall (i.e.

Original: R 10 @
Revised: R 10 @

Original: k , k = (1 , 2 , 5) )
Revised: k , k = (1 , 2 , 5) )

Original: as our evaluation metric, which indicates the probabilities of whether the correct answer stands in the top k candidates given 10 samples.
Revised: as our evaluation metric, which indicates the probabilities of whether the correct answer stands in the top k candidates given 10 samples.

Original: For the Douban benchmark, we also compute the values of mean average precision ( MAP ) and mean reciprocal rank ( MRR ), and precision at one ( P @1 ) since the context in this dataset may contain multiple positive responses.
Revised: For the Douban benchmark, we also compute the values of mean average precision ( MAP ) and mean reciprocal rank ( MRR ), and precision at one ( P @1 ) since the context in this dataset may contain multiple positive responses.

Original: B. C ONVERSATIONAL SEMANTIC ROLE LABELING
Revised: B. C ONVERSATIONAL SEMANTIC ROLE LABELING

Original: Training We follow the previous work (Wu et al., 2021) and solve the CSRL task as the sequence labeling problem.
Revised: Training We follow the previous work (Wu et al., 2021) and solve the CSRL task as the sequence labeling problem.

Original: We keep the training settings same to CSAGN’s.
Revised: We keep the training settings same to CSAGN’s.

Original: The parameters of the model are initialized from the pre-trained BERT, and updated by Adam optimizer with a linear learning rate schedule.
Revised: The parameters of the model are initialized from the pre-trained BERT, and updated by Adam optimizer with a linear learning rate schedule.

Original: The margin values of SimCTG and SimDRC are set to 0.2 and 0.5, respectively.
Revised: The margin values of SimCTG and SimDRC are set to 0.2 and 0.5, respectively.

Original: The loss weight α in SimDRC is set to 0.2.
Revised: The loss weight α in SimDRC is set to 0.2.

Original: All hyper-parameters are selected on the development set.
Revised: All hyper-parameters are selected on the development set.

Original: The training process on the DuConv training set takes around 2 hours on two A100 GPUs.
Revised: The training process on the DuConv training set takes around 2 hours on two A100 GPUs.

Original: Evaluation Following (Xu et al., 2021; Wu et al., 2021)
Revised: Evaluation Following (Xu et al., 2021; Wu et al., 2021)

Original: [Table]
Revised: [Table]

Original: we report the F1 all , F1 intra and F1 inter scores over the (predicate, argument, label) tuples.
Revised: we report the F1 all , F1 intra and F1 inter scores over the (predicate, argument, label) tuples.

Original: The arguments are categorized into two types,
Revised: The arguments are categorized into two types,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., intra-arguments and cross-arguments, according to whether the argument appears in the same turn with the predicate or not.
Revised: e., intra-arguments and cross-arguments, according to whether the argument appears in the same turn with the predicate or not.

Original: Therefore, we calculate the F1 intra and F1 inter scores on intra- and cross-arguments, respectively.
Revised: Therefore, we calculate the F1 intra and F1 inter scores on intra- and cross-arguments, respectively.

Original: B. R ESULTS OF D IALO GPT ON RESPONSE GENERATION TASK C M ORE EVALUATION RESULTS
Revised: B. R ESULTS OF D IALO GPT ON RESPONSE GENERATION TASK C M ORE EVALUATION RESULTS

Original: [Table]
Revised: [Table]

Original: Table 6 shows the results of DialoGPT models on DailyDialog.
Revised: Table 6 shows the results of DialoGPT models on DailyDialog.

Original: Since there are no suitable DialoGPT models for Chinese, we only evaluate on the DailyDialog dataset here.
Revised: Since there are no suitable DialoGPT models for Chinese, we only evaluate on the DailyDialog dataset here.

Original: D M ORE I N - DEPTH A NALYSES D. V ISUALIZATION OF S ELF -A TTENTION W EIGHTS
Revised: D M ORE I N - DEPTH A NALYSES D. V ISUALIZATION OF S ELF -A TTENTION W EIGHTS

Original: In this part
Revised: In this part

Original: [Table]
Revised: [Table]

Original: we visualize the self-attention weights of vanilla BART and BART+SimDRC trained on DailyDialog.
Revised: we visualize the self-attention weights of vanilla BART and BART+SimDRC trained on DailyDialog.

Original: As shown in Figure
Revised: As shown in Figure

Original: [Table]
Revised: [Table]

Original: we can see that 1) in vanilla BART, all tokens are primarily attended to the dialogue representative token,
Revised: we can see that 1) in vanilla BART, all tokens are primarily attended to the dialogue representative token,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. 
 in our cases.
Revised: e. 
 in our cases.

Original: This would lead to the problem that all tokens receive much the same information and be nearby to each other on representations.
Revised: This would lead to the problem that all tokens receive much the same information and be nearby to each other on representations.

Original: This is exactly the problem of anisotropy ; 2) with the aid of SimDRC, the tokens are encouraged to be concentrated on the tokens within the same utterance and discriminative to tokens in different utterances.
Revised: This is exactly the problem of anisotropy ; 2) with the aid of SimDRC, the tokens are encouraged to be concentrated on the tokens within the same utterance and discriminative to tokens in different utterances.

Original: Therefore, we believe that SimDRC also essentially learns locality and isotropy on attention 
Revised: Therefore, we believe that SimDRC also essentially learns locality and isotropy on attention weights,

Original: . D. D IALOGUE S TATE E MBEDDING VS . S IM DRC
Revised: . D. D IALOGUE S TATE E MBEDDING VS . S IM DRC

Original: A straightforward idea to distinguish different utterance might be simply adding a dialogue state embedding (DiaSE)
Revised: A straightforward idea to distinguish different utterance might be simply adding a dialogue state embedding (DiaSE)

Original: (Wolf et al., 2019b; Xu et al., 2021) to token embeddings.
Revised: (Wolf et al., 2019b; Xu et al., 2021) to token embeddings.

Original: To this end, we introduce an additional randomly initialized dialogue state embedding layer into vanilla BART to differentiate different turns.
Revised: To this end, we introduce an additional randomly initialized dialogue state embedding layer into vanilla BART to differentiate different turns.

Original: During encoding, these additional embeddings are added with word embeddings and positional embeddings.
Revised: During encoding, these additional embeddings are added with word embeddings and positional embeddings.

Original: As shown in Figure 6(b), even with the help of the turn indicator, the problem of anisotropy still remains.
Revised: As shown in Figure 6(b), even with the help of the turn indicator, the problem of anisotropy still remains.

Original: We think the reason is that DiaSE is essentially another kind of “segment embedding” which is not capable of capturing deep semantic or conversational features
Revised: We think the reason is that DiaSE is essentially another kind of “segment embedding” which is not capable of capturing deep semantic or conversational features.

Original: . D. E XAMPLES OF G ENERATED T EXTS
Revised: D. E XAMPLES OF G ENERATED T EXTS

Original: [Table] [Table]
Revised: [Table] [Table]