Paper: Let Models Speak Ciphers: Multiagent Debate through Embeddings (ICLR-2024)

Reviewer comment (Reproducibility):
The method requires access to the output embeddings of the model and the full output probability, so cannot be directly applied to LLMs served via API.

Author response:
Our research is focused on open-source language models due to its transparency and reproducibility. While open-source models provide insight into their internal mechanisms, APIs often obscure the distinction between the AI model and the engineering components, e.g., incorporating specialized engineering for specific canned responses.

Reviewer comment (Reproducibility):
While the method is described in several sections, the definition of the method still didn't feel precise enough. For example, I assume that at each step of the autoregressive generation the input token embedding is the previously aggregated token embedding and not a single sampled token? Or do you still sample tokens for decoding and just convert the full sequence to aggregated embeddings after?

Author response:
We are grateful to the reviewer for pointing out the need for greater clarity. 
In our approach, at each step $t$ of the autoregressive generation, we utilize an aggregation of all previous $t−1$ generated token embeddings as the input for the next token generation. 

We have revised Section 3.2 of our manuscript to reflect this clarification. The updated notation, now denoted as $\overline{emb}^{(1:t-1)}$, has been introduced to clearly represent the aggregated input at each step of the generation process. Please refer to the revised version for a more detailed and precise explanation.

Reviewer comment (Experiments):
There is a lot of discussion on the effect of sampling temperature, including ablation experiments, that I am confused about. From my understanding of the method (e.g., Equation 2), there is no temperature parameter in the method. The only effect of temperature is in the final generation of the answer by one of the models. Therefore, I do not understand why two temperatures are reported, or the ablation in Figure 5.

Author response:
We apologize for any confusion caused by our initial presentation of Equation 2 and appreciate the opportunity to clarify. In Equation 2, our method generates a new embedding by calculating the weighted average of all token embeddings within the vocabulary set. Here, the “weight” for each token is determined by its respective probability, which is derived from the softmax of the logits. Similar to conventional natural language generation processes, the temperature parameter $T$ controls the smoothness of the probability distribution, thereby shaping the output embedding.
To enhance clarity and facilitate better understanding, we have revised notations and Equation 2 in our manuscript. We decided to break it down into two separate equations as follows:

$\bar{emb}^{(t)} = \sum_{i = 1}^{V} p_{vocab_i}^{(t)} \cdot emb_{vocab_i}$

where

$[p_{vocab_i}^{(t)}, \dots, p_{vocab_V}^{(t)}] = softmax \\{ logit(emb_{prompt}, \bar{emb}^{(1:t-1)} ) /T  \\}$




In addition to the equation, we have revised Section 3.2 of our manuscript to clarify on the temperature:

*”**Role of temperature.** The temperature $T$ in Eq.1 and Eq.3 controls the smoothness of the probability $p^{(t)}_{vocab_i}$. When $T \rightarrow 0$, both CIPHER's embedding generation and natural language generation result in greedy generation. In contrast, a large $T$ leads to a uniform averaging and sampling over the whole vocabulary set for CIPHER and natural language generation, respectively. We find that choosing proper temperatures for the debaters plays a pivotal role in the performance of CIPHER debate and natural language debate. Thus, to ensure fairness of our empirical evaluation, we utilize Bayesian optimization [1] to select the best performing temperatures for each method in our experiments in Section 4. Moreover, we conduct sensitivity analysis on the temperatures in Section 5.2.”*

[1] Bayesian Optimization: Open source constrained global optimization tool for Python


We believe these changes will effectively address the comments and improve the overall clarity of our method's description.

Reviewer comment (Experiments):
The experimental results are only on 4 datasets, the improvements are relatively small, and no confidence intervals are reported.

Author response:
We would like to clarify a minor discrepancy in the number of datasets mentioned. Our experiments were conducted on five datasets, including GSM8K, Formal Logic, High School Math, Professional Psychology, and Arithmetic. 


> the improvements are relatively small, and no confidence intervals are reported.


Our approach demonstrates consistent improvements of 1-3.5% across five datasets and six different models without any further training or fine-tuning.  Thus, we believe that the improvement is significant.

We ran 5 experiments for each method in Table 1 to obtain the confidence intervals for LLaMA2, reported as below:

| Method                   | GSM8K | H.S. Math | Psychology | Formal Logic | Arithmetic |
|--------------------------|-------|-----------|------------|--------------|------------|
| Single Answer            | 59.5±2.0  | 38.1±2.6      | 71.5±1.1       | 46.0±2.9         | 75.0±1.7       |
| Major@5   | 65.5±1.1  | 41.5±1.5      | **74.0±1.0**   | 44.4±2.3         | 77.6±1.4       |
| NLD    | 66.5±0.8  | 39.4±0.9      | 73.0±1.5      | 49.2±0.9         | 83.0±1.6       |
| CIPHER (ours)            | **67.5±0** | **41.5±0**  | **74.0±0**   | **52.4±0**     | **86.5±0**   |

Overall, *Single Answer* has the highest variance, ranging from 1 to 3% across five datasets. This increased variance is attributed to its token sampling process during token generation. *Major@5* and *NLD* methods reduce variance by ensembling diverse responses. However, they still rely on a similar token sampling process, leading to relatively high variance. In contrast, *CIPHER* is not affected by this issue, thanks to its deterministic embedding generation. For any given input, *CIPHER* consistently produces the same embeddings, which are then translated into identical texts via a nearest neighbor search across the vocabulary set.

It's important to note that running these baseline methods are time-consuming, regarding that we need at least 5 runs for each to obtain a valid standard error estimation. While we are working on confidence intervals for additional experiments, as of now, we can only provide these intervals for LLaMA2 in Table 1 as part of our rebuttal. We plan to include all the results in the final camera-ready version of our work.

Reviewer comment (Experiments):
I didn't see any discussion on validation set for hyperparameter setting, and according to the appendix it seems like different temperatures were used for the main baseline (NLD) and for CIPHER. Why is that? This raises some additional concerns about the experimental setup.

Author response:
As our approach focuses solely on inference without involving any training phase, the conventional use of a validation set for hyperparameter tuning was not applicable. Additionally, we swept over temperatures to guarantee we report the best of each method, hence no need to set a validation set.


>  according to the appendix it seems like different temperatures were used for the main baseline (NLD) and for CIPHER? Why is that? This raises some additional concerns about the experimental setup.

We employ Bayesian optimization to sweep over some temperature pairs for all the baselines and our method. For a fair comparison, we reported the best performance of each method in our paper. We also included an ablation study on the temperature, as reported in Figure 5, to give a deeper understanding of how the temperature affects debate performance.

As each method prefers different temperatures to obtain the best results, it is not fair to use the same temperature. For example, Majority Voting (major@5) usually requires higher temperatures compared to *Single Answer*, to have a more diverse response for the ensemble.

 That said, we still conducted some results on the same pairs of temperatures (in brackets) for both CIPHER and NLD baseline as follows:

| Method                   | GSM8K (0.1, 0.5) | H.S. Math (0.1, 0.2)| Psychology (0.3, 0.8) | Formal Logic (0.3, 0.5) | Arithmetic (0.2, 0.3)|
|--------------------------|-------|-----------|------------|--------------|------------|
| NLD    | **65.5**  | 38.2      | 68.5      | 39.7      | **74.5**       |
| CIPHER (ours)            | 65.0 | **40.0**  | **72.5**  | **44.4**    | 74.0   |


In general, CIPHER works better than NLD for these temperatures reported above, but again, this is not a good setting as each method favors different temperature settings. As discussed in Sec 5.2, CIPHER often prefers low and high-temperature agents to pair together. This explains its lower accuracy compared to NLD on two datasets, as shown in the table above, particularly in Arithmetic, where CIPHER prefers a large difference in temperatures.

Reviewer comment (Reproducibility):
In addition to point 2, the variants of the method are also not precisely described. For example, in Table 2: "This issue can be addressed by maintaining a bijection mapping between the vocabulary embeddings of the two models." I can guess how it was implemented, but would appreciate elaboration.

Author response:
Thanks for bringing this to our attention! In Section 4.2, we’ve added more implementation details for CIPHER debates between LLaMA-65B and LLaMA2-70B as follows:

*”While the tokenizer is shared between LLaMA-65B and LLaMA2-70B, they use distinct embeddings for each vocabulary. To tackle this issue, we keep a mapping vocab -> [$embed_{LLaMA1}$, $embed_{LLaMA2}$] for each vocabulary and compute the weighted average of the embeddings using the embeddings of the receiver. For example, to pass the message to LLaMA2-70B, we average over embed_llama2 during the CIPHER response generation from LLaMA1-65B debater. This guarantees that the output of LLaMA1-65B is encoded within the LLaMA2-70B’s token embedding space.”*

Reviewer comment (Experiments):
It would be interesting to further understand the effect of passing these aggregated embeddings to the model. The communication to later generated tokens is anyway through K,V embeddings of the self-attention that have some dense representations and are not limited to the token vocabulary. Some exploration on how the input embeddings impact the K,Vs could perhaps shed some light on how CIPHER modifies the communication.

Author response:
We thank the reviewer for the suggestion. 
We have included an analysis of attention heatmaps for the human language debate baseline (NLD) and CIPHER in Appendix B3 of the updated manuscript. The heatmap figures can be found in the latest version of the paper.

*“Fig 9 shows a comparison of attention heatmaps for CIPHER and NLD at the 45$^{th}$ decoder layer of LLaMA2-70B. These heatmaps correspond to the arithmetic question we used in Figure 1 during the last debate round. Specifically, we compute the similarity between the $q$ vector of the last token of the first agent and the $k$ vectors of its preceding 100 tokens. We can observe that NLD's heatmap exhibits uniform attention distribution, lacking intense focus on any particular segment. Conversely, CIPHER's heatmap shows some distinct bright spots, particularly around the 40$^{th}$ attention head and the 74$^{th}$ time steps. This suggests that the model's attention is highly focused on those areas, potentially indicating areas of higher relevance for the task.”*

Reviewer comment (Experiments):
The authors conducted experiments on five common reasoning datasets, can this method be tested on agent-related leaderboards.

Author response:
> The authors conducted experiments on five common reasoning datasets, can this method be tested on agent-related leaderboards

We kindly ask the reviewer to clarify which agent-related leaderboards might be relevant. If the leaderboard suits our framework, we would be happy to test and report it in our final version.

Reviewer comment (Experiments):
In formula 2, whether the response embedding will be adjusted, how are the results of different weights?

Author response:
> In formula 2, whether the response embedding will be adjusted, how are the results of different weights?

In Equation 2, our method generates a new embedding by calculating the weighted average of all token embeddings within the vocabulary set. Here, the “weight” for each token is determined by its respective probability, which is derived from the softmax of the logits. Similar to conventional natural language generation processes, the temperature parameter $T$ controls the smoothness of the probability distribution, thereby shaping the output embedding. We also added a discussion on the temperature in the revised version in Sec 3.2, as follows:

*”**Role of temperature.** The temperature $T$ in Eq.1 and Eq.3 controls the smoothness of the probability $p^{(t)}_{vocab_i}$. When $T \rightarrow 0$, both CIPHER's embedding generation and natural language generation result in greedy generation. In contrast, a large $T$ leads to a uniform averaging and sampling over the whole vocabulary set for CIPHER and natural language generation, respectively. We find that choosing proper temperatures for the debaters plays a pivotal role in the performance of CIPHER debate and natural language debate. Thus, to ensure fairness of our empirical evaluation, we utilize Bayesian optimization [1] to select the best performing temperatures for each method in our experiments in Section 4. Moreover, we conduct sensitivity analysis on the temperatures in Section 5.2.”*



To enhance clarity and facilitate better understanding, we have revised the notations and Equation 2 in our manuscript. We decided to break it down into two separate parts, now Eq.2 and Eq.3, as follows:


$\bar{emb}^{(t)} = \sum_{i = 1}^{V} p_{vocab_i}^{(t)} \cdot emb_{vocab_i}$

where

$[p_{vocab_i}^{(t)}, \dots, p_{vocab_V}^{(t)}] = softmax \\{ logit(emb_{prompt}, \bar{emb}^{(1:t-1)} ) /T  \\}$

Reviewer comment (Experiments):
Why are the results in table1 and table2 completely different, how many rounds are used in table1?

Author response:
> Why are the results in table1 and table2 completely different, how many rounds are used in table1？

The settings in Table 1 and Table 2 are different and are stated in the captions. Specifically, in Table 1, we facilitate debates between two identical LLaMA-family models, with temperature being the only difference between the two debaters. In this case, we report the accuracies of the final responses from the debater operating at the lower temperature. In Table 2, we facilitate debates between one LLaMA-65B (first version) and one LLaMA2-70B. In this case, we report the accuracies of the final round responses from the two debaters separately for a thorough analysis.
In both Table 1 and Table 2, the total number of rounds is 3.
We added this to the caption for Table 1 in our revised version.

Reviewer comment (Experiments):
Can this method be used for different models with the same tokenizer, for NLD, different models can communicate each other.

Author response:
> Can this method be used for different models with the same tokenizer, for NLD, different models can communicate each other.

Yes. In Table 2, we applied CIPHER debate between LLaMA-65B and LLaMA2-70B.

Reviewer comment (Experiments):
Limited Generalizability. As the authors described in the limitations, this method is only applicable to LLMs that share a common vocabulary. For different types of LLMs, aligning embeddings is a difficult task.

Author response:
As stated in the Limitation section, requiring a shared tokenizer is a limitation of our work and is out of the scope of this paper. Extending CIPHER debate to models with different tokenizations will require tokenization/embedding alignment, which would be an interesting direction for us to further explore. Such an alignment can potentially be tackled by utilizing methods in multimodality literature, such as [1, 2]. This literature deals with broader questions related to aligning various modalities.

We would like to note that, even though CIPHER debate can only be applied to models sharing the same tokenizer, we verify that it is effective in pushing the reasoning performance of the open-source models beyond the current state-of-the-art methods.

[1] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

[2] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Reviewer comment (Presentation):
From Figure 10, the language of CIPHER is still difficult to analyze.

Author response:
As described in the Introduction Section: “First, we do not necessarily need to understand the intermediate debates amongst LLMs. Second, natural language generation uses only one token to represent the model’s belief over the entire vocabulary, which risks losing information embedded within the model output logits.” 
The main takeaway of our approach is to trade interpretability for information preservation during the debates. Yet, we can still interpret the communications made by the debater operating at low temperatures (e.g., blue debater, now Figure 11 in the revised version) while using another debater with a higher temperature to gather more information. In practice, keeping at least one debater at a low temperature is necessary to collect the final result of the debate.

Reviewer comment (Experiments):
Which experiment result can support the statement "our approach can generalize across a wide array of LLMs, enabling even smaller LLMs to unlock the benefits of debate and achieve better performance than majority voting"? Is there any experiment of smaller LLMs like LLaMA-2 13B or others?

Author response:
Prior work in LLM debate and self-critique (e.g., [1, 2]) show that only some state-of-the-art LLMs, such as GPT-3.5, GPT4 are capable of taking advantage of debates and discussion. These models are over 100B parameters and closed-source.  In our work, we propose CIPHER that unleashes the potential of smaller open-source models in debates. By “generalize across a wide array of LLMs”, we mean that one can apply CIPHER to smaller sized models such as LLaMA-family ones, ranging from 7B (reported in Appendix B.2, Fig. 8a) to 70B (table 1). We chose to mainly use LLaMA2-70B in our experiments because it’s one of the largest and most up-to-date models at that time. Additionally, we showed the results across different model architectures in Fig 3 to evaluate the robustness and generalizability of our method.

[1] Self-refine: Iterative refinement with self-feedback

[2] Improving language model negotiation with self-play and in-context learning from AI feedback.

Reviewer comment (Evaluation):
Why is the performance of CIPHER worse than natural language debate when Round=1?

Author response:
In Table 2, when round=1, it means the model only gives a direct response. By choosing responses of the larger model (“LLaMA2-70B” columns) as the final answers, our method is not worse, but rather slightly better than the baseline at round=1. That said, CIPHER is designed for communication in debate settings. Thus, we need debate rounds to fully take advantage of the method. We added the results by round for ablation study purposes, to see how the performance changes over rounds.

Reviewer comment (Reproducibility):
It is unclear how the Convert-and-Aggregate function works, in particular how the responses from multiple debaters are distilled into a single response.

Author response:
We write the *Convert-and-Aggregate* function in the debate algorithm within a general framework, to accommodate the various distillation strategies in the current literature. In our revision, we made extra elaboration on what such a function can be in the context of LLM debates and also justify our choice. In general, we can apply random pick, majority voting [1], involving a judge as suggested in a concurrent work [2], and lower temperature (our method). We now clarify in Section 3.2 the following:

*“To close the debate, at Convert-and-Aggregate step (Line 6), we convert the embedding responses back to natural language using nearest neighbor search over the vocabulary set, then aggregate them to obtain the final response. In most cases, LLM debaters typically reach a consensus answer by the final round, as observed in [3]. When divergence in final responses occurs, majority voting [1] or random tie-breaking are often used. However, majority voting may not be suitable for open-ended questions (e.g., summarization tasks) where multiple correct answers exist, as in the game of 24 [4], and scenarios where debates involving only two agents. Thus, in our experiments, we select the response from the debater operating at the lowest temperature as the final answer. This approach is computationally efficient as it only requires running inference on one model at the final round. Meanwhile, it also achieves comparable accuracy with the best performing debater, as evidenced in Fig.5”*

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models

[2] Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

[3] Improving Factuality and Reasoning in Language Models through Multiagent Debate

[4] Tree of thoughts: Deliberate problem solving with large language models

Reviewer comment (Reproducibility):
The "Result collection" and "Metrics" paragraphs in Section 4.1 are the first time in the paper that differing temperatures are mentioned. If an optimization procedure is being used for temperature selection as part of the method, then this should be described in detail along with the rest of the method in Section 3.

Author response:
Thanks for pointing it out. We have incorporated the *Result collection* section along with the temperature selection procedure to Section 3.2, answered in the above question.
We also dedicated a section for the role of temperature and how it can affect the debate resulting in the new revision, Section 3.2 as follows:

*”**Role of temperature.** The temperature $T$ in Eq.1 and Eq.3 controls the smoothness of the probability $p^{(t)}_{vocab_i}$. When $T \rightarrow 0$, both CIPHER's embedding generation and natural language generation result in greedy generation. In contrast, a large $T$ leads to a uniform averaging and sampling over the whole vocabulary set for CIPHER and natural language generation, respectively. We find that choosing proper temperatures for the debaters plays a pivotal role in the performance of CIPHER debate and natural language debate. Thus, to ensure fairness of our empirical evaluation, we utilize Bayesian optimization [1] to select the best performing temperatures for each method in our experiments in Section 4. Moreover, we conduct sensitivity analysis on the temperatures in Section 5.2.”*

[1] Bayesian Optimization: Open source constrained global optimization tool for Python

Reviewer comment (Experiments):
The temperatures used should all be clearly reported, and whatever process is used for temperature selection should either also be applied to the other baseline methods where relevant, or ablated away in a separate experiment to highlight potential sensitivities to this hyperparameter, or both.

Author response:
We apologize for the confusion. The temperatures used for the reported accuracies are already included in Appendix D (tables 3, 4, 5). To further aid reproducibility, we added all other temperature settings in two additional tables (Table 6 and Table 7) in Appendix D.1. These new tables correspond to the temperatures depicted in Figure 4. 
We included a reference to this Appendix in the main text of our revised version, at the beginning of Section 4.1 “Experimental Setup”, for easier reference as follows:

*“For all the methods, we use Bayesian optimization to select temperatures, which are reported in Appendix D for reproducibility of our results.”*

For both CIPHER and the baselines, we swept over temperatures using Bayesian hyperparameter optimization to make sure that the accuracy reported is optimized for each method. Additionally, we also investigate the effect of temperature, as shown in Contour plots in Fig. 5 for an ablation study.

Reviewer comment (Writing):
In Section 4.1 you say “we select the response from **the debater operating at the lowest temperature** as the final answer”. But in Section 5.2 you say “To determine the optimal temperature pairs, we utilize Bayesian optimization (Nogueira, 2014–), and report the accuracies based on the final round response generated by **the first debater**”. These appear to be contradictory.

Author response:
Thank you for bringing this to our attention. When facilitating debates between identical models operating at different temperatures, we always put the lower temperature debater in the first place. So taking the first debater’s final round response is equivalent to taking the debater operating at the lowest temperature. However, we agree that the inconsistency in our presentation can bring confusion. In the revised version, we have unified the explanations of how our final responses are obtained.

Reviewer comment (Presentation):
In the caption for Figure 5 you say “best performance is achieved when temperature 1 is lower than temperature 2” but this is not at all apparent from these plots. The only clear take-away from them is that accuracy is high when temperature 1 is low.

Author response:
We have re-plotted Fig 5 with different temperature ranges, and changed the color scale to make the accuracies more distinguishable in the figures. The main takeaway from these plots is that CIPHER debate (Fig. 5, bottom row) usually prefers the debaters to have their temperatures wider spread apart in order to achieve optimal performance. Additionally, the optimal regions appear on the left side of the charts, where temperature 1 is less than temperature 2. These indicate that a good strategy is using various temperature agents, and choosing the response of the lower temperature agent as the final debate answer.

We updated the discussion in our new version and rewrote Sec 5.2 as follows:

*”[...] We investigate the advantages of allowing certain debaters to deviate from natural language during debates. More concretely, one debater operates at a lower temperature to ensure the final answer remains comprehensible to humans, while the other debater is tasked with conveying information that may deviate from natural language by generating its responses at a higher temperature. We employed Bayesian optimization [1] to identify promising pairs of temperature settings. We evaluate the effectiveness of these settings based on the accuracy of the final response from the first debater, referred to as temperature 1. This approach enabled us to examine the relationship between temperature settings and debate performance, allowing us to determine whether lower or higher temperature settings yield better results”*

*”[...] Notably, the optimal regions for CIPHER are mostly on the left side of the charts, indicating that CIPHER benefits the most when it pairs a low-temperature agent with a high-temperature one. At higher temperatures, the probability distribution of the tokens in the vocabulary becomes more uniform, allowing CIPHER's responses to lean towards less confident token choices. This effectively complements the information communicated by other CIPHER debaters operating at lower temperatures, where they focus on more confident tokens. Additionally, low-temperature agents are necessary for collecting results that can be easily understood by humans. Therefore, a good strategy for CIPHER is to employ various temperature agents and use the response of the
lower temperature agent as the final debate answer.”*

Reviewer comment (Writing):
Using “l” to refer to a token index rather than a layer index is confusing.

Author response:
We switched to $t$ for token indexing.

Reviewer comment (Presentation):
Using long names and blocky Courier-esque fonts for variables (e.g. "$\texttt{embresponse}$”) makes the text harder to read.

Author response:
We have revised the namings and fonts in equations and algorithms.

Reviewer comment (Writing):
Using the direct-sum symbol for concatenation is a nonstandard choice that may confuse readers.

Author response:
We changed the direct-sums to “concat”  to make it clear.

Reviewer comment (Presentation):
Captions for Table 1 and Table 2 are above their respective figures rather than below. These tables, captions, and their adjacent paragraphs are also extremely close together.

Author response:
Putting captions above the tables is actually required by the ICLR formatting guidelines.

> These tables, captions, and their adjacent paragraphs are also extremely close together.

We adjusted the paragraphs to make it clear to read.

Reviewer comment (Presentation):
The micro-parentheticals in Table 2 make the overall table hard to read without adding much; it is recommended to remove these or add them as supplemental information in the appendix.

Author response:
We have removed them to make the table clear

Reviewer comment (Presentation):
The heatmap plot in Figure 5 is very hard to interpret. Especially on the right side of the plot, the points are very sparse, leading to artifact-heavy interpolation. A different way of presenting this information is recommended.

Author response:
During our experiments, we explored the temperature pairs using Bayesian optimization, which naturally explores more in the high accuracy area, resulting in most of the points being gathered around the left side of the charts. It would be better to have more points on the right side of the plot, but due to the time constraint of this rebuttal phase, we were not able to have more experiments. Thus, we rescaled the heatmaps by exponential coloring and chose more meaningful axis ranges. We hope these changes make the plots easier to read.

Reviewer comment (Presentation):
The most unclear parts of the paper were related to the use of temperature, and the selection procedure for temperature. These should be described explicitly and clearly along with the rest of the method, rather than being scattered across the Results and Analysis sections.

Author response:
Thank you for bringing this to our attention. In accordance with this, we’ve moved the Result Collection part to Section 3. 
We also dedicated a section for the role of temperature and how it can affect the debate resulting in the new revision, Section 3.2 as follows:

*”**Role of temperature.** The temperature $T$ in Eq.1 and Eq.3 controls the smoothness of the probability $p^{(t)}_{vocab_i}$. When $T \rightarrow 0$, both CIPHER's embedding generation and natural language generation result in greedy generation. In contrast, a large $T$ leads to a uniform averaging and sampling over the whole vocabulary set for CIPHER and natural language generation, respectively. We find that choosing proper temperatures for the debaters plays a pivotal role in the performance of CIPHER debate and natural language debate. Thus, to ensure fairness of our empirical evaluation, we utilize Bayesian optimization [1] to select the best performing temperatures for each method in our experiments in Section 4. Moreover, we conduct sensitivity analysis on the temperatures in Section 5.2.”*

Reviewer comment (Writing):
The paper is more difficult to read than it needs to be due to poor notation and formatting, which should be updated to match the style guide where appropriate.

Author response:
We updated all the changes as suggested as follows