Paper: MAP's not dead yet: Uncovering true language model modes by conditioning away degeneracy (ICLR-2024)

Reviewer comment (Writing):
**Poor Presentation**: The presentation of this work is poorly written. Please proofread your manuscript before submission. Some examples are  
  * Section 1: distribution which the mode representsEikema & Aziz (2020) --- the citation should be included in parenthesis and there should be a white space.  
  * Section 1: training data data --> training data  
  * Footnote 1: the regurgitating the input --> the regurgitating of the input  
  * Section 2.1: distributions arbitrarily closely --> distributions arbitrarily close  
  * Section 3: translation model Tiedemann & Thottingal (2020) --> translation model (Tiedemann & Thottingal, 2020)  
  * Section 3: ROC stories dataset Mostafazadeh et al. (2016) --> ROC stories dataset (Mostafazadeh et al., 2016)  
  * Section 3: LLaMA model Touvron et al. (2023) --> LLaMA model (Touvron et al., 2023)

Author response:
Thank you for the detailed feedback here, we have addressed the problems in the updated draft.

Reviewer comment (Theory):
**Poor Theoretical Analysis**: The theoretical motivations presented in Section 2.1, 2.2, 2.3 are hard to follow. For instance, the authors write "For the SVO translation example", what is SVO? And there should be a proper citation. The mathematical derivations in those sections are not detailed enough, which clearly undermines the quality of this work.

Author response:
We motivated our problem and analysis via several examples related to various considerations for natural language processing in Section 2. SVO is a common abbreviation used by linguists and NLP researchers to characterize a language that typically arranges information in a “subject verb object” order in the sentences. We have reworded the writing to make it clear we were referring to the preceding example in the text.

Regarding the mathematical derivations not being enough, our goal in Section 2 is simply to show that the bad mode problem can arise in the absence of model error. This does not require an involved calculation, as it’s clear that a large set of valid outputs can easily be dominated by a small set of noise sequences, even with a tiny mixing rate. There is value in presenting this to the community, as it is an underappreciated point. Most prior work on the topic makes the inference that “bad mode” implies “model error”, which has led to a dearth of research on approximate-MAP methods.

Reviewer comment (Reproducibility):
**Lack of Details in Experiments**: There are necessary details missing, including how the outputs are obtained from the LM which are then used to train the classifier; number of epochs; number of training samples.

Author response:
The space constraints made it difficult for us to include extensive details for a lot of aspects of the paper’s contents. In the original draft, we included extensive details in the appendix as well to complement the claims, experiments, and arguments in the main body. 
However, we would like to point out that we attempted to provide sufficient details in the main body of the original draft- sections 5.2, 5.3, 3, and 3.1. Additionally, we provided more extensive details about our experiment setup in the Appendix sections G and D, as well as giving pointers to them in the Reproducibility section. 

To the specific question about training examples for the LLaMA-7B classifier, 14,000 training examples were produced using vanilla beam search on inputs from the databricks-dolly-15k dataset.The number of training examples was stated in the body of the text, but we have added it to Appendix G along with the other details to make it easier to find.

Thank you once again for your detailed feedback, we believe it has helped us produce a much better draft, especially through the addition of more evaluations of our system outputs.

Reviewer comment (Experiments):
**Limited Evaluations**: The evaluations are only considered up to 200 tokens which is too short under current literature, e.g. ChatGPT API has 4k length. I strongly suggest the authors to extend their evaluations up to at least 2k length.

Author response:
We agree that experiments using more computational resources would be necessary to establish ACBS as a state-of-the-art decoding method. However, we actually view the ACBS experiments as supporting the preceding sections, rather than the other way around. Prior work has suggested that we should abandon MAP-like methods in favor of sampling-based ones, due beam search’s failure to produce good outputs in tasks less constrained than machine translation, and its failure in even MT at large beam sizes. (For example, see Holtzman et al. (2020), Stahlberg and Byrne (2019), and Eikema and Aziz (2020)).

In this work, we use ACBS as a preliminary indication that the choice to drop MAP was premature, as we suggest in the title of the paper. Our ideal outcome from this paper wouldn’t be that everyone stops using nucleus sampling and switches to ACBS tomorrow. Instead, we hope to spur future work on improved MAP-like methods.

Reviewer comment (Presentation):
Missing experimental data in 5.3. The detailed experiment result about the comparison between ACBS and regular beam search should be compared in a table instead of directly stating the data in the paragraph. Otherwise, your results won't be convincing.

Author response:
Thanks for pointing out that the empirical comparison between regular beam search and ACBS doesn’t appear to be detailed from our in-text description in the original draft. In our updated draft, we report the results in Table 5, and additionally. 

Please see our overall response to the reviewers for a description of the other newly added evaluations.

Reviewer comment (Presentation):
In 3.2.1, figure 1a and figure 1b just represents the increase of empty sequence with source length, lack of the curve that shows the decrease of empty output with source length to better support the conclusion.

Author response:
We have re-added a figure which was cut to make this explicit. Figure 2 in the appendix now shows that the probability of the empty output declines with source length. Despite this, the rate at which the mode is empty increases, which is not surprising in light of our analysis in Section 2.

Reviewer comment (Theory):
The paper only gives two examples to illustrate that low-entropy distractor outputs and empty outputs have a high log prob in section 2 but does not provide enough mathematical reasoning, so the argument that the degenerate modal behavior is related to the entropy of the set of valid outputs is not strong enough.

Author response:
The entropy of a distribution is just the average of the negative log likelihood of a sample, so what we do in Section 2.1 is compare how two distributions with different entropies react to a small amount of noise. We did not have space to get into this, but in practice what one needs is high-entropy PLUS the distribution of valid outputs is not too “non-uniform”. Since we don’t have analytic access to the population distribution for realistic datasets, we aren’t trying to show that the bad mode problem only arises from noise in the data. Instead, we just want to give enough argumentation to show that the inference “bad model mode -> the model has mis-estimated the mode of the distribution” is not true. It is important that this point becomes internalized by the community of NLG researchers, as it strongly affects how we interpret prior findings such as Stahlberg and Byrne (2019).

Additionally, we now also refer to Ott et al. (2018) for another example of a low entropy distractor. In their work, adding a small fraction of training examples where the reference translation is a copy of the source sentence leads to an outsized impact on search outputs. This is just what our analysis predicts, but they do make the theoretical argument for why this _must_ happen in certain cases, depending on the noise rate and entropy/likelihood of the valid outputs.

Reviewer comment (Theory):
At the end of Section 3, how can we conclude that the degenerate modal behavior is related to the entropy of the set of valid outputs?

Author response:
> At the end of Section 3, how can we conclude that the degenerate modal behavior is related to the entropy of the set of valid outputs?
We cannot conclusively conclude that, but entropy is roughly equivalent to there being a larger set of valid answers. As we note, the degenerate mode behavior occurs more frequently for more open-ended prompts and longer source sentences for translation (see Figure 1a), but which we argue is suggestive, but not conclusive that the entropy plays a large role.

Reviewer comment (Experiments):
Exact search experiments are meaningful, but the result that exact search performs good is not relevant to Subsequent Chapters. Maybe their relevance should be emphasized by experiments in terms of computational cost.

Author response:
> Exact search experiments are meaningful, but the result that exact search performs good is not relevant to Subsequent Chapters. Maybe their relevance should be emphasized by experiments in terms of computational cost.
We open Section 5 by stating “ DFS-related approaches are too expensive to be practical in NLG applications, especially so for the conditioning variables that force high amounts of backtracking.” Does this help with the connection between sections? If not, we can emphasize this further in the writing.

Reviewer comment (Experiments):
The experiment to prove that ACBS not only benefit from removing empty outputs should be detailed.

Author response:
The experiment to prove that ACBS not only benefit from removing empty outputs should be detailed

The experiment to prove that ACBS not only benefit from removing empty outputs should be detailed
We in fact controlled for this by making our baseline stronger in empirical comparison! By “restrict our attention” in paragraph 2 of Section 4, we meant that we dropped any examples from our analysis for which beam search (baseline) produces an empty output. This gives the beam search baseline a large advantage, since it isn’t penalized for this failure mode, but ACBS still finds an improved output much more often. As we mentioned above, we have additional results about the quality of generated outputs evaluated by GPT-4 and humans to further support our claims.

Reviewer comment (Novelty):
The explanation of degeneracy in terms of low-entropy strings is not new, and the authors may not be aware of the following two papers: Ott, https://arxiv.org/abs/1803.00047 and Holtzman, https://arxiv.org/abs/2104.08315.

Author response:
We have added brief discussion of both papers you referenced in our updated draft. Ott et al.’s work aligns precisely with the behavior we would expect, although they don’t make the generalization to low-entropy noise in general.

Reviewer comment (Evaluation):
The evaluation of the attribute constrained beam search method for LLaMA consists of recording what percent of sentences get a higher reward according to the same reward model used in training, and giving a general subjective impression of sample outputs. I think this is a fairly weak evaluation, and it would be a lot better to elicit quality judgements from other people.

Author response:
We agree with your concern about evaluation based on our reward model. In our updated draft, we report additional metrics that support our claims. In Table 5, we additionally report the results of blinded pairwise comparisons made by GPT-4 and one of the authors. (In a camera-ready version we will add additional human annotators). See also our overall response for other evaluations we have added.

Reviewer comment (Presentation):
There is too much important information in the appendix, especially Algorithm 1, with many references from the text to the appendix.

Author response:
We agree that it is not ideal to put so much content in the Appendix, but we were not able to accommodate it in the main text. We have also cleaned up the writing style in several places per your remarks.

Reviewer comment (Writing):
Everyone has their own writing style, but I feel that there are too many exclamation points for an academic paper. There are even two sentences in a row with exclamation points on page 4, but this is probably just an editing error.

Author response:
We agree that it is not ideal to put so much content in the Appendix, but we were not able to accommodate it in the main text. We have also cleaned up the writing style in several places per your remarks.

Reviewer comment (Evaluation):
Table 2: how can truncated beam search be better than itself?

Author response:
We realize that the presentation of these results might be confusing. The way to read this is that B=5 and length ratio 0.8, ACBS wins 56.9 % of the time and TBS wins 35.5. % (remaining 7.6% are ties), and for B=20 and length ratio 0.8, ACBS wins 57.8 % of the time and TBS wins 29.1 % (remaining 13.1% are ties). We have  made this clearer in the updated draft.

Reviewer comment (Evaluation):
Table 3: why do the TBS translations seem shorter?

Author response:
The apparent discrepancy in length is due to the tokenizer splitting “It’s” into 3 tokens. For example:

['▁Andr', 'ada', '▁said', ':', '▁"', 'It', '’', 's', '▁not']  
['▁Andr', 'ada', '▁said', ':', '▁“', 'It', '▁does', '▁not', '.']  
(Both 9 tokens, excluding EOS)

Reviewer comment (Presentation):
Section 5.3.1: In the example, the source sentence in the prompt is "I love machine learning," but the output translates the source sentence "My eyes are clear." Is that really what happened?

Author response:
Our original example used the sentence “I love machine learning” in the prompt, but we switched it to “My eyes are clear” once we decided to submit to ICLR.

Reviewer comment (Theory):
**Unclear logic between Section 2 & 3.** In Section 3.2, the authors provide experimental results that "*the occurrence of empty sequences increases with source length*". The authors attribute the empty mode problem to that "*the entropy of valid outputs increases with input length, but the probability of the empty output does not decline enough*". I am confused that this may contradict the analysis in Section 2 that attributes the bad mode to "*low-entropy distractors*".

Author response:
The empty sequence is such a low-entropy distractor (more generally you could consider the set of truncated translations of 0-2 tokens to be the set of distractors).
The number of valid translations for a 30 token input will be enormous, so despite the fact that the probability of the low-entropy set declines (See the newly added Figure 2), the entropy of the good outputs increases too fast. Entropy increasing is the same thing as the average log-likelihood decreasing, so this could equivalently be stated as: The likelihood of a typical good translation decreases with length faster than the likelihood of the distractors decreases with length.

Reviewer comment (Evaluation):
**Experiments are not serious** – In Section 5.2.1, the authors compare their proposed attribute-conditional beam search with truncated beam search when a target length is provided. The comparison is based solely on the log-likelihood of the search results, without considering the evaluation of generated quality, such as BLEU scores. This evaluation seems insufficient, especially considering the background that a language model's mode can lead to degenerated results. Furthermore, there's no comparison between the proposed attribute-conditional beam search and standard beam search (without length truncating, evaluated with both likelihood and BLEU), which appears to be weird and not convincing.

Author response:
In Section 5.2.1, we primarily focused on likelihood because we wanted to emphasize that constrained beam search is not an appropriate method for performing length-constrained MAP finding since our approach can find higher likelihood sequences under the unspecified length-conditional distribution. While it is true that mode of the MT model might be of bad quality, in the sections prior to ACBS, we show that *conditional* modes ameliorate the bad mode problem. Hence, we only reported likelihood scores in our first draft. 

To further increase the confidence in this finding, we have added quantitative evaluations throughout our experiments. Please see our summary response for a description of the newly added evaluations.

Reviewer comment (Novelty):
**Concerns on the novelty** – The novelty of proposed attribute-conditional beam search is limited. Those attribute-conditional sampling methods is well-studied, and the authors only adapt it to the beam search.

Author response:
We see ACBS as an individual supporting piece of evidence to the overall story, rather than the main novel contribution of the paper. Namely it suggests that our insights and empirical results can be used to drive the design of future MAP-like decoding algorithms. Currently, the decoding algorithms in common use are either sampling based (top-k sampling, top-p sampling, locally typical, FUDGE), or try to maximize some score (Reward-augmented beam search, Monte-Carlo tree search, or the recent methods suggested by Deng and Raffel, 2023, and Mudgal et al., 2023). Here, we point towards another possible class of algorithms: those which     search for sequences of maximal likelihood under a constraint.

The community has largely stopped trying to improve on this class of algorithms, due to what we argue are misconceptions about the reason for degenerate modal sequences. As we argued in Section 2, MAP-like methods have certain desirable properties in terms of controlling noise. We hope that the results we share here will urge the community into further research on this class of algorithms.

Reviewer comment (Writing):
**The paper has too many typos** – Please check the format (should use ICLR 2024).

Author response:
Thank you for taking the time to review our paper, providing helpful feedback, and recognizing the detail in which we address the issue of modal sequences in LMs. We have tried to **eliminate the typos** in our newly posted draft and addressed some of your concerns as well.

Reviewer comment (Writing):
**The paper has too many typos** – wrong citing format.

Author response:
Thank you for taking the time to review our paper, providing helpful feedback, and recognizing the detail in which we address the issue of modal sequences in LMs. We have tried to **eliminate the typos** in our newly posted draft and addressed some of your concerns as well.