Paper: The Consensus Game: Language Model Generation via Equilibrium Search (ICLR-2024)

Reviewer comment (Writing):
To improve the clarity of the proposed method, it would be advantageous to include a running example that demonstrates the step-by-step process.

Author response:
Thank you for the suggestion. To improve clarity, we will include a pseudo-code with a running example in our revision.

Reviewer comment (Writing):
To enhance the quality of the paper, it is recommended to address minor typos and errors. For instance, in line 6 of the abstract, "a new, a training-free" should be corrected to "a new, training-free." Additionally, in section 2, line 6, "we may them" appears to be a typographical error and should be revised for clarity.

Author response:
Thank you for catching those typos. We will fix them in our revision.

Reviewer comment (Theory):
How is the payoff matrix defined for the consensus game?

Author response:
Signalling games are sequential in nature, as such, they are typically represented in tree-form as we show in Figure 1 (i.e., D and G both receive a reward of 1 each if D picks the correctness parameter that D was trying to communicate. In all other cases, they receive a reward of 0 each). More specifically, D and G receive a reward of (1, 1) each if and only if G picks the same correctness parameter (correct or incorrect) that D received. In all other cases, they receive (0, 0). So, for example, if D receives the signal "correct" and produces a natural language string "u". And if G, conditioned on "u", produces "correct", they both receive (1, 1). If G instead picks "incorrect", they both receive (0,0).

Reviewer comment (Experiments):
If we compare the proposed ER-based reranking method and previous baselines, there are actually two major differences. One is the process of finding the Nash equilibrium under the regularization part, the other is the combination of two normalized probabilities in Section 2.2. In the current evaluation, it is clear that the combination of these two can bring consistent improvement, but it's unclear what is the effect of each individual component. It would be helpful to have a baseline just combining the two normalized probabilities (e.g., via the MI approach).

Author response:
**What are the relative contributions of normalization and equilibrium search?**

This is a great question! To tease apart these two components of EQUILIBRIUM-RANKING, we have run a new ablation experiment that reranks using a pointwise-mutual-information-style product using $\pi_G^{(1)}$ and $\pi_D^{(1)}$ rather than unnormalized generator and discriminator probabilities. This new ablation (labled MI* in the table below) does improves performance over  baseline approaches but consistently underperforms the full EQUILIBRIUM-RANKING method. We will include these results in the revised version of the paper that we will upload in the next few days.


| Domain        | Model | ER-G     | ER-D     | MI\*     |
| ------------- | ----- | -------- | -------- | -------- |
| MMLU          | 7B    | 39.4     | **39.9** | 35.8     |
|               | 13B   | 44.9     | **45.1** | 43.1     |
| ARC-Easy      | 7B    | **71.6** | 71.5     | 71.04    |
|               | 13B   | 76.1     | **76.4** | 76.1     |
| ARC-Challenge | 7B    | **58.7** | 58.3     | 58.5     |
|               | 13B   | 61.1     | **61.4** | **61.4** |
| RACE-Middle   | 7B    | 63.2     | **63.5** | 62.8     |
|               | 13B   | 67.9     | **68.6** | 67.5     |
| RACE-High     | 7B    | 56.3     | **56.4** | 55.9     |
|               | 13B   | 62.1     | **62.8** | 62.2     |
| HHH           | 7B    | 71.5     | **71.5** | 69.7     |
|               | 13B   | 61.1     | 61.1     | 61.1     |

Reviewer comment (Reproducibility):
I understand the piKL algorithm is from Jacobs et al., 2022, but I would still suggest the authors provide a slightly more detailed description in the appendix. The lack of related details makes it very hard to understand how important the hyper-parameters are, and if the method is sensitive to these values.

Author response:
**More detailed discussion of piKL in the appendix?**

Thank you for your suggestion, we will do that!
More context behind the choice and properties of piKL can be found in our response to Reviewer Jsc5.

Reviewer comment (Theory):
I'm trying to understand equations (1) and (2) on page 5, but I can't really understand how the weights terms (i.e., $\lambda_{G}$  and $\eta_Gt$ part) work. Should there be an additional $1/t$ before the $Q$? This will also make it more similar to the algorithm in Jacob et al., 2022. Otherwise, can you explain why there is a difference?

Author response:
**Question 1 (Equations (1) and (2))**

The difference between the way our algorithm is written, and the way piKL-Hedge (Algorithm 1 of Jacob et al.) is written, is just cosmetic and amounts to a change of variable. The policies the two algorithms produce are equivalent. 

In the original piKL paper, Algorithm 1 (piKL-Hedge) keeps track of cumulative values in the variable named $\textsf{CV}^{(t)}$ on line 11. Then, the update is 

$$ \pi^{(t+1)}(a) \propto \exp \{ \frac{\eta \textsf{CV}^{(t)}(a) + t \lambda \eta \log \tau(a)} {1 + t\lambda\eta} \}$$

(where for us the anchor policy $\tau$ is $\pi^{(1)}$, the initial policy of the language model.) In our paper, we use the same formula above, but divide the numerator and denominator of the fraction inside the exponentiation by the same quantity $\eta t$. This results in an update of the form
$$ \pi^{(t+1)}(a) \propto \exp \{ \frac {(\frac{1}{t}\textsf{CV}^{(t)}(a)) + \lambda \log \tau(a) } {1/(\eta t) + \lambda } \}$$

In fact, in our paper we keep track of the average values, which we call $Q$ (indeed, observe in the display equation at the very bottom of page 4, the cumulated values $\sum\frac{1}{2}\pi(\cdots)$ are divided by $t$ on the left of the sum). So, in summary, our updates are equivalent to those of piKL and can be written as
$$ \pi^{(t+1)}(a) \propto \exp \{ \frac {Q^{(t)}(a) + \lambda \log \tau(a) } {1/(\eta t) + \lambda } \}$$
which is what we have in the paper.

The reason why we performed this change is that the numerical range of the $Q$ variables we keep track of remains small, whereas the cumulated utilities $\textsf{CV}$ can grow linearly in absolute value.

Reviewer comment (Experiments):
Since this is a training-free method, I'm wondering if the authors have tried on even larger models.

Author response:
**Question 2 (What about larger models?)**

Thank you for the suggestion. Unfortunately, due to the size of the datasets and the number of tasks involved, we have not been able to evaluate our approach on models larger than LLaMA-13B. We leave this to future work!

Reviewer comment (Writing):
Typo: on page 7, Race-high -> RACE-high

Author response:
**Typo**

Thanks for the catch. We will fix this in our revision.

Reviewer comment (Theory):
The paper could benefit from a more detailed explanation on the methodological. The process by which Equations (1) and (2) facilitate the attainment of the game's equilibrium is not sufficiently illuminated.

Author response:
**Methodological notes behind the choice of Equations (1) and (2)**

We use Equations (1) and (2) to iteratively refine the policies of the players (generator and discriminator). Such policy-refinement procedures typically go under the name of "learning dynamics" in the theory of learning in games. Methogologically, the learning dynamics we use in our paper were designed to achieve the following desiderata:
1. The update rules in Equation (1) and (2) are very easy to implement in practice, in that they only require elementary operations such as exponentiation and normalization. This enables us to efficiently implement the refinement operations in a framework such as NumPy or PyTorch in only a couple of lines of code.
2. The update rules guarantee strong _no-regret_ properties. Regret is the standard optimality metric for online learning, and intuitively tracks the discrepancy between realized utility and the best utility that could have been realized in hindsight. 
3. The update rules ensure that the refined policies do not get arbitrarily far from the initial policies that are produced by the language model. As explained in the paper, this is important to guarantee _reasonableness_.
4. By virtue of their no-regret properties, the update rules retain strong connections with game-theoretic notions of optimality. 

From a technical point of view, Equations (1) and (2) are a form of mirror descent, a generalization of gradient descent that accomodates more specialized geometries, such as the natural geometry of conditional probability distributions, such as in the case of this paper.

Reviewer comment (Novelty):
The reliance on the piKL algorithm for the learning dynamics of the consensus game may limit the methodological novelty of the paper.

Author response:
**On the reliance on piKL**

We disagree that reliance on piKL limits the methodological novelty of the paper, for two reasons:
1. The use of game-theoretic learning dynamics for language generation is novel. For historical context, prior uses of piKL were in the direction of providing interpolations between imitation learning (aka. behavioral cloning) and pure self-play learning in the context of multi-agent reinforcement learning. However, we realized that the strong mathematical properties of piKL dynamics could be used to help guide production of language, which led to this paper. (See also above for considerations that help explain the speicifc choice of using piKL dynamics.)
2. From a technical point of view, piKL dynamics were defined in the context of normal-form games (that is, simultaneous-action nonsequential games, such as rock-paper-scissors). Instead, we instantiate piKL as part of a sequential potential (coordination) game.

Reviewer comment (Theory):
The pursuit of a no-regret dynamic steers the system towards a coarse correlated equilibrium. The paper could be enriched by an exploration into the feasibility and potential advantages of converging to stronger equilibrium constructs, such as pure Nash or mixed Nash equilibria.

Author response:
**Why search for coarse correlated equilibria?**

The existing analysis of piKL-hedge provides a guarantee of convergence to a coarse correlated equilibrium. However, as we hinted at the very end of Section 2.2, we  believe that in the case at hand most no-regret dynamics (including our own) will actually converge to Nash equilibrium. This is because the consensus game is a *potential game*, and in potential games dynamics based on smooth regularization are known to lead to Nash equilibria. Unfortunately, piKL is not, strictly speaking, an instance of such a dynamics, due to the fact that entropic regularization does not guarantee Lipschitz-continuous gradients. (Very small perturbations of piKL dynamics could fix the issue, but introducing a negligible amount of $\epsilon$-exploration). In practice, we observe excellent convergence properties, and leave the question of ironing out these technical subtleties for future work.

We will add this discussion in the next version of the paper that we will upload in the next few days.

Reviewer comment (Evaluation):
A notable performance drop is observed on the HHH dataset when transitioning from the LLaMA-7B to LLaMA-13B backbone model. An elaboration on this counterintuitive outcome would be beneficial.

Author response:
**Drop in performance on the HHH dataset**

We first note that on this benchmark, other hyperparameter settings of EQUILIBRIUM-RANKING for LLaMA-13B results in the performance that is close to LLaMA-7B. The drop in performance for the larger model could be explained by the fact that HHH dataset tests the alignment of instruction-tuned chatbots. The LLaMA models that we use are not instruction-tuned which makes them poor at following prompts of the form listed in Appendix B. Furthermore, inverse-scaling (where, performance of larger models degrade) has been observed in similar tasks such as TruthfulQA [1].

[1] Lin, Stephanie, Jacob Hilton, and Owain Evans. "TruthfulQA: Measuring How Models Mimic Human Falsehoods." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

Reviewer comment (Writing):
Section 2.2 was far too short. Readers without any Game Theory background might not even know what regret means. A better intuition should be built, possibly requiring 2–3 more paragraphs. Some of this could be drawn from the experimental section, where although there is a lot of useful material, some of it could be moved to the appendix to make room.

Author response:
Thank you for your suggestion, we will improve section 2.2 to include a more detailed introduction in our revision.

Reviewer comment (Writing):
The term 'reasonableness' is a bit of a misleading word for the concept being captured. Maybe 'alignment' could be considered, but that has other implications in the LM space.

Author response:
"Reasonableness": We acknowledge this concern and are open to suggestions! As you pointed out, "alignment" does have usage in this space that could make it confusing.

Reviewer comment (Writing):
For Equations 1/2, it would be helpful to clarify that this update does not result in a backpropagation to the parameters of the LM. In current RLHF literature, such backpropagation generally happens, so it is easy for readers to get confused.

Author response:
We agree and will note this point in our revision.

Reviewer comment (Presentation):
It is a bit confusing to put ER-G and ER-D in the section on "Baselines."

Author response:
You are right. This subsection should be called "decoding methods".

Reviewer comment (Novelty):
The characterization of SC as significantly related to Contrastive Decoding is questionable. The idea of using a weaker LM as the contrastive seems significant as it enables the strong LM to avoid the weaker one's mistakes, which for reasoning problems are often systematic reasoning errors. It is unfortunate that CD was not compared against, though the paper with strong results for CD on reasoning problems may be quite recent.

Author response:
We agree our CD baseline is sufficiently different from other work that we should clarify in our revision. We were unable to compare against CD for reasoning [1] as it was concurrent with our work. We do note that we *can* provide a direct comparison to published CD numbers on ARC-E and ARC-C. On ARC-E, LLaMA-13B + ER-D (76.4) is competitive with the (**much larger**) LLaMA-65B + CD ($\beta=1.0$) (76.9). On ARC-C, LLaMA-13B + ER-D (61.4) outperforms LLaMA-65B + CD ($\beta=1.0$) (59.7).

Reviewer comment (Novelty):
The biggest challenge to the potential significance of this work is that it does not seem readily applicable for all use cases of LMs. It requires a fact-checking problem with a set of candidate answers, which means it cannot be used out of the box as a general decoder — something that would be highly valuable. Nevertheless, the set of applications it is relevant for is important and it is likely to find application given its ease of use.

Author response:
Our approach does require using LMs as "sequence-level" discriminators that evaluate complete generations. That said, we are still able to apply it to standard generation tasks by formulating them as generate-and-rerank problems (like we do in GSM8K and TruthfulQA) - that is, we first sample a set of candidates from the language model using top-p and top-k sampling then use EQUILIBRIUM-RANKING on top. It can thus already be applied to open-ended generation problems. We think that recent work in partial reward models [2, 3] is a promising direction for future work that could apply it directly to generation rather than discriminative reranking.

Reviewer comment (Evaluation):
There may be a subtle form of bias that could creep in with use of this method. The model is “trained” with both correct and incorrect answers at the root node, which is good, but over time model selection is likely to be performed on some dev set using only accuracy as the metric. Since generally only the correct branches matter for accuracy, could this be a possible problem?

Author response:
In the gameplay itself, the "correctness parameter" is always uniformly sampled from between "correct" and "incorrect". However, we haven't experimented with fitting this prior on a dev set, which as you pointed out could result in bias of the form you described.

Reviewer comment (Experiments):
Both ER-G and ER-D are quite competitive and there is no clear winner between the two. Is there some simple combination strategy you recommend, such as ensembling or averaging?

Author response:
We do not have strong intuition here but we think that additive or multiplicative ensembling are both great options for combining these two approaches!