Paper: Modeling Boundedly Rational Agents with Latent Inference Budgets (ICLR-2024)

Reviewer comment (Presentation):
**Clarity and Presentation**: The paper often comes across as convoluted, making it difficult for readers unfamiliar with the domain to grasp its core concepts. Clearer explanations, along with better-organized sections, would significantly improve comprehension.

Author response:
Thank you for your review!

**Clarity and Presentation:**

We will try to improve clarity of presentation in our revision, specifically as it relates to the plots. Are there specific aspects that you would like to see improved?

Reviewer comment (Novelty):
**Lack of Thorough Literature Review**: The paper does not delve deep into existing works, leaving readers unaware of the full landscape of related research. A dedicated literature review, even if placed in the appendix, would help contextualize the presented work better.

Author response:
**Lack of Thorough Literature Review:**

We discuss related work in section 2 as well as several others in sections 4-6. Are there papers that you think are missing or warrant further discussion?

Reviewer comment (Experiments):
**Quality of Experiments**: The experimental section lacks rigorous validation on diverse datasets or scenarios, or providing substantial comparisons with more state-of-the-art existing models or methodologies, potentially limiting the generalizability of the proposed method.

Author response:
**Quality of Experiments:**

We have included experiments in 3 very different domains as well as relevant state-of-the-art approaches for each of the domains. Are there specific comparisons with other models of bounded rationality that you think are missing?

Reviewer comment (Theory):
**The Modeling of $\beta$**: The paper models $\beta$ in a manner that appears static across the trajectory. However, in realistic scenarios, it would be logical to assume that the distribution of $\beta$ is adaptive (either in a Bayesian or frequentist manner) and may change at every time step, conditional on histories. This richer representation can offer a more meaningful interpretation of agent behaviors.

Author response:
**Modeling of $\beta$:**

This comment is not quite right: our approach does allow beta to vary per-step. However, we currently model these choices as independent; we do think that explicitly modeling state-dependence of beta is an interesting future direction.

Reviewer comment (Presentation):
There are a few ad hoc decisions buried in the middle, which make the story less clear.

Author response:
**Improving clarity:**

Are there specific decisions that you would like us to elaborate on? We will work towards improving clarity of our presentation in our revision.

Reviewer comment (Theory):
"consider again the trajectories depicted in Fig. 1(b–c), which differ only in the difficulty of the search problem, and not in the cost of the optimal trajectory at all." There are some pretty big assumptions hidden in here.

Author response:
**Figure 1:**

Fig. 1 was meant to be conceptual but we will improve the clarity of our assumptions in the next revision.

Reviewer comment (Presentation):
I am not a fan of the visualization in figure 6. It is quite hard to parse.

Author response:
**Figure 6:**

We have improved Figure 6 in our current revision. Please let us know if it is better.

Reviewer comment (Experiments):
I would really love to see some additional comparisons. Other than that, I find the paper to be a nice contribution.

Author response:
1) Yes, there are definitely methods that have been developed for specific applications. The paper you cited is interesting and relevant, but not directly applicable to the tasks we're studying. It assumes that rewards have fixed, meaningful scales known and communicated to human players, then studies deviation from optimal policy under those rewards. In our work, because we are using a 0-1 reward (in RSA) and a learned reward model (in chess), it will not make any predictions that are different from the base Boltzmann model. But, we do think extending our approach to accommodate CPT-type models would be an interesting direction for future work! We will add a discussion relating to this work in our final revision.

Reviewer comment (Evaluation):
The accuracy differences in the RSA and chess tasks are fairly marginal, and seem to indicate that it is difficult to jointly infer both latent budgets and intent in more complicated tasks.

Author response:
**Accuracy differences:**

1. Although the performance improvements are small in an absolute sense, in chess (which is sequential in nature), these improvements at a per-move level can quickly cascade at a trajectory level. 
2. In scientific applications, we often care about inferring skill parameters for their own sake, and not just to improve predictive accuracy. In this sense, we view it as a strength of L-IBM that it can infer these population-level parameter differences from only small differences in behavior. Indeed, in both language understanding & chess, we're not aware of past work that can automatically derive these differences from behavior alone; the ability to do so has direct applications in cognitive science (e.g., [1]) and computational linguistics (e.g., [2]).

[1] van Opheusden, Bas, et al. "Expertise increases planning depth in human gameplay." Nature (2023): 1-6.

[2] Franke, Michael, and Judith Degen. "Reasoning in reference games: Individual-vs. population-level probabilistic modeling." PloS one 11.5 (2016): e0154854.

Reviewer comment (Theory):
The modeling of the latent budget requires fairly strong assumptions about the underlying reasoning mechanism of the agent. In addition to potential misalignment of assumptions, this also leads to situations like Sec. 5.3 where a constant of proprotionality is approximated over the set of all natural strings which are potential additional sources of error.

Author response:
**Modeling Assumption:**

1. The key underlying assumption is that the inference procedure is an anytime algorithm. While not allowing arbitrary inference procedures, this still accommodates a wide variety of algorithms that are used in practice. Most importantly, even with these assumptions, our method successfully explains real behavioral data from humans.
3. The challenge in computing the normalizing constant in Sec. 5.3 is a property of all rational speech acts models, and not inherent to to latent budget modeling. As reference games admit an arbitrarily large natural language utterance set, it is indeed the case that we have to include some form of approximation. We use the same approximation that past work has shown to be effective. [1, 2]

[1] McDowell, Bill, and Noah Goodman. "Learning from omission." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

[2] Monroe, Will, et al. "Colors in context: A pragmatic neural model for grounded language understanding." Transactions of the Association for Computational Linguistics 5 (2017): 325-338.

Reviewer comment (Evaluation):
What exactly does beta_temp intuitively represent in the RSA task? This is not entirely clear to me, nor why the figures in Table 3a/b seem to show such uninformative results. Do you have any insight as to why the inferred beta_depth seems to be more informative than beta_temp (in the sense that it seems more correlated with player skill)?

Author response:
**$\beta_\mathrm{temp}$ and $\beta_\mathrm{runtime}$ in RSA:**

Intuitively, $\beta_\mathrm{temp}$ can be seen as a greediness parameter. A low $\beta_\mathrm{temp}$ indicates that the speaker is optimally "best-responding" to the listener and a high $\beta_\mathrm{temp}$ indicates that the speaker is choosing actions from a uniform distribution. Empirically, human language users are somewhere between the two extremes, and past work has found that tuning $\beta_\mathrm{temp}$ is necessary to get a good fit to human data [1]. Despite this, we find here that *individual human differences* in language production are not explained by *differences in $\beta_\mathrm{temp}$*, but they are explained by an L-IBM (inferring $\beta_\mathrm{runtime}$) that models individual participants as literal or pragmatic listeners.

[1] Frank, Michael C., and Noah D. Goodman. "Predicting pragmatic reasoning in language games." Science 336.6084 (2012): 998-998.

Reviewer comment (Evaluation):
What is the relationship between beta_puct and ELO rating in Sec. 6? For time control it seems reasonable that longer time budgets would enable more exploration, but it seems there would be a trade-off between exploration and exploitation with respect to ELO rating. So it's not clear to me what this relationship is expected to be.

Author response:
**Relationship between $\beta_\mathrm{puct}$ and ELO rating:**

$\beta_\mathrm{puct}$ (as used in popular strength-modeling approaches like [1]) doesn’t simply control exploration vs. exploitation. Instead, it biases exploration towards a specific policy prior. As $\beta_\mathrm{puct}$ tends to infinity, it is identical to playing the prior policy (effectively reducing to imitation learning). When $\beta_\mathrm{puct}$ is set to 0, it is equivalent to greedily picking the search Q-values. As it relates to ELO rating, what we are noticing here is that stronger players start deviating more from their base policy to instead depend more on their MCTS search Q-values. This is therefore indicating that stronger players rely *more* on search compared to weaker players.

We have added this discussions to our revision!

[1] Jacob, Athul Paul, et al. "Modeling strong and human-like gameplay with KL-regularized search." International Conference on Machine Learning. PMLR, 2022.