Paper:
A Framework for Testing Identifiability of Bayesian Models of Perception
Bayesian observer models are very effective in describing human performance in perceptual tasks, so much so that they are trusted to faithfully recover hidden mental representations of priors, likelihoods, or loss functions from the data. However, the intrinsic degeneracy of the Bayesian framework, as multiple combinations of elements can yield empirically indistinguishable results, prompts the question of model identifiability. We propose a novel framework for a systematic testing of the identifiability of a significant class of Bayesian observer models, with practical applications for improving experimental design. We examine the theoretical identifiability of the inferred internal representations in two case studies. First, we show which experimental designs work better to remove the underlying degeneracy in a time interval estimation task. Second, we find that the reconstructed representations in a speed perception task under a slow-speed prior are fairly robust.

Review:
In this article, the authors propose a framework for performing model comparison of Bayesian models on behavioral data. To do so, they summarize the Bayesian Decision Theory framework, pinpoint areas of non-identifiability, and outline the types of constraints that can be used to make each term in the Bayesian framework identifiable. They then make assumptions to constrain each term in the Bayesian framework, explore how differentiable parameter values are in their model, and apply the technique to two studies that use Bayesian decision theory to explain behavioral responses: time interval estimation and motion perception. Issues of identifiability of internal representations and processes have been prominent issues within cognitive science and psychology for decades. For example, Anderson (1978) analyzed when representational formats can mimic each other, proving that they are indistinguishable without some constraints on processes (whenever the representations make equivalent distinctions of their inputs).In recent years, it is one of the major drawbacks of the Bayesian framework that has hindered its widespread adoption (e.g., Jones & Love, 2011; Bowers & Davis, 2012 – the latter directly addresses speed perception as an example). Although the article touches on an issue of great theoretical importance, is well written, and generally does so in a reasonable and effective manner, I have two major reservations with it. First, it ignores the large previous literature on model comparison within cognitive science and psychology, which makes it difficult to evaluate its contribution appropriately. The general framework for model comparison proposed by the authors is not so different from some of this previous work (e.g., Pitt, Myung, & Zhang, 2002; Navarro, Pitt, & Myung, 2004). Second, the authors make assumptions that seem arbitrary at times and it is unclear how much of their framework depends on them. For example, why should $p_{meas}$ be defined by equation 3? Is defining the values of the parameters in Equation 12 really enough to define and capture all the characteristics of a cognitive or perceptual process? I understand that it is robust and can express a large class of sensory models. However, it is not clear that is a sufficient justification for the assumption and at the very least, the article should discuss and justify the assumptions used to constrain the Bayesian models so that they are identifiable. Minor comments:Line 53: “It is a trivial mathematical fact…” It might be more helpful to explain why it is true rather than declare it trivial. It might not be trivial for all readers.Line 73: I believe the second “nor” should be an “or”.Figure 2: What do the star and dashed lines denote? From the caption, it sounds like they both denote the true value. ReferencesAnderson, J. R. (1978). Arguments concerning representations for mental imagery. Psychological Review, 85, 249-277.Bowers, J. S., & Davis, C. J. (2012). Bayesian Just-So Stories in Psychology and Neuroscience. Psychological Bulletin, 138(3), 389-414.Jones, M., & Love, B. C. (2011). Bayesian fundamentalism or enlightenment? On the explanatory status and theoretical contributions of Bayesian models of cognition. Behavioral and Brain Sciences, 34, 169-188.Navarro, D. J., Pitt, M. A., & Myung, I. J. (2004). Assessing the distinguishability of models and the informativeness of data. Cognitive Psychology, 49, 47-84.Pitt, M. A., Myung, I. J., & Zhang, S. (2002). Toward a Method of Selecting Among Computational Models of Cognition. Psychological Review, 109(3), 472-491. (also see Myung, Pitt, Zhang, & Balasubramanian, 2000 NIPS 13).