Paper:
Confusions over Time: An Interpretable Bayesian Model to Characterize Trends in Decision Making
We propose Confusions over Time (CoT), a novel generative framework which facilitates a multi-granular analysis of the decision making process. The CoT not only models the confusions or error properties of individual decision makers and their evolution over time, but also allows us to obtain diagnostic insights into the collective decision making process in an interpretable manner. To this end, the CoT models the confusions of the decision makers and their evolution over time via time-dependent confusion matrices. Interpretable insights are obtained by grouping similar decision makers (and items being judged) into clusters and representing each such cluster with an appropriate prototype and identifying the most important features characterizing the cluster via a subspace feature indicator vector. Experimentation with real world data on bail decisions, asthma treatments, and insurance policy approval decisions demonstrates that CoT can accurately model and explain the confusions of decision makers and their evolution over time.

Review:
The authors propose a generative model for time-stamped K-categorical ratings specific to evaluator—item pairs.  The authors cover relevant background work in the crowdsourcing and human annotation literature in section 2.  In section 3, the authors describe the proposed CoT model which is based on clustering evaluators and items and associating a series of KxK confusion matrices with each (evaluator-cluster, item-cluster) pair.   Each series contains T confusion matrices, one for each discrete time step in the data, and a simple Markov chain is imposed over the series so that the matrices evolve smoothly through time.  In section 4, the authors describe a series of experiments where they compare the proposed model to ablations of the model and three existing baselines on prediction and estimation tasks using both real and synthetic data.
The authors motivate the proposed model with the setting in which items have “true” but unobserved labels/ratings and the observed labels/ratings given by evaluators are potentially incorrect.  This differs from the very common problem in recommendation systems or collaborative filtering where evaluators provide their subjective ratings but there is not assumed to be any “true” rating (e.g., users of Netflix giving 1-5 star ratings to movies). This seems like a common but underexplored setting that is worthy of further study within machine learning.  The authors are also right to highlight interpretability as a desired aspect of any machine learning solution that may yield post-hoc insights into common human biases and thus suggest corrective measures.  This paper does a good job of motivating the proposed model and situating it within the crowdsourcing and human annotation literature.  However the paper does not provide any coverage of two highly related areas of machine learning research.  The proposed generative model is a stochastic block model (SBM) [1, 2].  There is a vast literature on SBMs that includes many dynamic variants.  I think this paper must at least mention this literature to show that the proposed model is novel.  Furthermore, the setting of evaluator—item categorical ratings is very familiar to machine learning within the areas of collaborative filtering, recommendation systems, and matrix factorization yet the paper does not mention any of these areas.  Much of the writing is clear but the description of the model is not easy to read owing to some mistakes in the mathematical definitions and overly burdensome notation.   The binomial distribution defined on line 128 and throughout should be the Bernoulli distribution.  Similarly the multinomial distribution defined on line 138 and throughout should be the categorical distribution.  The uniform distribution has support on the real numbers and thus the uniform distribution defined on line 124 and throughout should be a categorical distribution.  Is V_{m,e} in equation 1 ever defined?  The description of the experiments is vague.  It is unclear where and when the true labels are used during training.  On line 211, it’s stated that the true labels are only used for evaluation but on line 228 it’s said that 5% are used during training.  It’s also not said how the Gibbs sampler is used to generate predictions.  Is the last sample taken as a point estimate of the latent parameters? Or are multiple samples being used?  It’s stated that the Gibbs sampler can be parallelized but not shown how.  Also, how exactly is BIC used to set the number of clusters? With cross-validation?  The experiments section in general needs to be more precise in order for the results to be understood and interpreted.   [1] Nowicki, K. and Snijders, T. A. B. Estimation and prediction for stochastic blockstructures. Journal of the American Statistical Association, 96(455):1077–1087, 2001.  [2] Kemp, C., Tenenbaum, J. B., Griffiths, T. L., Yamada, T., and Ueda, N. Learning systems of concepts with an infinite relational model. In Proceedings of the Twenty-First National Conference on Artificial Intelligence, 2006.