Paper:
Distributionally Robust Graphical Models
In many structured prediction problems, complex relationships between variables are compactly defined using graphical structures. The most prevalent graphical prediction methods-probabilistic graphical models and large margin methods-have their own distinct strengths but also possess significant drawbacks. Conditional random fields (CRFs) are Fisher consistent, but they do not permit integration of customized loss metrics into their learning process. Large-margin models, such as structured support vector machines (SSVMs), have the flexibility to incorporate customized loss metrics, but lack Fisher consistency guarantees. We present adversarial graphical models (AGM), a distributionally robust approach for constructing a predictor that performs robustly for a class of data distributions defined using a graphical structure. Our approach enjoys both the flexibility of incorporating customized loss metrics into its design as well as the statistical guarantee of Fisher consistency. We present exact learning and prediction algorithms for AGM with time complexity similar to existing graphical models and show the practical benefits of our approach with experiments.

Review:
Distributionally Robust Graphical Models  The authors suggest dealing with the structure prediction task using adversarial graphical model (AGM), a generative model trained using an adversary distribution, instead of the empirical one. Instead of focusing on the loss metric, the authors focus on the graphical model allowing more flexibility wrt the loss metric. The AGM algorithm has similar complexity to conditional random fields but is less limited in its loss metric and is Fisher consistent for additive loss metrics.   Following a complex mathematical transformation, the authors provide optimization of node and edge distribution of the graphical model but since this optimization is intractable, they restrict their method to tree-structural models or models with low treewidths. I would expect the authors to discuss this limitation of their algorithm.  I have several reservations regarding the experimental section for which the authors’ may wish to reply. First, only two prediction problems are evaluated; both with low treewhidth, a condition which suits the AGM more than its competitors and does not reflect most prediction problems. Second, different loss metrics are used for each of the problems. Third, while results for the first problem are reported as an average over 20 data splits, no details about the methodology of the second experiment are given. In addition, since no statistical significance was reported for the results of the latter experiment, I assume these results are reported on the entire data set. Fourth, instead of performing the parametric t-test to compare several algorithms on multiple (20) data sets, a non-parametric test (e.g., Friedman’s test with a post-hoc test) should have been exercised. Thus, whether the results in Table 1 are significant or not is due to a suitable test being exercised. Fifth, the AGM shows no advantage over its competitors for the two prediction tasks tested, and thus I would solicit the authors to discuss the method benefits, practicality, and generalizability to non-tree problems.  Major issues  Not all terms are defined (see e.g., Section 2.1 and 2.2)  The authors claim: “This two-stage prediction approach can create inefficiencies when learning from limited amounts of data since optimization may focus on accurately estimating probabilities in portions of the input space that have no impact on the decision boundaries of the Bayesian optimal action” – but this claim is controversial wrt the author selection in a generative model (i.e., graphical models) for prediction. The authors are asked to motivate their selection in the graphical model that required applying complex mathematical transformations in order to make it a prediction model.  Minor issues  56 – argmax vs argmin?  Include directed graphical models (Bayesian networks) in you your survey of probabilistic graphical models to complete it.  References: Capitalize “Bayesian”, conference/journal names, Arc; decide on a consistent use of author and journal names  Quality: good Clarity: average Originality: average Significance: poor  The authors response has raised my opinion about the experimental part of the paper and thus also about the paper itself.