Paper:
Cooperative Graphical Models
We study a rich family of distributions that capture variable interactions significantly more expressive than those representable with low-treewidth or pairwise graphical models, or log-supermodular models. We call these cooperative graphical models. Yet, this family retains structure, which we carefully exploit for efficient inference techniques. Our algorithms combine the polyhedral structure of submodular functions in new ways with variational inference methods to obtain both lower and upper bounds on the partition function. While our fully convex upper bound is minimized as an SDP or via tree-reweighted belief propagation, our lower bound is tightened via belief propagation or mean-field algorithms. The resulting algorithms are easy to implement and, as our experiments show, effectively obtain good bounds and marginals for synthetic and real-world examples.
Probabilistic inference in high-order discrete graphical models has been an ongoing computational challenge, and all existing methods rely on exploiting specific structure: either low-treewidth or pairwise graphical models, or functional properties of the distribution such as log-submodularity. Here, we aim to compute approximate marginal probabilities in complex models with long-range variable interactions that do not possess any of these properties. Instead, we exploit a combination of structural and functional properties in new ways.
The classical example of image segmentation may serve to motivate our family of models: we would like to estimate a posterior marginal distribution over k labels for each pixel in an image. A common approach uses Conditional Random Fields on a pixel neighborhood graph with pairwise potentials that encourage neighboring pixels to take on the same label. From the perspective of the graph, this model prefers configurations with few edges cut, where an edge is said to be cut if its endpoints have different labels. Such cut-based models, however, short-cut elongated structures (e.g. tree branches), a problem known as shrinking bias. Jegelka and Bilmes [1] hence replace the bias towards short cuts (boundaries) by a bias towards configurations with certain higher-order structure: the cut edges occur at similar-looking pixel pairs. They group the graph edges into clusters (based on, say, color gradients across the endpoints), observing that the true object boundary is captured by few of these clusters. To encourage cutting edges from few clusters, the cost of cutting an edge decreases as more edges in its cluster are cut. In short, the edges "cooperate". In Figure 1, each pixel takes on one of two labels (colors), and cut edges are indicated by dotted lines. The current configuration cuts three red edges and one blue edge, and has lower probability than the configuration that swaps X 3,1 to gray, cutting only red edges. Such a model can be implemented by an energy (cost) h(#red edges cut) + h(#blue edges cut), where e.g. h(u) = √ u. Similar cooperative models can express a preference for shapes [2].
While being expressive, such models are computationally very challenging: the nonlinear function on pairs of variables (edges) is equivalent to a graphical model of extremely high order (up to the number of variables). Previous work hence addressed only MAP inference [3,4]; the computation of marginals and partition functions was left as an open problem. In this paper, we close this gap, even for a larger family of models.
We address models, which we call cooperative graphical models, that are specified by an undirected graph G = (V, E): each node i ∈ V is associated with a random variable X i that takes values in X = {1, 2, . . . , k}. To each vertex i ∈ V and edge {i, j}, we attach a potential function θ i : X → R and θ i,j : X 2 → R, respectively. Our distribution is then
where we call y : X n → {0, 1} E the disagreement variable 1 , defined as y i,j = x i = x j . The term ν : X n → R ≥0 is the base-measure and allows to encode constraints, e.g., conditioning on some variables. With f ≡ 0 we obtain a Markov random field.
Probabilistic inference in our model class (1) is very challenging, since we make no factorization assumption about f . One solution would be to encode P (x) as a log-linear model via a new variable z ∈ {0, 1} E and constraints ν(x, z) = y(x) = z , but this in general requires computing exponential-sized sufficient statistics from z. In contrast, we make one additional key assumption that will enable the development of efficiently computable variational lower and upper bounds: we henceforth assume that f : {0, 1} E → R is submodular, i.e., it satisfies
where the min and max operations are taken element-wise. For example, the pairwise potentials θ i,j are submodular if θ i,j (0, 0) + θ i,j (1, 1) ≤ θ i,j (0, 1) + θ i,j (1, 0). In our introductory example, f is submodular if h is concave. As opposed to [3], we do not assume that f is monotone increasing. Importantly, even if f is submodular, P (x) neither has low treewdith, nor is its logarithm sub-or supermodular in x, properties that have commonly been exploited for inference.
Contributions. We make the following contributions: (1) We introduce a new family of probabilistic models that can capture rich non-submodular interactions, while still admitting efficient inference. This family includes pairwise and certain higher-order graphical models, cooperative cuts [1], and other, new models. We develop new inference methods for these models; in particular, (2) upper bounds that are amenable to convex optimization, and (3) lower bounds that we optimize with traditional variational methods. Finally, we demonstrate the efficacy of our methods empirically.

Review:
This paper provides variational lower/upper bound of partition of a special class of graphical models called `cooperative graphical model’. Cooperative graphical model is a probability distribution which is similar with pairwise graphical models but its energy function contains some extra submodular function of disagreement variable. The main technique for formulating variational lower/upper bound of cooperative graphical model is finding pairwise graphical model and its associate polytope which upper/lower bound the original cooperative graphical model. The author also provides algorithms for suggested bounds and experimental results for image segmentation.
I think the proposed graphical model is interesting and novel. The authors provide a lower bound and a fully convex upper bound for partition function. In particular, variational upper/lower bound are formulated by using pairwise graphical models, e.g. mean field approximation and SDP, and therefore, one can apply existing variational methods for pairwise graphical models for cooperative graphical models. The author also provides algorithms, Frank-Wolfe, PGD and BP, for finding upper/lower bound. The experimental results shows that cooperative graphical models and its variational methods performs better than pairwise graphical models for image segmentation.  Variational bounds provided in the paper are formulated by bounding the cooperative graphical model by pairwise models and use the existing variational bounds for pairwise models to bound cooperative graphical model. However, as the author applies two bounds consecutively, the suggested bound might be loose.  Lower bound is not tractable. Since the optimization over a polytope U(f) is not tractable, the variational lower bound is also intractable. The author suggests an algorithm based on the block coordinate descent which finds the lower bound for the suggested variational lower bound, however, the algorithm only explore the finite points of U(f) and it may not provide a tight lower bound.  The technique of constructing B(f) and U(f) seems generic. Can this technique applied to a wider class of f or other probability distribution? Furthermore, the optimization objectives (3), (4) for the variational upper/lower bound does not seems to be dependent with the submodularity of f (while the algorithms do).  Typos  - In page 2, line 59~60, for all x,y -> for all y,y^\prime - In page 5, procedure Linear-Oracle, `)’ is missing for procedure 4,5 - In page 6, line 211, U(F) -> U(f) - In page 5, line 179, smoth -> smooth