We study the problem of projecting a dis tribution onto (or finding a maximum likeli hood distribution among) Markov networks of bounded tree-width. By casting it as the combinatorial optimization problem of find ing a maximum weight hypertree, we prove that it is NP-hard to solve exactly and pro vide an approximation algorithm with a prov able performance guarantee. Estimating model structure from data remains a dif ficult problem. A typical approach involves heuristic search using some structure scoring metric, such as the Bayesian score. While this general problem is known to be hard [Chi96] (and the typical approach reflects this fact), hardness of many restricted learning sce narios is not well-understood, nor is the potential for guaranteed approximation algorithms. We focus here on a slightly simpler problem, in which regularization is attained solely by limiting the model to a restricted "concept" class and a maximum likelihood model is sought. Casting the associated estimation problem in purely combinatorial terms permits us to obtain both hardness results and provable guarantees of approxi mation accuracy. In 1968, Chow and Liu [CL68] provided a rigorous analysis for finding maximum likelihood Markov trees, casting it as a problem of finding a maximum-weight tree, and thus providing an efficient and exact algo rithm for it. We would like to generalize the work of Chow and Liu to the problem of learning a maximum likelihood Markov network of bounded complexity. In Section 2 we discuss how tree-width is in many ways the appropriate measure of complexity. Accordingly, we analyze the problem of learning a maximum likeli hood Markov network of bounded tree-width. In fact, we study a somewhat more general problem. The maximum likelihood distribution is a distribution minimizing the information divergence from the em pirical distribution. Finding a maximum likelihood distribution can thus be seen as a special case of the problem of projecting a target distribution onto a con cept class, i.e. finding the distribution from within the class that minimizes the information divergence from the target. Such projections have applications beyond finding the maximum likelihood distribution. In this paper, we use this framework and discuss the prob lem of projecting a target distribution onto the class of Markov networks of bounded tree-width. Similarly to the work of Chow and Liu, we are able to formalize the projection problem as a combina torial optimization problem on graphs. We show that projecting a distribution onto Markov networks of bounded tree-width is equivalent to finding a maximum-weight hypertree. This equivalence gives rise to global, integer programming-based approxima tion algorithms with provable performance guarantees. This contrasts with previously suggested local-search heuristics for the same problem [Mal91] . The equiva lence also allows us to study the computational hard ness of the learning problem. We show that learning a maximum likelihood Markov network of bounded tree width is NP-hard, even for tree-width two. Complementary to this presentation is a paper on ap proximation algorithms for the maximum-weight hy pertree problem [KS01], which details the algorithms mentioned here. The algorithms are motivated by the problems discussed here, and some results from Sec tion 3 are quoted in [KSOl].
As shown by Srebro REF , it is an NP-hard problem to find the optimal hypergraph.
Maximum Likelihood Bounded Tree-Width Markov Networks
ArXiv
ArXiv
Computer Science Mathematics