Many practitioners who use EM and related al gorithms complain that they are sometimes slow. When does this happen, and what can be done about it? In this paper, we study the general class of bound optimization algorithms-includ ing EM, Iterative Scaling, Non-negative Matrix Factorization, CCCP -and their relationship to direct optimization algorithms such as gradient based methods for parameter learning. We de rive a general relationship between the updates performed by bound optimization methods and those of gradient and second-order methods and identify analytic conditions under which bound optimization algorithms exhibit quasi-Newton behavior, and under which they possess poor, first-order convergence. Based on this analysis, we consider several specific algorithms, inter pret and analyze their convergence properties and provide some recipes for preprocessing input to these algorithms to yield faster convergence be havior. We report empirical results supporting our analysis and showing that simple data pre processing can result in dramatically improved performance of bound optimizers in practice.
Convergence properties of such general "bound optimization" algorithms have been discussed by REF .
On the Convergence of Bound Optimization Algorithms
ArXiv
ArXiv
Computer Science Mathematics