Many statistical learning problems in NLP call for local model search methods . But accuracy tends to suffer with current techniques , which often explore either too narrowly or too broadly : hill-climbers can get stuck in local optima , whereas samplers may be inefficient . We propose to arrange individual local optimizers into organized networks . Our building blocks are operators of two types : ( i ) transform , which suggests new places to search , via non-random restarts from already-found local optima ; and ( ii ) join , which merges candidate solutions to find better optima . Experiments on grammar induction show that pursuing different transforms ( e.g. , discarding parts of a learned model or ignoring portions of training data ) results in improvements . Groups of locally-optimal solutions can be further perturbed jointly , by constructing mixtures . Using these tools , we designed several modular dependency grammar induction networks of increasing complexity . Our complete system achieves 48.6 % accuracy ( directed dependency macro-average over all 19 languages in the 2006/7 CoNLL data ) -more than 5 % higher than the previous state-of-the-art . Statistical methods for grammar induction often boil down to solving non-convex optimization problems . Early work attempted to locally maximize the likelihood of a corpus , using EM to estimate probabilities of dependency arcs between word bigrams ( Paskin 2001a ; 2001b ) . That parsing model has since been extended to make unsupervised learning more feasible ( Klein and Manning , 2004 ; Headden et al . , 2009 ; Spitkovsky et al . , 2012b ) . But even the latest techniques can be quite error-prone and sensitive to initialization , because of approximate , local search . In theory , global optima can be found by enumerating all parse forests that derive a corpus , though this is usually prohibitively expensive in practice . A preferable brute force approach is sampling , as in Markov-chain Monte Carlo ( MCMC ) and random restarts ( Hu et al . , 1994 ) , which hit exact solutions eventually . Restarts can be giant steps in a parameter space that undo all previous work . At the other extreme , MCMC may cling to a neighborhood , rejecting most proposed moves that would escape a local attractor . Sampling methods thus take unbounded time to solve a problem ( and ca n't certify optimality ) but are useful for finding approximate solutions to grammar induction ( Cohn et al . , 2011 ; Mareček and Žabokrtský , 2011 ; Naseem and Barzilay , 2011 ) . We propose an alternative ( deterministic ) search heuristic that combines local optimization via EM with non-random restarts . Its new starting places are informed by previously found solutions , unlike conventional restarts , but may not resemble their predecessors , unlike typical MCMC moves . We show that one good way to construct such steps in a parameter space is by forgetting some aspects of a learned model . Another is by merging promising solutions , since even simple interpolation ( Jelinek and Mercer , 1980 ) of local optima may be superior to all of the originals . Informed restarts can make it possible to explore a combinatorial search space more rapidly and thoroughly than with traditional methods alone . We proposed several simple algorithms for combining grammars and showed their usefulness in merging the outputs of iterative and static grammar induction systems . Unlike conventional system combination methods , e.g. , in machine translation ( Xiao et al . , 2010 ) , ours do not require incoming models to be of similar quality to make improvements . We exploited these properties of the combiners to reconcile grammars induced by different views of data ( Blum and Mitchell , 1998 ) . One such view retains just the simple sentences , making it easier to recognize root words . Another splits text into many inter-punctuation fragments , helping learn word associations . The induced dependency trees can themselves also be viewed not only as directed structures but also as skeleton parses , facilitating the recovery of correct polarities for unlabeled dependency arcs . By reusing templates , as in dynamic Bayesian network ( DBN ) frameworks ( Koller and Friedman , 11 The so-called Yarowsky-cautious modification of the original algorithm for unsupervised word-sense disambiguation . 2009 , § 6.2.2 ) , we managed to specify relatively " deep " learning architectures without sacrificing ( too much ) clarity or simplicity . On a still more speculative note , we see two ( admittedly , tenuous ) connections to human cognition . First , the benefits of not normalizing probabilities , when symmetrizing , might be related to human language processing through the base-rate fallacy ( Bar-Hillel , 1980 ; Kahneman and Tversky , 1982 ) and the availability heuristic ( Chapman , 1967 ; Tversky and Kahneman , 1973 ) , since people are notoriously bad at probability ( Attneave , 1953 ; Kahneman and Tversky , 1972 ; Kahneman and Tversky , 1973 ) . And second , intermittent " unlearning " -though perhaps not of the kind that takes place inside of our transformsis an adaptation that can be essential to cognitive development in general , as evidenced by neuronal pruning in mammals ( Craik and Bialystok , 2006 ; Low and Cheng , 2006 ) . " Forgetful EM " strategies that reset subsets of parameters may thus , possibly , be no less relevant to unsupervised learning than is " partial EM , " which only suppresses updates , other EM variants ( Neal and Hinton , 1999 ) , or " dropout training " ( Hinton et al . , 2012 ; Wang and Manning , 2013 ) , which is important in supervised settings . Future parsing models , in grammar induction , may benefit by modeling head-dependent relations separately from direction . As frequently employed in tasks like semantic role labeling ( Carreras and Màrquez , 2005 ) and relation extraction ( Sun et al . , 2011 ) , it may be easier to first establish existence , before trying to understand its nature . Other key next steps may include exploring more intelligent ways of combining systems ( Surdeanu and Manning , 2010 ; Petrov , 2010 ) and automating the operator discovery process . Furthermore , we are optimistic that both count transforms and model recombination could be usefully incorporated into sampling methods : although symmetrized models may have higher cross-entropies , hence prone to rejection in vanilla MCMC , they could work well as seeds in multi-chain designs ; existing algorithms , such as MCMCMC ( Geyer , 1991 ) , which switch contents of adjacent chains running at different temperatures , may also benefit from introducing the option to combine solutions , in addition to just swapping them .

Challenge: Existing local model search methods tend to explore too narrowly or broadly because they are error-prone and initialization sensitive, and finding global optima is expensive.
Approach: They propose a search heuristic that combines local optimization via EM with non-random restarts which may not resemble their predecessors by using previously found solutions.
Outcome: The proposed methods show improvements in grammar induction experiments and a newly designed system outperforms the previous state-of-the-art by more than 5%.