Recent developments in statistical modeling of various linguistic phenomena have shown that additional features give consistent performance improvements . Quite often , improvements are limited by the number of features a system is able to explore . This paper describes a novel progressive training algorithm that selects features from virtually unlimited feature spaces for conditional maximum entropy ( CME ) modeling . Experimental results in edit region identification demonstrate the benefits of the progressive feature selection ( PFS ) algorithm : the PFS algorithm maintains the same accuracy performance as previous CME feature selection algorithms ( e.g. , Zhou et al . , 2003 ) when the same feature spaces are used . When additional features and their combinations are used , the PFS gives 17.66 % relative improvement over the previously reported best result in edit region identification on Switchboard corpus ( Kahn et al . , 2005 ) , which leads to a 20 % relative error reduction in parsing the Switchboard corpus when gold edits are used as the upper bound . Conditional Maximum Entropy ( CME ) modeling has received a great amount of attention within natural language processing community for the past decade ( e.g. , Berger et al . , 1996 ; Reynar and Ratnaparkhi , 1997 ; Koeling , 2000 ; Malouf , 2002 ; Zhou et al . , 2003 ; Riezler and Vasserman , 2004 ) . One of the main advantages of CME modeling is the ability to incorporate a variety of features in a uniform framework with a sound mathematical foundation . Recent improvements on the original incremental feature selection ( IFS ) algorithm , such as Malouf ( 2002 ) and Zhou et al . ( 2003 ) , greatly speed up the feature selection process . However , like many other statistical modeling algorithms , such as boosting ( Schapire and Singer , 1999 ) and support vector machine ( Vapnik 1995 ) , the algorithm is limited by the size of the defined feature space . Past results show that larger feature spaces tend to give better results . However , finding a way to include an unlimited amount of features is still an open research problem . In this paper , we propose a novel progressive feature selection ( PFS ) algorithm that addresses the feature space size limitation . The algorithm is implemented on top of the Selective Gain Computation ( SGC ) algorithm ( Zhou et al . , 2003 ) , which offers fast training and high quality models . Theoretically , the new algorithm is able to explore an unlimited amount of features . Because of the improved capability of the CME algorithm , we are able to consider many new features and feature combinations during model construction . To demonstrate the effectiveness of our new algorithm , we conducted a number of experiments on the task of identifying edit regions , a practical task in spoken language processing . Based on the convention from Shriberg ( 1994 ) and Charniak and Johnson ( 2001 ) , a disfluent spoken utterance is divided into three parts : the reparandum , the part that is repaired ; the inter-regnum , which can be filler words or empty ; and the repair / repeat , the part that replaces or repeats the reparandum . The first two parts combined are called an edit or edit region . An example is shown below : interregnum It is , you know , this is a tough problem . This paper presents our progressive feature selection algorithm that greatly extends the feature space for conditional maximum entropy modeling . The new algorithm is able to select features from feature space in the order of tens of millions in practice , i.e. , 8 times the maximal size previous algorithms are able to process , and unlimited space size in theory . Experiments on edit region identification task have shown that the increased feature space leads to 17.66 % relative improvement ( or 3.85 % absolute ) over the best result reported by Kahn et al . ( 2005 ) , and 10.65 % relative improvement ( or 2.14 % absolute ) over the new baseline SGC algorithm with all the variables from Zhang and Weng ( 2005 ) . We also show that symbolic prosody labels together with confidence scores are useful in edit region identification task . In addition , the improvements in the edit identification lead to a relative 20 % error reduction in parsing disfluent sentences when gold edits are used as the upper bound .

Challenge: Although it is known models with a larger number of features can achieve better performance, applicable feature space is limited with most models.
Approach: They propose a progressive feature selection algorithm that allows a conditional maximum entropy model to be efficiently trained with an unlimited number of features.
Outcome: The proposed training algorithm can improve over baseline models by increasing feature space on the edit region identification task.