Query segmentation is the process of taking a user 's search-engine query and dividing the tokens into individual phrases or semantic units . Identification of these query segments can potentially improve both document-retrieval precision , by first returning pages which contain the exact query segments , and document-retrieval recall , by allowing query expansion or substitution via the segmented units . We train and evaluate a machine-learned query segmentation system that achieves 86 % segmentationdecision accuracy on a gold standard set of segmented noun phrase queries , well above recently published approaches . Key enablers of this high performance are features derived from previous natural language processing work in noun compound bracketing . For example , token association features beyond simple N-gram counts provide powerful indicators of segmentation . Billions of times every day , people around the world communicate with Internet search engines via a small text box on a web page . The user provides a sequence of words to the search engine , and the search engine interprets the query and tries to return web pages that not only contain the query tokens , but that are also somehow about the topic or idea that the query terms describe . Recent years have seen a widespread recognition that the user is indeed providing natural language text to the search engine ; query tokens are not independent , unordered symbols to be matched on a web document but rather ordered words and phrases with syntactic relationships . For example , Zhai ( 1997 ) pointed out that indexing on single-word symbols is not able to distinguish a search for " bank terminology " from one for " terminology bank . " The reader can submit these queries to a current search engine to confirm that modern indexing does recognize the effect of token order on query meaning in some way . Accurately interpreting query semantics also depends on establishing relationships between the query tokens . For example , consider the query " two man power saw . " There are a number of possible interpretations of this query , and these can be expressed through a number of different segmentations or bracketings of the query terms : One simple way to make use of these interpretations in search would be to put quotation marks around the phrasal segments to require the search engine to only find pages with exact phrase matches . If , as seems likely , the searcher is seeking pages about the large , mechanically-powered two-man saws used by lumberjacks and sawyers to cut big trees , then the first segmentation is correct . Indeed , a phrasal search for " two man power saw " on Google does find the device of interest . So does the second interpretation , but along with other , less-relevant pages discussing competitions involving " two-man handsaw , two-woman handsaw , power saw log bucking , etc . " The top document returned for the third interpretation , meanwhile , describes a man on a rampage at a subway station with two cordless power saws , while the fourth interpretation finds pages about topics ranging from hockey 's thrilling two-man power play advantage to the man power situation during the Second World War . Clearly , choosing the right segmentation means finding the right documents faster . Query segmentation can also help if insufficient pages are returned for the original query . A technique such as query substitution or expansion ( Jones et al . , 2006 ) can be employed using the segmented units . For example , we could replace the sexist " two man " modifier with the politically-correct " two person " phrase in order to find additional relevant documents . Without segmentation , expanding via the individual words " two , " " man , " " power , " or " saw " could produce less sensible results . In this paper , we propose a data-driven , machinelearned approach to query segmentation . Similar to previous segmentation approaches described in Section 2 , we make a decision to segment or not to segment between each pair of tokens in the query . Unlike previous work , we view this as a classification task where the decision parameters are learned discriminatively from gold standard data . In Section 3 , we describe our approach and the features we use . Section 4 describes our labelled data , as well as the specific tools used for our experiments . Section 5 provides the results of our evaluation , and shows the strong gains in performance possible using a wide set of features within a discriminative framework . We have developed a novel approach to search query segmentation and evaluated this approach on actual user queries , reducing error by 56 % over a recent comparison approach . Gains in performance were made possible by both leveraging recent progress in feature engineering for noun compound bracketing , as well as using a flexible , discriminative incorporation of association information , beyond the decisionboundary tokens . We have created and made available a set of manually-segmented user queries , and thus provided a new testing platform for other researchers in this area . Our initial formulation of query segmentation as a structured learning problem , and our leveraging of association statistics beyond the decision boundary , also provides powerful tools for noun compound bracketing researchers to both move beyond three-word compounds and to adopt discriminative feature weighting techniques . The positive results achieved on this important application should encourage further inter-disciplinary collaboration between noun compound interpretation and information retrieval researchers . For example , analysing the semantics of multiword expressions may allow for more-focused query expansion ; knowing to expand " bank manager " to include pages describing a " manager of the bank , " but not doing the same for non-compositional phrases like " real estate " or " private investigator , " requires exactly the kind of techniques being developed in the noun compound interpretation community . Thus for query expansion , as for query segmentation , work in natural language processing has the potential to make a real and immediate impact on search-engine technology . The next step in this research is to directly investigate how query segmentation affects search performance . For such an evaluation , we would need to know , for each possible segmentation ( including no segmentation ) , the document retrieval performance . This could be the proportion of returned documents that are deemed to be relevant to the original query . Exactly such an evaluation was recently used by Kumaran and Allan ( 2007 ) for the related task of query contraction . Of course , a dataset with queries and retrieval scores may serve for more than evaluation ; it may provide the examples used by the learning module . That is , the parameters of the contraction or segmentation scoring function could be discriminatively set to optimize the retrieval of the training set queries . A unified framework for query contraction , segmentation , and expansion , all based on discriminatively optimizing retrieval performance , is a very appealing future research direction . In this framework , the size of the training sets would not be limited by human annotation resources , but by the number of queries for which retrieved-document relevance judgments are available . Generating more training examples would allow the use of more powerful , finer-grained lexical features for classification .

Challenge: Because users provide natural language texts to the search engine, choosing the right segmentations helps for faster retrieval, and when original queries return insufficient pages.
Approach: They create a manually-segmented user query dataset and propose a classification-based query segmentation system which uses features such as token associations beyond simple N-gram counts.
Outcome: The proposed system achieves 86% segmentation-decision accuracy on a set of segmented noun phrase queries which is 56% fewer errors than a comparison approach.