We approached the problem as learning how to order documents by estimated relevance with respect to a user query . Our support vector machines based classifier learns from the relevance judgments available with the standard test collections and generalizes to new , previously unseen queries . For this , we have designed a representation scheme , which is based on the discrete representation of the local ( lw ) and global ( gw ) weighting functions , thus is capable of reproducing and enhancing the properties of such popular ranking functions as tf.idf , BM25 or those based on language models . Our tests with the standard test collections have demonstrated the capability of our approach to achieve the performance of the best known scoring functions solely from the labeled examples and without taking advantage of knowing those functions or their important properties or parameters . Our work is motivated by the objective to bring closer numerous achievements in the domains of machine learning and classification to the classical task of ad-hoc information retrieval ( IR ) , which is ordering documents by the estimated degree of relevance to a given query . Although used with striking success for text categorization , classification-based approaches ( e.g. those based on support vector machines , Joachims , 2001 ) have been relatively abandoned when trying to improve ad hoc retrieval in favor of empirical ( e.g. vector space , Salton & McGill , 1983 ) or generative ( e.g. language models ; Zhai & Lafferty 2001 ; Song & Croft ; 1999 ) , which produce a ranking function that gives each document a score , rather than trying to learn a classifier that would help to discriminate between relevant and irrelevant documents and order them accordingly . A generative model needs to make assumptions that the query and document words are sampled from the same underlying distributions and that the distributions have certain forms , which entail specific smoothing techniques ( e.g. popular Dirichlet-prior ) . A discriminative ( classifier-based ) model , on the other side , does not need to make any assumptions about the forms of the underlying distributions or the criteria for the relevance but instead , learns to predict to which class a certain pattern ( document ) belongs to based on the labeled training examples . Thus , an important advantage of a discriminative approach for the information retrieval task , is its ability to explicitly utilize the relevance judgments existing with standard test collections in order to train the IR algorithms and possibly enhance retrieval accuracy for the new ( unseen ) queries . Cohen , Shapire and Singer ( 1999 ) noted the differences between ordering and classification and presented a two-stage model to learn ordering . The first stage learns a classifier for preference relations between objects using any suitable learning mechanism ( e.g. support vector machines ; Vapnik , 1998 ) . The second stage converts preference relations into a rank order . Although the conversion may be NP complete in a general case , they presented efficient approximations . We limited our first study reported here to linear classifiers , in which conversion can be performed by simple ordering according to the score of each document . However , approaching the problem as " learning how to order things " allowed us to design our sampling and training mechanisms in a novel and , we believe , more powerful way . Our classifier learns how to compare every pair of documents with respect to a given query , based on the relevance indicating features that the documents may have . As it is commonly done in information retrieval , the features are derived from the word overlap between the query and documents . According to Nallapati ( 2004 ) , the earliest formulation of the classic IR problem as a classification ( discrimination ) problem was suggested by Robertson and Sparck Jones ( 1976 ) , however performed well only when the relevance judgments were available for the same query but not generalizing well to new queries . Fuhr and Buckley ( 1991 ) used polynomial regression to estimate the coefficients in a linear scoring function combining such well-known features as a weighted term frequency , document length and query length . They tested their " description-oriented " approach on the standard small-scale collections ( Cranfield , NPL , INSPEC , CISI , CACM ) to achieve the relative change in the average precision ranging from -17 % to + 33 % depending on the collection tested and the implementation parameters . Gey ( 1994 ) applied logistic regression in a similar setting with the following results : Cranfield +12 % , CACM +7.9 % , CISI -4.4 % , however he did not test them on new ( unseen by the algorithm ) queries , hypothesizing that splitting documents into training and testing collections would not be possible since " a large number of queries is necessary in order to train for a decent logistic regression approach to document retrieval . " Instead , he applied a regression trained on Cranfield to CISI collection but with a negative effect . Recently , the approaches based on learning have reported several important breakthroughs . Fan et al . ( 2004 ) applied genetic programming in order to learn how to combine various terms into the optimal ranking function that outperformed the popular Okapi formula on robust retrieval test collection . Nallapati ( 2004 ) made a strong argument in favor of discriminative models and trained an SVM-based classifier to combine 6 different components ( terms ) from the popular ranking functions ( such as tf.idf and language models ) to achieve better than the language model performance in 2 out of 16 test cases ( figure 3 in Nallapati , 2004 ) , not statistically distinguishable in 8 cases and only 80 % of the best performance in 6 cases . There have been studies using past relevance judgements to optimize retrieval . For example , Joachims ( 2002 ) applied Support Vector Machines to learn linear ranking function from user click-throughs while interfacing with a search engine . In this study , we have developed a representation scheme , which is based on the discretization of the global ( corpus statistics ) and local ( document statistics ) weighting of term overlaps between queries and documents . We have empirically shown that this representation is flexible enough to learn the properties of the popular ranking functions : tf.idf , BM25 and the language models . The major difference of our work from Fan et al . ( 2004 ) or Nallapati ( 2004 ) or works on fusion ( e.g. Vogt & Cottrell , 1999 ) is that we did not try to combine several known ranking functions ( or their separate terms ) into one , but rather we learn the weighting functions directly through discretization . Discretization allows representing a continuous function by a set of values at certain points . These values are learned by a machine learning technique to optimize certain criteria , e.g. average precision . Another important motivation behind using discretization was to design a representation with high dimensionality of features in order to combine our representation scheme with Support Vector Machines ( SVM ) ( Vapnik , 1998 ) , which are known to work well with a large number of features . SVM contains a large class of neural nets , radial margin separation ( RBF ) nets , and polynomial classifiers as special cases . They have been delivering superior performance in classification tasks in general domains , e.g. in face recognition ( Hearst , 1998 ) , and in text categorization ( Joachims , 2001 ) . Another important distinction of this work from the prior research is that we train our classifier not to predict the absolute relevance of a document d with respect to a query q , but rather to predict which of the two documents d1 , d2 is more relevant to the query q. The motivation for this distinction was that all the popular evaluation metrics in information retrieval ( e.g. average precision ) are based on document ranking rather than classification accuracy . This affected our specially designed sampling procedure which we empirically discovered to be crucial for successful learning . We have also empirically established that our combination of the representation scheme , learning mechanism and sampling allows learning from the past relevance judgments in order to successfully generalize to the new ( unseen ) queries . When the representation was created without any knowledge of the top ranking functions and their parameters , our approach reached the known top performance solely through the learning process . When our representation was taking advantage of functions that are known to perform well and their parameters , the resulting combination was able to slightly exceed the top performance on large test collections . The next section formalizes our Discretization Based Learning ( DBL ) approach to Information Retrieval , followed by empirical results and conclusions . We explored learning how to rank documents with respect to a given query using linear Support Vector Machines and discretization-based representation . Our approach represents a family of discriminative approaches , currently studied much less than heuristic ( tf.idf , bm25 ) or generative approaches ( language models ) . Our experiments indicate that learning from relevant judgments available with the standard test collections and generalizing to new queries is not only feasible but can be a source of improvement . When tested with a popular standard collection , our approach achieved the performance of the best well-known techniques ( BM25 and language models ) , which have been developed as a result of extensive past experiments and elaborate theoretical modeling . When combined with the best performing ranking functions , our approach added a small ( 2 - 3 % ) , but statistically significant , improvement . Although practical significance of this study may be limited at the moment since it does not demonstrate a dramatic increase in retrieval performance in large test collections , we believe our findings have important theoretical contributions since they indicate that the power of discriminative approach is comparable to the best known analytical or heuristic apporaches . This work also lays the foundation for extending the discriminative approach to " richer " representations , such as those using word n-grams , grammatical relations between words , and the structure of documents . Our results also indicate that gw.lw family , which includes practically all the popular " bag of words " ranking formulas such as tf.idf , BM25 or language models , has almost reached its upper limit and other classes of representations and ranking formulas need to be explored in order to accomplish significant performance break-troughs . Of course , using only few test cases ( topics sets and collections ) is a limitation of this current study , which we are going to address in our future research . We view our approach as a complement , rather than competitive , to the analytical approaches such as language models . Our approach can be also used as an explorative tool in order to identify important relevance-indicating features , which can be later modeled analytically . We believe that our work and the ones referred in this paper may bring many of the achievements made in a more general area of classification and machine learning closer to the task of rank ordered information retrieval , thus making retrieval engines more helpful in reducing the information overload and meeting people 's needs .

Challenge: Even with the recent successes of classification models, their applications to information retrieval have been abandoned.
Approach: They propose to use a machine learning-based discriminative model to obtain richer features further can be used for information retrieval.
Outcome: They show that the proposed approach can reach the performance of well-studied heuristics such as TF-IDF and generalizes well to unseen queries.