Testing a decision-theoretic approach to the evaluation of information retrieval systems
With information overload a real problem, especially on the Internet, there has been much interest in developing effective and efficient information retrieval (IR) systems. The various information retrieval approaches will require accurate evaluation to justify the requisite substantial development and implementation investment. Recently, a comprehensive and integrated evaluation model has been proposed and illustrated. By analyzing the evaluation measures using the analytic hierarchy process (AHP), the model transforms IR evaluation into a multi-criteria decision making (MCDM) problem, which assesses both the IR outcome and the interactive IR process. This paper extends that research by refining the evaluation model and by testing the research question through mathematical testing and simulation. The tests confirm the need to include both process and outcome criteria in any IR evaluation and prove the superiority of the proposed decision-theoretic approach over the traditional evaluation methodologies that focus on the IR outcome alone.
evaluation information retrieval system analytic hierarchy process decision making information search process multi-criteria model