trec_eval is the standard tool used by the TREC community for evaluating an ad hoc retrieval run, given the results file and a standard set of judged results. ------------------------------------------------------------------------------ Installation: Should be as easy as typing "make" in the source directory. If you wish the trec_eval binary to be placed in a standard location, alter the first line of Makefile appropriately. ------------------------------------------------------------------------------ Testing: sample input and output files are included in the directory test. "make quicktest" will perform some sample simple evaluations and compare the results. ------------------------------------------------------------------------------ Usage: Most options can be ignored. The only one most folks will need is the "-q" flag, to indicate whether to output official results for individual queries as well as the averages over all queries. Official TREC usage might be something like trec_eval -q -c -M1000 official_qrels submitted_results to ensure correct evaluation if submitted_results doesn't have results for all queries, or returns more than 1000 documents per query. If you wish to output only one particular measure: trec_eval -m measure[.params] rel_info_file results_file ------------------------------------------------------------------------------ Change Log (only recent) ------------------------------------------------------------------------------ 12/31/08 comments and documentation of Zscore file format corrected. trec_eval.c get_zscores.c 2/25/08 Version 9.0alpha. Complete rewrite of entire trec_eval (needed for a long time!). Complete separation of individual measure calculations - computers are now fast enough so can afford recalculation of lots of intermediate values. Should be much easier to add measures to, and much easier to add new input file formats with associated measures. Parameters for measures (eg, cutoffs for P) can be specified on the command line. Choice of measures can be specifed on the command line. An initial set of preference evaluation measures (with their own input rel_info format) have been added. Help now gives targeted measure and format descriptions. Try trec_eval -h -m all_prefs -R prefs to get info on preference measures and formats, for instance. All internal calculations are in double rather than float. Yields minor variations in output at rare times; mostly when going from a double percentage to a corresponding doc cutoff (eg, in iprec_at_recall). All globally known procedure names or variables now begin with 'te_' to allow incorporation of procedures in other programs. Measures added: ndcg, ndcg_cut, set_F, success, map_avgjg, P_avgjg, various preference evaluation measures. Measures renamed: Rprec-mult_* was *R-prec set_P was exact_prec set_recall was exact_recall set_relative_P was exact_relative_prec set_recall was exact_recall set_map was exact_unranked_avg_prec gm_map was gm_ap 11pt_avg was 11-pt_avg P_* was P* recall_* was recall* relative_P_* was relative_P* iprec_at_recall_* was ircl_prn.* Measures dropped for now: 3-pt_avg avg_doc_prec avg_relative_prec exact_relative_unranked_avg_prec map_at_R int_map exact_int_R_rcl_prec int_map_at_R unranked_avg_prec* relative_unranked_avg_prec* rcl_at_142_nonrel fallout_recall_* int_*R-prec micro_prec micro_recall micro_bpref bpref variants time base measures. Input formats added: prefs - allows expression of preferences qrels_prefs - same as standard qrels, except treated as prefs qrels_jg - same as standard qrels, except allows judgment sets from multiple users (judgment groups). Version 8.1, Added infAP, minor bug fixes 7/24/06 Improved infAP comments (implementation verified by Yilmaz). trec_eval_help.c: allow longer measure explanations. 6/27/06 get_opt.c Fixed error message 6/22/06 Added measure infAP (Aslam et al) to allow judging only sample of pools. -1 for rel in qrels file interpreted as pool doc not judged. 6/22/06 trvec_teval.c: fixed bugs in calculation of bpref if multiple relevance levels were used and a non-default relevance level was given. (Eg. A doc with rel level of 2 was counted as unjudged rather than judged nonrel if a relevance level of 3 was needed to consider relevant.) 4/5/06 Changed comments in README, trec_eval.c, trec_eval_help.c files which incorrectly claimed queries with no relevant docs are ignored (this was true with very old versions of trec_eval). Now reads that queries with no relevance information are ignored. Giorgio Di Nunzio and Nicola Ferro, ------------------------------------------------------------------------------ Version 8.0, full bpref bug fix, see file bpref_bug. I decided to up the version number since bpref results are incompatible with previous results (though the changes are small). ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Files: Main procedure: trec_eval.c --- Procedures to read input files of various formats: formats.c Mapping names of input file formats to input procedures get_qrels.c Read the standard judged documents (qrels format) get_qrels_jg.c Read qrels format with multiple judgment groups per query get_prefs.c Read preferences judgments instead of doc judgments - see formats.c for full description. get_qrels_prefs.c Read qrels_jg format file, interpret as prefs file. get_trec_results.c Read the standard result file (trec_results format). --- Procedures to merge rel_info and results from input form into form that measures can easily use, if they wish: form_res_rels.c 'qrels' and 'trec_results' into RES_RELS format. form_res_rels_jg.c 'qrels_jg' and 'trec_results' into RES_RELS_JG format. form_pref_counts.c ('prefs' or 'qrels_prefs') and 'trec_results' format --- The actual measures: measures.c Associates measure name with parameters and init, calculation, accumulation, printing procedures meas_*.c Common procedures used by many measures for init, acc, printing. m_.c measure specific procedures --- Miscellaneous: Makefile Compile and test trec_eval README This file CHANGELOG Recent changes test Directory of collection of sample input and output for trec_eval trec_eval.h Basic evaluation structures. functions.h Prototype decorations of measure procedures. sysfunc.h common.h bpref_bug: Description of bug in bpref that existed in trec_eval versions 6 through 7.3. ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Adding a new measure. Assuming it uses standard input files: 1. In m_.c, write any needed measure specific procedures needed to initialize measure calculate measure accumulate measure (adding one topic's value to summary value over topics) calculate the ending average for a measure. print single print final query measure value (and cleanup if needed) Most measures require only a new calculate measure procedure - the other procedures are generic and already implemented depending on the type of measure (has cutoffs and value for each cutoff, parameters, is a float, etc). See functions.h to see fit for these generic procedures. 2. In same file, construct TREC_MEAS te_meas* entry pointing to above procedures and any default cutoffs or parameters. 3. Add pointer to that TREC_MEAS entry in "measures.c" 4. Add measure source file to Makefile ------------------------------------------------------------------------------ Adding a new file format 1. Implement reading of new format in get_.c, with returned top level output of type ALL_REL_INFO or ALL_RESULTS. The individual topic returned values will be in a format dependent form which will be passed to the appropriate measures. 2. Add format to appropriate format list in formats.c 3. Add measures to take advantage of format (see above) 4. To use, invoke trec_eval with -R or -T values, and -m measures that are appropriate.