10-3 Optional alternative reading

10-3 Moffat et al. Strategic system comparisons via targeted relevance judgments

10-3 Guiver, John, Stefano Mizzaro, and Stephen Robertson. A few good topics: Experiments in topic set reduction for retrieval evaluation.

10-3 Soboroff, I. and Nicholas, C. and Cahan, P. Ranking retrieval systems without relevance judgments.

26-Sep Optional Alternative Reading

26-Sep Kazai, Gabriella, et al. An analysis of systematic judging errors in information retrieval. CIKM 2012.

26-Sep B. Carterette, I. Soboroff. The Effect of Assessor Errors on IR System Evaluation. SIGIR 2010.

26-Sep Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management 36, pp. 697-716, 2000.

19-Sep Optional Alternative Reading

19-Sep Mark Smucker and Charles LA Clarke. Time-based calibration of effectiveness measures.

19-Sep Google’s Search Quality Rating Guidelines. Version 1.0.

19-Sep Tefko Saracevic. 2007. Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part III: Behavior and Effects of Relevance

12-Sep Optional Alternative Reading

12-Sep Sanderson. Test collection based evaluation of information retrieval systems, Ch. 4

12-Sep Harman. Information Retrieval Evaluation Ch. 2

12-Sep Kelly Ch. 10