11-14 Optional alternative reading

11-14 Gabriella Kazai et al. User Intent and Assessor Disagreement in Web Search Evaluation

11-14 Falk Scholer et al. Quantifying test collection quality based on the consistency of relevance judgements.

11-14 Falk Scholer, et al. The Effect of Threshold Priming and Need for Cognition on Relevance Calibration and Assessment.

11-7 Optional alternative reading

11-7 Ajanki et al. (2009). Can eyes reveal interest? Implicit queries from gaze patterns

11-7 Moshfeghi, Yashar, et al. Understanding relevance: an fMRI study

11-7 Vo, T., & Gedeon, T. Reading Your Mind: EEG during Reading Task

10-31 Mark Sanderson. CHAPTER 5 - Test collection based evaluation of information retrieval systems, Foundations and Trends in Information Retrieval, pp. 308-318.

10-31 Smucker, Mark D., James Allan, and Ben Carterette. A comparison of statistical significance tests for information retrieval evaluation. CIKM’07.

10-31 M. Sanderson and J. Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, SIGIR’05

10-31 Optional Alternative Reading

10-24 Optional alternative reading

10-24 Moffat, Alistair, Falk Scholer, and Paul Thomas. Models and metrics: IR evaluation as a user process.

10-24 Oard et al. Building an information retrieval test collection for spontaneous conversational speech. SIGIR 2004, pp. 41-48.

10-24 Maria Eskevich et al. Creating a data collection for evaluating rich speech retrieval. LREC’12.

10-17 Lydia Chilton et al. Task search in a human computation market

10-17 W. John Wilbur and Won Kim. Improving a gold standard: treating human relevance judgments of MEDLINE document pairs

10-17 Azzah Al‐Maskari and Mark Sanderson. A review of factors influencing user satisfaction in information retrieval

10-17 Optional Alternative Reading

10-10 Optional Alternative Reading

10-10 A Chouldechova, D Mease. Differences in search engine evaluations between query owners and non-owners. WSDM’13.

10-10 P Organisciak, M Efron, K Fenlon, M Senseney. Evaluating rater quality and rating difficulty in online annotation activities. ASIS&T’12.

10-10 Harris, Christopher G., and Padmini Srinivasan. Applying human computation mechanisms to information retrieval. ASIS&T 2012.

10-3 Optional alternative reading

10-3 Moffat et al. Strategic system comparisons via targeted relevance judgments

10-3 Guiver, John, Stefano Mizzaro, and Stephen Robertson. A few good topics: Experiments in topic set reduction for retrieval evaluation.

10-3 Soboroff, I. and Nicholas, C. and Cahan, P. Ranking retrieval systems without relevance judgments.

26-Sep Optional Alternative Reading

26-Sep Kazai, Gabriella, et al. An analysis of systematic judging errors in information retrieval. CIKM 2012.

26-Sep B. Carterette, I. Soboroff. The Effect of Assessor Errors on IR System Evaluation. SIGIR 2010.

26-Sep Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management 36, pp. 697-716, 2000.

19-Sep Optional Alternative Reading

19-Sep Mark Smucker and Charles LA Clarke. Time-based calibration of effectiveness measures.

19-Sep Google’s Search Quality Rating Guidelines. Version 1.0.

19-Sep Tefko Saracevic. 2007. Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part III: Behavior and Effects of Relevance

12-Sep Optional Alternative Reading

12-Sep Sanderson. Test collection based evaluation of information retrieval systems, Ch. 4

12-Sep Harman. Information Retrieval Evaluation Ch. 2

12-Sep Kelly Ch. 10

Kelly FnTIR 2009 Ch. 2-4.

Diane Kelly. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval 3.1—2 (2009). READ ONLY CHAPTERS 2-4, pp. 9-30.

Croft, Metzler, Strohman Ch. 8

Bruce Croft, Don Metzler, and Trevor Strohman. 2009. Search Engines: Information Retrieval in Practice. Ch. 8: Evaluating Search Engines. 

Voorhees 2001

Ellen Voorhees. 2001. The Philosophy of Information Retrieval Evaluation.

Webster & Watson

Webster and Watson. 2002. Analyzing the past to prepare for the future: Writing a literature review