Methods for Evaluating Interactive Information Retrieval Systems with Users

Provides a foundation on which those new to Interactive Information Retrieval (IIR) can make more informed choices about how to design and conduct IIR evaluations with human subjects. The primary goal is to catalogue and compile material related to the IIR evaluation method into a single source. It begins by providing historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems. It proceeds to describe the major components of interactive information retrieval system evaluation and goes on to describe different experimental designs and sampling strategies. From there, it presents core instruments and data collection techniques and measures, explains basic data analysis techniques and reviews and discusses previous studies. It also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval. It concludes with a discussion of outstanding challenges and future research directions.