Ranking -- Use and Usability

Abstract

Ranking has recently attracted much attention, conceptually as well as algorithmically and in its uses in major Internet search engines. A core goal of ranking and related techniques on the Web is to help people find what they are looking for -- and (depending on the application) to suggest to them things that they didn't know they were looking for, but might still find interesting. Any evaluation of these techniques should therefore consider such deployment scenarios. The paper starts from the observation that search engines and recommender systems generally provide users with some ranking on resources, and that standard evaluations rest on a comparison of this system output with an assumed mental representation of the user's ``true ranking''. This is followed by an overview of i) where and how ranking is used by the operators of a Web site or similar service, ii) how ranking is used by the end users of that site or service, and how such usage is measured, and iii) how and according to which criteria this usage and the success as well as the quality of ranking are measured. The paper demonstrates how an interdisciplinary approach can sharpen the view of challenges and promises of this user-oriented analysis of intelligent information access.