[X] Close
You are about to erase all the values you have customized, search history, page format, etc.
Click here to RESET all values       Click here to GO BACK without resetting any value
Item 1 of about 1
1. Coiera EW, Vickland V: Is relevance relevant? User relevance ratings may not predict the impact of Internet search on decision outcomes. J Am Med Inform Assoc; 2008 Jul-Aug;15(4):542-5
PDF icon [Fulltext service] Download fulltext PDF of this article and others, as many as you want.

  • [Source] The source of this record is MEDLINE®, a database of the U.S. National Library of Medicine.
  • [Title] Is relevance relevant? User relevance ratings may not predict the impact of Internet search on decision outcomes.
  • OBJECTIVE: A common measure of Internet search engine effectiveness is its ability to find documents that a user perceives as 'relevant'.
  • This study sought to test whether user provided relevance ratings for documents retrieved by an Internet search engine correlate with the decision outcome after use of a search engine.
  • DESIGN: 227 university students were asked to answer four randomly assigned consumer health questions, then to conduct an Internet search on one of two randomly assigned search engines of different performance, and to again answer the question.
  • MEASUREMENTS: Participants were asked to provide a relevance score for each document retrieved as well as a pre and post search answer to each question.
  • RESULTS: User relevance rankings had little or no predictive power.
  • Relevance rankings were unable to predict whether the user of a search engine could correctly answer a question after search and could not differentiate between two search engines with statistically different performance in the hands of users.
  • Only when users had strong prior knowledge of the questions, and the decision task was of low complexity, did relevance appear to have modest predictive power.
  • CONCLUSIONS: User provided relevance rankings taken in isolation seem to be of limited to no value when designing a search engine that will be used in a general-purpose setting.
  • Relevance rankings may have a place in situations in which experts provide rankings, and decision tasks are of complexity commensurate with the abilities of the raters.
  • A more natural metric of search engine performance may be a user's ability to accurately complete a task, as this removes the inherent subjectivity of relevance rankings, and provides a direct and repeatable outcome measure which directly correlates with the performance of the search technology in the hands of users.
  • [MeSH-major] Decision Making. Information Storage and Retrieval / standards. Internet
  • [MeSH-minor] Humans. Medical Informatics

  • [Email] Email this result item
    Email the results to the following email address:   [X] Close
  • [Cites] Stud Health Technol Inform. 2001;84(Pt 1):348-52 [11604761.001]
  • [Cites] Int J Med Inform. 2003 Jul;70(2-3):309-16 [12909183.001]
  • [Cites] JAMA. 1998 Oct 21;280(15):1347-52 [9794316.001]
  • [Cites] J Med Internet Res. 2008;10(1):e2 [18244893.001]
  • [Cites] J Am Med Inform Assoc. 2005 May-Jun;12(3):315-21 [15684126.001]
  • [Cites] J Med Internet Res. 2005;7(5):e52 [16403716.001]
  • [Cites] J Am Med Inform Assoc. 2005 May-Jun;12(3):296-8 [15684123.001]
  • (PMID = 18436902.001).
  • [ISSN] 1067-5027
  • [Journal-full-title] Journal of the American Medical Informatics Association : JAMIA
  • [ISO-abbreviation] J Am Med Inform Assoc
  • [Language] eng
  • [Publication-type] Journal Article; Research Support, Non-U.S. Gov't
  • [Publication-country] United States
  • [Other-IDs] NLM/ PMC2442252
  •  go-up   go-down


Advertisement





Advertisement