바로가기메뉴

본문 바로가기 주메뉴 바로가기

logo

  • P-ISSN1013-0799
  • E-ISSN2586-2073

An Experimental Study on the Effect of Domain Expertise on the Consistency of Relevance Judgements

Journal of the Korean Society for Information Management, (P)1013-0799; (E)2586-2073
2021, v.38 no.3, pp.1-22
https://doi.org/10.3743/KOSIM.2021.38.3.001
Stacey Scholten (Yonsei University)
Sung-Been Moon (Yonsei University)

  • Downloaded
  • Viewed

Abstract

An online experiment was conducted to test the subject-knowledge view of relevance theory in order to find evidence of a conceptual basis for relevance. Six experts in Library and Information Science (LIS), nine Master’s students of LIS, and twelve non-experts judged the relevance of 14 abstracts within and outside of the LIS domain. Consistency among the judges was calculated by joint-probability agreement (PA) and interclass correlation coefficients (ICC). When using PA to analyze the judgements, non-experts had a higher consensus regardless of the task or division of groups. However, ICC calculations found Master’s candidates had a higher level of consensus than non-experts within LIS, although the experts did not; and the agreement rates on the non-LIS task for all groups were only poor to moderate. It was only when the groups were analyzed as two groups (experts including Master’s candidates and non-experts) that the expected trend of higher consistency among experts in the LIS task was seen.

Submission Date
2021-08-16
Revised Date
2021-09-08
Accepted Date
2021-09-18

Journal of the Korean Society for Information Management