Relevance Judgment Convergence Degree -- A Measure of Inconsistency among Assessors for Information Retrieval
Authors:
Dengya Zhu,
Shastri L Nimmagadda,
Kok Wai Wong,
Torsten Reiners
Abstract:
Relevance judgment of human assessors is inherently subjective and dynamic when evaluation datasets are created for Information Retrieval (IR) systems. However, a small group of experts' relevance judgment results are usually taken as ground truth to "objectively" evaluate the performance of the IR systems. Recent trends intend to employ a group of judges, such as outsourcing, to alleviate the pot…
▽ More
Relevance judgment of human assessors is inherently subjective and dynamic when evaluation datasets are created for Information Retrieval (IR) systems. However, a small group of experts' relevance judgment results are usually taken as ground truth to "objectively" evaluate the performance of the IR systems. Recent trends intend to employ a group of judges, such as outsourcing, to alleviate the potentially biased judgment results stemmed from using only a single expert's judgment. Nevertheless, different judges may have different opinions and may not agree with each other, and the inconsistency in human relevance judgment may affect the IR system evaluation results. In this research, we introduce a Relevance Judgment Convergence Degree (RJCD) to measure the quality of queries in the evaluation datasets. Experimental results reveal a strong correlation coefficient between the proposed RJCD score and the performance differences between the two IR systems.
△ Less
Submitted 8 August, 2022;
originally announced August 2022.
An Integrated Search Framework for Leveraging the Knowledge-Based Web Ecosystem
Authors:
Dengya Zhu,
Shastri Lakshman Nimmagadda,
Torsten Reiners,
Amit Rudra
Abstract:
The explosion of information constrains the judgement of search terms associated with Knowledge-Based Web Ecosystem (KBWE), making the retrieval of relevant information and its knowledge management challenging. The existing information retrieval (IR) tools and their fusion in a framework need attention, in which search results can effectively be managed. In this article, we demonstrate the effecti…
▽ More
The explosion of information constrains the judgement of search terms associated with Knowledge-Based Web Ecosystem (KBWE), making the retrieval of relevant information and its knowledge management challenging. The existing information retrieval (IR) tools and their fusion in a framework need attention, in which search results can effectively be managed. In this article, we demonstrate the effective use of information retrieval services by a variety of users and agents in various KBWE scenarios. An innovative Integrated Search Framework (ISF) is proposed, which utilises crawling strategies, web search technologies and traditional database search methods. Besides, ISF offers comprehensive, dynamic, personalized, and organization-oriented information retrieval services, ranging from the Internet, extranet, intranet, to personal desktop. In this empirical research, experiments are carried out demonstrating the improvements in the search process, as discerned in the conceptual ISF. The experimental results show improved precision compared with other popular search engines.
△ Less
Submitted 9 December, 2020;
originally announced December 2020.
