-
What Can Natural Language Processing Do for Peer Review?
Authors:
Ilia Kuznetsov,
Osama Mohammed Afzal,
Koen Dercksen,
Nils Dycke,
Alexander Goldberg,
Tom Hope,
Dirk Hovy,
Jonathan K. Kummerfeld,
Anne Lauscher,
Kevin Leyton-Brown,
Sheng Lu,
Mausam,
Margot Mieskes,
Aurélie Névéol,
Danish Pruthi,
Lizhen Qu,
Roy Schwartz,
Noah A. Smith,
Thamar Solorio,
**gyan Wang,
Xiaodan Zhu,
Anna Rogers,
Nihar B. Shah,
Iryna Gurevych
Abstract:
The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time…
▽ More
The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review -- manuscripts, reviews, discussions -- are largely text-based, Natural Language Processing has great potential to improve reviewing. As the emergence of large language models (LLMs) has enabled NLP assistance for many new tasks, the discussion on machine-assisted peer review is picking up the pace. Yet, where exactly is help needed, where can NLP help, and where should it stand aside? The goal of our paper is to provide a foundation for the future efforts in NLP for peer-reviewing assistance. We discuss peer review as a general process, exemplified by reviewing at AI conferences. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance, illustrated by existing work. We then turn to the big challenges in NLP for peer review as a whole, including data acquisition and licensing, operationalization and experimentation, and ethical issues. To help consolidate community efforts, we create a companion repository that aggregates key datasets pertaining to peer review. Finally, we issue a detailed call for action for the scientific community, NLP and AI researchers, policymakers, and funding bodies to help bring the research in NLP for peer review forward. We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.
△ Less
Submitted 10 May, 2024;
originally announced May 2024.
-
Exploring task-based query expansion at the TREC-COVID track
Authors:
Thomas Schoegje,
Chris Kamphuis,
Koen Dercksen,
Djoerd Hiemstra,
Toine Pieters,
Arjen de Vries
Abstract:
We explore how to generate effective queries based on search tasks. Our approach has three main steps: 1) identify search tasks based on research goals, 2) manually classify search queries according to those tasks, and 3) compare three methods to improve search rankings based on the task context. The most promising approach is based on expanding the user's query terms using task terms, which sligh…
▽ More
We explore how to generate effective queries based on search tasks. Our approach has three main steps: 1) identify search tasks based on research goals, 2) manually classify search queries according to those tasks, and 3) compare three methods to improve search rankings based on the task context. The most promising approach is based on expanding the user's query terms using task terms, which slightly improved the NDCG@20 scores over a BM25 baseline. Further improvements might be gained if we can identify more specific search tasks.
△ Less
Submitted 16 November, 2020; v1 submitted 23 October, 2020;
originally announced October 2020.
-
REL: An Entity Linker Standing on the Shoulders of Giants
Authors:
Johannes M. van Hulst,
Faegheh Hasibi,
Koen Dercksen,
Krisztian Balog,
Arjen P. de Vries
Abstract:
Entity linking is a standard component in modern retrieval system that is often performed by third-party toolkits. Despite the plethora of open source options, it is difficult to find a single system that has a modular architecture where certain components may be replaced, does not depend on external sources, can easily be updated to newer Wikipedia versions, and, most important of all, has state-…
▽ More
Entity linking is a standard component in modern retrieval system that is often performed by third-party toolkits. Despite the plethora of open source options, it is difficult to find a single system that has a modular architecture where certain components may be replaced, does not depend on external sources, can easily be updated to newer Wikipedia versions, and, most important of all, has state-of-the-art performance. The REL system presented in this paper aims to fill that gap. Building on state-of-the-art neural components from natural language processing research, it is provided as a Python package as well as a web API. We also report on an experimental comparison against both well-established systems and the current state-of-the-art on standard entity linking benchmarks.
△ Less
Submitted 2 June, 2020;
originally announced June 2020.
-
Dealing with Label Scarcity in Computational Pathology: A Use Case in Prostate Cancer Classification
Authors:
Koen Dercksen,
Wouter Bulten,
Geert Litjens
Abstract:
Large amounts of unlabelled data are commonplace for many applications in computational pathology, whereas labelled data is often expensive, both in time and cost, to acquire. We investigate the performance of unsupervised and supervised deep learning methods when few labelled data are available. Three methods are compared: clustering autoencoder latent vectors (unsupervised), a single layer class…
▽ More
Large amounts of unlabelled data are commonplace for many applications in computational pathology, whereas labelled data is often expensive, both in time and cost, to acquire. We investigate the performance of unsupervised and supervised deep learning methods when few labelled data are available. Three methods are compared: clustering autoencoder latent vectors (unsupervised), a single layer classifier combined with a pre-trained autoencoder (semi-supervised), and a supervised CNN. We apply these methods on hematoxylin and eosin (H&E) stained prostatectomy images to classify tumour versus non-tumour tissue. Results show that semi-/unsupervised methods have an advantage over supervised learning when few labels are available. Additionally, we show that incorporating immunohistochemistry (IHC) stained data provides an increase in performance over only using H&E.
△ Less
Submitted 16 May, 2019;
originally announced May 2019.
