How Many Annotators Do We Need? -- A Study on the Influence of Inter-Observer Variability on the Reliability of Automatic Mitotic Figure Assessment
Authors:
Frauke Wilm,
Christof A. Bertram,
Christian Marzahl,
Alexander Bartel,
Taryn A. Donovan,
Charles-Antoine Assenmacher,
Kathrin Becker,
Mark Bennett,
Sarah Corner,
Brieuc Cossic,
Daniela Denk,
Martina Dettwiler,
Beatriz Garcia Gonzalez,
Corinne Gurtner,
Annika Lehmbecker,
Sophie Merz,
Stephanie Plog,
Anja Schmidt,
Rebecca C. Smedley,
Marco Tecilla,
Tuddow Thaiwong,
Katharina Breininger,
Matti Kiupel,
Andreas Maier,
Robert Klopfleisch
, et al. (1 additional authors not shown)
Abstract:
Density of mitotic figures in histologic sections is a prognostically relevant characteristic for many tumours. Due to high inter-pathologist variability, deep learning-based algorithms are a promising solution to improve tumour prognostication. Pathologists are the gold standard for database development, however, labelling errors may hamper development of accurate algorithms. In the present work…
▽ More
Density of mitotic figures in histologic sections is a prognostically relevant characteristic for many tumours. Due to high inter-pathologist variability, deep learning-based algorithms are a promising solution to improve tumour prognostication. Pathologists are the gold standard for database development, however, labelling errors may hamper development of accurate algorithms. In the present work we evaluated the benefit of multi-expert consensus (n = 3, 5, 7, 9, 11) on algorithmic performance. While training with individual databases resulted in highly variable F$_1$ scores, performance was notably increased and more consistent when using the consensus of three annotators. Adding more annotators only resulted in minor improvements. We conclude that databases by few pathologists and high label accuracy may be the best compromise between high algorithmic performance and time investment.
△ Less
Submitted 8 January, 2021; v1 submitted 4 December, 2020;
originally announced December 2020.
Simphony: An open-source photonic integrated circuit simulation framework
Authors:
Sequoia Ploeg,
Hyrum Gunther,
Ryan M. Camacho
Abstract:
We present Simphony, a free and open-source software toolbox for abstracting and simulating photonic integrated circuits, implemented in Python. The toolbox is both fast and easily extensible; plugins can be written to provide compatibility with existing layout tools, and device libraries can be easily created without a deep knowledge of programming. We include several examples of photonic circuit…
▽ More
We present Simphony, a free and open-source software toolbox for abstracting and simulating photonic integrated circuits, implemented in Python. The toolbox is both fast and easily extensible; plugins can be written to provide compatibility with existing layout tools, and device libraries can be easily created without a deep knowledge of programming. We include several examples of photonic circuit simulations with novel features and demonstrate a speedup of more than 20x over a leading commercially available software tool.
△ Less
Submitted 23 July, 2020;
originally announced September 2020.