Analysis and Validation of Image Search Engines in Histopathology
Authors:
Isaiah Lahr,
Saghir Alfasly,
Peyman Nejat,
Jibran Khan,
Luke Kottom,
Vaishnavi Kumbhar,
Areej Alsaafin,
Abubakr Shafique,
Sobhan Hemati,
Ghazal Alabtah,
Nneka Comfere,
Dennis Murphee,
Aaron Mangold,
Saba Yasir,
Chady Meroueh,
Lisa Boardman,
Vijay H. Shah,
Joaquin J. Garcia,
H. R. Tizhoosh
Abstract:
Searching for similar images in archives of histology and histopathology images is a crucial task that may aid in patient matching for various purposes, ranging from triaging and diagnosis to prognosis and prediction. Whole slide images (WSIs) are highly detailed digital representations of tissue specimens mounted on glass slides. Matching WSI to WSI can serve as the critical method for patient ma…
▽ More
Searching for similar images in archives of histology and histopathology images is a crucial task that may aid in patient matching for various purposes, ranging from triaging and diagnosis to prognosis and prediction. Whole slide images (WSIs) are highly detailed digital representations of tissue specimens mounted on glass slides. Matching WSI to WSI can serve as the critical method for patient matching. In this paper, we report extensive analysis and validation of four search methods bag of visual words (BoVW), Yottixel, SISH, RetCCL, and some of their potential variants. We analyze their algorithms and structures and assess their performance. For this evaluation, we utilized four internal datasets ($1269$ patients) and three public datasets ($1207$ patients), totaling more than $200,000$ patches from $38$ different classes/subtypes across five primary sites. Certain search engines, for example, BoVW, exhibit notable efficiency and speed but suffer from low accuracy. Conversely, search engines like Yottixel demonstrate efficiency and speed, providing moderately accurate results. Recent proposals, including SISH, display inefficiency and yield inconsistent outcomes, while alternatives like RetCCL prove inadequate in both accuracy and efficiency. Further research is imperative to address the dual aspects of accuracy and minimal storage requirements in histopathological image search.
△ Less
Submitted 8 June, 2024; v1 submitted 6 January, 2024;
originally announced January 2024.
When is a Foundation Model a Foundation Model
Authors:
Saghir Alfasly,
Peyman Nejat,
Sobhan Hemati,
Jibran Khan,
Isaiah Lahr,
Areej Alsaafin,
Abubakr Shafique,
Nneka Comfere,
Dennis Murphree,
Chady Meroueh,
Saba Yasir,
Aaron Mangold,
Lisa Boardman,
Vijay Shah,
Joaquin J. Garcia,
H. R. Tizhoosh
Abstract:
Recently, several studies have reported on the fine-tuning of foundation models for image-text modeling in the field of medicine, utilizing images from online data sources such as Twitter and PubMed. Foundation models are large, deep artificial neural networks capable of learning the context of a specific domain through training on exceptionally extensive datasets. Through validation, we have obse…
▽ More
Recently, several studies have reported on the fine-tuning of foundation models for image-text modeling in the field of medicine, utilizing images from online data sources such as Twitter and PubMed. Foundation models are large, deep artificial neural networks capable of learning the context of a specific domain through training on exceptionally extensive datasets. Through validation, we have observed that the representations generated by such models exhibit inferior performance in retrieval tasks within digital pathology when compared to those generated by significantly smaller, conventional deep networks.
△ Less
Submitted 14 September, 2023;
originally announced September 2023.
