$\mu$-Bench: Vision-Language Benchmark for Microscopy Understanding

Safe and ethical use of biomedical data: We prioritize safe and ethical research practices while creating $\mu$-Bench . All public datasets with patient-derived histopathology images had already been de-identified by the dataset’s original authors in compliance with applicable privacy laws and institutional guidelines. The public histopathology image data and metadata were reviewed, and it was determined that it was not possible to identify any individual from the de-identified data. The Stanford Institutional Review Board guidelines were reviewed and discussed, and the use of the images was determined not to be human subjects research.

The $\mu$-Bench cognition images and questions were voluntarily submitted by a small (<10) number of biology/pathology users alpha-testing a free web chat application. There was no intervention, experiment vs. control group, or research question during the alpha testing. There was no greater risk of using the app than other internet apps. At registration, users agreed to the service terms, which included releasing image and text data under CC-BY-SA-4.0.

Consent and data usage: We thank and respect the original dataset authors and use data according to the original copyright and license. $\mu$-Bench was developed with both academic and commercial research in mind. Many datasets are a version of CC-BY-4.0 to allow both academic and commercial usage. However, some data restricts commercial applications via non-commercial clauses or CC-BY-NC-4.0-related licenses. While creating $\mu$-Bench  we significantly improved the data by performing expert review, quality control/standardization, expert labeling with biomedical ontology codes, and creating multiple-choice VQA questions and captions for each image. When the original dataset license is CC-BY-4.0, we release our $\mu$-Bench versions under a permissive CC-BY-SA-4.0 to foster a transparent and collaborative benchmark. For the subset with CC-BY-NC-SA-4.0, we respect the original license and release these data under CC-BY-NC-SA-4.0.

Bias and data diversity: We recognize that AI models, including VLMs, can perpetuate or exacerbate biases in the training data, and incorporating diverse data may mitigate these biases. Diversity is key to understanding biological processes and how they vary across biological sex, ethnicity, or other factors. When possible, we annotate cell lines with age, sex, ethnicity, and other metadata from Cellosaurus or the Cell Line Ontology. We consciously include diverse microscopy images from multiple institutions across various organisms, modalities, and biological states to ensure $\mu$-Bench provides an accurate and fair performance assessment. However, images of human samples and brightfield/fluorescence microscopy are more common in the field and thus over-represented in $\mu$-Bench .

Potential negative societal impacts: AI models trained on biomedical data have the potential for far-reaching impacts on society, both positive and negative. Potential negative impacts include biased performance across different demographic groups and reinforcing existing disparities in research and healthcare. We are committed to mitigating these risks by ensuring the dataset’s diversity and continuously reviewing the ethical implications of our work. Additionally, we will engage with the broader research community to identify and address any emerging ethical concerns.

We are committed to ongoing improvements of $\mu$-Bench to prioritize diverse and representative microscopy data. We will review and update $\mu$-Bench in response to evolving ethical standards and technological advancements.

The authors have no conflicts of interest to disclose.

This research was supported by NIH grants (NIH#P30AG066515 to JJN), the Chan Zuckerberg Initiative Neurodegeneration Challenge Pairs Pilot Project to SYL (2020-221724, 5022), the Wu Tsai Knight Initiative Innovation grant (#KIG 102) to SYL, Arc Institute to AL, Quad Fellowship to JB, and NSF Graduate Research Fellowship (#DGE-2146755) and Stanford Graduate Fellowship to AU. SYL is a Chan Zuckerberg Biohub – San Francisco Investigator.

We thank all the domain experts for testing the web app and submitting questions that were eventually used in $\mu$-Bench cognition. We highlight the contributions from Pedro Guedes-Dias, Andrew Moore, and Julian Perez. We appreciate feedback from Josiah Aklilu and Orr Zohar on earlier manuscript versions.

The benchmark can be downloaded via HuggingFace Datasets at the following url.

$\mu$-Bench is organized into three main categories:

1. 1.

   $\mu$-Bench Perception (Coarse-grained): Containing basic questions about the type of biomedical field of study (domain), subdomain, microscopy modality, submodality, and stain.

2. 2.

   $\mu$-Bench Perception (Fine-grained): Identification or questions regarding a biological cell, cellular process, subcellular or tissue structures, biological state (normal/abnormal), etc.

3. 3.

   $\mu$-Bench Cognition (Reasoning): Expert-generated questions that typically require visual-based reasoning or integrating knowledge about the micrograph’s composition and subject to deduce an answer.

4. 4.

   $\mu$-Bench Object Detection (Localization): Bounding box detection of common biological objects, with an easy and hard data split.

Table 2 shows descriptive statistics for $\mu$-Bench Perception while Table 4 shows statistics for $\mu$-Bench Reasoning.

$\mu$-Bench Perception contains 17,235 microscopy images collected from 25 unique datasets of 96 different cell and tissue types across light, fluorescence, and electron microscopy. If defined, only images from each dataset’s test set are selected; otherwise, a random 15 percent split is created. Each cell/tissue class is sub-sampled to a maximum of 200 sub-classes per class (if the cell type is less than the maximum, the full set is used). Each cell/tissue type is densely annotated (then propagated to each instance of the same type) by an expert, as shown in Figure 18

Table 10 provides summary statistics of domain coverage. Overall, the benchmark covers 8,637 biology images and 8,678 pathology images across 12 subdomains. Similarly, Table 11 shows summary statistics of microscopy modalities covered by $\mu$-Bench perception, including 10,864 images for light microscopy, 5,618 for fluorescence microscopy, and 833 images for electron microscopy across 8 microscopy imaging submodalities.

$\mu$-Bench Perception (Coarse-grained): Hierarchical metadata for each of the 17,235 perception images and task-specific templates (shown in Table 17) are used to create 5 coarse-grained questions and captions regarding microscopy modality, submodality, domain, subdomain, and staining technique. The use of hierarchical metadata enables the generation of options within each hierarchical level. For example, for microscopy submodality, we leveraged microscopy modality metadata (shown in Figure13 ) to randomly sample submodality options within a microscopy modality (e.g. differential interference contrast microscopy within light microscopy). A total of 86,175 (17,235x5) coarse-grained questions are generated using this approach, leveraging domain metadata Figure 14 as well as staining metadata Figure 15). Section H provides 10 random examples of coarse-grained data points (two per type).

$\mu$-Bench Perception (Fine-grained): Metadata for each of the 25 unique datasets and custom prompts (Table 2) are used to generate 13 unique tasks, which are 13 (17,235 unique question-image pairs) closed VQA style questions. of VQA fine-grained tasks.

Perception dataset: source details: Table LABEL:table:sources lists the dataset names, licenses, and corresponding DOIs or URLs for dataset used. The majority of these datasets were sourced from open data repositories, such as Zenodo, Dataverse, Dryad, BBBC, EMPIAR, Kaggle, and various project websites.

Perception dataset: forming questions for evaluation Each sample has a question and set of candidate answers. We describe how to evaluate this VQA task in appendix D.

Cognition dataset: generation details

The cognition dataset was collected during alpha-testing of a web application chat interface. A group of 6 biology and pathology experts were invited to interact with a web application as they wished during their daily routines (invitation is shown in Figure 20). There was no specific research question, intervention, or experimental vs. control group; this was a free service. User registration was voluntary and required reviewing and accepting the terms of service. The terms of service discussed that this was not a research study and that the risk of harm is insignificant and would be similar to any VLM chatbot. The terms of service indicated that uploading images indicated the user had copyright or permission, that images did not contain offensive content, and that the images and text could be released with a CC-BY-SA-4.0 license.

Figure 7) shows a screenshot of the web application interface. A typical usage involved uploading a microscopy image, providing context about the image (experiment details), and asking a question. The web app processed the submission using GPT-4V and provided an answer in real time. Users were encouraged to provide feedback on the answer and a correct answer (if known). Users were encouraged to ask questions that required complex visual reasoning, advanced biomedical knowledge, or could be considered challenging for humans. Random samples for cognition questions are shown in Section J

For datasets with segmentation annotations, we copy the segmentation annotations and also convert them to object detection annotations, giving 3641 images [50, 54, 61, 65, 62].

We define an easy and hard split. The easy split contains the ‘cell’ class from Burgess et al. [50] dataset and ‘nucleus’ class from the Held et al. [54]. Here the image contains only the target object, meaning that simple foreground / background segmentation would work well. The hard split contains the ‘nucleus’ class from Burgess et al. [50], the ‘nucleus’ class from opencell [61], the ‘mitochondria’ class from Wu et al. [65], and the ‘gland’ class from Sirinukunwattana et al. [62]. Here, the target object must be separated from surrounding visual information, and this is challenging.

Although some datasets have been used in the biomedical computer vision community [58, 59], others do not have well-established performance baselines. Thus, we evaluate a supervised linear model’s baseline classification performance for the fine-grained and coarse-grained perception tasks. This establishes that the questions in our benchmark are solvable using image features.

The images in $\mu$-Bench represent the test subset of the original data. The original data were split into train, validation, and test subsets according to the published splits. If no published splits were available, which was the case for many previously unused datasets, we created train/val/test splits (0.7/0.1/0.2) stratifying samples by the class label. We use DINOv2 S14 features (384 dims) because they are robust and can train strong linear models on many natural image classification datasets. We extract DinoV2-S14 features from the train/val subsets and train a logistic regression classifier to determine the performance of a linear classifier for these tasks. We use PCA dimensionality reduction (0.95% var) and whitening on the features with Optuna for hyperparameter tuning. Baseline performance was determined using the best dummy classifier, predicting based on a random sampling of prior probabilities, the most common class, etc.

Figure 6 illustrates that most datasets provide sufficient signal to train a robust linear classifier with DinoV2 feature representations primarily learned from natural images. Our results show a weighted average accuracy of 0.86 across all classification tasks. The lowest performance was observed in the classification of mitosis stages in darkfield microscopy (50.96%). In contrast, the highest performance was seen in the classification of synthetic white blood cells in bright microscopy and the organisms and structures in electron microscopy, with a weighted average accuracy of 100%. Lastly, 58.33 % of the tasks achieved a balanced accuracy greater than 80%. These results show a weighted average of 86.71% across all classification tasks with the upper and lower performance range we could expect from a VLM.

Three out of four tasks are formulated as multiple-choice visual question answering: coarse-grained perception, fine-grained perception, and cognition. We describe their evaluation here, while the fourth task of object detection is described separately in section E.1. For each sample, $i$, we have an image, $\textbf{x}_{i}$ a question string, $q$, and a set of $k$ candidate answer strings $\{a_{ij}\}_{j=1}^{k}$.

For the autoregressive models, $f_{A}$, we first generate a query string, $t$, from the question and candidate answer strings, $t(q,\{a_{ij}\}_{j=1}^{k})$, using the template in Figure 19. The prompt text instructs the response to start with a letter for one of the multiple choice answers, (‘A’, ‘B’, $\dots$). We pass the prompt string and image to the model and decode the output, $y=f_{A}(x_{i},t)$, where $y$ is the output string. We have two strategies for matching the response string, $y$, to the candidate answers $\{a_{ij}\}_{j=1}^{k}$. First, for each $j$, we check whether the answer string is in the output: $a_{ij}\in y$ with lowercase string matching. This is important because models do not always follow the instructions to output the multiple choice letter and instead return the answer. If there were no matches, then we extract the first character from $y$. If it is one of (‘A’, ‘B’, $\dots$) – as instructed in the prompt – then that it is assigned to the corresponding answer, $a_{ij}$. Otherwise, we mark the answer as incorrect.

The contrastive models have a vision encoder $E_{x}$ and text caption encoder $E_{c}$. We first compute the image embedding, $z_{x_{i}}=E_{x}(\textbf{x}_{i})$. Then for each candidate answer, $a_{ij}$, we form a caption, $c$, as $c(a_{ij})$, using a text template that is suitable for CLIP-like models. The templates are in table 17 for the fine-grained perception task, and table 18 for the coarse-grained perception task. For the cognition task, each caption is the concatenation of the question string, $q$, and the candidate answer string, $a_{ij}$. We get the embedding for each caption, $z_{c_{ij}}=E_{c}(c_{ij})$ for $j\in[1,k]$, and then compute the cosine similarity score for each caption, $s_{ij}=z_{c_{ij}}\cdot z_{x_{i}}$ for $j\in[1,k]$. The $j$ with the largest $s_{ij}$ is the final prediction. If argmax($s_{ij}$) has the same index as the correct answer the question is marked as correct, incorrect otherwise.

Code for evaluation is made public through our repository: eVLLM.

We evaluate the two models that support localization: QwenVL [21] and PaliGemma [43], and follow their user guide for prompting. For ‘QwenVL’ the prompt is ‘‘Detect {class_name}’’. For PaliGemma the prompt is ‘‘Detect {class_name}; {class_name}’’, where the repeated class name indicates that multiple instances may be predicted. Our early experiments found that PaliGemma would sometimes fail to localize any instances using detection prompting, but would localize them with segmentation prompting. So if PaliGemma does return zero instances, we prompt for segmentation ‘‘Segment {class_name}; {class_name}’’ and extract the bounding box. The Burgess et al. dataset has two classes, so we prompt the model one at a time. Both models output detection predictions as a string with a standardized structure, which we parse using regex.

We use the GRIT localization metric [47] because it is well-motivated and has previously been used in VLM evaluations by QwenVL [21]. The score is:

There are $M$ ground truth boxes, and $P$ predicted boxes, which are matched using the Hungarian algorithm on the IoU metric. $G_{missed}$ is the number of predicted boxes not matched to a ground truth box. Intuitively, this metric measures the average IoU for matched boxes, while penalizing making too many predictions using $G_{missed}$ (similar to the precision metric). Note that we cannot use the more typical mAP score from object detection because they depend on a threshold for controlling the false-positive rate, which these VLMs do not support.

table 5 summarizes object detection for the datasets having object localization annotations.

Overall, the localization scores are very poor, which is expected since both models are generalist and object localization has received relatively less attention in autoregressive VLMs. Looking at the splits:

• •

  Easy split. Although both models have higher scores for the ‘cell’ class in Burgess et al., they still fall below 80, and the task is extremely easy. The story is similar for ‘nucleus’ in Held et al., but for QwenVLM, the scores are even lower.

• •

  Hard split. All models perform poorly on the hard split. In Burgess et al ‘nucleus’, PaliGemma scores around 30, however qualitative inspection shows that in most cases, the bounding box predicts the entire cell, which encapsulates the nucleus. Similarly, in Wu et al., the mitochondria class scores more than 20 for both models, but qualitative inspection shows that the prediction is usually a box around the entire image. We find the same pattern in ‘Sirinukunwattana et al.’ for ‘gland’ detection.

In the results section 5.2, we consider PLIP, which was fine-tuned from OpenCLIP, and QuiltNet, which was fine-tuned from CLIP, both using pathology data. Since we have benchmark results for all these models, our evaluation can evaluate the impact of fine-tuning on pathology data (moving from OpenCLIP/CLIP to PLIP/QuiltNet). We showed that pathology fine-tuning can improve performance on the pathology subset of $\mu$-Bench (the fine-grained perception), where we filter for all samples from the histopathology or H&E imaging modality. However, the fine-tuned models have worse overall performance on $\mu$-Bench(which includes the pathology subset).

We proposed to create ‘merged models’ by combining the base model (OpenCLIP or CLIP) with their fine-tuned models (PLIP or QuiltNet) with weight merging. Specifically, following [48], for a base model with weights $\theta_{B}$ and tuned model $\theta_{T}$ (which have the same architecture), the merged model weights $\theta_{M}$ are :

That is, we are linearly interpolating each model weight independently, with a single fixed constant $\alpha$. We arbitrarily set $\alpha=0.5$ in our experiments, but tuning that constant could lead to better overall results.

We now show more comprehensive results in table 1, which is the main results table but includes our merged models, M-PLIP and M-QuiltNet; table 7 is the same but with the pathology-only split. In all cases (M-PLIP and M-QuiltNet), the merged models outperform the tuned models (PLIP and QuiltNet), while also outperforming the base models (OpenCLIP and CLIP) in almost all cases. For fine-grained perception (the task that is most in distribution for PLIP and QuiltNet training), the merged models become among the strongest performing overall models, and on pathology-specific fine-grained perception, the merged models outperform BiomedCLIP.

While prior evaluations show that general contrastive VLMs have some biology and pathology knowledge (a finding also reported in [25] and [44]), most specialist models analyzed in this work were fine-tuned. To this end, we analyzed the performance on pathology-only tasks and found similar rankings.

Table table 9 provides a breakdown of model parameters and training data (with dataset size), specialist models include their base model.

Error bars represent 95% confidence intervals (CI) computed via nonparametric bootstrap** using the SciPy $stats\text{.}bootstrap$ function with 1000 resamplings and default settings. No data were excluded from the analyses.

Figure 9 presents a breakdown of perception coarse-grained results by task. It reveals that autoregressive generalist models perform well across all tasks, as indicated by the overall averages. Notably, while GPT-4o dominates most tasks, PaliGemma excels in domain identification, achieving the best performance in that specific area.

Figure 10 shows a breakdown of biology-specific perception fine-grained results by task. While GPT-4o dominates in some tasks, other models excel in specific tasks. For instance, ALIGN outperforms all models in molecular colocalization, while BiomedCLIP has the best performance in mitochondrial morphology classification.

Figure 11 shows a breakdown of pathology-specific perception fine-grained results by task. This breakdown reveals that while GPT-4o has the best performance in three tasks, specialist models still outperform it in amyloid morphology [a] and Pap smear grading.