Why isn't there a competition or benchmarking for microscopes?

Hardware
,
1

An honest question. I am currently comparing spinning disc to light sheet and I am wondering why it is practically impossible to find image data of the same specimen from both light sheet and spinning disc.

Is it impossible to come up with a standardized comparison?

The individual researcher usually has only limited access to equipment and does not have the time and resources for a thorough comparison.

I only found a single recent paper that compares two microscopes based on one specimen:

Ryu, Youngjae, Yoonju Kim, Sang-Joon Park, Sung Rae Kim, Hyung-Jun Kim, and Chang Man Ha. “Comparison of Light-Sheet Fluorescence Microscopy and Fast-Confocal Microscopy for Three-Dimensional Imaging of Cleared Mouse Brain.” Methods and Protocols 6, no. 6 (December 2023): 108. Comparison of Light-Sheet Fluorescence Microscopy and Fast-Confocal Microscopy for Three-Dimensional Imaging of Cleared Mouse Brain.

1 Like
2

Hi @macromeer, welcome to the forum, it’s a good question.

You hit the nail on the head with:

While not “impossible” in the strictest definition of the word, it’s extremely hard (like, even harder than it might appear) to do a rigorous, fair comparison between two different microscopes. Can’t overemphasize that. Moreover, there are multiple things that one might wish to standardize when making the comparison (SNR, speed, irradiance and/or bleaching), and that can affect interpretations. Also, even measuring the metrics of performance (resolution, contrast, SNR, bleaching, toxicity) is not trivial.

I like the concept of a benchmark or competition, but honestly my first thought is that doing this all correctly is so much more than simply setting some parameters in the software, that I’m not sure it would be practically all that useful to compare measurements made by different researchers at different institutions on different (physical) samples, etc… There’s just way too many ways that could lead to erroneous conclusions.

This leaves us in a place where the most useful thing that you can generally do is compare the practical utility of two or more specific microscopes (and operators) you make have direct access to (and resist the temptation to broadcast and generalize those findings as representative of an entire class or “type” of microscope technology or product). The paper you link can be seen as something like that. Those authors are happier with their (specific) LSFM than their (specific) spinning disc for the purpose of imaging whole cleared mouse brain. That’s great! :slight_smile: But I would caution anyone from taking much more away from that paper. Note that they used different objective lenses, different cameras, different pixel sizes & sampling rates, different exposure times, different laser power, etc… It’s all fine if what one wants to say is “if I sit down at this exact microscope vs that exact microscope and work with what I’m given without changing too many parameters, which image generally looks qualitatively better?” but I’m not sure I’d go much farther than that.

As a specific example of things that require more investigation take their resolution measurements:

scope objective theoretical resolution @ 525nm actual measured FWHM
light sheet 2.5x/0.12 NA ~2.7µm 6.04 µm
spinning disc 4x/0.2 NA ~1.6µm 7.43 µm

My first question would be: why was the measured resolution for the spinning disc so much worse than theory? (it was just a bead on a coverslip). It makes me suspect some sort of error. (note also that they used a 4µm bead that was well above the diffraction limit for both objective lenses to make these measurements)

In any case, making truly careful comparisons that are actually generalizeable is just incredibly hard

2 Likes
3

Thank you very much! How about using a standardized specimen as used for calibration and comparing the captured image data, e.g. by applying the same image segmentation pipeline (or an ensemble) and corresponding metrics. Similar to the evaluation of machine learning models.

4

We compared different microscopes/imaging modalities in the paper below. We discuss the difficulties of fairly comparing microscopes and explain the (great!) lengths we went through to develop and validate our method. It’s an old paper now but the principles still apply.

Murray, J.M. et al. (2007) Evaluating performance in three-dimensional fluorescence microscopy. J. Microsc. 228, 390–405

3 Likes
5

yeah, highly recommend @jennifer’s paper as an excellent example for just how much thought needs to go into a fair comparison of two microscopes. (it gets even harder when trying to compare modalities with dramatically different illumination strategies light LSFM and confocal).

as for the standardized specimen you mention, it’s also a great idea. the beads described in Jennifer’s paper were specially designed. John Murray also published a strategy for getting exactly 240 FPs in a viral capsid (An icosahedral virus as a fluorescent calibration standard: a method for counting protein molecules in cells by fluorescence microscopy - PubMed), which can be used in theory. there’s argolights and stuff like that.

2 Likes