Motivation
Cell counting is contingent on having a focused image. Since our objective lens configuration has a fixed focal length, we can vary its height to bring the image to focus. However, the working range in which the image is focused is rather small. In this blog, we will discuss how we quantify the extent of focus, and discuss our efforts and findings as relating to focusing.
Without sharply defined edges, the cell counting performance drop sharply. Consider the following two images. Clearly, the object being imaged is more or less the same, but the extent of focus is clearly worse in the second image.
When we run cell detection on both of them, we can see a marked drop in the number of cells of almost two-thirds. We can also observe that the cell detection model is not picking up some of the cells in the background. Looking again at the original images, it seems that the first image is not quite focused too, and the dynamic Otsu’s threshold seems to have discarded the cells in the background.
Looking at a different, well-focused image (below), we can see a similar issue, in which cells are left out: note the number of grey dots leftover in the 4th image; the cells that were counted are highlighted in green. There are still many left over cells.
As a caveat, the cell solution was not well-mixed, and there is obvious clumping present. Nevertheless, this concern is not relevant for the problem of missing cells.
There are a few problems we’d like to solve: (1) how do we quantify the extent of focus? (2) how do we know that we are at the position of maximum focus? (3) even if the image is not quite focused, how do we process the image such that these cells won’t be left out? (4) for a well-focused image, it seems that some smaller cells are left out; how can we avoid this?
In the following sections, we will examine in detail (1) how we can improve the thresholding process, (2) how we can quantify the extent of focus, and (3) a scheme to autofocus the image, along with (4) our concluding thoughts
1. Improving thresholding
At the very beginning of the project cycle – before we started dismantling microscopes, assembling 3D printer stages, and designing cell counting algorithms and designing web user interfaces – we prepared yeast samples to observe under a biological microscope, and took pictures for initial study.
These images were taken at 600X. With a biological microscope, we could use both the coarse and fine control knobs in order to bring the image into focus. The image is a backlit bright-field image. We’re not exactly sure why the background seems yellowish, but we guess that there could be a glass filter inside the aperture between the slide stage and the light source.
We first crop out a square in the centre. This will discard the blurry cells along the edges, along with the black borders. We also load the image as greyscale, since there is not much meaningful colour information anyway.
Let’s take a look at the histogram of the pixel intensities:
These histograms correspond to subplots 3 and 4 in the above figure. The thresholding has set the blank pixels to 0, and the rest of the pixels are treated as actual cells.
Let’s re-plot them in a log scale:
We can clearly see the effect of the low-pass filter now! With this preliminary understanding, we can try using Otsu’s thresholding with the same image to understand what it does.
The working principle of Otsu’s thresholding was described in a separate blog post (under Cell Counting Model)
- suggestion for DIY histogram (i.e. static thresholds, or dynamically shift threshold and observe the change in cell count)
- after applying these changes, what are the new cell counts / debug plots?
2. Quantifying focus
We refer to information taken from a blog [1] extending the paper “Analysis of focus measure operators for shape-from-focus” [2]. It offers several methods, and we decided to try the Laplacian-based operator. It is the same as applying an edge-detection kernel, which is an approximation of the second spatial derivative. Since the blog reports that such methods are sensitive to noise (which we quantify elsewhere), we also apply a Gaussian blur to smooth out the image first before applying the edge-detection kernel.
To illustrate what this does, let’s compare the following three images.
We observe that the edge detection helps to bring out the features of the original image. Intuitively, if there are more edges in the image, then the image is likely to be more focused. If the image is very blurry, then the image is likely to be out of focus.
We can perform the following sanity check, by applying difference strengths of blur, and computing the energy of the edge-detection output (summing the pixel values of output)
It seems that our intuitions were correct. However, the “energy” value would depend on the size of the image: a larger image would have more edges, and would have a higher “energy”.
To make focus comparable across images, we can take the variance of the pixel values. Since pixel values are distributed between [0,255], the more “energy” there is in the edge detection image, the larger the variance will be. Besides, this would set a scaling limit on the maximum variance, which allows us to make meaningful comparisons across image sizes.
Replotting:
These values seem a lot more manageable now. Hence, we are rather confident in our use of the variance of the edge-detection output as a proxy for the focus of the image.
For the following experiment, we are focusing on a slide of glass. We perform a scan with an interval of 0.025mm, acquiring an image and computing the focus at each step.
The range of focus is somewhat narrow, and a naive gradient descent method is unlikely to work, since there seems to be outliers at various points. Interestingly, these outliers appear on both sides of the central peak. If there was actually interesting edges inside the glass, we would expect outliers to only appear on one side of the peak.
While not motivated by theoretical considerations, we can fit the central peak to a Gaussian:
The quality of fit is not great, most likely due to the outliers, which we don’t have an a priori reason for which we can exclude them: without more information, the outliers are probably real artifacts worthy of further investigation.
Nevertheless, we find a sigma of about 0.015, which when converted to full width at half maximum (FWHM) is about 0.034 mm. This suggests that we could sample the focus every 0.015mm, which would always allow us to find the peak. We also know that the focusing position at ~1.1mm is reliable (for this situation), since we checked the image for that.
The next step would be to repeat the scan, perhaps on a different image, and manually inspect the outliers.
[insert plot]
However, we can see that even with a small scan interval of 0.01mm, there is no well-defined peak; there’s only one data point that represents a position of maximum focus.
We suggest that this could have to do with whether the image is darkfield or brightfield.
3. An autofocusing algorithm
Our initial algorithm was to scan over a distance (e.g. 2mm) in small intervals (e.g. 0.1mm). We would then find the position of maximum focus, and then move the microscope lens to the position of maximum focus.
However, since it takes time (~0.5s) to acquire the image from the camera, the smaller the interval, the longer it would take. For a scan distance of 2mm, it would take 200s for a scan interval of 0.01 mm.
Hence, we tried a simple gradient descent algorithm:
Since we are searching for a local maximum, we want to move in the direction of gradient increase; where x_1 represents the updated position, and x_0 is the original position; alpha represents a small constant, and the gradient represents the local gradient.
However, as we can observe from the above plot, there is quite a bit of random noise, and the local gradient does not necessarily represent point to the direction of the global maximum.
Hence, we settled with a simple scan algorithm to first estimate the position of the global maximum, and later perform a smaller scan to refine the estimate.
4. Concluding Thoughts
In conclusion, we have directed a significant amount of effort towards improving our cell-counting model, and also accounting for the scenario in which the image is not focused. We have studied metrics to quantify the extent of an image’s focus, and also applied these metrics towards an autofocusing routine.
While it’s unlikely that our autofocusing routine can replace manually focusing the image, it is still functional and can be used as a substitute.
At the same time, through this process, we have realised the importance of lighting on the cell counting. We also write more about our efforts to control the lighting in another blog.
References
[1] “CVIA – Focus Measure,” Google.com. [Online]. Available: https://sites.google.com/view/cvia/focus-measure. [Accessed: 16-Jul-2021].
[2] S. Pertuz, D. Puig, and M. A. Garcia, “Analysis of focus measure operators for shape-from-focus,” Pattern Recognit., vol. 46, no. 5, pp. 1415–1432, 2013.