Single Image Estimation of Cell Migration Direction by Deep Circular Regression

Abstract

In this paper we study the problem of estimating the migration direction of cells based on a single image. To the best of our knowledge, there is only one related work that uses a classification CNN for four classes (quadrants). This approach does not allow detailed directional resolution. We solve the single image estimation problem using deep circular regression with special attention to cycle-sensitive methods. On two databases we achieve an average accuracy of $\sim$17^∘, which is a significant improvement over the previous work.

1 Introduction

Cell images are widely used in biomedical research. Information extraction from cell images is a crucial prerequisite for answering challenging biomedical questions. Important cell image analysis tasks include segmentation, classification, and tracking [6, 17]. In this work we study the problem of single image estimation of cell migration direction (SIECMD), i.e. estimating the future direction of cell migration from a single image of a cell, see Figure 1 for examples.

Cell migration direction can be easily obtained from cell tracking in videos [7]. In contrast, performing this estimation by using a single image is much more challenging. There are, however, biological indications that this estimation is possible: A cell’s first reaction to a chemical cue, which promotes migration towards or away of the gradient, is to polarize and extend protrusions in the direction of migration. The cell uses these protrusions to exert forces to the substrate, causing the cell to migrate in the direction of these protrusions, see [12] for a detailed explanation. The study [4] shows that the migration direction (of an adhering mammalian cell) is influenced by its enforced polarity, i.e. a large front and a small rear. Consequently, there exists a clear relationship between the motility and shape of individual migrating cells [8].

In this work we use deep circular regression to learn shape features for SIECMD. A detailed prediction of motion direction from single images is of relevance for various biomedical applications. One such example are histological probes from biopsies or other human tissue. These samples can only be obtained in small quantities and preclude live cell analysis. Here, SIECMD may advance our understanding on single cell dynamics under physiological and pathologcial (e.g. cancer) conditions. A second example, where a prediction of motion analysis may be advantageous, is for high-throughput screening. To probe the effectiveness of pharmaceutical compounds on cell motility is time and data intensive. Replacing single cell tracking data with kinetic information inferred from SIECMD may significantly augment the throughout of such assays. Finally, the estimated migration could be even used for applications beyond biomedical data. One possible application is to complement the cellular Potts model (CPM), an established framework for cell simulation originally proposed in [1] (a current toolbox can be found in [15]). The estimated migration direction from SIECMD could be integrated into CPM to enable a more robust simulation.

The task of SIECMD has been hardly studied so far. To our best knowledge, the only relevant work is done by Nishimoto et al. [9]. It is based on a CNN of 14 layers designed for solving a 4-class classification problem (upper left, upper right, lower left, lower right; i.e. one class for each quadrant). We consider this formulation to be a major drawback of that work, as it represents only a very coarse sampling of the entire space of directions (on a unit circle) and thus does not allow for a more detailed directional resolution. The thesis work [19] also considers SIECMD. It extends the approach in [9] by a variable number of classes (instead of constant 4) that is tuned during the training phase. However, this number is selected using information from two consecutive images at $t$ and $t+1$, thus not applicable to a single-image context. In addition, the estimated migration direction has not been separately evaluated, but is only used to support cell tracking in videos with low frame rate. Another single-image problem is that of classifying the cell migration modes (continuous, discontinuous) [2].

We solve the problem of SIECMD using deep regression with special attention to the circular nature of direction data. We present the details of our methods in Section 2. The experimental results and discussions follow in Section 3. Then, further discussion concludes the paper.

Figure 1: Example cell images with ground truth cell migration direction (red arrow).

2 Methods

To design a deep neural network for estimating the cell migration direction we need to specify several details (direction encoding, activation function, loss function). For each of them there are various choices. In the following we present these design options and a way of finding an optimal configuration. In particular, dealing with directions (angles) means working with circular data. Because of the special nature of circular data, traditional techniques meant for Euclidean space may not be applicable. Instead, cycle-sensitive methods are needed [5, 11].

2.1 Direction encoding

We consider two schemes of direction encoding: 1) an angle $\alpha\in[0,2\pi)$. 2) a point on the unit circle $(x,y),\ x^{2}+y^{2}=1$. The choice of encoding also determines the number of output neurons (1 or 2) of a deep neural network for migration direction estimation.

2.2 Activation functions

ReLU is used for the activation function except the last (output) layer. There, we define for the angle direction encoding the following activation function:

$$\varphi_{cyclic}(x)\ =\ x\bmod 2\pi$$ (1)

For the circle direction encoding we consider two options: the identity function and the Sigmoid function:

$$\varphi_{identity}(x)\ =\ x;\quad\varphi_{sigmoid}(x)\ =\ \frac{e^{x}-1}{e^{x}+1}$$ (2)

In the case of modulo operator and Sigmoid function the activation function has the purpose of normalization (to $[0,2\pi)$ and $[-1,1]$, respectively).

2.3 Loss functions

For the angle direction encoding we define two loss functions together with their quadratic variants to compare the network output and ground truth:

$$\delta_{linear}(\alpha,\beta) = |\alpha-\beta|;\quad\delta_{linear}^{2}(\alpha,\beta)\ =\ (\delta_{linear}(\alpha,\beta))^{2}$$ (3)

$$\delta_{cyclic}(\alpha,\beta) = \min(|\alpha-\beta|,2\pi-|\alpha-\beta|)$$ (4)

$$\delta_{cyclic}^{2}(\alpha,\beta) = (\delta_{cyclic}(\alpha,\beta))^{2}$$ (5)

The loss functions (4) and (5) take the circular nature of direction data into account and are thus expected to perform better than the two in (3). In addition, we define another loss function that is also cycle-sensitive:

$$\delta_{cos}(\alpha,\beta)\ =\ -\cos(\alpha-\beta)$$ (6)

For the circle direction encoding we define two simple loss functions:

$$\delta_{dist}((x_{1},y_{1}),(x_{2},y_{2})) = |x_{1}-x_{2}|+|y_{1}-y_{2}|$$ (7)

$$\delta_{dist}^{2}((x_{1},y_{1}),(x_{2},y_{2})) = (\delta_{dist}((x_{1},y_{1}),(x_{2},y_{2})))^{2}$$ (8)

While all the loss functions above are easy to understand, the one in (6) has a mathematical interpretation in terms of maximum likelihood estimation. Given a machine learning model (CNN in our case) $\alpha=y(\mathbf{w},x)$ with parameters $\mathbf{w}$ for predicting an angle $\alpha$ for an input cell image $x$, the uncertainty of the decision $\alpha$ can be modeled using a probability distribution, particularly the normal distribution. For circular data this corresponds to the von Mises distribution [5, 11]:

$$p(t\ |\ \mathbf{w},x,\kappa)\ =\ \frac{e^{\kappa\cos(t-y(\mathbf{w},x))}}{2\pi I_{0}(\kappa)}$$ (9)

where $\kappa>0$ is its concentration, parameter $I_{0}(\kappa)$ is the modified Bessel function of the first kind of order 0. We use the training data $\mathbf{x}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}$ with the related ground truth $\mathbf{t}=\{t_{1},\ldots,t_{N}\}$ to determine the unknown parameters $\mathbf{w}$ by maximizing the likelihood function:

$$p(\mathbf{t}\ |\ \mathbf{w},x,\kappa)\ =\ \prod_{n=1}^{N}\frac{e^{\kappa\cos(t_{k}-y(\mathbf{w},x_{k}))}}{2\pi I_{0}(\kappa)}\ =\ \frac{1}{\{2\pi I_{0}(\kappa)\}^{N}}\prod_{n=1}^{N}e^{\kappa\cos(t_{k}-y(\mathbf{w},x_{k}))}$$ (10)

To simplify the optimization, we use the log likelihood function:

$$\ln p(\mathbf{t}\ |\ \mathbf{w},x,\kappa)\ =\ \kappa\sum_{n=1}^{N}\cos(t_{k}-y(\mathbf{w},x_{k}))-N\ln(2\pi I_{0}(\kappa))$$ (11)

Maximizing the (log) likelihood function is thus equivalent to minimizing the following error function:

$$-\sum_{n=1}^{N}\cos(t_{k}-y(\mathbf{w},x_{k}))$$ (12)

which exactly corresponds to the loss function (6).

2.4 Deep neural networks

In Sections 2.1–2.3 we described several encoding schemes, activation functions (for the output layer), and loss functions to the studied. In total there are nine combinations. We introduce a small-size CNN to probe the best combination (configuration) and then only use this optimal configuration to test the performance of larger networks. This probing CNN consists of two convolutional layers followed by max pooling layers. The head of this model consists of three dense (fully connected) layers ending with one or two neurons, depending on the direction encoding (see Table 1).

Fine-tuning pre-trained neural networks to a specific task minimizes the training effort and leads to good results [18, 20]. Therefore, we explore several backbones of prominent neural networks (YOLOv8, ResNet50 [3], and EfficientNet [13]). Of these networks, we have chosen the smallest architecture in each case. To test these models for our task, we only used the backbone (i.e. all layers up to first fully-connected layers). Our head was adapted to the complexity of the foreign architectures (see Table 2). The backbones of these networks were pre-trained on known datasets and therefore only need minimal re-training. To do this, we first trained the entire model for 10 epochs, then freeze the weights of the respective backbone and trained the model for further 50 epochs.

Layer	Output	# Parameters
Input layer	(128, 128)	0
Conv2D (5x5), activation=ReLU	(124, 124, 16)	416
MaxPooling2D	(62, 62, 16)	0
Conv2D (3x3), activation=ReLU	(60, 60, 32)	4640
MaxPooling2D	(30, 30, 32)	0
Flatten	(28800)	0
Dense, activation=ReLU	(256)	7373056
Dense, activation=ReLU	(16)	4112
Dense, activation=$\varphi$	($\#$)	$\#$

Table 1: Probing CNN for finding the best configuration. Cell image size: $128\times 128$. Symbol $\#$ indicates that the value there depends on the direction encoding.

Layer	Output	# Parameters
Backbone
GlobalAveragePooling2D	(#)	0
Dense, activation=ReLU	(1024)	#
Dense, activation=ReLU	(256)	262400
Dense, activation=$\varphi$	($\#$)	$\#$

Table 2: Fine-tuning architecture. Backbone refers to the foreign model that is connected to our customized head with the fully connected layers. Symbol $\#$ indicates that the value there depends on the direction encoding or the size of the previous layers.

2.5 Test-time augmentation

Since SIECMD is a challenging task, we also explore the potential of test-time augmentation (TTA) for ensemble effect [10, 16]. For this propose multiple images are randomly generated for an input cell image by rotations. The migration direction estimated for each rotated image is corrected by the applied rotation. For a total number $n$ of cell images (one original, $n-1$ rotated versions) we thus yield $n$ migration direction angles that are fused to obtain the final result, see Figure 2 for an example. We applied a heuristic averaging method: 1) We sort all predictions to receive a sorted ascending list $\{p_{1},...,p_{n}\}$. 2) We calculate the distance $d=p_{n}-p_{1}$. 3) We add $2\pi$ to the smallest prediction $p_{1}$ and start with these new predictions at 1). Our goal is to find the list of predictions with the smallest possible difference $d$. On that resulting list we finally calculate the mean and then use the modulo operator to get the averaging result $\in[0,2\pi)$.

Figure 2: TTA for migration direction estimation ($n=10$).

Dataset	Encoding	Activation	Loss	$\mathbf{E}_{deg}$	$\pm$
NIH3T3	1N	$\varphi_{cyclic}$	$\delta_{linear}$	97.81	0.51
		$\varphi_{cyclic}$	$\delta_{linear}^{2}$	97.82	0.59
		$\varphi_{cyclic}$	$\delta_{cyclic}$	97.86	0.54
		$\varphi_{cyclic}$	$\delta_{cyclic}^{2}$	97.83	0.53
		$\varphi_{cyclic}$	$\delta_{cos}$	66.54	12.10
	2N	$\varphi_{identity}$	$\delta_{dist}$	33.27	1.79
		$\varphi_{identity}$	$\delta_{dist}^{2}$	34.29	2.20
		$\varphi_{sigmoid}$	$\delta_{dist}$	30.53	2.21
		$\varphi_{sigmoid}$	$\delta_{dist}^{2}$	30.51	0.86
U373	1N	$\varphi_{cyclic}$	$\delta_{linear}$	87.05	1.73
		$\varphi_{cyclic}$	$\delta_{linear}^{2}$	87.06	1.77
		$\varphi_{cyclic}$	$\delta_{cyclic}$	86.97	1.71
		$\varphi_{cyclic}$	$\delta_{cyclic}^{2}$	87.11	1.85
		$\varphi_{cyclic}$	$\delta_{cos}$	33.79	3.76
	2N	$\varphi_{identity}$	$\delta_{dist}$	21.59	1.27
		$\varphi_{identity}$	$\delta_{dist}^{2}$	22.70	1.63
		$\varphi_{sigmoid}$	$\delta_{dist}$	22.92	2.15
		$\varphi_{sigmoid}$	$\delta_{dist}^{2}$	21.90	1.05

Table 3: Migration direction estimation by the probing CNN. The column ”Encoding” indicates one or two output neurons. The column $\mathbf{E}_{deg}$ shows the mean deviation in degrees (13)

with the standard deviation in column ”$\pm$”.

Dataset	Backbone	Pre-training	$\mathbf{E}_{deg}$	$\pm$
NIH3T3	YOLOv8	coco	23.77	0.64
	EfficientNet	imagenet	17.27	2.26
	Resnet50	imagenet	20.07	0.53
U373	YOLOv8	coco	18.19	2.73
	EfficientNet	imagenet	16.93	3.69
	Resnet50	imagenet	23.77	4.59

Table 4: Migration direction estimation by large neural networks using the optimal configuration derived from Table 5.

Dataset	Testset Rotation	$\mathbf{E}_{deg}$	$\pm$
NIH3T3	Original image	17.27	2.26
	+ 1 rotated version	17.41	2.48
	+ 5 rotated versions	17.33	1.88
	+ 9 rotated versions	17.24	2.15
	+ 13 rotated versions	17.18	2.17
U373	Original image	16.93	3.69
	+ 1 rotated version	17.59	2.95
	+ 5 rotated versions	17.77	2.53
	+ 9 rotated versions	17.57	2.86
	+ 13 rotated versions	17.47	2.59

Table 5: Migration direction estimation by TTA (EfficientNet backbone, pre-training on imagenet).

3 Experimental validation

3.1 Datasets

Our performance evaluation is based two datasets used in [9] that contain time-lapse phase-contrast microscopic images of cell migration in different ways.

• NIH3T3. Fibroblasts were seeded onto dishes made of glass. Following a night of incubation, an inverted microscope was used to track the migration of the cells. Phase-contrast footage was captured using CCD cameras.

• U373. This dataset was part of the ISBI (International Symposium on Biomedical Imaging) cell tracking challenge in 2015 [14].

Example images can be found in Figure 1, where the left (right) block shows images from NIH3T3 (U373). The NIH3T3 (U373) dataset contains 830 (1011) images with 170 (260) distinct ground truth angles. Note that in [9] results are also reported on a third dataset hTERT-RPE1. Although available to us, however, this dataset was already in a form prepared for the 4-class problem so that no angle ground truth could be obtained. For this reason this dataset was not used for our study.

The tracking information given in the datasets consists of the position $(x,y)$ for each cell in the frames. With this information, it was possible to obtain a ground truth angle for all cells that migrated a distance of more than 5$\mu$m during image acquisition.

We multiplied the training data using augmentation. This includes a random combination of rotation $\in[0,2\pi)$, $x$ and $y$ shift $\in[-0.2,0.2]$ percent of the image size, scaling with factor $\in[-0.1,0.1]$, and vertical and horizontal mirroring.

3.2 Experimental results

We used a 4-fold validation, where each fold was randomly generated and split into 40% training, 10% validation and 50% test. For the evaluation, we used the mean deviation over all predictions $\alpha_{i}$ and corresponding ground truth $\beta_{i}$ as a metric for the migration direction estimation accuracy $\mathbf{E}_{deg}$:

$$\mathbf{E}_{deg} = \frac{1}{n}\sum_{i=1}^{n}\min(|\alpha_{i}-\beta_{i}|,\ 2\pi-|\alpha_{i}-\beta_{i}|)$$ (13)

The performance of the probing CNN is shown in Table 5. When considering both datasets, the configuration: circle encoding, activation function $\varphi_{sigmoid}$, loss function $\delta_{dist}^{2}$, can be considered optimal, as marked in color. The loss function $\delta_{cos}$ has a sound mathematical interpretation, but does not perform best (for these two datasets). The reason may be the discontinuous nature of the cyclic activation function in (1). It is not compatible with (stochastic) gradient descent-type optimization algorithms and needs specialized handling.

Then, we use this optimal configuration to test the performance of three networks YOLOv8, ResNet50, and EfficientNet. The results are given in Table 5. For both datasets, the EfficientNet model performs best in terms of mean accuracy and standard deviation. Using this model we are able to achieve a mean accuracy of $\sim$17^∘​ (marked in color) on both datasets.

Finally, using EfficientNet we further study the performance of TTA for $n=2,6,10,14$, see Table 5. In term of mean and standard deviation, NIH3T3 benefits from TTA. For U373 only the standard deviation declines when using TTA. We estimate the maximum estimation error by (mean + 3 $\times$ standard deviation) for the case without & with TTA ($n=14$). This results in 24.05^∘ & 23.69^∘ for NIH3T3 and 28.00^∘ & 25.24^∘ for U373, respectively. In this regard TTA is advantageous for U373. Overall, it can be concluded that TTA has a positive effect.

3.3 Comparison with previous work [9]

The major drawback of the only related work for solving SIECMD [9] is the very coarse sampling of directions (on a unit circle) in four discrete classes only, which does not allow for a more detailed directional resolution. Given the predicted class, without extra knowledge, the best possible angle output is the one in the middle of the related quadrant, e.g. $\frac{1}{4}\pi$ if the first quadrant is predicted. This output causes an average inaccuracy relative to the (unknown) ground truth:

• $\frac{1}{8}\pi$: if the predicted quadrant is correct.

• $\frac{1}{2}\pi$: if the predicted quadrant is a direct neighbor of the correct quadrant.

• $\frac{7}{8}\pi$: if the predicted quadrant is not a direct neighbor of the correct quadrant.

Here we assume equal probability of an angle within the ground truth quadrant. The maximal inaccuracy is $\frac{1}{4}\pi$, $\frac{3}{4}\pi$, and $\pi$, respectively. In [9] a classification accuracy of 87.89% (NIH3T3) and 81.76% (U373) is reported. Since no error percentage for the incorrect quadrants is specified, we exemplarily use $1/3$ of the remainder percentage for each of the incorrect quadrants to obtain the average inaccuracy:

$$\mbox{NIH3T3}: 87.89\%\cdot\frac{1}{8}\pi+4.04\%\cdot(\frac{1}{2}\pi+\frac{1}{2}\pi+\frac{7}{8}\pi)=18.56\%\cdot\pi\ (33.41^{\circ})$$

$$\mbox{U373}: 81.86\%\cdot\frac{1}{8}\pi+6.05\%\cdot(\frac{1}{2}\pi+\frac{1}{2}\pi+\frac{7}{8}\pi)=21.58\%\cdot\pi\ (38.84^{\circ})$$

A comparison with Table 5 clearly shows an improved accuracy of angle estimation by our approach. Similarly, the average maximal inaccuracy can be estimated by:

$$\mbox{NIH3T3}: 87.89\%\cdot\frac{1}{4}\pi+4.04\%\cdot(\frac{3}{4}\pi+\frac{3}{4}\pi+\pi)=29.04\%\cdot\pi\ (52.28^{\circ})$$

$$\mbox{U373}: 81.86\%\cdot\frac{1}{8}\pi+6.05\%\cdot(\frac{1}{2}\pi+\frac{1}{2}\pi+\frac{7}{8}\pi)=31.05\%\cdot\pi\ (55.89^{\circ})$$

A comparison with our results on maximum estimation error (see Section 3.2) shows superior performance over the previous work [9] in terms of maximal inaccuracy.

4 Conclusion

In this paper have we studied the problem of estimating the migration direction of cells based on a single image. Our solution, based on deep circular regression with special attention to cycle-sensitive methods, has achieved an average accuracy of $\sim$17^∘​ on two datasets. This performance represents a significant improvement over the only previous work on this problem. The outcome of this work provides foundation for the realization of applications such as those outlined in the introduction, which were previously not possible. The focus of our future work will therefore be on investigating such applications.