**A Hierarchical Framework for Mitosis Detection in Time-Lapse Phase Contrast Microscopy Image Sequences of Stem Cell Populations**

Anan Liu1\*, Kang Li2 and Tong Hao<sup>3</sup>

<sup>1</sup>*School of Electronic Information Engineering, Tianjin University* <sup>2</sup>*Microsoft Corporation* <sup>3</sup>*School of Chemical Engineering and Technology, Tianjin University* 1,3*China* <sup>2</sup>*USA*

#### **1. Introduction**

354 Medical Imaging

Nahrendorf M, Keliher E, Marinelli B, Waterman P, Feruglio PF, Fexon L, Pivovarov M,

Niikura N, Costelloe CM, Madewell JE, Hayashi N, Yu TK, Liu J, et al (2011). FDG-PET/CT

Ntziachristos V, Ripoll J, Wang LV, Weissleder R. 2005 Looking and listening to light: the evolution of whole-body photonic imaging. *Nat Biotechnol* 23:313–320 Ouellet V, Tiedemann K, Mourskaia A, Fong JE, Tran-Thanh D, Amir E, Clemons M, Perbal B,

Roodman GD. (2004). Mechanisms of bone metastasis. *N Engl J Med* 15;350 (16):1655-64. Rybak LD, Rosenthal DI. (2001). Radiological imaging for the diagnosis of bone metastases.

channel system versus dual-modality PET-CT. *Eur Radiol* 17(4):939-49 Snoeks TJ, Khmelinskii A, Lelieveldt BP, Kaijzel EL, Löwik CW (2011). Optical advances in

skeletal imaging applied to bone metastases. *Bone* 48(1):106-14

therapeutic opportunities. *Nat Rev Endocrinol* 7(4):208-18

metastatic breast cancer. *Oncol Rep* 21(5):1153-9

Schmidt GP, Schoenberg SO, Schmid R, Stahl R, Tiling R, Becker CR, Reiser MF, Baur-

Suva LJ, Washam C, Nicholas RW, Griffin RJ.(2011). Bone metastasis: mechanisms and

Takenaka D, Ohno Y, Matsumoto K, Aoyama N, Onishi Y, Koyama H, Nogami M,

FDG-PET/CT, and bone scintigraphy. *J Magn Reson Imaging* 30(2):298-308 Trinkaus M, Ooi WS, Amir E, Popovic S, Kalina M, Kahn H, Singh G, Gainford MC,

Vassiliou V, Andreopoulos D, Frangos S, Tselis N, Giannopoulou E, Lutz S. (2011). Bone

Medicine Imaging Modalities. Clin *Oncol (R Coll Radiol)* [Epub ahead of print] Wahl RL, Jacene H, Kasamon Y, Lodge MA. (2009). From RECIST to PERCIST: Evolving

Welch HG, Black WC. (2010). Overdiagnosis in cáncer. *J Natl Cancer Inst*. 5;102(9):605-13 Weilbaecher KN, Guise TA, McCauley LK. (2011). Cancer to bone: a fatal attraction. *Nat Rev* 

Withofs N, Grayet B, Tancredi T, Rorive A, Mella C, Giacomelli F, et al. (2011). 18F-

Wu LM, Gu HY, Zheng J, Xu X, Lin LH, Deng X, Zhang W, Xu JR. (2011). Diagnostic value

Ye XC, Choueiri M, Tu SM, Lin SH.(2007). Biology and clinical management of prostate

review and meta-analysis. *J Magn Reson Imaging* 34(1):128-35

cancer bone metástasis. *Front Biosci* 1;12:3273-86

using targeted probes. *Proc Natl Acad Sci USA* 27;107(17):7910-5

primary breast cáncer. *Oncologist*; 16 (8):1111-9

*Q J Nucle Med* 45(1):53-64

1:122S-50S

*Cancer* 2011 Jun;11(6):411-25

*Med Commun* 32: 168-76

Swirski FK, Pittet MJ, Vinegoni C, Weissleder R.(2010). Hybrid PET-optical imaging

Compared with conventional imaging in the detection of distant metastases of

Komarova SV, Siegel PM. (2011). CCN3 impairs osteoblast and stimulates osteoclast differentiation to favor breast cancer metastasis to bone. *Am J Pathol* 178(5):2377-88

Melnyk A. (2007). Screening for bone metastases: whole-body MRI using a 32-

Yoshikawa T, Matsumoto S, Sugimura K. (2009). Detection of bone metastases in non-small cell lung cancer patients: comparison of whole-body diffusion-weighted imaging (DWI), whole-body MR imaging without and with DWI, whole-body

Clemons M. (2009). Examination of the mechanisms of osteolysis in patients with

Metastases: Assessment of Therapeutic Response through Radiological and Nuclear

Considerations for PET response criteria in solid tumors. *J Nucl Med* 50 Suppl

fluridePET/CT for assessing bone involvement in prostate and breast cancers. *Nucl* 

of whole-body magnetic resonance imaging for bone metastases: a systematic

Measurement of the proliferative behaviors of cells in vitro is important to many biomedical applications ranging from basic biological research to advanced applications, such as drug discovery, stem cell manufacturing, and tissue engineering. Critical to such measurement is accurate identification and localization of mitosis, which is the process whereby a eukaryotic cell separates the chromosomes in its cell nucleus into two identical sets in two daughter nuclei. In the early years, it is possible to manually identify incidents of mitosis because mitotic cells in culture tend to retract, round up, and exhibit intensified surrounding halos under phase contrast illumination (Fig.1) for short-period, small-scale studies. Recently, many cell proliferation assays have been developed for high throughput cell imaging and analysis. Especially, phase contrast microscopy is a superior imaging modality because it enables continuous monitoring of live cells without requiring destructive methods of cell manipulation, such as cell lysis and staining. Consequently, the need for extended-time observation and the proliferation of high-throughput imaging have made automated image analysis mandatory.

The state-of-the-art mitosis detection methods can be categorized into three classes, tracking-based, tracking-free, and hybrid approaches. Tracking-based approaches rely on cell tracking to determine individual cell trajectories, and then identify mitosis based on the temporal progression of cell features along their trajectories (Al-Kofahi et al., 2006; Bise et al., 2009; Debeir et al., 2005). The dependency on cell tracking is a severe burden because tracking *per se* is a challenging task. Tracking-free approaches alleviate this burden and can detect mitosis directly in an image sequence. One representative technique was proposed by Li et al (Li, Kanade, Chen, Miller & Campbell, 2008), which applies a cascade classifier to classify volumetric sliding windows of an image sequence with 3D Haar-like features. Major drawbacks of this approach include the requirement of a large amount of training

<sup>\*</sup>Corresponding Author

proposed framework has high generalization ability and can be straightforwardly adapted to

<sup>357</sup> A Hierarchical Framework for Mitosis Detection in Time-Lapse

The rest of the paper is structured as follows. In Section 2, we introduce the hierarchical framework for mitosis event detection. Then, the experimental methods and results are respectively detailed in Section 3 and 4. At last, the conclusion and future work are presented.

As shown in Fig.2, the proposed hierarchical framework for mitosis event detection consists of three steps: *Mitosis Candidate Extraction*, *Mitosis Identification*, and *Mitosis Localization*. We

will present the technical details of each step in the subsequent sections.

Phase Contrast Microscopy Image Sequences of Stem Cell Populations

different cell types.

**2. Mitosis detection framework**

Fig. 2. Mitosis detection framework

Fig. 1. Mitosis Sample. All the frames in one mitosis sequence are concatenated into one figure to show the visual pattern transition. One mitosis sequence can be visually divided into three stages, before mitosis, during mitosis, and after mitosis. Especially, all mitotic regions within the stage of during mitosis appear like 8 shape.

data and the lack of location specificity of detection. Hybrid approaches aim to construct a self-contained solution by leveraging the advantages of the previous two methods. These approaches typically consist of candidate sequence detection, sequence feature extraction, and classification as consecutive steps. To detect mitosis candidates, earlier methods (Eccles & Klevecz, 1986) apply thresholding and morphological filtering to extract bright halos surrounding potentially mitotic cells in each image, and then group the extracted regions in successive images based on their spatial relationship. Subsequently, to identify mitosis, Eccles et al (Eccles & Klevecz, 1986) employed a ring shape detector to locate the mother and two daughter cells; Gallardo et al (Gallardo et al., 2004) adopted a hidden Markov model to classify candidates based on temporal patterns of cell shape and appearance features; Liang et al (Liang et al., 2007) utilized conditional random field model to identify mitosis with shape and texture features in the mitotic regions. However, these methods can only recognize mitosis event without precise spatial and temporal localization during cell proliferation because of the lack of the ability to analyze the latent structure of mitosis progression.

Different from previous work simultaneously identifying and localizing mitosis event, we divided the problem of mitosis event detection into two sub-tasks: 1) mitosis identification: identifying the visual pattern transition from 0 shape mother cell to 8 shape mother cell and finally to two separate daughter cells in Fig.1; 2) mitosis localization: localizing the starting point, the first 8 shape pattern, and ending point, the last 8 shape pattern, during mitosis progression in Fig. 1. The proposed framework follows the spirit of the hybrid approach. It takes a phase contrast microscopy image sequence as input, and automatically pinpoints the time point at which mitosis occurs. First, model-based microscopy image preconditioning (Li & Kanade, 2009) and volumetric segmentation are utilized to extract spatiotemporal sub-regions in the input image sequence where mitosis potentially occurred. Then, a hidden conditional random field classifier (Quattoni et al., 2007) is applied for mitosis identification. By making significant extension on our previous work on mitosis sequence identification (Liu et al., 2010a;b), a conditional random field model (Lafferty et al., 2001) is implemented for mitosis localization. The first two stages jointly optimize the recall and precision for mitosis identification while the third stage pinpoints the time point at which mitosis occurs, minimizing the mean error and standard deviation of mitosis location. The superiority lies in three aspects: 1) nonnegative mixed-norm preconditioning method can avoid over segmentation and *ad hoc* image processing; 2) volumetric region grow method can avoid the explicit cell tracking; 3) latent contextual information can be discovered by hidden conditional random field and conditional random field classifiers for both sub-tasks. Consequently the

proposed framework has high generalization ability and can be straightforwardly adapted to different cell types.

The rest of the paper is structured as follows. In Section 2, we introduce the hierarchical framework for mitosis event detection. Then, the experimental methods and results are respectively detailed in Section 3 and 4. At last, the conclusion and future work are presented.

## **2. Mitosis detection framework**

2 Will-be-set-by-IN-TECH

Fig. 1. Mitosis Sample. All the frames in one mitosis sequence are concatenated into one figure to show the visual pattern transition. One mitosis sequence can be visually divided into three stages, before mitosis, during mitosis, and after mitosis. Especially, all mitotic

data and the lack of location specificity of detection. Hybrid approaches aim to construct a self-contained solution by leveraging the advantages of the previous two methods. These approaches typically consist of candidate sequence detection, sequence feature extraction, and classification as consecutive steps. To detect mitosis candidates, earlier methods (Eccles & Klevecz, 1986) apply thresholding and morphological filtering to extract bright halos surrounding potentially mitotic cells in each image, and then group the extracted regions in successive images based on their spatial relationship. Subsequently, to identify mitosis, Eccles et al (Eccles & Klevecz, 1986) employed a ring shape detector to locate the mother and two daughter cells; Gallardo et al (Gallardo et al., 2004) adopted a hidden Markov model to classify candidates based on temporal patterns of cell shape and appearance features; Liang et al (Liang et al., 2007) utilized conditional random field model to identify mitosis with shape and texture features in the mitotic regions. However, these methods can only recognize mitosis event without precise spatial and temporal localization during cell proliferation because of

Different from previous work simultaneously identifying and localizing mitosis event, we divided the problem of mitosis event detection into two sub-tasks: 1) mitosis identification: identifying the visual pattern transition from 0 shape mother cell to 8 shape mother cell and finally to two separate daughter cells in Fig.1; 2) mitosis localization: localizing the starting point, the first 8 shape pattern, and ending point, the last 8 shape pattern, during mitosis progression in Fig. 1. The proposed framework follows the spirit of the hybrid approach. It takes a phase contrast microscopy image sequence as input, and automatically pinpoints the time point at which mitosis occurs. First, model-based microscopy image preconditioning (Li & Kanade, 2009) and volumetric segmentation are utilized to extract spatiotemporal sub-regions in the input image sequence where mitosis potentially occurred. Then, a hidden conditional random field classifier (Quattoni et al., 2007) is applied for mitosis identification. By making significant extension on our previous work on mitosis sequence identification (Liu et al., 2010a;b), a conditional random field model (Lafferty et al., 2001) is implemented for mitosis localization. The first two stages jointly optimize the recall and precision for mitosis identification while the third stage pinpoints the time point at which mitosis occurs, minimizing the mean error and standard deviation of mitosis location. The superiority lies in three aspects: 1) nonnegative mixed-norm preconditioning method can avoid over segmentation and *ad hoc* image processing; 2) volumetric region grow method can avoid the explicit cell tracking; 3) latent contextual information can be discovered by hidden conditional random field and conditional random field classifiers for both sub-tasks. Consequently the

regions within the stage of during mitosis appear like 8 shape.

the lack of the ability to analyze the latent structure of mitosis progression.

As shown in Fig.2, the proposed hierarchical framework for mitosis event detection consists of three steps: *Mitosis Candidate Extraction*, *Mitosis Identification*, and *Mitosis Localization*. We will present the technical details of each step in the subsequent sections.

Fig. 2. Mitosis detection framework

that compensates for a nonzero background level, nonuniform illumination, and spatial sensitivity variations of the detector; and 3) a noise model *n*(·) that accounts for imaging

<sup>359</sup> A Hierarchical Framework for Mitosis Detection in Time-Lapse

with *g*(*x*, *y*) being the observed image, and *f*(*x*, *y*) being the ideal object image that we want to retrieve, which could represent the optical path length distribution in the object, fluorescence intensities, or an phenomenological image that simply facilitates object segmentation. Under the assumption of additive noise, the model above can be expressed succinctly in a linear

where *g* denotes a vectoried representation of the observed image, which is formed by concatenating the image pixels in raster order. The vectors *f* , *b*, *n* are defined likewise for the object, bias, and noise, respectively. The imaging model is expressed as a matrix-vector multiplication between the *N* × *N* transfer matrix *H* and *f* , which is adequate for representing a wide range of microscopy image formation processes. Depending on the physics of phase contrast microscopy, we choose the image formation model, *H*, defined by an effective point

where *δ*(·) is a Dirac delta function, and *α*, *β* are scaling factors. The EPSF approximates the imaging function of phase contrast optics, which accounts for the formation of halo effects

With the image model specified in Eq.2 and the assumption of additive noise, we need to compute the ideal object image *f* given an observed image *g*. This inverse problem can be tackled through a two-step process. Then an observed image which is unfriendly for computer analysis can be transformed into the one that facilitates automated object segmentation and

*a.Bias Elimination*:As the first step of preconditioning, a bias-corrected image can be obtained by estimating the bias field from an image. Under the assumption that the bias field is smooth

where *p* = (*p*0, *p*1, *p*2,...)� is the coefficient vector, and *X* is a matrix of *N* rows and (*K* +

the optimal coefficients can be computed firstly by solving the over-determined linear system

has a closed-form solution *p*<sup>∗</sup> = (*X*�*X*)−1*X*�*g*. The bias-corrected image is then computed

*Xp* <sup>=</sup> *<sup>g</sup>*. This amounts to solving the least-squares problem *<sup>p</sup>*<sup>∗</sup> <sup>=</sup> arg min*<sup>p</sup>* ||*Xp* <sup>−</sup> *<sup>g</sup>*||<sup>2</sup>

and spatially slowly varying, it can be modeled as a *K*-th order polynomial surface:

<sup>−</sup>(*x*<sup>2</sup>+*y*<sup>2</sup>)/*σ*<sup>2</sup>

*g*(*x*, *y*) = *n*(*h*(*f*(*x*, *y*)) + *b*(*x*, *y*)), (1)

*g* = *H f* + *b* + *n* ≈ *H f* + *constant*, (2)

<sup>1</sup> <sup>−</sup> *<sup>β</sup>e*−(*x*<sup>2</sup>+*y*<sup>2</sup>)/*σ*<sup>2</sup>

*b* = *Xp* (4)

*<sup>i</sup>* ,...). To eliminate the bias,

<sup>2</sup>, which

*<sup>i</sup>* , *xiyi*, *<sup>y</sup>*<sup>2</sup>

<sup>2</sup> ) (3)

and detection noise. The model can be written as:

Phase Contrast Microscopy Image Sequences of Stem Cell Populations

algebraic framework, given by

spread function (or EPSF) in (Li, 2009):

*(b) Nonnegative mixed-norm preconditioning*

1)(*K* + 2)/2 columns with the *i*-th row being (1, *xi*, *yi*, *x*<sup>2</sup>

around imaged cells.

measurement.

as *g*<sup>∗</sup> = *g* − *Xp*∗.

*EPSF*(*x*, *y*) ∝ *δ*(*x*, *y*) − (*αe*

Fig. 3. Outputs by the key steps of the proposed framework. (a) Original image sequence; (b) Outputs of image preconditioning; (c) Outputs of mitosis candidate extraction; (d) Outputs of mitosis identification; (e) Outputs of mitosis localization.

#### **2.1 Mitosis candidate extraction**

This step serves the purposes of eliminating "easily" negative regions in the image sequence where mitosis is unlikely to occur and extracting temporally continuous sub-sequences with potential mitosis to facilitate subsequent sequence classification. The algorithm consists of two sub-steps, image preconditioning and sequence segmentation.

#### **2.1.1 Image preconditioning**

Image segmentation in microscopy, especially in non-fluorescence optical microscopy modalities, is notoriously challenging due to inherent optical artifacts. Rather than treating phase contrast microscopy images as general natural images and rushing into the image processing warehouse for solutions, we study the properties of phase contrast optics to model its image formation process. Li et al's research revealed that the phase contrast imaging system can be relatively well explained by a linear imaging model (Li, 2009; Li & Kanade, 2009; Yin et al., 2010). With this model, a quadratic optimization function with sparseness and smoothness regularizations can be formulated to enhance cell regions and transforms the original image into an ideal image with zero background and nonzero foreground regions that correspond to potential mitotic cells as shown in Fig.3.(b). The detailed method contains following two steps.

#### *(a) Algebraic image model*

The generic model for microscopy images consists of three components: 1) an imaging model *h*(·) that represents the image formation process of the microscope; 2) an additive bias *b*(*x*, *y*) 4 Will-be-set-by-IN-TECH

Fig. 3. Outputs by the key steps of the proposed framework. (a) Original image sequence; (b) Outputs of image preconditioning; (c) Outputs of mitosis candidate extraction; (d) Outputs

This step serves the purposes of eliminating "easily" negative regions in the image sequence where mitosis is unlikely to occur and extracting temporally continuous sub-sequences with potential mitosis to facilitate subsequent sequence classification. The algorithm consists of

Image segmentation in microscopy, especially in non-fluorescence optical microscopy modalities, is notoriously challenging due to inherent optical artifacts. Rather than treating phase contrast microscopy images as general natural images and rushing into the image processing warehouse for solutions, we study the properties of phase contrast optics to model its image formation process. Li et al's research revealed that the phase contrast imaging system can be relatively well explained by a linear imaging model (Li, 2009; Li & Kanade, 2009; Yin et al., 2010). With this model, a quadratic optimization function with sparseness and smoothness regularizations can be formulated to enhance cell regions and transforms the original image into an ideal image with zero background and nonzero foreground regions that correspond to potential mitotic cells as shown in Fig.3.(b). The detailed method contains

The generic model for microscopy images consists of three components: 1) an imaging model *h*(·) that represents the image formation process of the microscope; 2) an additive bias *b*(*x*, *y*)

of mitosis identification; (e) Outputs of mitosis localization.

two sub-steps, image preconditioning and sequence segmentation.

**2.1 Mitosis candidate extraction**

**2.1.1 Image preconditioning**

following two steps.

*(a) Algebraic image model*

that compensates for a nonzero background level, nonuniform illumination, and spatial sensitivity variations of the detector; and 3) a noise model *n*(·) that accounts for imaging and detection noise. The model can be written as:

$$g(\mathbf{x}, y) = n(h(f(\mathbf{x}, y)) + b(\mathbf{x}, y)),\tag{1}$$

with *g*(*x*, *y*) being the observed image, and *f*(*x*, *y*) being the ideal object image that we want to retrieve, which could represent the optical path length distribution in the object, fluorescence intensities, or an phenomenological image that simply facilitates object segmentation. Under the assumption of additive noise, the model above can be expressed succinctly in a linear algebraic framework, given by

$$
\log = Hf + b + n \approx Hf + \text{constant},\tag{2}
$$

where *g* denotes a vectoried representation of the observed image, which is formed by concatenating the image pixels in raster order. The vectors *f* , *b*, *n* are defined likewise for the object, bias, and noise, respectively. The imaging model is expressed as a matrix-vector multiplication between the *N* × *N* transfer matrix *H* and *f* , which is adequate for representing a wide range of microscopy image formation processes. Depending on the physics of phase contrast microscopy, we choose the image formation model, *H*, defined by an effective point spread function (or EPSF) in (Li, 2009):

$$EPSF(\mathbf{x}, y) \propto \delta(\mathbf{x}, y) - (ae^{-(\mathbf{x}^2 + y^2)/\sigma\_1^2} - \beta e^{-(\mathbf{x}^2 + y^2)/\sigma\_2^2}) \tag{3}$$

where *δ*(·) is a Dirac delta function, and *α*, *β* are scaling factors. The EPSF approximates the imaging function of phase contrast optics, which accounts for the formation of halo effects around imaged cells.

#### *(b) Nonnegative mixed-norm preconditioning*

With the image model specified in Eq.2 and the assumption of additive noise, we need to compute the ideal object image *f* given an observed image *g*. This inverse problem can be tackled through a two-step process. Then an observed image which is unfriendly for computer analysis can be transformed into the one that facilitates automated object segmentation and measurement.

*a.Bias Elimination*:As the first step of preconditioning, a bias-corrected image can be obtained by estimating the bias field from an image. Under the assumption that the bias field is smooth and spatially slowly varying, it can be modeled as a *K*-th order polynomial surface:

$$b = \mathbf{X}p\tag{4}$$

where *p* = (*p*0, *p*1, *p*2,...)� is the coefficient vector, and *X* is a matrix of *N* rows and (*K* + 1)(*K* + 2)/2 columns with the *i*-th row being (1, *xi*, *yi*, *x*<sup>2</sup> *<sup>i</sup>* , *xiyi*, *<sup>y</sup>*<sup>2</sup> *<sup>i</sup>* ,...). To eliminate the bias, the optimal coefficients can be computed firstly by solving the over-determined linear system *Xp* <sup>=</sup> *<sup>g</sup>*. This amounts to solving the least-squares problem *<sup>p</sup>*<sup>∗</sup> <sup>=</sup> arg min*<sup>p</sup>* ||*Xp* <sup>−</sup> *<sup>g</sup>*||<sup>2</sup> <sup>2</sup>, which has a closed-form solution *p*<sup>∗</sup> = (*X*�*X*)−1*X*�*g*. The bias-corrected image is then computed as *g*<sup>∗</sup> = *g* − *Xp*∗.

incorporate long range dependencies for inference. On the other hand, different from other discriminative models, like Conditional Random Fields(CRFs) (Lafferty et al., 2001), HCRF use intermediate hidden variables to model the latent structure of the input domain and can infer a single label for the entire input sequence with the training data not explicitly labeled. Therefore, several literatures show that HCRF always outperforms HMMs and CRFs

<sup>361</sup> A Hierarchical Framework for Mitosis Detection in Time-Lapse

Mitosis identification can be formulated as the problem of predicting a label *y* given an

frame in the observation sequence and *y* is a member of a sequence-wise label set *Y* ≡ (*Mitosis*, *Non* − *mitosis*). HCRF model also consists of a vector of hidden variables h ≡

Given the definitions of the sequence-wise label *y*, the observation sequence X, the hidden

*<sup>p</sup>*(*y*, <sup>h</sup>|X, *<sup>θ</sup>*) = <sup>1</sup>

*f*1,*p*(*j*, *y*, *hj*, X)*θ*1,*<sup>p</sup>* + ∑

where *f*1,*<sup>p</sup>* and *f*2,*<sup>p</sup>* denote the *P*<sup>1</sup> node feature functions and *P*<sup>2</sup> edge feature functions respectively; *θ*1,*p*, *θ*2,*<sup>p</sup>* are the components of *θ*, corresponding to node and edge parameters. For mitosis identification, we define the two kinds of node feature functions, *f*1,1 and *f*1,2, as

*<sup>f</sup>*1,1(*j*, *hj*, <sup>X</sup>) = *<sup>φ</sup>*(x<sup>j</sup> ), *<sup>f</sup>*1,2(*j*, *hj*, *<sup>y</sup>*) = 1, if *hj* ∈ H and *<sup>y</sup>* is *Mitosis*,

where *φ*(x<sup>j</sup> ) is the feature descriptor of frame x*j*. Correspondingly, *θ*1,*<sup>p</sup>* consists of two parts, *θ*1,1 for *f*1,1 measuring the compatibility between observation x<sup>j</sup> and hidden state *hj*, and *θ*1,2 for *f*1,2 measuring the compatibility between latent variable *hj* and segment-wise label *y*. We

*<sup>f</sup>*2,1(*j*, *<sup>k</sup>*, *<sup>y</sup>*, *hj*, *hk*, <sup>X</sup>) = 1, if (*hj*, *hk*) ∈ T and *<sup>y</sup>* is *Mitosis*,

where T denotes all possible hidden state transitions from *hj* to *hk* within a mitosis sequence. Correspondingly, *θ*2,1 for *f*2,1 measures the compatibility between an edge with hidden state

*<sup>t</sup>*=<sup>1</sup> where *ht* ∈ H captures the latent structure of the input domain and H is the set of potential hidden states. A graphical representation of the HCRF model *G* = (*V*, *E*) where

> *<sup>Z</sup>* ∑ h

> > (*j*,*k*)∈*E*

*<sup>t</sup>*=1, where x*<sup>t</sup>* denotes the feature descriptor of the *t*-th

,h, X; *θ*)) is a partition function; *ψ*(*y*,h, X; *θ*) ∈ R is a

∑ *p*∈*P*<sup>2</sup>

*exp*(*ψ*(*y*,h, X; *θ*)) (8)

0, otherwise. (10)

0, otherwise. (11)

*f*2,*p*(*j*, *k*, *y*, *hj*, *hk*, X)*θ*2,*<sup>p</sup>* (9)

(Gunawardana et al., 2005; Quattoni et al., 2007; Wang et al., 2006).

Phase Contrast Microscopy Image Sequences of Stem Cell Populations

*V* denotes the vertex set and *E* denotes the edge set, is shown in Fig.4 (a).

h

*<sup>p</sup>*(*y*|X, *<sup>θ</sup>*) = ∑

<sup>∈</sup>*Y*,h∈H *exp*(*ψ*(*y*�

*j*∈*V*

also define the edge feature function as follows:

*hi* and *hj* and the segment-wise label *y*.

∑ *p*∈*P*<sup>1</sup>

potential function parameterized by *θ* as:

*ψ*(*y*,h, X; *θ*) = ∑

variables h and the model parameters *θ*, the HCRF model can be defined by:

observation sequence <sup>X</sup> ≡ {x*t*}*<sup>T</sup>*

{*ht*}*<sup>T</sup>*

where *Z* = ∑*y*�

follows:

*b.Object Reconstruction via Convex Optimization*: The second step of preconditioning reconstructs the object from the bias-corrected image. It is achieved by minimizing:

$$O(f) = ||g^\* - Hf||\_2^2 + \gamma \text{Smoothness}(f) + \beta \text{Spparity}(f), \quad \text{s.t.} f \ge 0 \tag{5}$$

where || · ||<sup>2</sup> denotes a L2 norm; the positive coefficients *γ* and *β* control the importance; the first term promotes data fidelity; the second and third terms encourage the spatial smoothness and sparseness of the reconstructed image, respectively.

By employing an L2 (Tikhonov) smoothness term (Tikhonov, 1977) and a weighted L1 sparseness term (Donoho, 2004), Eq.5 can be transformed into:

$$O\_1(f) = ||\mathfrak{g}^\* - Hf||\_2^2 + \gamma ||\mathfrak{R}f||\_2^2 + \beta ||\mathcal{W}f||\_1 \quad \text{s.t.} \, f \ge 0 \tag{6}$$

The smoothness term penalizes the L2 norm of the Laplacian of *f* , where the *N* × *N* matrix *R* represents an algebraic Laplacian operator with symmetric boundary condition. The sparseness term penalizes the weighted L1 norm of *f* , where *W* is a diagonal matrix with positive weights *w*<sup>1</sup> ... *wN* on the diagonal and zeros elsewhere.

By rewriting *O*1(*f*) in terms of the symmetric positive definite matrix *Q* = *H*�*H* + *γR*�*R* and the vector *l* = −*H*�*g*∗, and letting *w* denote the weight vector (*w*<sup>1</sup> ... *wN*), the minimization problem can be expressed as the following nonnegative-constrained quadratic program:

$$f^\* = \arg\min\_f \frac{1}{2} f^\top Q f + \left(l + \beta w\right)^\top f, \quad s.t. f \ge 0 \tag{7}$$

We utilized the simple and efficient iterative algorithm, the sparseness-enhanced multiplicative update (SEMU) algorithm (Li & Kanade, 2009), to solve the object function above.

#### **2.1.2 Sequence segmentation**

After preconditioning, 3D seeded region growing (Silvela & Portillo, 2001) is applied to the transformed image sequences to extract spatiotemporal sub-regions that correspond to candidate mitosis sequences (Fig.3(c)). The algorithm relies on two automatically-determined thresholds: a seeding threshold computed by Otsu's optimal thresholding algorithm (Otsu, 1979)is used to detect seeds; and a lower threshold determined by Rosin's unimodal thresholding algorithm (Rosin, 2001) is used as the stopping criterion of region growing.

#### **2.2 Mitosis identification**

With the extracted candidate mitosis sequences, the goal of mitosis identification is to classify the candidate mitosis sequences detected in the previous step as mitosis or not as shown in Fig.3(d). To take advantages of the spatiotemporal context and the flexible constellation of latent structure during mitosis progression, the hidden conditional random field (HCRF) classifier is utilized to model the visual pattern transition shown in Fig.1.

HCRF is a powerful discriminative model for temporal inference (Quattoni et al., 2007). Comparing to generative models, like Hidden Markov Models (HMMs) (Rabiner, 1989), HCRF does not assume observations are independent of each other and therefore can 6 Will-be-set-by-IN-TECH

*b.Object Reconstruction via Convex Optimization*: The second step of preconditioning

where || · ||<sup>2</sup> denotes a L2 norm; the positive coefficients *γ* and *β* control the importance; the first term promotes data fidelity; the second and third terms encourage the spatial smoothness

By employing an L2 (Tikhonov) smoothness term (Tikhonov, 1977) and a weighted L1

<sup>2</sup> <sup>+</sup> *<sup>γ</sup>*||*R f* ||<sup>2</sup>

The smoothness term penalizes the L2 norm of the Laplacian of *f* , where the *N* × *N* matrix *R* represents an algebraic Laplacian operator with symmetric boundary condition. The sparseness term penalizes the weighted L1 norm of *f* , where *W* is a diagonal matrix with

By rewriting *O*1(*f*) in terms of the symmetric positive definite matrix *Q* = *H*�*H* + *γR*�*R* and the vector *l* = −*H*�*g*∗, and letting *w* denote the weight vector (*w*<sup>1</sup> ... *wN*), the minimization problem can be expressed as the following nonnegative-constrained quadratic program:

We utilized the simple and efficient iterative algorithm, the sparseness-enhanced multiplicative update (SEMU) algorithm (Li & Kanade, 2009), to solve the object function

After preconditioning, 3D seeded region growing (Silvela & Portillo, 2001) is applied to the transformed image sequences to extract spatiotemporal sub-regions that correspond to candidate mitosis sequences (Fig.3(c)). The algorithm relies on two automatically-determined thresholds: a seeding threshold computed by Otsu's optimal thresholding algorithm (Otsu, 1979)is used to detect seeds; and a lower threshold determined by Rosin's unimodal thresholding algorithm (Rosin, 2001) is used as the stopping criterion of region growing.

With the extracted candidate mitosis sequences, the goal of mitosis identification is to classify the candidate mitosis sequences detected in the previous step as mitosis or not as shown in Fig.3(d). To take advantages of the spatiotemporal context and the flexible constellation of latent structure during mitosis progression, the hidden conditional random field (HCRF)

HCRF is a powerful discriminative model for temporal inference (Quattoni et al., 2007). Comparing to generative models, like Hidden Markov Models (HMMs) (Rabiner, 1989), HCRF does not assume observations are independent of each other and therefore can

classifier is utilized to model the visual pattern transition shown in Fig.1.

<sup>2</sup> + *γSmoothness*(*f*) + *βSparsity*(*f*), *s*.*t*. *f* ≥ 0 (5)

<sup>2</sup> + *β*||*W f* ||1, *s*.*t*. *f* ≥ 0 (6)

<sup>2</sup> *<sup>f</sup>* �*Q f* + (*<sup>l</sup>* <sup>+</sup> *<sup>β</sup>w*)� *<sup>f</sup>* , *<sup>s</sup>*.*t*. *<sup>f</sup>* <sup>≥</sup> <sup>0</sup> (7)

reconstructs the object from the bias-corrected image. It is achieved by minimizing:

*<sup>O</sup>*(*f*) = ||*g*<sup>∗</sup> <sup>−</sup> *H f* ||<sup>2</sup>

and sparseness of the reconstructed image, respectively.

sparseness term (Donoho, 2004), Eq.5 can be transformed into:

positive weights *w*<sup>1</sup> ... *wN* on the diagonal and zeros elsewhere.

*<sup>f</sup>* <sup>∗</sup> <sup>=</sup> arg min*<sup>f</sup>*

above.

**2.1.2 Sequence segmentation**

**2.2 Mitosis identification**

1

*<sup>O</sup>*1(*f*) = ||*g*<sup>∗</sup> <sup>−</sup> *H f* ||<sup>2</sup>

incorporate long range dependencies for inference. On the other hand, different from other discriminative models, like Conditional Random Fields(CRFs) (Lafferty et al., 2001), HCRF use intermediate hidden variables to model the latent structure of the input domain and can infer a single label for the entire input sequence with the training data not explicitly labeled. Therefore, several literatures show that HCRF always outperforms HMMs and CRFs (Gunawardana et al., 2005; Quattoni et al., 2007; Wang et al., 2006).

Mitosis identification can be formulated as the problem of predicting a label *y* given an observation sequence <sup>X</sup> ≡ {x*t*}*<sup>T</sup> <sup>t</sup>*=1, where x*<sup>t</sup>* denotes the feature descriptor of the *t*-th frame in the observation sequence and *y* is a member of a sequence-wise label set *Y* ≡ (*Mitosis*, *Non* − *mitosis*). HCRF model also consists of a vector of hidden variables h ≡ {*ht*}*<sup>T</sup> <sup>t</sup>*=<sup>1</sup> where *ht* ∈ H captures the latent structure of the input domain and H is the set of potential hidden states. A graphical representation of the HCRF model *G* = (*V*, *E*) where *V* denotes the vertex set and *E* denotes the edge set, is shown in Fig.4 (a).

Given the definitions of the sequence-wise label *y*, the observation sequence X, the hidden variables h and the model parameters *θ*, the HCRF model can be defined by:

$$p(y|\mathbf{X},\boldsymbol{\theta}) = \sum\_{h} p(y,h|\mathbf{X},\boldsymbol{\theta}) = \frac{1}{Z} \sum\_{h} \exp(\psi(y,h,\mathbf{X};\boldsymbol{\theta})) \tag{8}$$

where *Z* = ∑*y*� <sup>∈</sup>*Y*,h∈H *exp*(*ψ*(*y*� ,h, X; *θ*)) is a partition function; *ψ*(*y*,h, X; *θ*) ∈ R is a potential function parameterized by *θ* as:

$$\psi(y,\boldsymbol{h},\mathbf{X};\boldsymbol{\theta}) = \sum\_{j \in V} \sum\_{p \in P\_1} f\_{1,p}(j, y, h\_j, \mathbf{X}) \theta\_{1,p} + \sum\_{(j,k) \in E} \sum\_{p \in P\_2} f\_{2,p}(j, k, y, h\_j, h\_k, \mathbf{X}) \theta\_{2,p} \tag{9}$$

where *f*1,*<sup>p</sup>* and *f*2,*<sup>p</sup>* denote the *P*<sup>1</sup> node feature functions and *P*<sup>2</sup> edge feature functions respectively; *θ*1,*p*, *θ*2,*<sup>p</sup>* are the components of *θ*, corresponding to node and edge parameters.

For mitosis identification, we define the two kinds of node feature functions, *f*1,1 and *f*1,2, as follows:

$$f\_{1,1}(j\_\prime h\_{\dot{j}\prime} \mathbf{X}) = \phi(\mathbf{x}\_{\dot{j}}), \quad f\_{1,2}(j\_\prime h\_{\dot{j}\prime} \mathbf{y}) = \begin{cases} 1, \text{if } h\_{\dot{j}} \in \mathcal{H} \text{ and } \boldsymbol{y} \text{ is } \text{Mitsis}, \\ 0, \text{otherwise.} \end{cases} \tag{10}$$

where *φ*(x<sup>j</sup> ) is the feature descriptor of frame x*j*. Correspondingly, *θ*1,*<sup>p</sup>* consists of two parts, *θ*1,1 for *f*1,1 measuring the compatibility between observation x<sup>j</sup> and hidden state *hj*, and *θ*1,2 for *f*1,2 measuring the compatibility between latent variable *hj* and segment-wise label *y*. We also define the edge feature function as follows:

$$f\_{2,1}(j,k,y,h\_j,h\_{k'}\mathbf{X}) = \begin{cases} 1, \text{if } (h\_j, h\_k) \in \mathcal{T} \text{ and } y \text{ is } M\text{its}\text{is},\\ 0, \text{otherwise.} \end{cases} \tag{11}$$

where T denotes all possible hidden state transitions from *hj* to *hk* within a mitosis sequence. Correspondingly, *θ*2,1 for *f*2,1 measures the compatibility between an edge with hidden state *hi* and *hj* and the segment-wise label *y*.

annotation for accurate mitosis localization. Furthermore, CRF can incorporate long range dependencies between observations and their labels without the assumption of generative dynamic Bayesian network models that the observations are independent given the values of

<sup>363</sup> A Hierarchical Framework for Mitosis Detection in Time-Lapse

The task of mitosis localization can be considered as the problem of predicting a sequence of

frame-wise label set L of all possible labels. For one mitosis event, we designate L contains three members, before mitosis (BM), during mitosis (DM), after mitosis (AM). Therefore, the first and the last frames of a segment with label DM within one mitosis sequence respectively correspond to the start and end of mitosis event. CRF is an undirected probabilistic graphical

Given the definitions of the frame-wise label sequence L, the observation sequence X, and

1,*q*(*j*, *lj*, X)*γ*1,*<sup>q</sup>* + ∑

respectively; *γ*1,*<sup>q</sup>* and *γ*2,*<sup>q</sup>* are the components of *γ*, corresponding to node and edge

between observation *xj* and frame-wise label *lj*. We also define the edge feature function as

2,1(*j*, *<sup>k</sup>*, *lj*, *lk*, <sup>X</sup>) = 1, if (*lj*, *lk*) ∈ T �

where T � denotes all possible state transitions from *lj* to *lk* within a mitosis sequence and *γ*2,1 measures the compatibility between an edge linking label *lj* and *lk* and the frame *xj* and *xk*.

Given *Tr*� = {(X1, L1),(X2, L2),...,(XM, LM)} be a set of training examples, the model

where *M* is the total number of training sequences. The first term in the objective function is the log-likelihood. The second term denotes the Gaussian prior with variance *σ*2. Learning the parameters of a CRF model is a convex problem so that the global optimality *γ*∗ = arg max<sup>γ</sup> *L*(*γ*) can be guaranteed and obtained with gradient ascent algorithm, such as BFGS

log *<sup>p</sup>*(Li|Xi, *<sup>γ</sup>*) <sup>−</sup> <sup>1</sup>

*<sup>t</sup>*=<sup>1</sup> , where *lt* is a member of a

2,*q*(*j*, *k*, *lj*, *lk*, X)*γ*2,*<sup>q</sup>* (15)

1,1 measures the compatibility

*<sup>Z</sup>*� *exp*(*ϕ*(L, <sup>X</sup>; *<sup>γ</sup>*)) (14)

, X; *γ*)) is a partition function; *ϕ*(L, X; *γ*) ∈ R is a potential

∑ *q*∈*Q*<sup>2</sup> *f* �

1,1(*j*, *lj*, X) = *φ*(x<sup>j</sup> ) (16)

,

0, otherwise. (17)

<sup>2</sup>*σ*<sup>2</sup> ||*γ*||<sup>2</sup> (18)

(*j*,*k*)∈*E*�

2,*<sup>q</sup>* denote the *Q*<sup>1</sup> node feature functions and *Q*<sup>2</sup> edge feature functions

*<sup>t</sup>*=<sup>1</sup> given an observation sequence <sup>X</sup> <sup>=</sup> {x*t*}*<sup>T</sup>*

the model parameters *γ*, the CRF model can be mathematically formulated by:

*<sup>p</sup>*(L|X, *<sup>γ</sup>*) = <sup>1</sup>

*f* �

)) as shown in Fig.4 (b).

Phase Contrast Microscopy Image Sequences of Stem Cell Populations

hidden variables.

labels <sup>L</sup> <sup>=</sup> {*lt*}*<sup>T</sup>*

model (*G*� = (*V*�

where *f* �

follows:

, *E*�

where *Z*� = ∑<sup>L</sup> *exp*(*ϕ*(L

1,*<sup>q</sup>* and *f* �

function parameterized by *γ* as:

*ϕ*(L, X; *γ*) = ∑

*j*∈*V*�

∑ *q*∈*Q*<sup>1</sup> *f* �

parameters. We define the node feature function as follows:

where *φ*(x<sup>j</sup> ) is the feature descriptor of frame *xj* and *γ*1,1 for *f* �

parameters can be learned by optimizing the objective function:

*M* ∑ *i*=1

*L*(*γ*) =

*f* �

Given *Tr* = {(X1, *y*1),(X2, *y*2),...,(XN, *yN*)} be a set of training examples, the model parameters can be learned by optimizing the objective function (Lafferty et al., 2001):

$$L(\theta) = \sum\_{i=1}^{N} \log p(y\_i | \mathbf{X}\_{i\prime}\theta) - \frac{1}{2\sigma^2} ||\theta||^2 \tag{12}$$

where *N* is the total number of training sequences. The first term in Eq.12 is the log-likelihood of the data; the second term is the log of a Gaussion prior with variance *<sup>σ</sup>*2, i.e., *<sup>p</sup>*(*θ*) <sup>∼</sup> *exp*( <sup>1</sup> <sup>2</sup>*σ*<sup>2</sup> ||*θ*||2). The optimal model parameter *<sup>θ</sup>*<sup>∗</sup> <sup>=</sup> arg max*<sup>θ</sup> <sup>L</sup>*(*θ*) can be obtained with gradient ascent algorithm, such as BFGS (Avriel, 2003). Given an unseen test sequence X, the best corresponding label *y*∗ can be computed by

$$y^\* = \arg\max\_y p(y|\mathbf{X}, \theta^\*)\tag{13}$$

Fig. 4. A comparison of graphical structure. (a)HCRF; (b)CRF

#### **2.3 Mitosis localization**

Comparing to previous work only focusing on coarse level mitosis localization (Gallardo et al., 2004; Liang et al., 2007), it is more important to localize the start and end of mitosis event in spatio-temporal domain which will benefit real-time quantitative analysis of mitotic rate and the correlation between the mother cell and the daughter cells during cell division (Li, Miller, Chen, Kanade, Weiss & Campbell, 2008). Different from our previous work only considering the start and end of mitosis as a spatial visual pattern like 8 shape (Liu et al., 2010b), we regard mitosis progression as the spatio-temporal visual pattern changes from 0 shape mother cell to 8 shape cell and finally to two separate daughter cells. Consequently we propose to model the visual pattern transition between adjacent frames with Conditional Random Model (CRF), the most notably discriminative model proposed by Lafferty et al(Lafferty et al., 2001), to localize the start and end of one mitosis event as shown in Fig.3(e). Comparing to Support Vector Machine (Liu et al., 2010b) using only spatial information for mitotic cell modeling, CRF can take good advantages of spatio-temporal context to refine the frame-wise 8 Will-be-set-by-IN-TECH

Given *Tr* = {(X1, *y*1),(X2, *y*2),...,(XN, *yN*)} be a set of training examples, the model

where *N* is the total number of training sequences. The first term in Eq.12 is the log-likelihood of the data; the second term is the log of a Gaussion prior with variance *<sup>σ</sup>*2, i.e., *<sup>p</sup>*(*θ*) <sup>∼</sup>

Comparing to previous work only focusing on coarse level mitosis localization (Gallardo et al., 2004; Liang et al., 2007), it is more important to localize the start and end of mitosis event in spatio-temporal domain which will benefit real-time quantitative analysis of mitotic rate and the correlation between the mother cell and the daughter cells during cell division (Li, Miller, Chen, Kanade, Weiss & Campbell, 2008). Different from our previous work only considering the start and end of mitosis as a spatial visual pattern like 8 shape (Liu et al., 2010b), we regard mitosis progression as the spatio-temporal visual pattern changes from 0 shape mother cell to 8 shape cell and finally to two separate daughter cells. Consequently we propose to model the visual pattern transition between adjacent frames with Conditional Random Model (CRF), the most notably discriminative model proposed by Lafferty et al(Lafferty et al., 2001), to localize the start and end of one mitosis event as shown in Fig.3(e). Comparing to Support Vector Machine (Liu et al., 2010b) using only spatial information for mitotic cell modeling, CRF can take good advantages of spatio-temporal context to refine the frame-wise

log *<sup>p</sup>*(*yi*|X*i*, *<sup>θ</sup>*) <sup>−</sup> <sup>1</sup>

<sup>2</sup>*σ*<sup>2</sup> ||*θ*||2). The optimal model parameter *<sup>θ</sup>*<sup>∗</sup> <sup>=</sup> arg max*<sup>θ</sup> <sup>L</sup>*(*θ*) can be obtained with gradient ascent algorithm, such as BFGS (Avriel, 2003). Given an unseen test sequence X,

<sup>2</sup>*σ*<sup>2</sup> ||*θ*||<sup>2</sup> (12)

*<sup>y</sup>*<sup>∗</sup> <sup>=</sup> arg max*<sup>y</sup> <sup>p</sup>*(*y*|X, *<sup>θ</sup>*∗) (13)

parameters can be learned by optimizing the objective function (Lafferty et al., 2001):

*L*(*θ*) =

Fig. 4. A comparison of graphical structure. (a)HCRF; (b)CRF

**2.3 Mitosis localization**

the best corresponding label *y*∗ can be computed by

*exp*( <sup>1</sup>

*N* ∑ *i*=1 annotation for accurate mitosis localization. Furthermore, CRF can incorporate long range dependencies between observations and their labels without the assumption of generative dynamic Bayesian network models that the observations are independent given the values of hidden variables.

The task of mitosis localization can be considered as the problem of predicting a sequence of labels <sup>L</sup> <sup>=</sup> {*lt*}*<sup>T</sup> <sup>t</sup>*=<sup>1</sup> given an observation sequence <sup>X</sup> <sup>=</sup> {x*t*}*<sup>T</sup> <sup>t</sup>*=<sup>1</sup> , where *lt* is a member of a frame-wise label set L of all possible labels. For one mitosis event, we designate L contains three members, before mitosis (BM), during mitosis (DM), after mitosis (AM). Therefore, the first and the last frames of a segment with label DM within one mitosis sequence respectively correspond to the start and end of mitosis event. CRF is an undirected probabilistic graphical model (*G*� = (*V*� , *E*� )) as shown in Fig.4 (b).

Given the definitions of the frame-wise label sequence L, the observation sequence X, and the model parameters *γ*, the CRF model can be mathematically formulated by:

$$p(\mathbf{L}|\mathbf{X},\gamma) = \frac{1}{Z'} \exp(\varrho(\mathbf{L},\mathbf{X};\gamma))\tag{14}$$

where *Z*� = ∑<sup>L</sup> *exp*(*ϕ*(L, X; *γ*)) is a partition function; *ϕ*(L, X; *γ*) ∈ R is a potential function parameterized by *γ* as:

$$\varrho(\mathbf{L}, \mathbf{X}; \gamma) = \sum\_{j \in V'} \sum\_{q \in Q\_1} f'\_{1,q}(j, l\_j, \mathbf{X}) \gamma\_{1,q} + \sum\_{(j,k) \in E'} \sum\_{q \in Q\_2} f'\_{2,q}(j, k, l\_j, l\_{k'}, \mathbf{X}) \gamma\_{2,q} \tag{15}$$

where *f* � 1,*<sup>q</sup>* and *f* � 2,*<sup>q</sup>* denote the *Q*<sup>1</sup> node feature functions and *Q*<sup>2</sup> edge feature functions respectively; *γ*1,*<sup>q</sup>* and *γ*2,*<sup>q</sup>* are the components of *γ*, corresponding to node and edge parameters. We define the node feature function as follows:

$$(f'\_{1,1}(\mathbf{j}\_{\prime}l\_{\mathbf{j}\prime}\mathbf{X}) = \boldsymbol{\phi}(\mathbf{x}\_{\mathbf{j}\prime})\tag{16}$$

where *φ*(x<sup>j</sup> ) is the feature descriptor of frame *xj* and *γ*1,1 for *f* � 1,1 measures the compatibility between observation *xj* and frame-wise label *lj*. We also define the edge feature function as follows:

$$f\_{2,1}'(j,k\_{\prime}l\_{j\prime}l\_{k\prime}\mathbf{X}) = \begin{cases} 1, \text{if } (l\_{j\prime}l\_k) \in \mathcal{T}^{\prime}, \\ 0, \text{otherwise}. \end{cases} \tag{17}$$

where T � denotes all possible state transitions from *lj* to *lk* within a mitosis sequence and *γ*2,1 measures the compatibility between an edge linking label *lj* and *lk* and the frame *xj* and *xk*.

Given *Tr*� = {(X1, L1),(X2, L2),...,(XM, LM)} be a set of training examples, the model parameters can be learned by optimizing the objective function:

$$L(\gamma) = \sum\_{i=1}^{M} \log p(L\_i | X\_{i\nu} \gamma) - \frac{1}{2\sigma^2} ||\gamma||^2 \tag{18}$$

where *M* is the total number of training sequences. The first term in the objective function is the log-likelihood. The second term denotes the Gaussian prior with variance *σ*2. Learning the parameters of a CRF model is a convex problem so that the global optimality *γ*∗ = arg max<sup>γ</sup> *L*(*γ*) can be guaranteed and obtained with gradient ascent algorithm, such as BFGS

**3.3 Experiments**

classifier as we did previously (Liu et al., 2010b).

at time t. To select the best window size *w*

truth.

**4. Experimental results**

**4.1 Mitosis identification**

*(a)Mitosis Identification*: The proposed model-based microscopy image preconditioning and sequence segmentation methods are implemented on the original image sequences to extract mitotic candidate sequences. Then, an HCRF classifier is learned with manually labeled mitotic and non-mitotic candidate sequences by the previous step for mitosis identification. In HCRF model, we can incorporate long range dependencies controlled by a sequence window length *w* which defines the amount of past and future observations to be used when predicting the state at time t (*w* = 0 indicates only the current observation is used). To select the best window size *w*, we tuned this parameter and trained the corresponding HCRF classifiers to compare their performances with Area Under Curve (AUC) of ROC curve. After obtaining the best window size, we randomly selected one image sequence as training set and tested on the other four image sequences. Four different outcomes of the mitosis identification algorithm are examined: 1) true positive (TP): the identified sequence contains one mitosis event; 2) false positive (FP): the identified sequence does not contain any mitosis event; 3) true negative (TN): the discarded sequence does not contain any mitosis event; 4) false negative (FN): the discarded sequence contains mitosis events. Then, Precision (the ratio of TP to TP+FP) and recall (the ratio of TP to TP+FN), and the F1 score (a harmonic mean of precision and recall) are used as quantitative metrics for measuring the accuracy of mitosis identification. To evaluate the performance of the proposed method, it is quantitatively compared with two different methods: 1) to show the superiority of HCRF for sequence classification, it is compare to CRF model (Liang et al., 2007) which can not discover the latent structure of the entire sequence for classification; 2) to validate the advantage of integrating temporal information, it is also compared to a frame-by-frame classification approach using a support vector machine (SVM)

<sup>365</sup> A Hierarchical Framework for Mitosis Detection in Time-Lapse

Phase Contrast Microscopy Image Sequences of Stem Cell Populations

*(b)Mitosis Localization*: With the identified mitosis sequences, we manually annotate the three states, before mitosis, during mitosis, and after mitosis, with label 1, 2, and 3 correspondingly. Then, a CRF classifier is trained for mitosis localization. In CRF model, we can also incorporate long range dependencies with a sequence window length *w* which defines the amount of past and future observations to be used when predicting the state

corresponding CRF classifiers to compare their performances with Area Under Curve of ROC curve. After getting the best window size, we selected the same training set for model learning and tested on the other four image sequences. To evaluate the accuracy of mitosis localization, we calculated the mean and standard deviation (SD) of the timing error of the start and end of the state of during mitosis. The timing error is defined as the absolute difference of the first/last frame with label 2 by the proposed method against the manually-labeled ground

With preconditioning and volumetric region growing, we extracted candidate mitosis sequences in each input sequence. This step achieved 100% recall with relatively low precision around 42%. To improve precision, we trained HCRF models with GIST features to refine the identification results. To optimize the performance of HCRF model, 4 HCRF classifiers were

, we tuned this parameter and trained the

(Avriel, 2003). Given a testing sequence X, with the optimal model parameters *γ*∗, the label L∗ of the input sequence is

$$L^\* = \arg\max\_L p(L|\mathbf{X}, \gamma^\*)\tag{19}$$

#### **3. Experimental method**

#### **3.1 Data**

Five phase contrast image sequences of C3H10T1/2 mouse mesenchymal stem cell populations (American Type Culture Collection, Manassas, VA) were acquired, each containing 1000 images. The multipotent C3H10T1/2 stem cells serve as a model for the adult human mesenchymal stem cells that can differentiate into a variety of cell types. The growing environment consists of Dulbecco's Modified Eagle's Media (DMEM; Invitrogen, Carlsbad, CA), 10% fetal bovine serum (Invitrogen, Carlsbad, CA) and 1% penicillin streptomycin (PS; Invitrogen, Carlsbad, CA). The cells were observed during growth under a Zeiss Axiovert 135TV inverted microscope with a 5×, 0.15 N.A. objective and phase contrast optics. Time-lapse image acquisition was performed every 5 minutes using a 12-bit Qimaging Retiga EXi Fast 1394 CCD camera at 500ms exposure with a gain of 1.01. Each image consists of 1392×1040 pixels with a resolution of 19 *μ*m/pixel.

#### **3.2 Feature representation**

From Fig.1, it is obvious that mitotic regions typically do not have rich features, and their shapes and appearances are highly irregular and flexible. To represent the feature of each frame within one mitosis sequence, we computed the GIST descriptor for each frame to formulate *φ*(xi). GIST is a well-known visual feature for image classification and can represent all levels of processing, from low-level features (e.g., color, spatial frequencies) to intermediate image properties (e.g., surface, volume) and high-level information (e.g., objects, activation of semantic knowledge) (Oliva & Torralba, 2001). The superiorities of GIST for mitotic region description lie in three aspects: 1) GIST representation can bypass the segmentation of individual objects. Precise cell region segmentation in phase contrast microscopy is a notoriously difficult problem due to the phenomena of halo or shade-off. Consequently, shape features, like Histogram of Oriented Gradients (Dalal & Triggs, 2005) and Shape Context (Belongie et al., 2002), are usually not satisfactory descriptors for this task. Comparatively, the computation of GIST only uses spectral and coarsely localized information and therefore formulates a holistic representation of entire image without requirement to precisely extract edge or boundary in the image; 2) GIST is invariant to scale and rotation. Mitotic regions usually experience drastic changes in both size and orientation during mitosis progression. GIST physically represents the dominant spatial structure of an image including a set of perceptual dimensions, naturalness, openness, roughness, expansion, and ruggedness. Therefore GIST is a robust descriptor for mitotic regions; 3) GIST is a compact descriptor. GIST can be formulated from localized energy spectrum (spectrogram) with both intensity and local structure information. Due to the high dimension and redundancy of spectrogram, dimensionality reduction of Karhunen-Loeve Transform (KLT) is implemented on it to formulate a compact representation.

#### **3.3 Experiments**

10 Will-be-set-by-IN-TECH

(Avriel, 2003). Given a testing sequence X, with the optimal model parameters *γ*∗, the label

Five phase contrast image sequences of C3H10T1/2 mouse mesenchymal stem cell populations (American Type Culture Collection, Manassas, VA) were acquired, each containing 1000 images. The multipotent C3H10T1/2 stem cells serve as a model for the adult human mesenchymal stem cells that can differentiate into a variety of cell types. The growing environment consists of Dulbecco's Modified Eagle's Media (DMEM; Invitrogen, Carlsbad, CA), 10% fetal bovine serum (Invitrogen, Carlsbad, CA) and 1% penicillin streptomycin (PS; Invitrogen, Carlsbad, CA). The cells were observed during growth under a Zeiss Axiovert 135TV inverted microscope with a 5×, 0.15 N.A. objective and phase contrast optics. Time-lapse image acquisition was performed every 5 minutes using a 12-bit Qimaging Retiga EXi Fast 1394 CCD camera at 500ms exposure with a gain of 1.01. Each image consists of

From Fig.1, it is obvious that mitotic regions typically do not have rich features, and their shapes and appearances are highly irregular and flexible. To represent the feature of each frame within one mitosis sequence, we computed the GIST descriptor for each frame to formulate *φ*(xi). GIST is a well-known visual feature for image classification and can represent all levels of processing, from low-level features (e.g., color, spatial frequencies) to intermediate image properties (e.g., surface, volume) and high-level information (e.g., objects, activation of semantic knowledge) (Oliva & Torralba, 2001). The superiorities of GIST for mitotic region description lie in three aspects: 1) GIST representation can bypass the segmentation of individual objects. Precise cell region segmentation in phase contrast microscopy is a notoriously difficult problem due to the phenomena of halo or shade-off. Consequently, shape features, like Histogram of Oriented Gradients (Dalal & Triggs, 2005) and Shape Context (Belongie et al., 2002), are usually not satisfactory descriptors for this task. Comparatively, the computation of GIST only uses spectral and coarsely localized information and therefore formulates a holistic representation of entire image without requirement to precisely extract edge or boundary in the image; 2) GIST is invariant to scale and rotation. Mitotic regions usually experience drastic changes in both size and orientation during mitosis progression. GIST physically represents the dominant spatial structure of an image including a set of perceptual dimensions, naturalness, openness, roughness, expansion, and ruggedness. Therefore GIST is a robust descriptor for mitotic regions; 3) GIST is a compact descriptor. GIST can be formulated from localized energy spectrum (spectrogram) with both intensity and local structure information. Due to the high dimension and redundancy of spectrogram, dimensionality reduction of Karhunen-Loeve Transform (KLT) is implemented

<sup>L</sup><sup>∗</sup> <sup>=</sup> arg max <sup>L</sup> *<sup>p</sup>*(L|X, *<sup>γ</sup>*∗) (19)

L∗ of the input sequence is

**3. Experimental method**

**3.2 Feature representation**

1392×1040 pixels with a resolution of 19 *μ*m/pixel.

on it to formulate a compact representation.

**3.1 Data**

*(a)Mitosis Identification*: The proposed model-based microscopy image preconditioning and sequence segmentation methods are implemented on the original image sequences to extract mitotic candidate sequences. Then, an HCRF classifier is learned with manually labeled mitotic and non-mitotic candidate sequences by the previous step for mitosis identification. In HCRF model, we can incorporate long range dependencies controlled by a sequence window length *w* which defines the amount of past and future observations to be used when predicting the state at time t (*w* = 0 indicates only the current observation is used). To select the best window size *w*, we tuned this parameter and trained the corresponding HCRF classifiers to compare their performances with Area Under Curve (AUC) of ROC curve. After obtaining the best window size, we randomly selected one image sequence as training set and tested on the other four image sequences. Four different outcomes of the mitosis identification algorithm are examined: 1) true positive (TP): the identified sequence contains one mitosis event; 2) false positive (FP): the identified sequence does not contain any mitosis event; 3) true negative (TN): the discarded sequence does not contain any mitosis event; 4) false negative (FN): the discarded sequence contains mitosis events. Then, Precision (the ratio of TP to TP+FP) and recall (the ratio of TP to TP+FN), and the F1 score (a harmonic mean of precision and recall) are used as quantitative metrics for measuring the accuracy of mitosis identification. To evaluate the performance of the proposed method, it is quantitatively compared with two different methods: 1) to show the superiority of HCRF for sequence classification, it is compare to CRF model (Liang et al., 2007) which can not discover the latent structure of the entire sequence for classification; 2) to validate the advantage of integrating temporal information, it is also compared to a frame-by-frame classification approach using a support vector machine (SVM) classifier as we did previously (Liu et al., 2010b).

*(b)Mitosis Localization*: With the identified mitosis sequences, we manually annotate the three states, before mitosis, during mitosis, and after mitosis, with label 1, 2, and 3 correspondingly. Then, a CRF classifier is trained for mitosis localization. In CRF model, we can also incorporate long range dependencies with a sequence window length *w* which defines the amount of past and future observations to be used when predicting the state at time t. To select the best window size *w* , we tuned this parameter and trained the corresponding CRF classifiers to compare their performances with Area Under Curve of ROC curve. After getting the best window size, we selected the same training set for model learning and tested on the other four image sequences. To evaluate the accuracy of mitosis localization, we calculated the mean and standard deviation (SD) of the timing error of the start and end of the state of during mitosis. The timing error is defined as the absolute difference of the first/last frame with label 2 by the proposed method against the manually-labeled ground truth.

## **4. Experimental results**

#### **4.1 Mitosis identification**

With preconditioning and volumetric region growing, we extracted candidate mitosis sequences in each input sequence. This step achieved 100% recall with relatively low precision around 42%. To improve precision, we trained HCRF models with GIST features to refine the identification results. To optimize the performance of HCRF model, 4 HCRF classifiers were

0 0.2 0.4 0.6 0.8 1

Seq1: AUC = 0.92 Seq2: AUC = 0.93 Seq3: AUC = 0.91 Seq4: AUC = 0.86

False positive rate

mitosis identification, it is first applied to label the full sequence. Then, a candidate sequence is classified as mitosis if the number of frames assigned with labels 2 (during mitosis) is greater than a certain threshold. By varying the threshold, we plot the ROC curves to evaluate the performance of CRF models and obtained the optimal AUC of 0.78. Moreover, to show the advantage of integrating temporal information, we compare its performance to a frame-by-frame classification approach using a support vector machine (SVM) classifier. We implemented a mitotic cell detector using SVM with a radial basis function (RBF) kernel. The best parameters for the SVM models were selected by cross validation. The detector was applied independently to each frame of a candidate sequence. Different from the training strategy of CRF, we labeled the frames during mitosis as positive samples, and the other frames as negative samples. A candidate sequence is classified as mitosis if the number of frames assigned to be mitotic exceeds a certain threshold. By varying the threshold, we plot the ROC curves to evaluate the performance of SVM models and obtained the optimal AUC

To compare the overall classification performances of HCRF, CRF and SVM trained with GIST features, we utilize the balanced F score as a complementary metric to AUC. We separately computed the AUC and the best achievable F score of each sequence with each classifier. From Table.2, the results indicate that the HCRF classifier consistently outperformed both CRF and

Fig. 6. ROC of HCRF with GIST feature and window size = 2

HCRF:WindowSize=2;GIST

<sup>367</sup> A Hierarchical Framework for Mitosis Detection in Time-Lapse

Phase Contrast Microscopy Image Sequences of Stem Cell Populations

0

of 0.77.

0.1

0.2

0.3

0.4

0.5

True positive rate

0.6

0.7

0.8

0.9

1

learned by varying window size from 0 to 3. From Fig.5, the optimal window size can be set with 2 (AUC=0.929) by plotting the ROC curve of each model and comparing the AUC values.

Fig. 5. ROC curves of HCRF models with different sequence window sizes from 0 to 3.

Thereafter, the HCRF classifier was trained with GIST feature and window size 2 on the training set. The ROC curves of the test results on the four test sequences are shown in Fig.6. By choosing the threshold corresponding to the maximum F score, we identify the mitosis events in each input sequence. Fig.7 shows the four kinds of samples decided by our method. From Table 1, we can achieve an average precision of 0.83 and an average recall of o.92 with a corresponding maximum F score of 0.88.


Table 1. Precision and recall of mitosis identification by HCRF

To demonstrate the superiority of HCRF for sequence classification, we compared its performance to the CRF model trained with fully-labeled sequences. To utilize CRF for 12 Will-be-set-by-IN-TECH

learned by varying window size from 0 to 3. From Fig.5, the optimal window size can be set with 2 (AUC=0.929) by plotting the ROC curve of each model and comparing the AUC values.

ROC of HCRF(GIST)

0 0.2 0.4 0.6 0.8 1

WindowSize = 0 & AUC = 0.907 WindowSize = 1 & AUC = 0.899 WindowSize = 2 & AUC = 0.929 WindowSize = 3 & AUC = 0.912

False positive rate

Thereafter, the HCRF classifier was trained with GIST feature and window size 2 on the training set. The ROC curves of the test results on the four test sequences are shown in Fig.6. By choosing the threshold corresponding to the maximum F score, we identify the mitosis events in each input sequence. Fig.7 shows the four kinds of samples decided by our method. From Table 1, we can achieve an average precision of 0.83 and an average recall of o.92 with a

> Sequence No. Precision (%) Recall (%) *MaximumF* (%) 1 85 95 90 2 83 90 87 3 79 94 86 4 84 90 87 Average 83 92 88

To demonstrate the superiority of HCRF for sequence classification, we compared its performance to the CRF model trained with fully-labeled sequences. To utilize CRF for

Fig. 5. ROC curves of HCRF models with different sequence window sizes from 0 to 3.

0

corresponding maximum F score of 0.88.

Table 1. Precision and recall of mitosis identification by HCRF

0.1

0.2

0.3

0.4

0.5

True positive rate

0.6

0.7

0.8

0.9

1

Fig. 6. ROC of HCRF with GIST feature and window size = 2

mitosis identification, it is first applied to label the full sequence. Then, a candidate sequence is classified as mitosis if the number of frames assigned with labels 2 (during mitosis) is greater than a certain threshold. By varying the threshold, we plot the ROC curves to evaluate the performance of CRF models and obtained the optimal AUC of 0.78. Moreover, to show the advantage of integrating temporal information, we compare its performance to a frame-by-frame classification approach using a support vector machine (SVM) classifier. We implemented a mitotic cell detector using SVM with a radial basis function (RBF) kernel. The best parameters for the SVM models were selected by cross validation. The detector was applied independently to each frame of a candidate sequence. Different from the training strategy of CRF, we labeled the frames during mitosis as positive samples, and the other frames as negative samples. A candidate sequence is classified as mitosis if the number of frames assigned to be mitotic exceeds a certain threshold. By varying the threshold, we plot the ROC curves to evaluate the performance of SVM models and obtained the optimal AUC of 0.77.

To compare the overall classification performances of HCRF, CRF and SVM trained with GIST features, we utilize the balanced F score as a complementary metric to AUC. We separately computed the AUC and the best achievable F score of each sequence with each classifier. From Table.2, the results indicate that the HCRF classifier consistently outperformed both CRF and

0 0.2 0.4 0.6 0.8 1

WinodwSize = 0 & AUC = 0.916 WinodwSize = 1 & AUC = 0.933 WinodwSize = 2 & AUC = 0.927 WinodwSize = 3 & AUC = 0.916

> Seq1: AUC = 0.93 Seq2: AUC = 0.96 Seq3: AUC = 0.92 Seq4: AUC = 0.94

False positive rate

CRF:WindowSize=1;GIST

0 0.2 0.4 0.6 0.8 1

False positive rate

threshold corresponding to the maximum F score, we located the starting and ending points in each input sequence as shown in Fig.10. The experimental result shows that the proposed method can achieve an average localization error of 1.35 ± 1.46 frames for the start of mitosis

Fig. 8. ROC curve of CRF models with different sequence window lengths from 0 to 3.

ROC of CRF(GIST)

<sup>369</sup> A Hierarchical Framework for Mitosis Detection in Time-Lapse

Phase Contrast Microscopy Image Sequences of Stem Cell Populations

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Fig. 9. ROC of CRF with GIST feature and window size = 1.

True positive rate

True positive rate

Fig. 7. Mitosis identification results by HCRF.

SVM, with a best-case performance of 92% AUC and 90% maximum F score when precision and recall are 85% and 95% respectively.


Table 2. Comparison between HCRF, CRF and SVM

#### **4.2 Mitosis localization**

To localize the starting and ending points of mitosis events, we used CRF to label each frame of the identified mitosis candidates by HCRF. To optimize the performance of CRF model, 4 CRF classifiers were learned by varying window size from 0 to 3. From Fig.8, the results showed that the model trained with GIST features and window size 1 outperformed the others with the best AUC value of 0.933.

The CRF classifier was trained with GIST feature and window size 1 on the training set. The ROC curves of the test results on the four test sequences are shown in Fig.9. By choosing the 14 Will-be-set-by-IN-TECH

SVM, with a best-case performance of 92% AUC and 90% maximum F score when precision

Sequence No. AUC (%) Maximum F (%)

To localize the starting and ending points of mitosis events, we used CRF to label each frame of the identified mitosis candidates by HCRF. To optimize the performance of CRF model, 4 CRF classifiers were learned by varying window size from 0 to 3. From Fig.8, the results showed that the model trained with GIST features and window size 1 outperformed the others with

The CRF classifier was trained with GIST feature and window size 1 on the training set. The ROC curves of the test results on the four test sequences are shown in Fig.9. By choosing the

1 92 78 77 90 84 80 2 93 73 71 87 83 77 3 91 73 70 86 84 81 4 86 62 55 87 75 80

HCRF CRF SVM HCRF CRF SVM

Fig. 7. Mitosis identification results by HCRF.

Table 2. Comparison between HCRF, CRF and SVM

and recall are 85% and 95% respectively.

**4.2 Mitosis localization**

the best AUC value of 0.933.

Fig. 8. ROC curve of CRF models with different sequence window lengths from 0 to 3.

Fig. 9. ROC of CRF with GIST feature and window size = 1.

threshold corresponding to the maximum F score, we located the starting and ending points in each input sequence as shown in Fig.10. The experimental result shows that the proposed method can achieve an average localization error of 1.35 ± 1.46 frames for the start of mitosis

event and 1.68 ± 1.73 frames for the end of mitosis event. Moreover, the proposed method does not depend on empirical parameters, *ad hoc* image processing, or cell tracking and

<sup>371</sup> A Hierarchical Framework for Mitosis Detection in Time-Lapse

We would like to thank Prof. Takeo Kanade, Dr. Phil G. Campbell, Dr. Lee E. Weiss, Dr. Mei Chen, Dr. Eric D. Miller, Dai Fei Elmer Ker, Dr. Sung Eun Eom, Dr. Zhaozheng Yin, Jiyan Pan, and Ryoma Bise for their helpful discussions. This work was supported in part by the National Natural Science Foundation of China (61100124, 21106095), Tianjin Research Program of Application Foundation and Advanced Technology (10JCYBJC25500), China Postdoctoral Science Foundation (2011M500512), 2010/2011 Innovation Foundation of

Al-Kofahi, O., Radke, R., Goderie, S., Shen, Q., Temple, S. & Roysam, B. (2006). Automated

Belongie, S., Malik, J. & Puzicha, J. (2002). Shape matching and object recognition using shape

Bise, R., Li, K., Eom, S. & Kanade, T. (2009). Reliably tracking partially overlapping neural

*Image Analysis in Microscopy Histopathology and Endoscopy*, Vol. 12, pp. 67–77. Dalal, N. & Triggs, B. (2005). Histograms of oriented gradients for human detection, *Proc. 2005 IEEE Int. Conf. Computer Vision and Pattern Recognition (CVPR05)*, pp. 886–893. Debeir, O., Ham, P. V., Kiss, R. & Decaestecker, C. (2005). Tracking of migrating cells under

Donoho, D. L. (2004). For most large underdetermined systems of linear equations the

Eccles, B. A. & Klevecz, R. R. (1986). Automatic digital image analysis for identification

Gallardo, G., Yang, F., Ianzini, F., Mackey, M. & Sonka, M. (2004). Mitotic cell recognition with hidden Markov models, *Proc. SPIE: Medical Imaging*, Vol. 5367, pp. 661–668. Gunawardana, A., Mahajan, M., Acero, A. & Platt, J. C. (2005). Hidden conditional random

Lafferty, J. D., McCallum, A. & Pereira, F. C. N. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data., *ICML'01*, pp. 282–289. Li, K. (2009). *Large-Scale Stem Cell Population Tracking in Phase Contrast and DIC Microscopy*

with murine neural progenitor cells, *Cell Cycle* 5(3): 327–335. Avriel, M. (2003). *Nonlinear Programming: Analysis and Methods*, Dover Publishing.

contexts, *IEEE Trans. Pattern Anal. Mach. Intell.* pp. 509–522.

cell lineage construction: A rapid method to analyze clonal development established

stem cells in dic microscopy image sequences, *MICCAI Workshop on Optical Tissue*

phase-contrast video microscopy with combined mean-shift processes, *IEEE Trans.*

minimal l1-norm solution is also the sparsest solution, *Comm. Pure Appl. Math*

of mitotic cells in synchronous mammalian cell cultures, *Anal. Quant. Cytol. Histol.*

fields for phone classification, *Proc. Int. Conf. Speech Communication and Technology*,

*Image Sequences*, PhD thesis, Carnegie Mellon University, 5000 Forbes Avenue,

consequently can be straightforwardly adapted to different cell types.

Phase Contrast Microscopy Image Sequences of Stem Cell Populations

**6. Acknowledgement**

Tianjin University.

*Med. Imag.* 24: 697–711.

59: 1848–1853.

8: 138–147.

pp. 1117–1120.

Pittsburgh, PA.

**7. References**

Fig. 10. Mitosis localization results by CRF. (a) Both starting and ending points are labeled correctly; (b-c) The starting point is labeled correctly while the ending point is labeled wrongly; (d-e) The starting point is labeled wrongly while the ending point is labeled correctly; (f-i) Four cases of locating both starting and ending points wrongly.

event and 1.68 ± 1.73 frames for the end of mitosis event. From Fig.10, it is understandable that the start point can usually be localized more precisely than the end point because the appearances and shapes of mitotic cells always change drastically when two daughter cells separate with each other physically.

#### **5. Conclusion**

We propose a hierarchical framework of mitosis event detection for cell populations imaged with time-lapse phase contrast microscopy. The method consists of three stages: mitosis candidate extraction, mitosis identification, and mitosis localization, which jointly optimize recall and precision for mitosis identification and minimize the mean error and standard deviation of mitosis location. The proposed detection method achieved two kinds of excellent results in very challenging image sequences of multipolar-shaped C3H10T1/2 mesenchymal stem cells. For mitosis identification, we can achieve an average precision of 0.83 and a recall of o.92 with a corresponding maximum F score of 0.88. For mitosis location, the proposed method can achieve an average localization error of 1.35 ± 1.46 frames for the start of mitosis event and 1.68 ± 1.73 frames for the end of mitosis event. Moreover, the proposed method does not depend on empirical parameters, *ad hoc* image processing, or cell tracking and consequently can be straightforwardly adapted to different cell types.

#### **6. Acknowledgement**

16 Will-be-set-by-IN-TECH

Fig. 10. Mitosis localization results by CRF. (a) Both starting and ending points are labeled correctly; (b-c) The starting point is labeled correctly while the ending point is labeled wrongly; (d-e) The starting point is labeled wrongly while the ending point is labeled

event and 1.68 ± 1.73 frames for the end of mitosis event. From Fig.10, it is understandable that the start point can usually be localized more precisely than the end point because the appearances and shapes of mitotic cells always change drastically when two daughter cells

We propose a hierarchical framework of mitosis event detection for cell populations imaged with time-lapse phase contrast microscopy. The method consists of three stages: mitosis candidate extraction, mitosis identification, and mitosis localization, which jointly optimize recall and precision for mitosis identification and minimize the mean error and standard deviation of mitosis location. The proposed detection method achieved two kinds of excellent results in very challenging image sequences of multipolar-shaped C3H10T1/2 mesenchymal stem cells. For mitosis identification, we can achieve an average precision of 0.83 and a recall of o.92 with a corresponding maximum F score of 0.88. For mitosis location, the proposed method can achieve an average localization error of 1.35 ± 1.46 frames for the start of mitosis

correctly; (f-i) Four cases of locating both starting and ending points wrongly.

separate with each other physically.

**5. Conclusion**

We would like to thank Prof. Takeo Kanade, Dr. Phil G. Campbell, Dr. Lee E. Weiss, Dr. Mei Chen, Dr. Eric D. Miller, Dai Fei Elmer Ker, Dr. Sung Eun Eom, Dr. Zhaozheng Yin, Jiyan Pan, and Ryoma Bise for their helpful discussions. This work was supported in part by the National Natural Science Foundation of China (61100124, 21106095), Tianjin Research Program of Application Foundation and Advanced Technology (10JCYBJC25500), China Postdoctoral Science Foundation (2011M500512), 2010/2011 Innovation Foundation of Tianjin University.

#### **7. References**


**0**

**18**

Alain Beaulieu

*Canada*

*Royal Military College of Canada*

**Safety of Interactive Image-Guided Surgery**

In this chapter, we discuss Interactive Image-Guided Surgery (IIGS) and present the current state of the art in the analysis and evaluation of this kind of computer-supported medical procedure. We also present the results of our research to evaluate the safety of IIGS. During our discussion of the current state of the art, we review the various approaches that have been proposed to analyze and evaluate IIGS systems including their strengths and weaknesses. The original hypothesis that initiated our research was that: "it is possible to improve the safety analysis of computer-guided surgery if we model this kind of system as a process control model." The hypothesis was first elaborated in an initial research planning meeting while we were investigating a better way of developing a Verification and Validation method for IIGS. During the evolution of this research, this hypothesis has evolved significantly from a basic idea into a modelling language and a protocol that have been evaluated using a

For many decades, medical imaging has been rapidly evolving as one of the most useful applications of scientific and engineering knowledge to improve the quality of health care. Medical imaging provides technology that enables physicians to see subcutaneous structures

The use of medical imaging increases the observability of areas of interest inside the patient. Increased observability allows for more complete diagnosis and better planning of surgical procedures leading to potential improvements in surgical outcome. The first use of medical imaging with clinical intent can be traced back to January 1896 in Birmingham, United Kingdom where an x-ray was used to plan the removal of a needle from a woman's hand Viergever (1998). Medical imaging for clinical procedures has therefore been around for over 100 years. The first use of medical images was static, but it demonstrated the usefulness of medical imaging in planning surgeries. Medical imaging progressed over the next 100 years with improvements in the quality of existing modalities and image processing, as

<sup>1</sup> The debate on the degree of invasiveness of ionizing and non-ionizing radiation is ongoing. Although it is clear that their side effects cannot be overlooked *Assessment and Management of Cancer Risks from Radiological and Chemical Hazards* (1998); on Radiological Protection (1960-Present); Profio (1979); Tenforde (1979), medical imaging is far less invasive and risky than open exploratory surgery Hindus

particular set of IIGS systems used for orthopedic surgery.

and tissues without the use of more invasive procedures1.

**2. Using medical imaging for surgery**

**1. Introduction**

(2001).

