**3. Empirical transition matrix state-space & commenges' test approach**

#### **3.1 Mathematical formulation**

Multi-state models are known to provide a relevant framework for modelling complex event histories. Quantities of interest are mainly the transition probabilities

**Figure 7.** *Violation counts in bins across all patients.*

that can be estimated by the empirical transition matrix, that is also referred to as the Aalen-Johansen estimator [30, 31]. Such multi-state models have had a wide range of applications for modelling complex courses of a disease over the course of time and across applications in medical research (Beyersmann et al. [32], Munoz-Price et al. [33], Andersen & Keiding [34]). We now utilise the Empirical Transition Matrix (etm) approach to model multi-state models of [22] and derive inference tests for such models using the approach of Commenges [25, 35, 36] with a particular focus on Commenges' test derived in earlier work [25].

Define patient states as follows, any state can transition into any other state (**Figure 8**).


A number of different approaches (etm on ICU time, etm on bin time, and log-rank type tests as in Commenges [25], will be used to investigate the difference between good and poor trackers in terms of a devised a 3 state transition formulation as defined below. Differences between transition probabilities between states for the good and poor trackers will be evaluated using Commenges' [25] chi square test.

The mathematics is well described in the work of Allignol [22], adapted to more complex scenarios in [23]; and in Commenges' approach [25, 35, 36]. The mathematical formulation of the ETM state-space approach and Commenges' test are given for general frameworks as follows.

Consider a stochastic process ð Þ *Xt* with finite state space *S* ¼ f g 1, … ,*K* where sample paths are right-continuous, and the stochastic process is assumed to be

**Figure 8.** *State-space system.*

*Modelling Agitation-Sedation (A-S) in ICU: An Empirical Transition and Time to Event… DOI: http://dx.doi.org/10.5772/intechopen.105480*

time-inhomogeneous Markov. The transition hazard from state *i* to state *j*, *i* 6¼ *j* is defined as

$$a\_{ij}(t)dt = P(X\_{t+dt} = j \mid X\_t = i).$$

The cumulative hazard transitions are defined as,

$$\begin{aligned} A\_{\vec{\eta}}(t) &= \int\_0^t a\_{\vec{\eta}}(t)dt \\ A\_{\vec{\mu}}(t) &= -\sum\_{j \neq i} A\_{\vec{\eta}j}(t) . \end{aligned}$$

Define

$$P\_{\vec{\eta}}(s, t) = P(X\_t = j | X\_s = i), \text{for } i, j \in \mathbb{S}, s \le t$$

as the probability that an individual who is in state *i* at time *s* is in state *j* at time *t*.

The (K + 1)x(K + 1) probability transition matrix with elements *Pij*ð Þ *s*, *t* can then be obtained from the transition hazards through product integration. Let *Nij*ð Þ*t* be the number of observed direct transitions from state I to state j up to time t and let *Yi*ð Þ*t* be the number of individuals under observation in state I just before time t. The Nelson-Aalen estimator is used to estimate the non-diagonal elements of the matrix of cumulative hazards as follows,

$$\hat{A}\_{\vec{\eta}}(t) = \int\_0^t \frac{dN\_{\vec{\eta}}(u)}{Y\_i(u)}, i \neq j$$

and the diagonal elements *<sup>A</sup>*^*ii*ð Þ*<sup>t</sup>* are obtained as above. The product integration relationship below leads to an estimate of the probability transition matrix as follows,

$$
\hat{P}(s, t) = \prod\_{s < t\_k \le t} \left( I + \Delta \hat{\mathcal{A}}(t\_k) \right),
$$

where the product is taken over all possible transition times in time interval (s,t]. An estimator for the covariance of the empirical transition matrix is given by,

$$\widehat{\operatorname{cov}}\big(\hat{\mathbf{P}}(s,t)\big) = \int\_{s}^{t} \hat{\mathbf{P}}(u,t)^{T} \bigotimes \hat{\mathbf{P}}(s,u-) \widehat{\operatorname{cov}}\big(d\hat{\mathbf{A}}(u)\big) \hat{\mathbf{P}}(u,t) \bigotimes \hat{\mathbf{P}}(s,u-), \,^{T}$$

which is a (K + 1)<sup>2</sup> x (K + 1)2 matrix, with number or rows and columns equal (K + 1)<sup>2</sup> .

## **3.2 Commenges' test formulation**

We now utilise the framework of the generalised Cochran–Mantel–Haenszel (CMH) test for (I x J x K) tables. The CMH test is based on the hypergeometric distribution. The CMH and the test of Commenges' for the specific case here, where we have three states (of violations) and two strata (good versus poor trackers) is now described.

In our application then we have 2 � 3 � 3 tables. For each k = 1, 2, 3 we have a 2 � 3 table, where the elements are counts nijk (k denotes the departure state j, so the rows, I, are the WPB strata, the columns, J, are the entry states I and the slices, K, are the departure states j), as tabulated below.


Assume that the row and column marginals are fixed. This implies that there are (I-1) by (J-1) values that are free to vary. For the cell nijk the expected value is then given by ni+k x n+jk. We are able to express all cell counts by the vector **nk** and express all expected cell counts by **uk**. The covariance matrix, denoted by **Vk** then has elements,

$$cov\left(n\_{j|k}, n\_{i'j'k}\right) = \frac{n\_{i+k}(\delta\_{ii'}n\_{++k} - n\_{i'+k})n\_{+jk}\left(\delta\_{jj'}n\_{++k} - n\_{+j'k}\right)}{n\_{++k}^2(n\_{++k} - 1)},$$

where *δab* ¼ 1 when a = b, and 0 otherwise. Assuming rows and columns are unordered we sum over the K strata to obtain,

$$n = \sum n\_k, \quad u = \sum u\_k, \quad V = \sum V\_k.$$

The generalised CMH statistic is then given by,

$$X^2 = \left(\mathfrak{n} - \mathfrak{u}\right)^{\prime \mathcal{V}^{-1}}(\mathfrak{n} - \mathfrak{u}),$$

which follows a chi square distribution with (I-1) by (J-1) degrees of freedom. This test is implemented in R via the mantelhaen.test function. Commenges [25] adapts this concept, with a test which differs to the generalised CMH test in that it does not sum over the K strata, before calculating the relevant chi squared test statistic.

Commenges' test [25] is as follows,

$$X\_k^2 = (\mathfrak{n}\_k - \mathfrak{u}\_k)^{\prime V^{-1}} (\mathfrak{n}\_k - \mathfrak{u}\_k),$$

where *X*<sup>2</sup> *<sup>k</sup>* is chi-square with (I-1) by (J-1) degrees of freedom.

The required total chi squared statistic is then simply obtained by taking *<sup>X</sup>*<sup>2</sup> <sup>¼</sup> <sup>P</sup>*X*<sup>2</sup> *<sup>k</sup>* which is itself disctributed as a chi square distribution with K(I-1) by (J-1) degrees of freedom.

## **3.3 Results of the ETM analysis and Commenges' test on transition states**

Conditionally on the number of patients in each state at each step we have 3 x 9 = 27 independent contingency tables (i.e., number of departure states j by the number of time points k–1, recall we have 10, 10% bins for a patient's time in ICU) and each of these tables has dimension 2 � 3 (good/poor tracker by the number of states i).

The corresponding three specific strata tables are given in **Tables 5**–**7**. For example, for departure state j = 0 and time point k = 2 we have a two by three contingency table with an overall total of 7<sup>ϕ</sup> violations (labelled <sup>ϕ</sup> in **Table 5**); the latter informs that, at time k = 2 there are 5**<sup>ϕ</sup>** good WPB based trackers that depart from state j = 0


*Modelling Agitation-Sedation (A-S) in ICU: An Empirical Transition and Time to Event… DOI: http://dx.doi.org/10.5772/intechopen.105480*

**Table 5.**

*Departure state j = 0: WPB strata [9].*

and enter state 0. Similarly there are two (2**<sup>ϕ</sup>** ) WPB based poor trackers that depart state j = 0 and enter state 1 (**Table 5**).

Three 2 x 3 contingency tables (one for each departure state j) are thus created.

Estimated transition probabilities for the 3 state process are then plotted using the 'xyplot' function from the lattice package in R. In the resultant plots (**Figures 9**–**11**), the vertical y-axis represents the transition probability value, which is represented by the solid line in each plot region. The numbers in the coloured bar above each plot defines the transition (e.g., 1 2 means transition probability from state 1 to state 2). The dotted lines around the solid line represent the confidence bands based on the covariance as calculated by the etm function. The horizontal x-axis shows the the


**Table 6.**

*Departure state j = 1: WPB strata [9].*

time i.e., 10% bins (i.e. 2–10, because no transitions occur at time one being the initial state). For each tracker status and possible piairs of state transitions there are three plots, given in the following order, good trackers, poor trackers. **Figure 11** displays the probability of being in each of the 3 states (0, 1, 2) given the initial state is state 0.

Our procedure results in three 2 x 3 contingency tables (one for each departure state j), see **Tables 8**–**10**. The chi squared statistic as derived in [25] can now be calculated in that for the Commenges test the same chi squared calculation is made for each state specific table separately (i.e., without summing over k). The results in this case are χ<sup>2</sup> (1) = 6.046, χ<sup>2</sup> (2) = 2.269 and χ<sup>2</sup> (3) = 9.280. Each of these follows a chi


*Modelling Agitation-Sedation (A-S) in ICU: An Empirical Transition and Time to Event… DOI: http://dx.doi.org/10.5772/intechopen.105480*

**Table 7.**

*Departure state j = 2: WPB strata [9].*

square distribution with 2 degrees of freedom with associated p-values of 0.049, 0.322 and 0.010 (**Table 11**). Kang WPB (2013) [9].

Summing these threeχ <sup>2</sup> (j), j = 1,2,3 statistics gives a value of χ2 =17.59 with 6 degrees of freedom and an associated p-value of 0.007 (**Table 11**). The underlying null hypothesis is that the two nominal variables (strata: good or poor tracker and entry state: 0, 1, or 2) are conditionally independent in each stratum (departure state j; 0, 1 or 2), assuming no three-way interaction. The low p-value of 0.007 suggests that this hypothesis be rejected, i.e., the two variables are not conditionally independent. Thus the Commenges test shows that there is a statistically significant difference between the good versus poor tracker WPB strata, and that this difference is mainly

**Figure 9.** *Transition probability profiles for* WPB good trackers*.*

due to transitions out of states 0 and 2, which agrees with the trends based on a graphical inspection of **Figures 9**–**11**.

The same procedure and related Commenges' test is then applied to each of the 3 remaining good/poor tracker definitions of Kang [10, 11, 14] for the three-state context studied in this chapter. These results are reported in **Table 12**.

Kang et al., WCORR [10].

Chase et al. [14].


Rudge et al. [11].


*Modelling Agitation-Sedation (A-S) in ICU: An Empirical Transition and Time to Event… DOI: http://dx.doi.org/10.5772/intechopen.105480*

Transition probability profiles of being in each state as time progresses, given start state 0, for the remaining 3 studies of [10, 14, 11] are given in **Figures 12**–**14**. In summary, **Figures 9**–**14** illustrate the trend that good trackers tend to have higher probability of transitioning into state 0 than poor trackers, and the good trackers tend to have lower probability of transitioning into state two than poor trackers, where state two indicates that more violations (>3 violations) are occurring, and state 0 indicates few violations are occurring.

Notably also, the probability of transitioning into state 2 overall appears to increase as ICU time increases. This is most likely because poor tracking patients tend to have longer ICU times, and so, as time goes on, it is only poor trackers transitions that are being estimated. By categorising patients according to total ICU time (≤64, >64) as discussed earlier (**Figures 5** and **6**, **Table 4**) some of this could be accounted for. The results obtained are still consistent, as shown in the etm profiles using ICU time (≤64, >64) in **Figures 15** and **16**, respectively. The corresponding ETM probabilities are determined according to etm in R [21] and associated state and strata specific plots given in **Figures 15** and **16**.

**Figure 10.** *Transition probability profiles for* WPB poor trackers*.*

#### **Figure 11.**

*Probability of being in each state as time progresses given start state 0. Top is good trackers, bottom is poor trackers:* WPB based.


#### **Table 8.**

*Departure state j = 0 summed over all time points k, k = 2, … ,10.*


#### **Table 9.**

*Departure state j = 1 summed over all time points k, k = 2, … ,10.*
