**2. Applications**

This section presents a number of applications of non-invasive monitoring methods for individuals.

#### **2.1 Monitoring health status of patients in the intensive care unit on the basis of realtime measured physiological variables and advanced modelling technology**

In cardiac surgery it would be very helpful to have a system that provides an early alert if there is a high probability that a patient will be disconnected from ventilation during the

Non-Invasive Methods for Monitoring Individual Bioresponses in Relation to Health Management 145

parameters of which were used as the features of the examples. One of the two possible class labels was assigned to each example. Training examples for each classifier were labelled positive (+1) if the moment when the patient became stable started within the first nine hours after admission and were labelled negative (-1) otherwise. The classification performance can be calculated by the aROC (area under the receiver operating characteristic

Table 1 gives the obtained aROCs for each experiment with the GP. The middle column contains the results obtained when using a logistic regression (LOGREG) model, included here as a baseline for performance. The increase in performance for all GP models versus the LOGREG models was found to be significant, except for the model based on admission. So, although logistic regression techniques are commonly used in medical applications, other classifiers might lead to better results. This was, among others, also concluded by Sakai et al. (2007) and Erol et al. (2005). It is also shown that the approach including dynamic information (MAR) performs better than the model purely based on admission information

> aROC LOGREG GP Avgstd (20) 0.628 0.713 MAR 0.591 0.708

This application shows that taking into account dynamic information in analysing time series of patients can be of added value when monitoring the health status of individual

Disease management is becoming increasingly important in our current society, especially considering the growing population of elderly and immune compromised people. There is a general agreement that sepsis and the systemic inflammatory response syndrome (SIRS) are characterized by an inability to regulate the inflammatory response. The cause of this perturbation is still unknown. So far research did not result in a dramatic reduction of the high mortality rates which for critically ill patients in intensive care units where sepsis and

As strategies for the early treatment of sepsis mostly failed, this fully justifies the development of a novel biosensor array at the heart of an on line early warning monitoring system for prediction of disease evolution and subsequent adaptation of life saving therapy. Complex biological processes are involved in these phenomena and therefore it is a challenge to quantify infection and inflammation processes in real-time. It is expected however that the use of biosensors in combination with real-time signal analysis allows

Developing such new sensing techniques requires an interdisciplinary approach between engineers, immunologists, medical experts and sensor developers. In a first step, we aimed at demonstrating the proof of principle in animal experiments (pigs). In a next step, the

**2.2 Dynamic algorithms of biomarkers for monitoring infection/inflammation** 

curve) for each classifier.

(in terms of the aROC).

Table 1. Classification results

SIRS remain major causes of death.

monitoring infection/inflammation processes in real-time.

developed methodologies might be transferred to human patients.

patients.

**processes** 

next day since this would lead to a more optimal planning in the Intensive Care Unit (ICU). It was shown that alterations in vital signals are relevant to patient management (Rivera-Fernandez et al., 2007), so we wanted to use the trends of some of those vital signals during the first hours of ICU stay to predict a short or prolonged length of stay from early on. All living organisms are characterised by the fact that they are complex, individually different time-variant and dynamic (so called CITD systems) (Quanten et al., 2006). Consequently, it is expected that taking these characteristics into account will lead to better models of the physiological signals of intensive care patients. So far, univariate and multivariate autoregressive analyses as well as the calculation of the cepstrum of physiological variables have been applied in several medical studies (Wada et al., 1988; Curcie and Craelius, 1997) to analyze individual patients. For making classifications using many variables at the same time, several data mining techniques are available. However, in most cases no dynamic information about the patients is taken into account when applying the data mining approach. Several attempts on temporal feature extraction for time series classification have been made (Verduijn et al., 2007). We describe here a study in which information of patients' dynamics was used to predict the timeframe when the conditions to start weaning of mechanical ventilation are reached.

#### **2.1.1 Analysis of time series of patient data**

Physiological variables, such as heart rate (bpm), systolic arterial blood pressure (mmHg), systolic pulmonary pressure (mmHg), blood temperature (°C) and oxygen saturation are routinely monitored in these patients and can be sampled frequently when Patient Data Management System are used. In this example, we show results of a total of 203 patients that were followed in the Intensive Care unit of the University Hospital of Leuven. More information can be found in the work of Van Loon et al. (2010).

Besides the mean and standard deviations of the signals (Avgstd), more advanced time series models can be applied that allow quantifying the dynamics of these physiological variables such as univariate and multivariate autoregressive (AR) models.

Dynamic features that are extracted from time series of patient data, can be used in a next step to determine the status of individual patients when applying them to machine learning techniques such as Support Vector Machines or Gaussian Processes.

Gaussian processes (GP), a type of kernel method, are a machine learning technique that has been successfully used to model and forecast real dynamic systems. In probabilistic binary classification the task is to determine for an unlabeled test input vector the probability of belonging to a given class when a training set is given. The training set is comprised of training input vectors and their corresponding binary class labels (+1 if the input vector belongs to the class, -1 otherwise).

The considered task in the presented example can be restated as follows: Predict the probability that the patient will begin to satisfy the stability criteria within each of the following time frames (classes): class 1: earlier than nine hours after admission; class 2: later than nine hours after admission. This nine hour threshold was chosen such that the resulting classes contained roughly the same amount of patients. This division also conforms to an intuitive classification used by intensivists into patients that recover quickly and those that require prolonged ICU stays. Data from each patient, collected during the first four hours ICU stay, were used to generate the different time-series models, the parameters of which were used as the features of the examples. One of the two possible class labels was assigned to each example. Training examples for each classifier were labelled positive (+1) if the moment when the patient became stable started within the first nine hours after admission and were labelled negative (-1) otherwise. The classification performance can be calculated by the aROC (area under the receiver operating characteristic curve) for each classifier.

Table 1 gives the obtained aROCs for each experiment with the GP. The middle column contains the results obtained when using a logistic regression (LOGREG) model, included here as a baseline for performance. The increase in performance for all GP models versus the LOGREG models was found to be significant, except for the model based on admission. So, although logistic regression techniques are commonly used in medical applications, other classifiers might lead to better results. This was, among others, also concluded by Sakai et al. (2007) and Erol et al. (2005). It is also shown that the approach including dynamic information (MAR) performs better than the model purely based on admission information (in terms of the aROC).


Table 1. Classification results

144 Health Management – Different Approaches and Solutions

next day since this would lead to a more optimal planning in the Intensive Care Unit (ICU). It was shown that alterations in vital signals are relevant to patient management (Rivera-Fernandez et al., 2007), so we wanted to use the trends of some of those vital signals during the first hours of ICU stay to predict a short or prolonged length of stay from early on. All living organisms are characterised by the fact that they are complex, individually different time-variant and dynamic (so called CITD systems) (Quanten et al., 2006). Consequently, it is expected that taking these characteristics into account will lead to better models of the physiological signals of intensive care patients. So far, univariate and multivariate autoregressive analyses as well as the calculation of the cepstrum of physiological variables have been applied in several medical studies (Wada et al., 1988; Curcie and Craelius, 1997) to analyze individual patients. For making classifications using many variables at the same time, several data mining techniques are available. However, in most cases no dynamic information about the patients is taken into account when applying the data mining approach. Several attempts on temporal feature extraction for time series classification have been made (Verduijn et al., 2007). We describe here a study in which information of patients' dynamics was used to predict the timeframe when the conditions to start weaning of

Physiological variables, such as heart rate (bpm), systolic arterial blood pressure (mmHg), systolic pulmonary pressure (mmHg), blood temperature (°C) and oxygen saturation are routinely monitored in these patients and can be sampled frequently when Patient Data Management System are used. In this example, we show results of a total of 203 patients that were followed in the Intensive Care unit of the University Hospital of Leuven. More

Besides the mean and standard deviations of the signals (Avgstd), more advanced time series models can be applied that allow quantifying the dynamics of these physiological

Dynamic features that are extracted from time series of patient data, can be used in a next step to determine the status of individual patients when applying them to machine learning

Gaussian processes (GP), a type of kernel method, are a machine learning technique that has been successfully used to model and forecast real dynamic systems. In probabilistic binary classification the task is to determine for an unlabeled test input vector the probability of belonging to a given class when a training set is given. The training set is comprised of training input vectors and their corresponding binary class labels (+1 if the input vector

The considered task in the presented example can be restated as follows: Predict the probability that the patient will begin to satisfy the stability criteria within each of the following time frames (classes): class 1: earlier than nine hours after admission; class 2: later than nine hours after admission. This nine hour threshold was chosen such that the resulting classes contained roughly the same amount of patients. This division also conforms to an intuitive classification used by intensivists into patients that recover quickly and those that require prolonged ICU stays. Data from each patient, collected during the first four hours ICU stay, were used to generate the different time-series models, the

mechanical ventilation are reached.

belongs to the class, -1 otherwise).

**2.1.1 Analysis of time series of patient data** 

information can be found in the work of Van Loon et al. (2010).

variables such as univariate and multivariate autoregressive (AR) models.

techniques such as Support Vector Machines or Gaussian Processes.

This application shows that taking into account dynamic information in analysing time series of patients can be of added value when monitoring the health status of individual patients.

#### **2.2 Dynamic algorithms of biomarkers for monitoring infection/inflammation processes**

Disease management is becoming increasingly important in our current society, especially considering the growing population of elderly and immune compromised people. There is a general agreement that sepsis and the systemic inflammatory response syndrome (SIRS) are characterized by an inability to regulate the inflammatory response. The cause of this perturbation is still unknown. So far research did not result in a dramatic reduction of the high mortality rates which for critically ill patients in intensive care units where sepsis and SIRS remain major causes of death.

As strategies for the early treatment of sepsis mostly failed, this fully justifies the development of a novel biosensor array at the heart of an on line early warning monitoring system for prediction of disease evolution and subsequent adaptation of life saving therapy.

Complex biological processes are involved in these phenomena and therefore it is a challenge to quantify infection and inflammation processes in real-time. It is expected however that the use of biosensors in combination with real-time signal analysis allows monitoring infection/inflammation processes in real-time.

Developing such new sensing techniques requires an interdisciplinary approach between engineers, immunologists, medical experts and sensor developers. In a first step, we aimed at demonstrating the proof of principle in animal experiments (pigs). In a next step, the developed methodologies might be transferred to human patients.

Non-Invasive Methods for Monitoring Individual Bioresponses in Relation to Health Management 147

Fig. 1. Criterion for early detection of survival/non-survival based on SAA: stability of model. Overview of a-parameters of AR-models for all surviving (blue) and non-surviving

Also for the biomarker IL-6, there was a significant difference in a-parameters between survivors and non-survivors (mean S = -0.1272, mean NS = -0.5751, p=0.0085). Fig. 2 shows a

Fig. 2. Criterion for early detection of survival/non-survival based on IL-6: trend vs noise. Overview of a-parameters of AR-models for all surviving (blue) and non-surviving (red)

(red) pigs

pigs

scatter plot of the a-parameters for all pigs.

#### **2.2.1 Data generation**

The aim of the experiments was to quantify the dynamics of 3 cytokines and 4 acute-phase proteins (APP) before and after infection by *Actinobacillus pleuropneumoniae* in pigs*.* More specific, the cytokines, tumor necrosis factor-alpha (TNF-α), interleukin-6 (IL-6) and interleukin-10 (IL-10), and the acute-phase proteins, C-reactive protein (CRP), haptoglobin (Hp), major acute phase protein (MAP) and serum-amyloid A (SAA), were analysed. In total, 22 pigs were infected. The blood sampling frequency that was used for the experiments was as follows: 1 sample/day: before infection, 1 sample/2 hours: starting from 2 hours before infection to end. The higher sampling frequency after infection was necessary to determine the dynamics of the response on the infection and to measure the entire course of the biomarker response. Due to the measuring frequencies, the focus of the modelling analysis was on the period starting with the infection of the pigs until the end of the experiment.

#### **2.2.2 Modelling of biomarker responses to infection**

In this example, the dynamics of all biomarkers were modelled using univariate autoregressive (AR) models. These time series models were defined as follows (Taylor et al., 2007):

$$\mathbf{y}(\mathbf{k})\mathbf{A}(\mathbf{z}^{-1}) = \mathbf{e}(\mathbf{k})\tag{1}$$

Where y(k) is the considered biomarker, <sup>1</sup> A(z ) is the polynomial of the model parameters and e(k) is additive noise, a serially uncorrelated sequence of random variables with variance σ2 that accounts for measurement noise, modelling errors and effects of unmeasured inputs to the process (assumed to be a zero mean). For the modelling the order of the AR models were ranged from 1 to 2, resulting in 2 possible AR models per biomarker for every pig. Since no significant results were found for second order AR models, all AR models described in this report are simple first order models which can be written in the time series form:

$$\mathbf{y(k)} = -\mathbf{a}\_1 \mathbf{y(k-1)} + \mathbf{e(k)} \tag{2}$$

As the results of SAA and IL-6 were most significant, we will focus in this example on these two biomarkers. In a first step, models were developed for each individual pig. In a second step, the individual models of the pigs were compared to develop model-based criteria for the early detection of survival/non-survival.

#### **2.2.3 Monitoring criteria**

All the modelling results described below will focus on the prediction of disease outcome (survival vs. non-survival). Since many pigs died shortly after the infection, only the data of a short period after infection were used for the development of the TF- and AR-models. To obtain comparable models, samples of the same time interval were used for the calculation of these models. More specifically, for all pigs data were used starting from the moment of infection until 16 hours after infection.

For the biomarker SAA a significant difference between survivors (S) and non-survivors (NS) was found for the a-parameters of the AR models (mean S = -0.7915, mean NS = -1.3204, p = 0.04). Fig. 1 shows a scatter plot of the a-parameters of the different pigs.

The aim of the experiments was to quantify the dynamics of 3 cytokines and 4 acute-phase proteins (APP) before and after infection by *Actinobacillus pleuropneumoniae* in pigs*.* More specific, the cytokines, tumor necrosis factor-alpha (TNF-α), interleukin-6 (IL-6) and interleukin-10 (IL-10), and the acute-phase proteins, C-reactive protein (CRP), haptoglobin (Hp), major acute phase protein (MAP) and serum-amyloid A (SAA), were analysed. In total, 22 pigs were infected. The blood sampling frequency that was used for the experiments was as follows: 1 sample/day: before infection, 1 sample/2 hours: starting from 2 hours before infection to end. The higher sampling frequency after infection was necessary to determine the dynamics of the response on the infection and to measure the entire course of the biomarker response. Due to the measuring frequencies, the focus of the modelling analysis was on the period starting with the infection of the pigs until the end of the

In this example, the dynamics of all biomarkers were modelled using univariate autoregressive (AR) models. These time series models were defined as follows (Taylor et al.,

Where y(k) is the considered biomarker, <sup>1</sup> A(z ) is the polynomial of the model parameters and e(k) is additive noise, a serially uncorrelated sequence of random variables with variance σ2 that accounts for measurement noise, modelling errors and effects of unmeasured inputs to the process (assumed to be a zero mean). For the modelling the order of the AR models were ranged from 1 to 2, resulting in 2 possible AR models per biomarker for every pig. Since no significant results were found for second order AR models, all AR models described in this report are simple first order models which can be written in the

As the results of SAA and IL-6 were most significant, we will focus in this example on these two biomarkers. In a first step, models were developed for each individual pig. In a second step, the individual models of the pigs were compared to develop model-based criteria for

All the modelling results described below will focus on the prediction of disease outcome (survival vs. non-survival). Since many pigs died shortly after the infection, only the data of a short period after infection were used for the development of the TF- and AR-models. To obtain comparable models, samples of the same time interval were used for the calculation of these models. More specifically, for all pigs data were used starting from the moment of

For the biomarker SAA a significant difference between survivors (S) and non-survivors (NS) was found for the a-parameters of the AR models (mean S = -0.7915, mean NS = -1.3204,

p = 0.04). Fig. 1 shows a scatter plot of the a-parameters of the different pigs.

<sup>1</sup> y(k)A(z ) e(k) (1)

y(k) a y(k 1) e(k) <sup>1</sup> (2)

**2.2.1 Data generation** 

experiment.

time series form:

**2.2.3 Monitoring criteria** 

2007):

**2.2.2 Modelling of biomarker responses to infection** 

the early detection of survival/non-survival.

infection until 16 hours after infection.

Fig. 1. Criterion for early detection of survival/non-survival based on SAA: stability of model. Overview of a-parameters of AR-models for all surviving (blue) and non-surviving (red) pigs

Also for the biomarker IL-6, there was a significant difference in a-parameters between survivors and non-survivors (mean S = -0.1272, mean NS = -0.5751, p=0.0085). Fig. 2 shows a scatter plot of the a-parameters for all pigs.

Fig. 2. Criterion for early detection of survival/non-survival based on IL-6: trend vs noise. Overview of a-parameters of AR-models for all surviving (blue) and non-surviving (red) pigs

Non-Invasive Methods for Monitoring Individual Bioresponses in Relation to Health Management 149

diagnostic tests (Bates et al., 1999). The nurses have stated their expectations of Personal Digital Assistant (PDA) devices in the nursing practice (Nilsson et al., 2007). The computerized versions of the pain scales and questionnaires have lately been introduced successfully in pain assessment practice (Wincent et al. 2003). It has been reported in the literature that the computerized version of the self-report pain questionnaires is not altering the response of the patients (Caro et al., 2001). It is also decreasing the number of the missing responses by obligating the patients to answer before proceeding to the next question (Caro et al., 2001). Compared to the paper versions, the electronic pain self-report questionnaires and electronic diaries are offering considerable advantages: 1) completeness of data (Hanscom et al., 2002); 2) entered data are date and time stamped (Burton et al., 2007); 3) saving of time and reduction of errors from entering written data manually to the

During the Painvision project (www.painvision.be), a digital scale was developed for data collection. This project took place in a specific geriatric centre, and was approved by a medical ethical committee. In this pilot study facial images of a bedside two-camera system were linked to the pain scores of the digital device (a tablet PC with a touch screen, Fig. 3, a commercial version has been introduced to the market by BioRICS nv. more details can be found on www.assessmentscales.com) carried out at the bedside by a nurse.

Fig. 3. Two implementations of the assessment scales (pictures courtesy of BioRICS nv

As input for the digital device, 3 valid and reliable scales were chosen: the Pain Assessment Checklist for Seniors with Limited Ability to Communicate (PACSLAC), the Discomfort Scale - Dementia of Alzheimer Type, and the Faces Pain Scale Revised. After an informed consent was signed by a relative, two nurses tracked nineteen bedridden patients, with limited ability to communicate, for 6 random days, in which 6 assessment sessions were performed at clinically interesting moments (before – during – after care, before - after manipulation, and at rest). The usability was more concretely evaluated by ten other professional caregivers of this geriatric centre, via the 'think aloud method' and a

database for the analysis Ryan et al., 2002.

(www.biorics.com)

These results show that there tend to be differences in dynamics of biomaekers between surviving and non-surviving pigs after infection. Although many more experiments are needed in order to confirm the actual findings, it is expected that the modelling results could be the first steps towards the development of an objective individualised method for a sensor-based early detection of sepsis and inflammation in animals and later on in humans. Combining the modelling approach with novel biosensors should allow monitoring the health status of animals and human patients in real-time and could form the basis of health management systems.

#### **2.3 Pain management in elderly suffering from dementia**

Pain in elderly at the latest stages of dementia is an underestimated factor for discomfort and quality of living. Being able to objectively measure pain allows caregivers to adjust the treatment of these patients with limited communication skils. In this regard, a big number of pen-paper observational pain scales have been developed by various researchers. However, data handling from these scales is not very easy and their use is limited by the amount of time that a caregiver can spend on a patient. To account for this, the Painvision (www.painvision.be) consortium has developed an electronic version of three popular observational pain scales (see section 2.3.1 below) that is currently sold by BioRICS nv (a K.U.Leuven spin-off, www.biorics.com).

The overall objective of the Painvision consortium was to develop an automatic pain detection system based on cameras. Since facial expression is a well-known and reliable indicator of pain, the algorithms of the system exploit this relationship and estimate the pain level of the observed patient. Continuous information about the pain level of a patient can subsequently be used to evaluate the medical as well as the physical treatment that the patient is receiving, their comfort level throughout the day and eventually their quality of life.

#### **2.3.1 Electronic pain observational scales (assessment scales)**

Pen-paper observational scales have proven to be useful to assess pain in severely demented elderly, but also have a lot of disadvantages. Their usability is often limited, they are timeconsuming because of their length, difficulties in calculating scores, and post processing required to evaluate the pain evolution. Also the timing and timing patterns of the indicators could be very valuable to pain assessment, but cannot be grasped using pen-paper assessment instruments. The American Pain Society has indicated the importance of the pain assessment and suggested the guidelines for improving its quality (Max et al., 1995). In case a patient is not able to report his/her pain experience verbally, it is recommended to measure the painrelated behaviours (e.g., grimacing, restlessness, vocalisation, etc.). Demented elderly have limited ability to communicate verbally, as a result of which self report of pain is difficult. Therefore, different observational pain assessment scales have been developed and validated for this group of patients (Herr et al., 2006). Despite the introduction of new technologies in healthcare, to the authors' knowledge, the scores for all pain scales for severely demented elderly are still obtained manually on the paper.

Computerized technologies are already introduced in home care (Koch, 2006) and gaining high satisfaction response among patients (Chae et al., 2001; Lind et al., 2008) . The computerized-assisted decision systems are utilized in clinical practice (Mikulich, et al. 2001). The information systems are reducing costs and improving quality in managing

These results show that there tend to be differences in dynamics of biomaekers between surviving and non-surviving pigs after infection. Although many more experiments are needed in order to confirm the actual findings, it is expected that the modelling results could be the first steps towards the development of an objective individualised method for a sensor-based early detection of sepsis and inflammation in animals and later on in humans. Combining the modelling approach with novel biosensors should allow monitoring the health status of animals and human patients in real-time and could form the basis of health

Pain in elderly at the latest stages of dementia is an underestimated factor for discomfort and quality of living. Being able to objectively measure pain allows caregivers to adjust the treatment of these patients with limited communication skils. In this regard, a big number of pen-paper observational pain scales have been developed by various researchers. However, data handling from these scales is not very easy and their use is limited by the amount of time that a caregiver can spend on a patient. To account for this, the Painvision (www.painvision.be) consortium has developed an electronic version of three popular observational pain scales (see section 2.3.1 below) that is currently sold by BioRICS nv

The overall objective of the Painvision consortium was to develop an automatic pain detection system based on cameras. Since facial expression is a well-known and reliable indicator of pain, the algorithms of the system exploit this relationship and estimate the pain level of the observed patient. Continuous information about the pain level of a patient can subsequently be used to evaluate the medical as well as the physical treatment that the patient is receiving, their comfort level throughout the day and eventually their quality of

Pen-paper observational scales have proven to be useful to assess pain in severely demented elderly, but also have a lot of disadvantages. Their usability is often limited, they are timeconsuming because of their length, difficulties in calculating scores, and post processing required to evaluate the pain evolution. Also the timing and timing patterns of the indicators could be very valuable to pain assessment, but cannot be grasped using pen-paper assessment instruments. The American Pain Society has indicated the importance of the pain assessment and suggested the guidelines for improving its quality (Max et al., 1995). In case a patient is not able to report his/her pain experience verbally, it is recommended to measure the painrelated behaviours (e.g., grimacing, restlessness, vocalisation, etc.). Demented elderly have limited ability to communicate verbally, as a result of which self report of pain is difficult. Therefore, different observational pain assessment scales have been developed and validated for this group of patients (Herr et al., 2006). Despite the introduction of new technologies in healthcare, to the authors' knowledge, the scores for all pain scales for severely demented

Computerized technologies are already introduced in home care (Koch, 2006) and gaining high satisfaction response among patients (Chae et al., 2001; Lind et al., 2008) . The computerized-assisted decision systems are utilized in clinical practice (Mikulich, et al. 2001). The information systems are reducing costs and improving quality in managing

management systems.

life.

**2.3 Pain management in elderly suffering from dementia** 

**2.3.1 Electronic pain observational scales (assessment scales)** 

(a K.U.Leuven spin-off, www.biorics.com).

elderly are still obtained manually on the paper.

diagnostic tests (Bates et al., 1999). The nurses have stated their expectations of Personal Digital Assistant (PDA) devices in the nursing practice (Nilsson et al., 2007). The computerized versions of the pain scales and questionnaires have lately been introduced successfully in pain assessment practice (Wincent et al. 2003). It has been reported in the literature that the computerized version of the self-report pain questionnaires is not altering the response of the patients (Caro et al., 2001). It is also decreasing the number of the missing responses by obligating the patients to answer before proceeding to the next question (Caro et al., 2001). Compared to the paper versions, the electronic pain self-report questionnaires and electronic diaries are offering considerable advantages: 1) completeness of data (Hanscom et al., 2002); 2) entered data are date and time stamped (Burton et al., 2007); 3) saving of time and reduction of errors from entering written data manually to the database for the analysis Ryan et al., 2002.

During the Painvision project (www.painvision.be), a digital scale was developed for data collection. This project took place in a specific geriatric centre, and was approved by a medical ethical committee. In this pilot study facial images of a bedside two-camera system were linked to the pain scores of the digital device (a tablet PC with a touch screen, Fig. 3, a commercial version has been introduced to the market by BioRICS nv. more details can be found on www.assessmentscales.com) carried out at the bedside by a nurse.

Fig. 3. Two implementations of the assessment scales (pictures courtesy of BioRICS nv (www.biorics.com)

As input for the digital device, 3 valid and reliable scales were chosen: the Pain Assessment Checklist for Seniors with Limited Ability to Communicate (PACSLAC), the Discomfort Scale - Dementia of Alzheimer Type, and the Faces Pain Scale Revised. After an informed consent was signed by a relative, two nurses tracked nineteen bedridden patients, with limited ability to communicate, for 6 random days, in which 6 assessment sessions were performed at clinically interesting moments (before – during – after care, before - after manipulation, and at rest). The usability was more concretely evaluated by ten other professional caregivers of this geriatric centre, via the 'think aloud method' and a

Non-Invasive Methods for Monitoring Individual Bioresponses in Relation to Health Management 151

Initially the incoming frame is passed to the face detector if the previous image in the sequence didn't contain a face. On the other hand when the previous frame did contain a face, a tracker will be used to relocate faces. At this stage it is known if the frame contains face(s) and their location. After detecting/tracking the face a rough 3D position of the head is known. However the normalization step requires an accurate estimation of the pose. Therefore the normalization step is headed by the pose estimation step. This step will iteratively estimate a more accurate pose. Once the pose in the incoming frame is known a normalization step can be preformed to bring the face to a frontal and fixed scale face. Next a 2D active appearance model (AAM) will be fitted. The parameters of this adaptive model describe both texture and shape information of the face. Now AAM parameters, the normalized face image, landmarks in the original and normalized image can be used in

It should be noted that the normalized texture is a forward warp of the original image pixels rather than a synthetic AAM instance. Although the warp could be deformed and incomplete, detailed texture features are preserved since the original texture is used. For instance wrinkles and person specific spots remain visible in the normalized image. These clues could be crucial in classification problems. Next, Fig. 5 is presenting the algorithm result for the detection of an 'open mouth' that is a pain indicator for demented

Fig. 5. Example of the different algorithm steps (i.e. face detection, 3D rigid fit, pose normalisation, AAM mask fit). From the different AAM parameters (top plot), an index is extracted (middle plot) that results in the detection of an open mouth

extracting pain related information.

elderly.

(bottom plot)

questionnaire. They performed a digital pain assessment twice, with an interval of four weeks, on two patients with severe dementia. Usability criteria were learnability, efficiency of use, number of manipulation errors and satisfaction.

The digital device allows the nurse to record facial indicator events, such as frequency and duration, as they occur in real time. Subsequently, the scores are calculated automatically. The digital information is stored in a database, improving administration and allowing database applications. The ten professional caregivers stated that the tool was easy to learn. After the second measurement their assessment time was reduced with approximately 50%, the number of detected manipulation errors was up to four times lower, and the general satisfaction has significantly increased (p =0.04).

The Digital Pain Labelling Tool provides data completeness, reduces errors from the manual pen paper pain scores and offers easier and faster analysis of the patient's current condition. These findings illuminate the potential of implementing the computerized observational pain scale in nursing practice.

#### **2.3.2 Automatic pain identification using a two-camera system**

The Painvision project focused on the development of an algorithm for automatic estimation of the pain levels of elderly suffering from dementia by use of a two-camera system. The cameras are directed to the face of the patient that is bedridden. Initially, the face of the person is detected in both videos and, subsequently, pain indicators are automatically extracted. This procedure is presented in Fig. 4 and the different steps are explained in more detail in the following.

Fig. 4. Block diagram of the video processing algorithm for automatic estimation of pain in demented elderly based on visual information

questionnaire. They performed a digital pain assessment twice, with an interval of four weeks, on two patients with severe dementia. Usability criteria were learnability, efficiency

The digital device allows the nurse to record facial indicator events, such as frequency and duration, as they occur in real time. Subsequently, the scores are calculated automatically. The digital information is stored in a database, improving administration and allowing database applications. The ten professional caregivers stated that the tool was easy to learn. After the second measurement their assessment time was reduced with approximately 50%, the number of detected manipulation errors was up to four times lower, and the general

The Digital Pain Labelling Tool provides data completeness, reduces errors from the manual pen paper pain scores and offers easier and faster analysis of the patient's current condition. These findings illuminate the potential of implementing the computerized observational

The Painvision project focused on the development of an algorithm for automatic estimation of the pain levels of elderly suffering from dementia by use of a two-camera system. The cameras are directed to the face of the patient that is bedridden. Initially, the face of the person is detected in both videos and, subsequently, pain indicators are automatically extracted. This procedure is presented in Fig. 4 and the different steps are explained in more

of use, number of manipulation errors and satisfaction.

satisfaction has significantly increased (p =0.04).

**2.3.2 Automatic pain identification using a two-camera system** 

Video sequence

Use tracking? Detect face Track face

> Normalisation (pose, scale)

Pose detection (3D masking)

Apply mask (2D AAM)

> Good quality?

demented elderly based on visual information

Extract parameters

Fig. 4. Block diagram of the video processing algorithm for automatic estimation of pain in

Classifiers Pain

Indicators

pain scale in nursing practice.

detail in the following.

Discard image

Initially the incoming frame is passed to the face detector if the previous image in the sequence didn't contain a face. On the other hand when the previous frame did contain a face, a tracker will be used to relocate faces. At this stage it is known if the frame contains face(s) and their location. After detecting/tracking the face a rough 3D position of the head is known. However the normalization step requires an accurate estimation of the pose. Therefore the normalization step is headed by the pose estimation step. This step will iteratively estimate a more accurate pose. Once the pose in the incoming frame is known a normalization step can be preformed to bring the face to a frontal and fixed scale face. Next a 2D active appearance model (AAM) will be fitted. The parameters of this adaptive model describe both texture and shape information of the face. Now AAM parameters, the normalized face image, landmarks in the original and normalized image can be used in extracting pain related information.

It should be noted that the normalized texture is a forward warp of the original image pixels rather than a synthetic AAM instance. Although the warp could be deformed and incomplete, detailed texture features are preserved since the original texture is used. For instance wrinkles and person specific spots remain visible in the normalized image. These clues could be crucial in classification problems. Next, Fig. 5 is presenting the algorithm result for the detection of an 'open mouth' that is a pain indicator for demented elderly.

Fig. 5. Example of the different algorithm steps (i.e. face detection, 3D rigid fit, pose normalisation, AAM mask fit). From the different AAM parameters (top plot), an index is extracted (middle plot) that results in the detection of an open mouth (bottom plot)

Non-Invasive Methods for Monitoring Individual Bioresponses in Relation to Health Management 153

focuses on REM sleep because this state resembles most to wakefulness and is being linked to dreaming and memory consolidation (Karni et al., 1994; Tilley & Empson, 1978; Takahara et al., 2008). In the same direction, during the course of the EASI project, an algorithm was developed that is automatically detecting periods of REM sleep. Additionally, the algorithm is contributing to the discussion of whether dreams occur only during REM sleep or not, by exploiting the concept of Additional Heart Rate and its link to emotions (Myrtek, 2004) during sleep. An example of the algorithm output is

1 2 3 4 5 6 7

Sleep stage Detected awakening

Time (h)

Fig. 6. Manual scoring of the sleep stages and the output of the developed algorithm for

Fig. 7. Example of the algorithm output for the REM detection algorithm

presented in Fig. 7.

Short Awakening

SWS

awakening detection

S2

S1 REM Awakening
