**8.4. Which parameters should be present?**

In the past many authors have not known how to present their cardiac output data from validations studies in a meaningful and useful manner. When presenting data on a scatter plot one should include the number of data points in the plot. Attention also needs to be given the scale used on the axes so that false impressions of the spread of the data are avoided. Ideally the axes should be of equal scale and range from zero to the maximum value of cardiac output. If a regression line is added, the equation of line should be shown. Correlation analysis is not required unless serial data that shows trending is being used.

be shown. Also, time plots provide only graphical evidence and an objective measure of

Minimally Invasive Cardiac Output Monitoring in the Year 2012

http://dx.doi.org/10.5772/54413

69

**Figure 14.** Time plot showing the relationship between test and reference cardiac output readings over time. Data pairs come from a single patient collected at intervals during surgery. The test method follows changes in reference cardiac output despite the test method under-reading by approximately 0.75 L/min. Thus, reliable trending ability is

The variable commonly used to assess trending in statistical analysis is delta cardiac out‐ put (∆CO), the difference between successive readings, or the change in cardiac output

Bland-Altman analysis does not show trending, so other analytical methods are used. There is limited consensus on which analytical method should be used [37]. In clinical trials con‐

The four quadrant plot is simply a scatter plot showing delta cardiac output (∆CO) for the test method against the reference method. Because the changes in cardiac output are used, the x and y axes pass through zero (0,0) at the centre of the plot. The delta data points should lie along the line of identity (y=x) if good trending is present (Figure 15). The earliest reference to

cordance using a four-quadrant plot has become the standard method.

this method appeared in the mid 1990's [48,49].

trending is also needed.

demonstrated in the patient.

(COb-COa).

**8.7. The four-quadrant plot**

Similar issues apply to the Bland-Altman plot. In particular, the range of cardiac output on the x-axis and the range of values for bias need to be appropriate. If several plots comparing data from several devices or patient groups are shown the scales on each plot should be equivalent.

The important data measured using the Bland-Altman analysis are:


The study size and percentage error at least should be presented with the Bland-Altman plot.

### **8.5. Percentage error and the 30%**

The percentage error is calculated using the formula "1.96 x standard deviation of the bias / mean cardiac output" and is expressed as a percentage. It represents a normalized version of the limits of agreement. The percentage error enables one to compare data from different studies when the ranges of cardiac outputs are different. Even today many authors still fail to present percentage error.

Following a meta-analysis of data from cardiac output studies published pre-1997 that used Bland-Altman analysis we proposed that when the percentage error was less than 28.4%, it was reasonable to accept the new test method. However, the reference method had to be thermodilution with an estimated precision was ±20% [39]. Our work lead to the 30% bench‐ mark for percentage error quoted in many publications over the last a decade. An error-gram was published in our 1999 paper to allow for adjustment to this threshold when reference methods of different precision errors were used.

#### **8.6. Showing reliable trending ability**

To assess the trending ability of a new monitor against a reference method one uses seri‐ al cardiac output readings. The simplest way to show trending is to plotting the test and reference methods together against time (Figure 14). However, time plots only show data from a single subject, but to confirm reliable trending data from several subjects needs to be shown. Also, time plots provide only graphical evidence and an objective measure of trending is also needed.

**Figure 14.** Time plot showing the relationship between test and reference cardiac output readings over time. Data pairs come from a single patient collected at intervals during surgery. The test method follows changes in reference cardiac output despite the test method under-reading by approximately 0.75 L/min. Thus, reliable trending ability is demonstrated in the patient.

#### **8.7. The four-quadrant plot**

**8.4. Which parameters should be present?**

**i.** The mean bias,

68 Artery Bypass

or Limits of Agreement,

methods of different precision errors were used.

**8.6. Showing reliable trending ability**

**iii.** The mean cardiac output and

**8.5. Percentage error and the 30%**

present percentage error.

required unless serial data that shows trending is being used.

The important data measured using the Bland-Altman analysis are:

**iv.** A calculated parameter called the percentage error.

In the past many authors have not known how to present their cardiac output data from validations studies in a meaningful and useful manner. When presenting data on a scatter plot one should include the number of data points in the plot. Attention also needs to be given the scale used on the axes so that false impressions of the spread of the data are avoided. Ideally the axes should be of equal scale and range from zero to the maximum value of cardiac output. If a regression line is added, the equation of line should be shown. Correlation analysis is not

Similar issues apply to the Bland-Altman plot. In particular, the range of cardiac output on the x-axis and the range of values for bias need to be appropriate. If several plots comparing data from several devices or patient groups are shown the scales on each plot should be equivalent.

**ii.** The standard deviation of the bias which is presented as the 95% confidence intervals

The study size and percentage error at least should be presented with the Bland-Altman plot.

The percentage error is calculated using the formula "1.96 x standard deviation of the bias / mean cardiac output" and is expressed as a percentage. It represents a normalized version of the limits of agreement. The percentage error enables one to compare data from different studies when the ranges of cardiac outputs are different. Even today many authors still fail to

Following a meta-analysis of data from cardiac output studies published pre-1997 that used Bland-Altman analysis we proposed that when the percentage error was less than 28.4%, it was reasonable to accept the new test method. However, the reference method had to be thermodilution with an estimated precision was ±20% [39]. Our work lead to the 30% bench‐ mark for percentage error quoted in many publications over the last a decade. An error-gram was published in our 1999 paper to allow for adjustment to this threshold when reference

To assess the trending ability of a new monitor against a reference method one uses seri‐ al cardiac output readings. The simplest way to show trending is to plotting the test and reference methods together against time (Figure 14). However, time plots only show data from a single subject, but to confirm reliable trending data from several subjects needs to The variable commonly used to assess trending in statistical analysis is delta cardiac out‐ put (∆CO), the difference between successive readings, or the change in cardiac output (COb-COa).

Bland-Altman analysis does not show trending, so other analytical methods are used. There is limited consensus on which analytical method should be used [37]. In clinical trials con‐ cordance using a four-quadrant plot has become the standard method.

The four quadrant plot is simply a scatter plot showing delta cardiac output (∆CO) for the test method against the reference method. Because the changes in cardiac output are used, the x and y axes pass through zero (0,0) at the centre of the plot. The delta data points should lie along the line of identity (y=x) if good trending is present (Figure 15). The earliest reference to this method appeared in the mid 1990's [48,49].

with the upper right (i.e. positive changes) and lower left (i.e. negative changes) quadrants of concordance. The concordance rate is 98% as one data point lie outside these quadrants.

Minimally Invasive Cardiac Output Monitoring in the Year 2012

http://dx.doi.org/10.5772/54413

71

When performing concordance analysis one needs to know what is an acceptable rate? In a re‐ cent publication on trend analysis, we analyzed data from nine studies that used concordance analysis. From this data we concluded that for good trending ability to be shown against ther‐

Concordance analysis and the four quadrant plot have limitations. The changes in cardiac output between the test and reference methods can be very different yet concord if both have the same direction of change and the magnitude of the change in cardiac output plays no part in the analysis other than determining what data is excluded. To address these issues we developed a method of concordance analysis based on converting the data to polar coordinates. The polar angle represented agreement whilst the radius represented the magnitude of change in cardiac output [50]. The polar data is generated from the ∆CO(test) and ∆CO(reference).

**Figure 16.** The polar plot displays ∆CO data. The axis of the plot lies at 0-degree (and 180-degrees). It is equivalent to the line of identity y=x on the scatter plot (figure 12), except that the plot has been rotated clockwise by 45-degrees. Concord‐ ance limits are draw at ±30-degrees. A circular exclusion zone of 0.5 L/min is draw at the centre. Data points that lies within

modilution as a reference method the concordance rate should be 92% or above [37].

Descriptions on how to draw polar plots are found in our paper.

**8.9. Polar plots**

**Figure 15.** Four quadrant scatter plot comparing changes in test and reference cardiac output (∆CO) readings. The plot is divided into four quadrants about the x and y axis that cross at the centre (0,0). Data points lie along the line (dashed) of identity y = x. A square exclusion zone is drawn at the centre to remove statistical noise. Concordance analysis is performed by counting the number of data points remaining after central zone exclusion that lie within the two quadrants of agreement (upper right and lower left). In the plot 98% of the data concords, thus trending ability is very good. Supra-sternal and oesophageal Doppler were being compared.

The concordance is measured as the proportion of data points in which either both methods change in a positive direction (i.e. increase and lie within the right upper quadrant) or change in a negative direction (i.e. decrease and lie within the left lower quadrant). Data points that do not concord (i.e. change in different direction) lie within the upper left or lower right quadrants. The concordance rate is the percentage of data points that are in concordance or agree regarding the direction of change of cardiac output.

#### **8.8. The central exclusion zone**

One of the main problems encountered when using the four quadrant plot is that data points close to its centre, which represent relatively small cardiac output changes, often do not concord because random error effects are of similar magnitude to the cardiac output changes. This phe‐ nomenon results in statistical noise that adversely affects the concordance rate. Perrino and col‐ leagues introduced a central exclusion zone to reduce the level of these random error effects [49].

Receiver operator characteristic (ROC) curve analysis of Perrino and colleagues data was performed to predict the most desirable exclusion zone [48]. For a mean cardiac output of 5.0 L/min these author recommended an exclusion zone of 0.75 L/min or 15%. In the above example it can seen that after central zone exclusion of data, most of the remaining data lie with the upper right (i.e. positive changes) and lower left (i.e. negative changes) quadrants of concordance. The concordance rate is 98% as one data point lie outside these quadrants.

When performing concordance analysis one needs to know what is an acceptable rate? In a re‐ cent publication on trend analysis, we analyzed data from nine studies that used concordance analysis. From this data we concluded that for good trending ability to be shown against ther‐ modilution as a reference method the concordance rate should be 92% or above [37].

#### **8.9. Polar plots**

**Figure 15.** Four quadrant scatter plot comparing changes in test and reference cardiac output (∆CO) readings. The plot is divided into four quadrants about the x and y axis that cross at the centre (0,0). Data points lie along the line (dashed) of identity y = x. A square exclusion zone is drawn at the centre to remove statistical noise. Concordance analysis is performed by counting the number of data points remaining after central zone exclusion that lie within the two quadrants of agreement (upper right and lower left). In the plot 98% of the data concords, thus trending ability is

The concordance is measured as the proportion of data points in which either both methods change in a positive direction (i.e. increase and lie within the right upper quadrant) or change in a negative direction (i.e. decrease and lie within the left lower quadrant). Data points that do not concord (i.e. change in different direction) lie within the upper left or lower right quadrants. The concordance rate is the percentage of data points that are in concordance or

One of the main problems encountered when using the four quadrant plot is that data points close to its centre, which represent relatively small cardiac output changes, often do not concord because random error effects are of similar magnitude to the cardiac output changes. This phe‐ nomenon results in statistical noise that adversely affects the concordance rate. Perrino and col‐ leagues introduced a central exclusion zone to reduce the level of these random error effects

Receiver operator characteristic (ROC) curve analysis of Perrino and colleagues data was performed to predict the most desirable exclusion zone [48]. For a mean cardiac output of 5.0 L/min these author recommended an exclusion zone of 0.75 L/min or 15%. In the above example it can seen that after central zone exclusion of data, most of the remaining data lie

very good. Supra-sternal and oesophageal Doppler were being compared.

agree regarding the direction of change of cardiac output.

**8.8. The central exclusion zone**

[49].

70 Artery Bypass

Concordance analysis and the four quadrant plot have limitations. The changes in cardiac output between the test and reference methods can be very different yet concord if both have the same direction of change and the magnitude of the change in cardiac output plays no part in the analysis other than determining what data is excluded. To address these issues we developed a method of concordance analysis based on converting the data to polar coordinates. The polar angle represented agreement whilst the radius represented the magnitude of change in cardiac output [50]. The polar data is generated from the ∆CO(test) and ∆CO(reference). Descriptions on how to draw polar plots are found in our paper.

**Figure 16.** The polar plot displays ∆CO data. The axis of the plot lies at 0-degree (and 180-degrees). It is equivalent to the line of identity y=x on the scatter plot (figure 12), except that the plot has been rotated clockwise by 45-degrees. Concord‐ ance limits are draw at ±30-degrees. A circular exclusion zone of 0.5 L/min is draw at the centre. Data points that lies within

these limits concord. Positive changes in cardiac output (∆CO) (right half) and negative ∆CO (left half) are presented on op‐ posite halves of the plot. The mean polar angle and radial limits of agreement for data have been omitted.

**Figure 17.** Half-moon polar plot showing the same data as the full-circle plot, but all within the same semi-circle. The mean polar angle and radial limits of agreement are now shown. A central exclusion zone circle removes data points where the changes in cardiac output are small. Trending of cardiac output is good because most of the data points lie within the 30-degrees of the polar axis (0-degrees). Concordance is performed by counting the percentage of data points that lie within this zone. Outcomes of the polar analysis are provided with the plots. (Graphs drawn using Sig‐

Minimally Invasive Cardiac Output Monitoring in the Year 2012

http://dx.doi.org/10.5772/54413

73

The exclusion zone is used for similar reasons as in the four quadrant plot. However, as the radial distance is mean cardiac output rather that the hypotenuse of a triangle bounded by two cardiac output readings reference and test, its 'size needs to be smaller by a ratio of 1 to 1.4. Thus, rather than using 0.75 L/min or 15% as in the four quadrant plot, we used 0.5 L/min.

If evidence based approaches are to be adopted when using MICOM devices in ones clinical practice then data from clinical validation studies will need to be critically reviewed. Market‐ ing information from most manufactures of MICOM devices provide lists of publications that they claim support their product. In reviewing such data one needs to ask the following ques‐

Study design is critical. (a) A sufficient number of patients should have been studied, though calculating the power of validation studies is not easy. Comparison of study size with other similar validation studies may help. (b) Type of patients and clinical setting effects results. Situations where a wide range of cardiac outputs and conditions (i.e. peripheral resistance) are encountered provide a rigorous test of performance. (c) Some of the early and more favourable validation studies using pulse contour devices were performed in cardiac surgery patients in whom haemodynamics were kept relatively stable. It was only when the same devices were tested in more labile liver transplant patients with cirrhosis that the problem with these devices

The different statistical methods used in validation have been systematically covered previ‐ ously. (a) If a simple test versus reference method comparison has been performed then only Bland-Altman analysis is needed, but make sure the outcomes of the analysis are properly presented, including the percentage error. (b) If a sophisticated study design that allows trending to be assessed has been used, then concordance analysis using the four quadrant plot, and possibly a polar analysis should have been used to show trending. Check that central exclusions zones have been applied to the ∆CO data. (c) Animal studies are slightly different because of extent and quality of data that can be collected, and it is reasonable to use regression

When interpreting the results of Bland-Altman analysis: (a) Make sure the precision error of the reference method is correct. Normally for thermodilution it is ±20%, but other modalities

ma Plot version 7.0).

tions:

analysis.

**8.10. Making sense of the outcomes**

**i.** Is the study design and data appropriate?

**iii.** Have the correct criteria been applied to results?

**ii.** Have the correct statistics been used?

and peripheral resistance became apparent [51].

**iv.** And are the conclusions correct?

Our earliest description of polar plots used a full 360-degree circle to show both positive and negative directional changes (Figure 16). The data points are seen to lie within narrow ±30 degree sectors about the polar axes signifying good trending ability. When 30-degree limits are used the allowable differences in size of ∆CO are limited to a ratio of 1 to 2, rather than just direction of change.

The half moon plot was later developed to show positive and negative ∆CO changes together (Figure 17).

The plot provides several parameters that describe trending:


**Figure 17.** Half-moon polar plot showing the same data as the full-circle plot, but all within the same semi-circle. The mean polar angle and radial limits of agreement are now shown. A central exclusion zone circle removes data points where the changes in cardiac output are small. Trending of cardiac output is good because most of the data points lie within the 30-degrees of the polar axis (0-degrees). Concordance is performed by counting the percentage of data points that lie within this zone. Outcomes of the polar analysis are provided with the plots. (Graphs drawn using Sig‐ ma Plot version 7.0).

The exclusion zone is used for similar reasons as in the four quadrant plot. However, as the radial distance is mean cardiac output rather that the hypotenuse of a triangle bounded by two cardiac output readings reference and test, its 'size needs to be smaller by a ratio of 1 to 1.4. Thus, rather than using 0.75 L/min or 15% as in the four quadrant plot, we used 0.5 L/min.
