Table 4. **Correlation variable-component**

Those correlations are also known as component loadings. A coefficient greater than 0.4 in absolute value is considered as significant (see, Stevens (1986) for a discussion). We can interpret PCA1 as being highly positively correlated with variables X1, X2 and X3, and weakly positively correlated to variables X4 and X5. So X1, X2 and X3 are the most important variables in the first principal component. PCA2, on the other hand, is highly positively correlated with X4 and X5, and weakly negatively related to X1 and X2. So X4 and X5 are most important in explaining the second principal component. Therefore, the name of the first component comes from variables X1, X2 and X3 while that of the second component comes from X4 and X5.

It can be shown that the coordinate of a variable on a component is the correlation coefficient between that variable and the principal component. This allows us to plot the reduced dimension representation of variables in the plane constructed from the first two components. Variables highly correlated with a component show a small angle. Figure 2 represents this graph for our dataset. For each variable we have plotted on the horizontal dimension its loading on component 1, on the vertical dimension its loading on component 2.

The graph also presents a visual aspect of correlation patterns among variables. The cosine of the angle θ between two vectors *x* and *y* is computed as:

$$<\text{x}, y> = \|\mathbf{x}\| \|y\| \cos(\mathbf{x}, y) \tag{28}$$

The Basics of Linear Principal Components Analysis 195

variable. It measures the proportion of the variance of a variable accounted for by the components. For example, in our example, the communality of the variable X1 is 0.9432+0.2412=0.948. This means that the first two components explain about 95% of the variance of the variable X1. This is quite substantial to enable us fully interpreting the variability in this variable as well as its relationship with the other variables. Communality can be used as a measure of goodness-of-fit of the projection. The communalities of the 5 variables of our data are displayed in Table 5. As shown by this Table, the first two components explain more than 80% of variance in each variable. This is enough to reveal the structure of correlation among the variables. Do not interpret as correlation the angle between two variables when at least one of them has a low communality. Using communality prevent potential biases that may arise by directly interpreting numerical and

> **Variables Value** X1 0.948 X2 0.920 X3 0.817 X4 0.970 X5 0.976

All these interesting results show that outcomes from normalized PCA can be easily interpreted without additional complicated calculations. From a visual inspection of the graph, we can see the groups of variables that are correlated, interpret the principal

A useful by product of PCA is factor scores. Factor scores are coordinates of subjects (individuals) on each component. They indicate where a subject stands on the retained component. Factor scores are computed as weighted values on the observed variables.

Factor scores can be used to plot a reduced representation of subjects. This is displayed by

How do we interpret the position of points on this diagram? Recall that this graph is a projection. As such some distances could be spurious. To distinguish wrong projections from real ones and better interpret the plot, we need to use that is called "the quality of representation" of subjects. This is computed as the squared of the cosine of the angle

2 2

*z z*

*<sup>i</sup> ki k*

2 1

(30)

*x* 

2

*s*

cos ( , ) *i i i j p*

graphical results yielded by the PCA.

Table 5. **Communalities of variables**

**6.2 Factor scores and their use in multivariate models** 

between a subject *<sup>i</sup> s* and a component *z* , following the formula:

2

*s z*

Results for our dataset are reported in Table 6.

components and name them.

Figure 3.

#### Fig. 2. **Circle of Correlation**

Replacing *x* and *y* with our transformed vectors yields:

$$\cos(\mathbf{x}, y) = \frac{\sum\_{i=1}^{n} (\mathbf{x}\_i - \overline{\mathbf{x}})(y\_i - \overline{y})}{\sqrt{\left(\sum\_{i=1}^{n} (\mathbf{x}\_i - \overline{\mathbf{x}})^2\right) \left(\sum\_{i=1}^{n} (y\_i - \overline{y})^2\right)}} = \rho(\mathbf{x}, y) \tag{29}$$

Eq.(29) shows the connection between the cosine measurement and the numerical measurement of correlation: the cosine of the angle between two variables is interpreted in terms of correlation. Variables highly positively correlated with each another show a small angle, while those are negatively correlated are directed in opposite sense, i.e. they form a flat angle. From Figure 2 we can see that the five variables hang together in two distinct groups. Variables X1, X2 and X3 are positively correlated with each other, and form the first group. Variables X4 and X5 also correlate strongly with each other, and form the second group. Those two groups are weakly correlated. In fact, Figure 2 gives a reduced dimension representation of the correlation matrix given in Table 2.

It is extremely important, however, to notice that the angle between variables is interpreted in terms of correlation only when variables are well-represented, that is they are close to the border of the circle of correlation. Remember that the goal of PCA is to explain multiple variables by a lesser number of components, and keep in mind that graphs obtained from that reduction method are projections that optimize global criterion (i.e. the total variance). As such some relationships between variables may be greatly altered. Correlations between variables and components supply insights about variables that are not well-represented. In a subspace of components, the quality of representation of a variable is assessed by the sumof-squared component loadings across components. This is called the communality of the 194 Principal Component Analysis

 

*x xy y x y x y xx yy*

1

*i*

*n*

2 1 1

Eq.(29) shows the connection between the cosine measurement and the numerical measurement of correlation: the cosine of the angle between two variables is interpreted in terms of correlation. Variables highly positively correlated with each another show a small angle, while those are negatively correlated are directed in opposite sense, i.e. they form a flat angle. From Figure 2 we can see that the five variables hang together in two distinct groups. Variables X1, X2 and X3 are positively correlated with each other, and form the first group. Variables X4 and X5 also correlate strongly with each other, and form the second group. Those two groups are weakly correlated. In fact, Figure 2 gives a reduced dimension

It is extremely important, however, to notice that the angle between variables is interpreted in terms of correlation only when variables are well-represented, that is they are close to the border of the circle of correlation. Remember that the goal of PCA is to explain multiple variables by a lesser number of components, and keep in mind that graphs obtained from that reduction method are projections that optimize global criterion (i.e. the total variance). As such some relationships between variables may be greatly altered. Correlations between variables and components supply insights about variables that are not well-represented. In a subspace of components, the quality of representation of a variable is assessed by the sumof-squared component loadings across components. This is called the communality of the

*n n i i*

*i i*

cos( , ) (,)

*i i*

2

(29)

Fig. 2. **Circle of Correlation**

Replacing *x* and *y* with our transformed vectors yields:

representation of the correlation matrix given in Table 2.

variable. It measures the proportion of the variance of a variable accounted for by the components. For example, in our example, the communality of the variable X1 is 0.9432+0.2412=0.948. This means that the first two components explain about 95% of the variance of the variable X1. This is quite substantial to enable us fully interpreting the variability in this variable as well as its relationship with the other variables. Communality can be used as a measure of goodness-of-fit of the projection. The communalities of the 5 variables of our data are displayed in Table 5. As shown by this Table, the first two components explain more than 80% of variance in each variable. This is enough to reveal the structure of correlation among the variables. Do not interpret as correlation the angle between two variables when at least one of them has a low communality. Using communality prevent potential biases that may arise by directly interpreting numerical and graphical results yielded by the PCA.


#### Table 5. **Communalities of variables**

All these interesting results show that outcomes from normalized PCA can be easily interpreted without additional complicated calculations. From a visual inspection of the graph, we can see the groups of variables that are correlated, interpret the principal components and name them.
