**1.3 Sample quantities**

Let **x**1, **x**2, … , **x***<sup>n</sup>* denote a sample of *n p*-dimensional random vectors

$$\mathbf{x}\_{i} = \begin{pmatrix} \mathbf{x}\_{1i}, \mathbf{x}\_{2i}, \dots \ \mathbf{x}\_{pi} \end{pmatrix}^{\prime}, \quad i = \mathbf{1}, \mathbf{2}, \dots, n. \tag{1}$$

Here, the transpose (<sup>0</sup> ) means that the vectors are being considered as column vectors. The sample *mean vector* is

$$\overline{\mathbf{x}} = \sum\_{i=1}^{n} \mathbf{x}\_i / n. \tag{2}$$

The *p* � *p* sample covariance matrix is denoted by

$$\mathbf{S} = \sum\_{i=1}^{n} (\mathbf{x}\_i - \overline{\mathbf{x}})(\mathbf{x}\_i - \overline{\mathbf{x}})'/(n-1). \tag{3}$$

## **1.4 Population quantitites**

The sample covariance matrix **S** estimates the true covariance matrix Σ of the random variables

$$\mathbf{X}\_1, \mathbf{X}\_2, \dots, \mathbf{X}\_p.$$

The true covariance matrix is

$$\sum = [\sigma\_{u,v}]\_{u,v=1,2,\dots,p},\tag{4}$$

where

$$
\sigma\_{uv} = \mathcal{C}[X\_u, X\_v],
\tag{5}
$$

the covariance of *Xu* and *Xv*, for *u* 6¼ *v*, *u*, *v* ¼ 1, 2, … , *p:* For *u* ¼ *v*, we have C½ �¼ *Xv*, *Xv* V½ � *Xv* , the variance of *Xv*.

## **1.5 Principal components**

The principal components of Σ are defined as *uncorrelated linear combinations of maximal variance.* Let us elaborate on this brief definition. First, a linear combination, say *LC*, of the *p* variables can be expressed as the vector product **a**<sup>0</sup> **x** of two vectors **a** and **x**, that is,

$$LC = \mathbf{a}'\mathbf{x} = a\_1\mathbf{x}\_1 + a\_2\mathbf{x}\_2 + \cdots + a\_p\mathbf{x}\_p. \tag{6}$$

Here, the vector **a** is a vector of scalars *a*1, *a*2, … , *ap* :

$$\mathbf{a}' = \begin{pmatrix} a\_1 \ a\_2 & \dots \ a\_p \end{pmatrix}. \tag{7}$$

These *a <sup>j</sup>* are the coefficients in the linear combination. Such linear combinations are called *variates.* Principal components are also called *latent variables.*

The variance V of a linear combination *LC* is

$$\mathcal{V}[LC] = \mathcal{V}[\mathbf{a}'\mathbf{X}] = \mathbf{a}'\Sigma\mathbf{a}.\tag{8}$$

This is estimated as **a**<sup>0</sup> **Sa***:* This is to be maximized over **a***:* The derivative with respect to the vector **a** is

$$
\partial \mathbf{a}' \mathbf{S} \mathbf{a} / \partial \mathbf{a} = \mathbf{S} \mathbf{a}.\tag{9}
$$

The solution is not unique: If **a** is a solution to this set of equations, so is *c***a**, where *c* is any scalar constant. Therefore, a constraint is required to obtain a meaningful solution. A reasonable such constraint is the condition **a**<sup>0</sup> **a** ¼ 1, that is, the squared length of the vector **a** equals 1. This is of course equivalent to the length of **a**, the quantity ffiffiffiffiffiffi **a**0 **<sup>a</sup>** <sup>p</sup> , being equal to 1.

A function incorporating the constraint, the *Lagrangian function,* is

$$L(\mathbf{S}, \mathbf{a}; \boldsymbol{\lambda}) = \mathbf{a}' \mathbf{S} \mathbf{a} + \boldsymbol{\lambda} (\mathbf{1} - \mathbf{a}' \mathbf{a}). \tag{10}$$

The partial derivatives of the function *L* with respect to a and *λ* are

$$
\partial \mathbf{L} / \partial \mathbf{a} = 2 \mathbf{S} \mathbf{a} - 2 \lambda \mathbf{a} \tag{11}
$$

and

$$
\partial \mathbf{L} / \partial \boldsymbol{\lambda} = \partial \boldsymbol{\lambda} (\mathbf{1} - \mathbf{a}' \mathbf{a}) / \partial \boldsymbol{\lambda} = \mathbf{1} - \mathbf{a}' \mathbf{a}.\tag{12}
$$

Setting these partial derivatives equal to zero gives the simultaneous linear equations

$$\mathbf{Sa} = \lambda \mathbf{a},\tag{13}$$

and the equation

$$\mathbf{a}'\mathbf{a} = \mathbf{1}.\tag{14}$$

The simultaneous linear equations can be written as

$$
\mathbf{S}\mathbf{a} - \lambda\mathbf{a} = \mathbf{0},
\tag{15}
$$

where **0** is the zero vector, the vector whose elements are all zeroes. Factoring out **a** on the right, we obtain

$$(\mathbf{S} - \lambda \mathbf{I})\mathbf{a} = \mathbf{0}.\tag{16}$$

For nontrivial solutions, the determinant of the coefficient matrix **S** � *λ***I** must be zero, that is, we must have detð Þ¼ **S** � *λ***I** 0*:* This condition is a polynomial equation of degree *p* in *λ*. Denote the *p* roots by *λ*<sup>1</sup> ≥ *λ*<sup>2</sup> ≥⋯≥*λp:* These roots are the *eigenvalues* (also called *latent values*). Their sum is the trace of **S**; their product is the determinant of **S***:*

The corresponding Eigen equations are

$$\mathbf{S}\mathbf{a}\_{j} = \lambda\_{j}\mathbf{a}\_{j}, \quad j = \mathbf{1}, \mathbf{2}, \dots, p. \tag{17}$$

## *1.5.1 Values of PCs in terms of Xs*

The *j*th principal component (PC), *Cj*, is the linear combination of the form

$$\mathbf{C}\_{j} = \mathbf{a}\_{j}^{\prime}\mathbf{x} = a\_{1j}\mathbf{x}\_{1} + a\_{2j}\mathbf{x}\_{2} + \cdots + a\_{pj}\mathbf{x}\_{p},\tag{18}$$

where **a**<sup>0</sup> *<sup>j</sup>* <sup>¼</sup> *<sup>a</sup>*1*<sup>j</sup>*, *<sup>a</sup>*2*<sup>j</sup>*, … , *apj :* That is to say, for *<sup>j</sup>* <sup>¼</sup> 1, 2, … , *<sup>p</sup>*, the value of the *<sup>j</sup>*th PC for Individual *i* is *cji* ¼ **a**<sup>0</sup> *j* **x***i*, *i* ¼ 1, 2, … , *n:*.

The equation for the *j*th PC in terms of the vector **x** ¼ *x*<sup>1</sup> *x*<sup>2</sup> … *xp* <sup>0</sup> is *cj* ¼ **a**0 *j* **x**, *j* ¼ 1, 2, … , *p:* Let **c** be the *p*-vector of values of the *p* PCs. Then, **c** ¼ **A**<sup>0</sup> **x**, where **A** ¼ **a1 a2** … **ap** is the *<sup>p</sup>* � *<sup>p</sup>* matrix whose columns are the eigenvectors.
