An Optimal Property of Principal Components

Abstract

Let $x' = (x_1, x_2, \cdots, x_p)$ be a random vector and let $E\lbrack x\rbrack = 0, E\lbrack xx'\rbrack = \Sigma = (\sigma_{ij})$ where we assume that $\Sigma$ is non-singular. Further let \[ \Sigma = T\Lambda T' = (t_1, t_2, \cdots, t_p) \left[ \begin{array}{c|c|c|c|c|c|c}\lambda_1 & 0 & \cdot & \cdot & \cdot & 0 \\ 0 & \lambda_2 & \cdot & \cdot & \cdot & 0 \\ \vdots & \vdots & \vdots & \vdots \\0 & 0 & \cdot & \cdot & \cdot & \lambda_p\end{array} \right] \] \[ \left[ \begin{array}{|c}t'_1 \\ t'_2 \\ \vdots \\ t'_p\end{array}\right] \] where $TT' = I$ and where we suppose that the eigenvalues of $\Sigma$ are in order of decreasing magnitude, that is $\lambda_1 \geqq \lambda_2 \geqq \cdots \geqq \lambda_p > 0$. The principal components of $x$, namely $u_1 = t'_1 x, u_2 = t'_2 x, \cdots, u_p = t_p' x$ were introduced by Hotelling (1933) who characterised them by certain optimal properties. Since then Girshick (1936), Anderson (1958) and Kullback (1959) have characterised the principal components by slightly different sets of optimal properties. Thus Anderson shows that $u_1$ is the linear function $\alpha_1' x$ having maximum variance subject to $\alpha_1' \alpha_1 = 1; u_2$ is the linear function $\alpha_2'x$ which is uncorrelated with $u_1$ and has maximum variance subject to $\alpha_2'\alpha_2 = 1$; and so on. The above mentioned characterisations have two properties in common; they introduce the principal components one by one and, more importantly, the optimal properties hold only with-in the class of linear functions of $x_1, x_2, \cdots, x_p$. In the following theorem the first $k$ principal components are characterized by an optimal property within the class of all random variables.