next up previous
Next: Single and Multiple ICA Up: tr01dl1 Previous: Variance criterion

Negentropy criterion

With PCA one obtains a decomposition like 1 with variance criterion and the constraints on the components is no correlation. Generalisations of these criterion and constraints has been a continuous endeavour and the literature is abondant about this subject. Recently the concept of Independent Components Analysis has been widely used in fMRI analysis. Initially the purpose was to look for Independent Components at a higher order than two (no correlation). The motivation was to recover non-gaussian latent variables (called sources) who generated the data by linear mixing of them, and supposed to be mutually independent. The link between this two concepts can be understood as follows. A measure of mutual independence [5] of the $p$ variables $y_j$ is the Kullback-Liebler divergence between the joint density and the product of the marginal densities :
$\displaystyle MI(y)=KL(f_y\vert\vert\prod_{j=1}^p f_{y_j})$ $\textstyle =$ $\displaystyle E_y(\ln( \frac{f_y(u)}{\prod_{j=1}^p f_{y_j}(u_j)})$ (5)
  $\textstyle =$ $\displaystyle \int f_y(u)\ln( \frac{f_y(u)}{\prod_{j=1}^p f_{y_j}(u_j)}du$ (6)

and is often termed mutual information. Negentropy itself can be defined as the Kullback-Liebler divergence between the density (single or vector variable) and the normal equivalent i.e. with the same mean and variance, or difference of the corresponding entropies 2:
\begin{displaymath}
N(y)=h(\phi_y)-h(f_y)=-E_\phi(\ln(\phi_y(u)) +E_f(\ln(f_y(u))=KL(f_y\vert\vert\phi_y)
\end{displaymath} (7)

The Negentropy can be related to the mutual information:
\begin{displaymath}
MI(y)=N(y)-\sum_{j=1}^pN(y_j)+\frac{1}{2} \ln(\frac{\prod_jV_{jj}}{det(V)}
\end{displaymath} (8)

then as the Negentropy is invariant by invertible transformation minimising the mutual information is equivalent to maximising negentropies of components which then are the least Gaussian variables obtained by linear transformation of the original vector variable. This assertion is in fact true is the third term is null as it is not invariant by invertible transformation. This term which would correspond to a PCA criterion is null when the data is spherised. One can see the analogy with Projection-Pursuit where projections of the data who are the least Gaussian are looked for often using also the negentropy as non-Gaussianity measure. The difference is that in Projection-Pursuit [8] the projections are not necessarily fixed to unidimensional ones.
next up previous
Next: Single and Multiple ICA Up: tr01dl1 Previous: Variance criterion
Didier Leibovici 2001-09-06