An ICA page - papers, code, demos, links
Explanation: ICA is about factoring probability distributions, and doing blind source separation. It is related to lots of other things - entropy and information maximisation, maximum likelihood density estimation (MLE), EM (expectation maximisation, which is MLE with hidden variables)and projection pursuit. It is basically a way of finding special linear (non-orthogonal) co-ordinate systems in multivariate data, using higher-order statistics in various ways. If you don't understand, read the papers below!
Applications: anywhere you have ensembles of multivariate data, eg: anywhere you might use PCA (Principal Components Analysis). Examples include blind separation (eg: of mixed speech signals), biomedical data processing (eg: of EEG [brainwave] data), finding `features' in data (eg: learning edge-detectors for ensembles of natural images).
Algorithms: There are many different algorithms, but often they are not so different really. Read on.
-Tony Bell
Papers.
A selection of papers that I have access to right now. There is no pretence to completeness, so sorry if you're omitted! Email me if you have something you'd like to include. Many other papers are collected at Paris Smaragdis's site listed below.
Quick guide to the papers:
Our main work is in B1 and B3: infomax/ICA for source separation and image
coding respectively. The related natural gradient approach (which we now
use) is in A1, a special case of the algorithm proposed in CU. The relations
between these algorithms and maximum likelihood (C1, PH) are gone through
in C2, MK, PP and O1 (so we'd better believe it!). The infomax origins
can be traced in NP. A nice review from the Finnish school appears in KA.
D1 is a book containing much material on ICA. Clever extensions and
analyses are appearing in G1, PN, T1, T2, and finally PP, where temporal
context gives better separation. Perhaps the best work on source
separation/deconvolution is in LA (a pun). Our work on EEGs and
ERPs is in M1 and M2.
[NOTE: All papers are PostScript file. Compressed versions are X-compressed. They have ".ps.Z" on the end, and need UNIX `uncompress' to make them ".ps" files.]
[A1] Amari S. Cichocki A. and Yang H.H. 1996. A new learning algorithm for blind signal separation, Advances in Neural Information Processing Systems 8, MIT press. Paper
[A2] Amari S-I. 1997. Natural Gradient works efficiently in learning.
submitted to Neural Computation Paper
[B1] Bell A.J. and Sejnowski T.J. 1995. An information maximisation approach
to blind separation and blind deconvolution, Neural Computation,
7, 6, 1129-1159 Abstract
, Paper (0.9MB),
Compressed (0.3MB)
(38 pages). [3 short conference papers on the same material: ICASSP
95 (1.4MB, 4 pages) , NIPS
94 (0.2MB, 8 pages), and NOLTA95
. ]
[B2] Bell A.J. and Sejnowski T.J. 1996a. Learning the higher-order structure of a natural sound, Network: Computation in Neural Systems, 7 Paper
[B3] Bell A.J. and Sejnowski T.J. 1996. The `Independent Components'
of natural scenes are edge filters, to appear in Vision Research,
[Please note that this is a draft] Paper
(1.5MB) , Compressed
(0.4MB) (27 pages).[*Compressed
tar-file of figures in case they don't print properly when you print
the paper].
[B4] Bell A.J. and Sejnowski T.J. 1996. Edges are the `independent components'of
natural scenes,
Advances in Neural Information Processing Systems 9, MIT press.
Paper
[C1] Cardoso J-F. and Laheld B. 1996. Equivariant adaptive source separation,
IEEE Trans. on Signal Proc., to appear Paper
[C2] Cardoso J-F, 1997. Infomax and maximum likelihood for blind separation,
to appear in IEEE Signal Processing Letters, Paper
[D1] Deco G. and Obradovic D. 1996. An information-theoretic approach to
neural computing, Springer-verlag Book
[G1] Girolami M. and Fyfe C. 1996. Negentropy and kurtosois as projection
pursuit indices provide generalised ICA algorithms. NIPS '96 working
paper Paper
[KA] Karhunen J. 1996. Neural approaches to independent component analysis
and source separation, Proc 4th European Symposium on Artificial Neural
Networks (ESANN '96). Paper
[LA] Lambert R.H. 1996. Multi-channel blind deconvolution: FIR matrix algebra
and separation of multipath mixtures, Ph.D. Thesis, Elec. Eng.,
Univ. of Southern California
[LB] Lee T-W., Bell A.J. and Lambert R. 1997. Blind separation of delayed
and convolved sources
in Proceedings of NIPS*9 (1996) Paper
[L2] Lee T-W., Bell A.J. and Orglmeister R. 1997. Blind Separation of Real
-World Signals,
in Proceedings of International Conference on Neural Networks (Houston),
ICNN '97 Paper
[L3] Lee T-W. Girolami M., Bell A.J. and Sejnowski T.J. 1998. A unifying
Information-theoretic framework for Independent Component Analysis.
International Journal on Mathematical and Computer Modeling, in press.
Paper
[MK] MacKay D. 1996. Maximum Likelihood and covariant algorithms for Independent
components analysis, DRAFT 3.1 Paper
[M1] Makeig S., Bell A.J., Jung T-P. and Sejnowski T.J. 1995. Independent
Component Analysis of Electroencephalographic Data, in Mozer M. et al (eds)
Advances in Neural Information Processing Systems 8, MIT press Paper
[M2] Makeig S., Jung T-P., Bell A.J., Ghahremani D. and Sejnowski T.J.
1997. Blind Separation of Event-related Brain Responses into Independent
Components, Proc. Natl. Acad. Sci. USA
to appear Paper
[NP] Nadal J-P. and Parga N. 1994. Non-linear neurons in the low-noise
limit: a factorial code maximises information transfer, Network, 4:295-312,
Paper
[O1] Olshausen B. 1996. Learning linear, sparse, factorial codes, MIT
AI-memo No. 1580 , aper
[PN] Parga N and Nadal J-P. 1996. Blind source separation with time-dependent
mixtures, Signal Processing, submitted. Paper
[PP] Pearlmutter B.A. and Parra L.C. 1996. A context-sensitive generalization
of ICA, Proc. ICONIP '96, Japan Paper
[PF] Platt J.C. and Faggin F. 1992. Networks for the separation of sources
that are superimposed
and delayed, NIPS 1992 Paper
[T1] Torkkola K. 1996a. Blind separation of delayed sources based on information maximisation, Proc. ICASSP, Atlanta, May 1996 Paper
[T2] Torkkola K. 1996b. Blind separation of convolved sources based on information maximisation, Paper
The following two papers are probably under-cited, containing, respectively, `early' theoretical and practical insights into algorithms which are part of the infomax/maximum-likelihood/natural-gradient family of ICA algorithms. Unfortunately we don't have them online. Does anyone else? Also, let us not forget the French originators: Herault/Jutten, Comon and others.
[PH] Pham D.T. Garrat P and Jutten C. 1992. Separation of a mixture
of independent sources through a maximum likelihood approach, in Proc.
EUSIPCO, p.771-774
[CU] Cichocki A., Unbehauen R., \& Rummert E. 1994. Robust learning
algorithm for blind separation of signals, Electronics Letters,
30, 17, 1386-1387
Code.
Basic ICA
code in MATLAB (as used in Bell and Sejnowski 1996) It's simple.
Scott
Makeig et al's MATLAB 4.2 routines for applying ICA to psychophysiological
data (tar-file)
Demos.
Demo of real-room blind separation/deconvolution of two sources
(by Te-Won Lee . Uses infomax/ICA
techniques in freq. domain with FIR matrix methods,
pioneered by Russ Lambert at USC. See our NIPS*96 poster - paper [LB] above.)
Links.
Hastily assembled selection of Web pages with more ICA papers and details.
Paris Smaragdis' page has a huge amount of material. It is more-or-less
a
superset of this page.
Paris Smaragdis'
ICA & BSS page (MIT) GO TO THIS PAGE. IT'S GREAT.
Allan Barros's
ICA page in Japan. Another huge collection of stuff.
Jean-François Cardoso
Andrzej Cichocki
Erkki Oja
Scott Makeig,
Shun-ichi
Amari
J.-P. Nadal : Publications
Mark
Girolami's Home Page
Homepage of Aapo Hyvärinen
NIPS*96
Signal Processing Workshop (several new papers).
The WWW homepage of Juha Karhunen
BSS Research
(Experiments at RIKEN in Japan)
11/96 -Tony Bell (tony@salk.edu), Computational
Neurobiology Lab , Salk Institute, San Diego.
back to T.Bell Home
Page