Distributed Representations and Spike Timing

Zoubin Ghahramani and Geoffrey Hinton
Gatsby Computational Neuroscience Unit
University College London

There is no general consensus about how the activities of spiking neurons combine to represent multiple properties of an object, such as its pose, deformation and illumination. We present a speculation about how spiking neurons can represent and operate on full probability distributions over the space of object properties. We start by assuming that each neuron encodes a basis function over the space of object properties. A neuron's activity (depolarization) represents a multiplicative coefficient on its basis function. The weighted bases combine additively across a population of neurons in the same group to form an energy landscape (i.e. negative log probability density). In this framework, since spikes cause a smooth depolarization in neurons downstream (the EPSP), the timing of all-or-none spikes can be used to represent real-valued coefficients.

This representation has several consequences: (1) Unlike disjunctive codes (Anderson and van Essen, 1994), here neurons with very broad spatio-temporal tuning curves (energy bases) can be combined to represent very sharp spatio-temporal densities. (2) The Bayesian operation of combining evidence from multiple sources with a prior density is additive in the energy domain, and therefore trivial to implement in these spiking neurons. (3) Spike timing can convey real values quickly and accurately without requiring precise coincidence detection, sub-threshold oscillations, or modifiable time delays (Hopfield, 1995). (4) Lateral connections are required for ``explaining away'' within a group: to correlate activities of nearby neurons so as to produce a desired density. That is, when multiple neurons have similar basis functions there are potentially many different ways of representing the same density over object properties. But once one neuron fires, its contribution to the energy landscape must be subtracted from what nearby neurons have to represent. (5) Noise models (uncertainty not inherent in the input) require the introduction of nonlinear activity decay within a group of neurons.