Getting past the “Deep Learning” neuron

If you learn about machine learning methods such as “deep learning”, you learn a neural model that does a remarkably simple calculation – it receives inputs from many sources, multiplies each input by a weight, takes a sum of the products, subtracts a threshold, and applies a function.
That model has led to amazing progress in voice recognition, scene recognition, and other areas.
But years have gone by, and now we know more about neurons.  We now know that each dendrite branch can detect sequences of inputs in time.

If neurons 1,2 and 3 all synapse on one dendrite branch of neuron 4, it is possible to show that neuron 4 will respond to the sequence (for example) 2,3,1 ,and not 1,2,3 or 3,2,1.  So neuron 4 might recognize a simple musical melody on each branch (assuming it had enough synapses on each branch from neurons in the auditory areas).

Since a neuron has many dendrites, and a dendrite can have several branches, and each branch can recognize a sequence, you have a computational unit now that can (and I’m sure will) be the basis of new and different types of neural networks.


Here are the ingredients of the recipe that allows a dendrite branch to recognize a sequence. (in the above diagram, synapse ‘1b’ is on a different branch than ‘1’.

The following points come from Sergey Alexashenko’s book “Cortical Circuitry

We now know that

1. The neuron has a new type of spike: Up to a point, as you increase the number of inputs on a specific point of a dendritic branch, the inputs are summed linearly, just as the traditional theory would suggest. However, at some point, once a certain threshold is reached, there is a spike in the local voltage. That spike in dendrites resembles a neuronal spike – inputs are summed until they reach a threshold, which triggers a spike. Additional inputs do not significantly increase the magnitude of the response beyond the spike amplitude. The important point here is that we are not talking about the regular neural spike that travels down the axon. That spike of course exists, but we are talking about spikes in dendrites.  These are also called NMDA spikes.

2. The threshold for a spike depends on time: if the inputs impinging on a dendrite via synapses are clustered in time, fewer are required to trigger an NMDA spike than if they are spaced apart in time. This is called ‘cooperativity’, and it works only within one branch of one dendrite. That means that if normally 10 synaptic inputs are needed to trigger an NMDA spike, a recent nearby NMDA spike can lower that threshold to 8 synaptic inputs.

3. Spatial proximity affects the likelihood of a dendrite spike: if for example, you have 10 inputs to the same point on a dendritic branch maybe that would trigger an NMDA spike, but 20 inputs are required if the inputs (synapses) are distributed along the length of the dendrite.

4. The threshold for making a dendrite spike varies along the length of the branch: For instance, it was found that the threshold for initiating a dendritic spike increases 5-fold from the tips of dendritic branches to the parts of the branches close to the soma. But the effect of the spike on the electrical charge of the soma increases 7-fold in the same direction. In other words, spikes that are close to the soma are a lot harder to trigger, but have a stronger effect on the cell body and so are more likely to set off an action potential.

5. The cutoff time before inputs don’t reinforce prior inputs varies spatially: Another way of saying this is that temporal summation increases as you go along the dendrite away from the soma – inputs that are close to the soma have a stronger effect when they are synchronized, but inputs on the tips of dendrites can be summed up without loss over relatively long periods of time.

So how does this lead to sequences? For one thing, neurons are more likely to fire when inputs arrive in a sequence approaching the soma, rather than when they arrive in a sequence moving further away from the soma. That makes sense – impulses arriving far away from the soma take more time to travel to the soma, so if impulses arrive on a dendritic branch in an order approaching the soma, they will arrive in a synchronized fashion, thus making peak voltage higher and increasing the probability of the neuron firing.
But apart from that, in the picture of the neuron above, if an input comes into synapse 1, then synapse 2, then 3, with each input triggering an NMDA spike, then the neuron is more likely to fire an action potential that travels down the axon and causes neurotransmitters to impinge on other neurons. Looking at a dendrite branch in the illustration, we see that the threshold out at point 1, which is far from the Soma, is lower than the threshold at point 3. So it is relatively easy to fire a dendrite spike at point 1. That spike will make it easier to trigger another spike at point 2, and the spikes at point 1 and point 2 will finally make it easier to trigger a spike at point 3. Remember that the dendrite spike at point 3 has a much larger impact than a spike at point 1 for getting the neuron as a whole to fire.

Sergey explains all this, and gives an example, in chapter 4 of his book.  He also has a model of how the cortex works in the later chapters of the book.

Alexashenko, Sergey. Cortical Circuitry (Kindle Locations 405-415). Cortical Productions. Kindle Edition.
A company that already has a product based on neurons that detect sequences is Numenta. It works differently than the model Sergey talks about, but it is a good site ( to visit and is on a research frontier that you can participate in.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: