Jeff’s theory describes an interaction between prediction and feedforward activation, whereby a cell is partially depolarised by coincident activity on its distal (predictive) dendrites. Predictive cells get a head start when they receive feedforward inputs, and are thus most likely to fire compared with the other cells in a column, as well as non-predictive cells in neighbouring columns.
For some reason, this is not completely implemented in NuPIC. The Spatial Pooler (SP) does not take prediction into account at all, and in fact acts as if it were one big cell with a single feedforward dendrite and no distal dendrites.
I propose the following changes from the existing (diagram below, left) to the proposed (right) CLA SP design:
1. Each cell now has its own feedforward dendrite.
2. Cells in a column have identical potential inputs (fanout).
3. Permanences initialised identically for all cells in a column.
4. Potential activation for each cell is sum of predictive and feedforward potential.
5. Cell with highest total activation provides column’s activation for inhibition.
6. Same cell is chosen for activation if column becomes active.
7. Feedforward permanences will diverge depending on correlations for each cell.
Anticipated Advantages of this Design
1. More Accurate Neural Model
The real neocortex has a feedforward dendrite per cell. The reason the cells share similar feedforward response is that the feedforward axons pass up through the column together, so they will form similar (but not identical) synapses with all the cells in the column.
Cells in a column will all have different histories of activation, so the permanences of their synapses with a given feedforward axon will not be identical. Each cell will learn to match its own activation history with the corresponding inputs.
In the real neocortex, prediction is implemented by a partial depolarisation of the cell membrane. This lowers the amount of feedforward potential needed to fire the cell. The cells with the highest total of predictive and feedforward potential will fire first and be in the SDR for the region.
2. More Informed Spatial Pooler Selection
The current SP ignores prediction, so it does not have the additional information which the region believes, i.e. what sequence are we in, and at what position? This is a significant factor in reducing ambiguity and constraining the space of likely patterns, which the real neocortex uses universally.
In addition, each cell now gets to tune its feedforward response more precisely to its actual inputs (i.e. the ones which occur in its sequence memory). Inputs which contribute to patterns found only in other cells’ sequences will be treated as noise and disconnected. This improves the overall noise suppression and spatial recognition.
3. Easier Reconstruction of Inputs
Because each cell has its own feedforward dendrite, the permanences on that dendrite will evolve to become a statistical representation of the bits associated with that cell. This makes it easier to reconstruct the inputs from the activation pattern (or from an SDR imposed on the region from above).
The current per-column dendrite represents the collective statistics of inputs to all the cells, and thus contains noise which confuses reconstruction at present.
4. Better Anomaly Detection
The added precision in reducing ambiguity through the use of sequence information in the SP will also improve anomaly detection. The region will be more sensitive to out-of-sequence events.
1. Resource Costs
Clearly, NuPIC will have a bigger memory requirement and a longer cycle time. In return, learning and prediction will improve in both quality and accuracy, so techniques such as swarming may decide which SP to use.
2. Slow Learning
It is possible that learning will slow as a result of this change, since only one cell’s dendrite will be updated per input record, instead of updating them all in parallel as at present.
This may be mitigated by copying updates to all cells in a column for the first N input records (or the first N activations of the column). This will hopefully replicate the current SP’s ability to learn the data. After that, we switch to the new per-cell updating policy to fine-tune the permanences.
I’ve been looking through the (Python) code to find out where all the changes need to be made. Here’s what I’ve found out:
The SP doesn’t know anything about TP’s. In fact, it thinks it has only one cell per column, or that a cell and a column are the same thing. That’s why it has one feedforward dendrite per column. The TP only knows about the SDR of columns, it doesn’t see inside the SP, so it can’t see any feedforward dendrites. So, if we want to do this, we have to connect the column and its cells (at least virtually). This would happen in the Region (which owns both).
Here’s how I think we should perform the hack (in Python):
In the SP:
1. Store a value for each column which is the predictive potential for the predicting cell (if any, so usually zero). Call this array _predictivePotentials.
2. Calculate the feedforward potential as usual, store in _overlaps, but then do _overlaps += _predictivePotentials.
3. Everything else is the same (or so the SP thinks!).
In the TP: No change!
In the Region:
For the first N steps, just use the standard SP and TP to learn good per-column dendrites from the data. After that, clone the column dendrites from the SP for the cells. Call this _cellFFDendrites.
1. Take the results from the TP (the predictive cells) and get their FF dendrites from _cellFFDendrites.
2. Overwrite the dendrites in the SP for those columns with the cell dendrites. The SP is now working with the per-cell dendrite.
_saveDendrites = SP._dendrites[TP._predictiveCols]
SP._dendrites[TP._predictiveCols] = _cellFFDendrites[_predictiveCells]
3. Update the SP’s _predictivePotentials:
SP._predictivePotentials = 0
SP._predictivePotentials[TP._predictiveCols] = TP._predictiveActivity[TP._predictiveCols]
4. (On the next step, we do SP before TP) Run SP as above.
5. Copy back out the (possibly refined) dendrites to the cells.
_cellFFDendrites[_predictiveCells] = SP._dendrites[TP._predictiveCols]
SP._dendrites[TP._predictiveCols] = _saveDendrites