Focus: Sparse Distributed Representations

  • Nov 24 / 2013
  • 0

Focus: Sparse Distributed Representations

Sparse Distributed Representations (SDRs) are one of the key principles of Jeff Hawkins’ Hierarchical Temporal Memory theory. The theory claims that communication within the neocortex, and thus the representation of information, is primarily in the form of SDRs. This article describes some of the key concepts and properties of SDRs, and goes into some detail which will be followed up in future articles. Acknowledgements for much of this material to Jeff, Subutai and the guys in Numenta, and to the members of the NuPIC mailing list (see for more). Some of the following is based on material in my upcoming book, Real Machine Intelligence with NuPIC – Using Neuroscience to Build Truly Intelligent Machines (preview the Introduction and Chapter 1 – Some Context for Machine Intelligence).

Neuroscience has long recognised that, in large, active regions containing many neurons, only a tiny fraction of them are observed to be active at once. In fact, everywhere you look in the neocortex, you see patterns of cell activation looking like this:


The reason for this is quite simple. Cells all over the region are each receiving a constantly-changing and unique stream of signals, both from lower regions (or the senses) and from other cells, near and far, in the neocortex. These varying levels of input will cause each cell to have a different rate of depolarisation (or rising potential). The first cell in a small area to reach its activation threshold will be the first of its peers to fire, and, in firing, it will cause inhibitory neurons surrounding it to fire too. These neurons have two effects: first, they reduce, or suppress, the activation potentials of the normal cells they connect to; and second, they trigger even more surrounding inhibitory neurons. This chain reaction spreads very, very quickly, effectively snuffing out the ability of any cells nearby to become active as well. The result is that the original cell, first to fire, now stands alone in a small area of quietude. The pattern we see all over the neocortex is due to this local inhibition being repeated over and over again.

So, we now understand how SDRs are produced in the brain. The next question we should ask is: why? What is so useful about SDRs that evolution has chosen them to be the “language” of the brain? What properties do SDRs have and how can these properties be exploited in cognitive processing?

Basic Properties of SDRs

The most basic properties of SDRs are the ones listed in the name, so let’s deal with those first. We’ll define each of them in turn, and then we’ll talk about the properties which result from having them all in combination. In the foregoing, please forgive me for interchangeably using the word bit, which is a term from computer science meaning a thing which can be a 1 (for “on”) or 0 (for “off”), instead of neuron, a cell in the brain which is either active (or “on”) or inactive (or “off”). This equivalence is used to make the diagrams and explanations easier, and is expressed directly in the NuPIC software.

SDRs are Sparse

SDRs are sparse. This means that each SDR is composed of many neurons (or bits), only a very few of which are active (1) at any time. The word sparse evokes an image of the vegetation in a drought-stricken landscape – trees and bushes are dotted around in an otherwise empty and barren savannah.

In NuPIC, we normally choose a sparseness of about 2%. In other words, for every neuron which is active, there are 49 others which are inactive. This seems to accord well with the numbers observed in the neocortex, and has been chosen because it seems from empirical experience to be the best choice for many kinds of problems. A typical region in NuPIC has 2048 columns, of which the 40 most active are selected to be on.

You might think that this is an inefficient way to represent information. Indeed, at any one time, most of the neurons are not participating in the representation, so perhaps they are not being utilised efficiently. This would be true, perhaps, if the brain was like a computer, where the 0’s cost almost as much as the 1’s (certainly to store, absolutely to use in calculations), and they are just as important to the encoding of information and its processing. But the brain is not a computer, and you’ll see that in fact the peculiar properties of SDRs give rise to significant power and flexibility.

SDRs are Distributed

SDRs are distributed. This means that the pattern of active cells is spread across the region in an approximately even and unpredictable manner. A second meaning is that there is no single cell, or group of cells, on which the whole thing depends – the significance of the pattern is also distributed across all the cells. The result is that the SDR is resilient to the failure or error of some of the neurons (real neurons live uncertain lives, and they cannot be relied upon to produce identical results again and again). Furthermore, you usually only need to check a sample of the cells to see if you have a certain pattern (more on this later).

SDRs are Representations

SDRs are representations. In the brain (and in NuPIC), the SDR conveys information about whatever the region is dealing with at the time. Each active cell is conveying some part of the information (we often can’t tell exactly what), and together they are representing something quite specific. We say that each active cell represents some semantic (information-bearing or meaningful) aspect of the overall message.

Together, these properties of the SDR allow for some profound advantages which we’re not used to seeing in the more traditional, dense representations familiar in computers. We’ll deal with these in turn.

Binary Depiction of SDRs

When showing SDRs, it often makes sense to depict an SDR as a list (or array) of bits which can be either 1 (on) or 0 (off). This is how SDRs are stored and processed in the NuPIC software. Note that the order of the bits in a binary SDR array is important (because each bit has a particular semantic meaning). Here’s a typical diagram showing two SDRs being compared:

0 0 1 0 0 0 1 0 0 .... 0 1 0 0 0 1 0 0 .... 0 0 0 0 1 0 0 0 0 0 1 0 [SDR A]
| | | | | | | | | .... | | | | | | | | .... | | | | | | | | | | | |
0 0 0 1 0 0 1 0 0 .... 0 1 0 0 0 1 0 0 .... 0 0 0 0 1 0 0 1 0 0 0 0 [SDR B]
| | | | | | * | | .... | * | | | * | | .... | | | | * | | | | | | |

The columns marked * are where the on-bits in A and B coincide.