A very important point to note here is that information flows down the conceptual hierarchy as well as up. If anything, this downward flow is even more significant. If, for example, we are reading from left to right and have already seen and recognized the letters “A,” “P,” “P,” and “L,” the “APPLE” recognizer will predict that it is likely to see an “E” in the next position. It will send a signal down to the “E” recognizer saying, in effect, “Please be aware that there is a high likelihood that you will see your ‘E’ pattern very soon, so be on the lookout for it.” The “E” recognizer then adjusts its threshold such that it is more likely to recognize an “E.” So if an image appears next that is vaguely like an “E,” but is perhaps smudged such that it would not have been recognized as an “E” under “normal” circumstances, the “E” recognizer may nonetheless indicate that it has indeed seen an “E,” since it was expected.
The neocortex is, therefore, predicting what it expects to encounter. Envisaging the future is one of the primary reasons we have a neocortex. At the highest conceptual level, we are continually making predictions—who is going to walk through the door next, what someone is likely to say next, what we expect to see when we turn the corner, the likely results of our own actions, and so on. These predictions are constantly occurring at every level of the neocortex hierarchy. We often misrecognize people and things and words because our threshold for confirming an expected pattern is too low.
In addition to positive signals, there are also negative or inhibitory signals which indicate that a certain pattern is less likely to exist. These can come from lower conceptual levels (for example, the recognition of a mustache will inhibit the likelihood that a person I see in the checkout line is my wife), or from a higher level (for example, I know that my wife is on a trip, so the person in the checkout line can’t be she). When a pattern recognizer receives an inhibitory signal, it raises the recognition threshold, but it is still possible for the pattern to fire (so if the person in line really is her, I may still recognize her).
The Nature of the Data Flowing into a Neocortical Pattern Recognizer
Let’s consider further what the data for a pattern looks like. If the pattern is a face, the data exists in at least two dimensions. We cannot say that the eyes necessarily come first, followed by the nose, and so on. The same thing is true for most sounds. A musical piece has at least two dimensions. There may be more than one instrument and/or voice making sounds at the same time. Moreover, a single note of a complex instrument such as the piano consists of multiple frequencies. A single human voice consists of varying levels of energy in dozens of different frequency bands simultaneously. So a pattern of sound may be complex at any one instant, and these complex instants stretch out over time. Tactile inputs are also two-dimensional, since the skin is a two-dimensional sense organ, and such patterns may change over the third dimension of time.
So it would seem that the input to a neocortex pattern processor must comprise two- if not three-dimensional patterns. However, we can see in the structure of the neocortex that the pattern inputs are only one-dimensional lists. All of our work in the field of creating artificial pattern recognition systems (such as speech recognition and visual recognition systems) demonstrates that we can (and did) represent two- and three-dimensional phenomena with such one-dimensional lists. I’ll describe how these methods work in chapter 7, but for now we can proceed with the understanding that the input to each pattern processor is a one-dimensional list, even though the pattern itself may inherently reflect more than one dimension.
We should factor in at this point the insight that the patterns we have learned to recognize (for example, a specific dog or the general idea of a “dog,” a musical note or a piece of music) are exactly the same mechanism that is the basis for our memories. Our memories are in fact patterns organized as lists (where each item in each list is another pattern in the cortical hierarchy) that we have learned and then recognize when presented with the appropriate stimulus. In fact, memories exist in the neocortex in order to be recognized.
The only exception to this is at the lowest possible conceptual level, in which the input data to a pattern represents specific sensory information (for example, image data from the optic nerve). Even this lowest level of pattern, however, has been significantly transformed into simple patterns by the time it reaches the cortex. The lists of patterns that constitute a memory are in forward order, and we are able to remember our memories only in that order, hence the difficulty we have in reversing our memories.
A memory needs to be triggered by another thought/memory (these are the same thing). We can experience this mechanism of triggering when we are perceiving a pattern. When we perceived “A,” “P,” “P,” and “L,” the “A P P L E” pattern predicted that we would see an “E” and triggered the “E” pattern that it is now expected. Our cortex is thereby “thinking” of seeing an “E” even before we see it. If this particular interaction in our cortex has our attention, we will think about “E” before we see it or even if we never see it. A similar mechanism triggers old memories. Usually there is an entire chain of such links. Even if we do have some level of awareness of the memories (that is, the patterns) that triggered the old memory, memories (patterns) do not have language or image labels. This is the reason why old memories may seem to suddenly jump into our awareness. Having been buried and not activated for perhaps years, they need a trigger in the same way that a Web page needs a Web link to be activated. And just as a Web page can become “orphaned” because no other page links to it, the same thing can happen to our memories.
Our thoughts are largely activated in one of two modes, undirected and directed, both of which use these same cortical links. In the undirected mode, we let the links play themselves out without attempting to move them in any particular direction. Some forms of meditation (such as Transcendental Meditation, which I practice) are based on letting the mind do exactly this. Dreams have this quality as well.
In directed thinking we attempt to step through a more orderly process of recalling a memory (a story, for example) or solving a problem. This also involves stepping through lists in our neocortex, but the less structured flurry of undirected thought will also accompany the process. The full content of our thinking is therefore very disorderly, a phenomenon that James Joyce illuminated in his “stream of consciousness” novels.
As you think through the memories/stories/patterns in your life, whether they involve a chance encounter with a mother with a baby carriage and baby on a walk or the more important narrative of how you met your spouse, your memories consist of a sequence of patterns. Because these patterns are not labeled with words or sounds or pictures or videos, when you try to recall a significant event, you will essentially be reconstructing the images in your mind, because the actual images do not exist.
If we were to “read” the mind of someone and peer at exactly what is going on in her neocortex, it would be very difficult to interpret her memories, whether we were to take a look at patterns that are simply stored in the neocortex waiting to be triggered or those that have been triggered and are currently being experienced as active thoughts. What we would “see” is the simultaneous activation of millions of pattern recognizers. A hundredth of a second later, we would see a different set of a comparable number of activated pattern recognizers. Each such pattern would be a list of other patterns, and each of those patterns would be a list of other patterns, and so on until we reached the most elementary simple patterns at the lowest level. It would be extremely difficult to interpret what these higher-level patterns meant without actually copying all of the information at every level into our own cortex. Thus each pattern in our neocortex is meaningful only in light of all the information carried in the levels below it. Moreover, other patterns at the same level and at higher levels are also relevant in interpreting a particular pattern because they provide context. True mind reading, therefore, would necessitate not just detecting the activations of the relevant axons in a person’s brain, but examining essentially her entire neocortex with all of its memories to understand these activations.