Note that neural net training will work even if the answers to the training trials are not all correct. This allows using real-world training data that may have an inherent error rate. One key to the success of a neural net–based recognition system is the amount of data used for training. Usually a very substantial amount is needed to obtain satisfactory results. As with human students, the amount of time that a neural net spends learning its lessons is a key factor in its performance.
Variations
Many variations of the above are feasible. For example:
There are different ways of determining the topology. In particular, the interneuronal wiring can be set either randomly or using an evolutionary algorithm.
There are different ways of setting the initial synaptic strengths.
The inputs to the neurons in layeri do not necessarily need to come from the outputs of the neurons in layeri–1. Alternatively, the inputs to the neurons in each layer can come from any lower layer or any layer.
There are different ways to determine the final output.
The method described above results in an “all or nothing” (1 or 0) firing called a nonlinearity. There are other nonlinear functions that can be used. Commonly a function is used that goes from 0 to 1 in a rapid but more gradual fashion. Also, the outputs can be numbers other than 0 and 1.
The different methods for adjusting the synaptic strengths during training represent key design decisions.
The above schema describes a “synchronous” neural net, in which each recognition trial proceeds by computing the outputs of each layer, starting with layer0 through layerM. In a true parallel system, in which each neuron is operating independently of the others, the neurons can operate “asynchronously” (i.e., independently). In an asynchronous approach, each neuron is constantly scanning its inputs and fires whenever the sum of its weighted inputs exceeds its threshold (or whatever its output function specifies).
10. Robert Mannell, “Acoustic Representations of Speech,” 2008, http://clas.mq.edu.au/acoustics/frequency/acoustic_speech.xhtml.
11. Here is the basic schema for a genetic (evolutionary) algorithm. Many variations are possible, and the designer of the system needs to provide certain critical parameters and methods, detailed below.
The Evolutionary Algorithm
Create N solution “creatures.” Each one has:
A genetic code: a sequence of numbers that characterize a possible solution to the problem. The numbers can represent critical parameters, steps to a solution, rules, etc.
For each generation of evolution, do the following:
Do the following for each of the N solution creatures:
Apply this solution creature’s solution (as represented by its genetic code) to the problem, or simulated environment. Rate the solution.
Pick the L solution creatures with the highest ratings to survive into the next generation.
Eliminate the (N – L) nonsurviving solution creatures.
Create (N – L) new solution creatures from the L surviving solution creatures by:
(1) Making copies of the L surviving creatures. Introduce small random variations into each copy; or
(2) Create additional solution creatures by combining parts of the genetic code (using “sexual” reproduction, or otherwise combining portions of the chromosomes) from the L surviving creatures; or
(3) Do a combination of (1) and (2).
Determine whether or not to continue evolving:
Improvement = (highest rating in this generation) – (highest rating in the previous generation).
If Improvement < Improvement Threshold then we’re done.
The solution creature with the highest rating from the last generation of evolution has the best solution. Apply the solution defined by its genetic code to the problem.
Key Design Decisions
In the simple schema above, the designer needs to determine at the outset:
Key parameters:
N
L
Improvement threshold.
What the numbers in the genetic code represent and how the solution is computed from the genetic code.
A method for determining the N solution creatures in the first generation. In general, these need only be “reasonable” attempts at a solution. If these first-generation solutions are too far afield, the evolutionary algorithm may have difficulty converging on a good solution. It is often worthwhile to create the initial solution creatures in such a way that they are reasonably diverse. This will help prevent the evolutionary process from just finding a “locally” optimal solution.
How the solutions are rated.
How the surviving solution creatures reproduce.
Variations
Many variations of the above are feasible. For example:
There does not need to be a fixed number of surviving solution creatures (L) from each generation. The survival rule(s) can allow for a variable number of survivors.
There does not need to be a fixed number of new solution creatures created in each generation (N – L). The procreation rules can be independent of the size of the population. Procreation can be related to survival, thereby allowing the fittest solution creatures to procreate the most.
The decision as to whether or not to continue evolving can be varied. It can consider more than just the highest-rated solution creature from the most recent generation(s). It can also consider a trend that goes beyond just the last two generations.
12. Dileep George, “How the Brain Might Work: A Hierarchical and Temporal Model for Learning and Recognition” (PhD dissertation, Stanford University, June 2008).
13. A. M. Turing, “Computing Machinery and Intelligence,” Mind, October 1950.
14. Hugh Loebner has a “Loebner Prize” competition that is run each year. The Loebner silver medal will go to a computer that passes Turing’s original text-only test. The gold medal will go to a computer that can pass a version of the test that includes audio and video input and output. In my view, the inclusion of audio and video does not actually make the test more challenging.
15. “Cognitive Assistant That Learns and Organizes,” Artificial Intelligence Center, SRI International, http://www.ai.sri.com/project/CALO.
16. Dragon Go! Nuance Communications, Inc., http://www.nuance.com/products/dragon-go-in-action/index.htm.
17. “Overcoming Artificial Stupidity,” WolframAlpha Blog, April 17, 2012, http://blog.wolframalpha.com/author/stephenwolfram/.
Chapter 8: The Mind as Computer
1. Salomon Bochner, A Biographical Memoir of John von Neumann (Washington, DC: National Academy of Sciences, 1958).
2. A. M. Turing, “On Computable Numbers, with an Application to the Entscheidungsproblem,” Proceedings of the London Mathematical Society Series 2, vol. 42 (1936–37): 230–65, http://www.comlab.ox.ac.uk/activities/ieg/e-library/sources/tp2-ie.pdf. A. M. Turing, “On Computable Numbers, with an Application to the Entscheidungsproblem: A Correction,” Proceedings of the London Mathematical Society 43 (1938): 544–46.