Delving into Deep Learning
The latest neural networks learn to see and hear, and maybe even dream.
Neural Nets and Neurology
These triumphs of neural networks might seem to be the definitive answer to the Minsky-Papert critique of perceptrons. Yet some of the questions raised 50 years ago have not gone away.
The foundation of the neural network methods is almost entirely empirical; there’s not much deep theory to direct deep learning. At best we have heuristic guidelines for choosing the number of layers, the number of neurons, the initial weights, the learning protocol. The impressive series of contests won by Hinton and his colleagues testifies to the effectiveness of their methods, but it also suggests that newcomers may have a hard time mastering those methods.
An immense space of network architectures remains to be explored, with a multitude of variations in topology, circuitry, and learning rules. Trial and error is not a promising tactic for finding the best of those alternatives.
Or is it? Trial and error certainly had a major role in building the most successful of all neural networks—those in our heads. And the long dialogue between biological and engineered approaches has been fascinating if not always fruitful. The biological model suggests ways to build better connectionist computers; the successes and failures of computational models inform our efforts to understand the brain.
In both of these projects, we have a ways to go. A machine that learns to distinguish cows from camels and cats from canines is truly a marvel. Yet any toddler can do the same without a training set of a million images.
- Dewdney, A. K. 1984. Computer recreations. Scientific American 251:22–34.
- Hinton, G. 2006. To recognize shapes, first learn to generate images. https://www.cs.toronto.edu/~hinton/absps/montrealTR.pdf
- Hinton, G. E., and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313:504–507.
- Hinton, G. E. 2007. Learning multiple layers of representation. Trends in Cognitive Sciences 11:428–434.
- Minsky, M. L., and S. A. Papert. 1969. Perceptrons: An Introduction to Computational Geometry. Cambridge, MA: MIT Press.
- Rosenblatt, F. 1962. Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Washington, D.C.: Spartan Books.
- Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning representations by back-propagating errors. Nature 323:533–536.