Reinforcement Learning: An Introduction,* by Rich Sutton and Andy Barto (MIT Press, 1998), is the standard textbook on the subject. Universal Artificial Intelligence,* by Marcus Hutter (Springer, 2005), is an attempt at a general theory of reinforcement learning. Arthur Samuel’s pioneering research on learning to play checkers is described in his paper “Some studies in machine learning using the game of checkers”* (IBM Journal of Research and Development, 1959). This paper also marks one of the earliest appearances in print of the term machine learning. Chris Watkins’s formulation of the reinforcement learning problem appeared in his PhD thesis Learning from Delayed Rewards* (Cambridge University, 1989). DeepMind’s reinforcement learner for video games is described in “Human-level control through deep reinforcement learning,”* by Volodymyr Mnih et al. (Nature, 2015).
Paul Rosenbloom retells the development of chunking in “A cognitive odyssey: From the power law of practice to a general learning mechanism and beyond” (Tutorials in Quantitative Methods for Psychology, 2006). A/B testing and other online experimentation techniques are explained in “Practical guide to controlled experiments on the Web: Listen to your customers not to the HiPPO,”* by Ron Kohavi, Randal Henne, and Dan Sommerfield (Proceedings of the Thirteenth International Conference on Knowledge Discovery and Data Mining, 2007). Uplift modeling, a multidimensional generalization of A/B testing, is the subject of Chapter 7 of Eric Siegel’s Predictive Analytics (Wiley, 2013).
Introduction to Statistical Relational Learning,* edited by Lise Getoor and Ben Taskar (MIT Press, 2007), surveys the main approaches in this area. My work with Matt Richardson on modeling word of mouth is summarized in “Mining social networks for viral marketing” (IEEE Intelligent Systems, 2005).
Chapter Nine
Model Ensembles: Foundations and Algorithms,* by Zhi-Hua Zhou (Chapman and Hall, 2012), is an introduction to metalearning. The original paper on stacking is “Stacked generalization,”* by David Wolpert (Neural Networks, 1992). Leo Breiman introduced bagging in “Bagging predictors”* (Machine Learning, 1996) and random forests in “Random forests”* (Machine Learning, 2001). Boosting is described in “Experiments with a new boosting algorithm,” by Yoav Freund and Rob Schapire (Proceedings of the Thirteenth International Conference on Machine Learning, 1996).
“I, Algorithm,” by Anil Ananthaswamy (New Scientist, 2011), chronicles the road to combining logic and probability in AI. Markov Logic: An Interface Layer for Artificial Intelligence,* which I cowrote with Daniel Lowd (Morgan & Claypool, 2009), is an introduction to Markov logic networks. The Alchemy website, http://alchemy.cs.washington.edu, also includes tutorials, videos, MLNs, data sets, publications, pointers to other systems, and so on. An MLN for robot mapping is described in “Hybrid Markov logic networks,”* by Jue Wang and Pedro Domingos (Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, 2008). Thomas Dietterich and Xinlong Bao describe the use of MLNs in DARPA’s PAL project in “Integrating multiple learning components through Markov logic”* (Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, 2008). “Extracting semantic networks from text via relational clustering,”* by Stanley Kok and Pedro Domingos (Proceedings of the Nineteenth European Conference on Machine Learning, 2008), describes how we used MLNs to learn a semantic network from the Web.
Efficient MLNs with hierarchical class and part structure are described in “Learning and inference in tractable probabilistic knowledge bases,”* by Mathias Niepert and Pedro Domingos (Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, 2015). Google’s approach to parallel gradient descent is described in “Large-scale distributed deep networks,”* by Jeff Dean et al. (Advances in Neural Information Processing Systems 25, 2012). “A general framework for mining massive data streams,”* by Pedro Domingos and Geoff Hulten (Journal of Computational and Graphical Statistics, 2003), summarizes our sampling-based method for learning from open-ended data streams. The FuturICT project is the subject of “The machine that would predict the future,” by David Weinberger (Scientific American, 2011).
“Cancer: The march on malignancy” (Nature supplement, 2014) surveys the current state of the war on cancer. “Using patient data for personalized cancer treatments,” by Chris Edwards (Communications of the ACM, 2014), describes the early stages of what could grow into CanceRx. “Simulating a living cell,” by Markus Covert (Scientific American, 2014), explains how his group built a computer model of a whole infectious bacterium. “Breakthrough Technologies 2015: Internet of DNA,” by Antonio Regalado (MIT Technology Review, 2015), reports on the work of the Global Alliance for Genomics and Health. Cancer Commons is described in “Cancer: A Computational Disease that AI Can Cure,” by Jay Tenenbaum and Jeff Shrager (AI Magazine, 2011).
Chapter Ten
“Love, actuarially,” by Kevin Poulsen (Wired, 2014), tells the story of how one man used machine learning to find love on the OkCupid dating site. Dataclysm, by Christian Rudder (Crown, 2014), mines OkCupid’s data for sundry insights. Total Recall, by Gordon Moore and Jim Gemmell (Dutton, 2009), explores the implications of digitally recording everything we do. The Naked Future, by Patrick Tucker (Current, 2014), surveys the use and abuse of data for prediction in our world. Craig Mundie argues for a balanced approach to data collection and use in “Privacy pragmatism” (Foreign Affairs, 2014). The Second Machine Age, by Erik Brynjolfsson and Andrew McAfee (Norton, 2014), discusses how progress in AI will shape the future of work and the economy. “World War R,” by Chris Baraniuk (New Scientist, 2014) reports on the debate surrounding the use of robots in battle. “Transcending complacency on superintelligent machines,” by Stephen Hawking et al. (Huffington Post, 2014), argues that now is the time to worry about AI’s risks. Nick Bostrom’s Superintelligence (Oxford University Press, 2014) considers those dangers and what to do about them.
A Brief History of Life, by Richard Hawking (Random Penguin, 1982), summarizes the quantum leaps of evolution in the eons BC. (Before Computers. Just kidding.) The Singularity Is Near, by Ray Kurzweil (Penguin, 2005), is your guide to the transhuman future. Joel Garreau considers three different scenarios for how human-directed evolution will unfold in Radical Evolution (Broadway Books, 2005). In What Technology Wants (Penguin, 2010), Kevin Kelly argues that technology is the continuation of evolution by other means. Darwin Among the Machines, by George Dyson (Basic Books, 1997), chronicles the evolution of technology and speculates on where it will lead. Craig Venter explains how his team synthesized a living cell in Life at the Speed of Light (Viking, 2013).