Instructor Spring 2019: Nicholas Dronen

About this Course

Neural networks have enjoyed several waves of popularity over the past half century. Each time they become popular, they promise to provide a general purpose artificial intelligence–a computer that can learn to do any task that you could program it to do. The first wave of popularity, in the late 1950s, was crushed by theoreticians who proved serious limitations to the techniques of the time. These limitations were overcome by advances that allowed neural networks to discover internal representations, leading to another wave of enthusiasm in the late 1980s. The second wave died out as more elegant, mathematically principled algorithms were developed (e.g., support-vector machines, Bayesian models). Around 2010, neural nets had a third resurgence. What happened over the past 20 years? Basically, computers got much faster and data sets got much larger, and the algorithms from the 1980s—with a few critical tweaks and improvements—appear to once again be state of the art, consistently winning competitions in computer vision, speech recognition, and natural language processing. The many accomplishments of the field have helped move research from academic journals into systems that improve our daily lives: apps that identify our friends in photos, automated vision systems that match or outperform humans in large-scale object recognition, phones and home appliances that recognize continuous, natural speech, self-driving cars, and software that translates from any language to any other language.
In this course, we’ll examine the history of neural networks and state-of-the-art approaches to deep learning. Students will learn to design neural network architectures and training procedures via hands-on assignments. Students will read current research articles to appreciate state-of-theart approaches as well as to question some of the hype that comes with the resurgence of popularity.

Prerequisites

The course is open to any students who have some background in cognitive science or artificial intelligence and who have taken introductory probability/statistics and linear algebra. Students must be competent in Python and NumPy – or be able to learn NumPy quickly.

Readings

Even though the lecture notes cover most of the material I care for you to know, the text will provide a more detailed and formal treatment of some of the topics. I’m very happy if you read the text in advance of class so that you can ask informed questions, or ask me to clarify material.

The primary text will be Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville. The text is available online by chapter in html, but it should serve as a good reference and is worth purchasing. If you wish additional background reading, consult:

  • Pattern Recognition and Machine Learning, Christopher Bishop
  • Deep Learning: Methods and Applications, Deng and Yu
  • Wikipedia is often a useful resource.

In the second half of the course, we’ll discuss current articles from the literature, all of which will be available on arXiv or other online sources.

The grandfather of the modern neural net field is Geoffrey Hinton from the University of Toronto (now at Google). He taught a Coursera class in 2012; it is a bit dated, but he gives such beautiful explanations and intuitions that his lectures are well worth viewing. Coursera has since deprecated the material, but it is available via YouTube. Many of his tutorials and invited talks are available on the web and I recommend viewing these talks over pretty much any other way you might spend your time.

Grades

Semester grades will be weighted with the following proportions :
5% Participation
40% Assignments
20% Midterm
5% Project proposals
30% Final project