rw-book-cover

Metadata

Highlights

The Bitter Lesson

Whenever you see one of these papers that train some kind of gigantic neural net model to do something you were not even sure a neural network could do, unquestionably pushing the state of the art and reconfiguring your ideas of what is possible, you get conflicting emotions. On the one hand: it is very impressive. Good on you for pushing AI forward. On the other hand: how could we possibly keep up? As an AI academic, leading a lab with a few PhD students and (if you’re lucky) some postdoctoral fellows, perhaps with a few dozen Graphics Processing Units (GPUs) in your lab, this kind of research is simply not possible to do. (View Highlight)

New highlights added 2024-05-20

(Sutton talks about the “Bitter Pill”, referring to the insight that simple methods that scale well always win the day when more compute becomes available [22].) (View Highlight)