What do you think?
Rate this book
Adaptive Computation and Machine Learning
Reinforcement Learning: An Introduction
Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications.
Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability.
The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.
Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability.
The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.
322 pages, Hardcover
Published March 1, 1998
About the author
Richard S. Sutton
6 books21 followersRichard S. Sutton is Professor of Computing Science and AITF Chair in Reinforcement Learning and Artificial Intelligence at the University of Alberta, and also Distinguished Research Scientist at DeepMind.
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 30 of 75 reviews
December 15, 2019
Takeouts: thoughts on
- Mone-Karlo methods,
- dynamic programming,
- learning from temporal differences.
- Mone-Karlo methods,
- dynamic programming,
- learning from temporal differences.
Shelved as 'read-part'
December 18, 2015Despite its age, this book is still the canonical introduction to reinforcement learning.
I'm reading parts as necessary — not sure if I'll ever read cover-to-cover. In any case this has been an indispensable resource in my research career.
From the outside, RL seems mathy and somewhat stilted; from the inside, there is a lot of room for creativity and the core concepts are quite straightforward. I credit this book (along with some incredibly talented mentors) for introducing me to that beautiful insider's view.
I'm reading parts as necessary — not sure if I'll ever read cover-to-cover. In any case this has been an indispensable resource in my research career.
From the outside, RL seems mathy and somewhat stilted; from the inside, there is a lot of room for creativity and the core concepts are quite straightforward. I credit this book (along with some incredibly talented mentors) for introducing me to that beautiful insider's view.
September 7, 2017
I' not finished this book but already want to leave a review. This is a very readable and still rigorous description of reinforcement learning. The main difference between this book and many others in the field of machine learning is that the author really tries to make his work approachable by others. Reading this is a joy, highly recommended.
July 24, 2018
One of the best book I ever read. A big step toward AI. The book inspired me to dig deeper.
A good supplementary would be an online course by Sutton's student and a former lead at Deepmind David Silver.
A good supplementary would be an online course by Sutton's student and a former lead at Deepmind David Silver.
December 6, 2020
این کتاب دروازه ای هست برای ورود به دنیای یادگیری تقویتی
یادگیری تقویتی نوعی یادگیری بر پایه پاداش و تنبیه ئه، شبیه به روشی که انسان یاد میگیره
یعنی کاری که براش سود داره رو بیشتر انجام میده و کاری که بهش آسیب میرسونه رو کمتر انجام میده
ویژگی خوب این کتاب فراگیر بودنشه. در هر ویرایش مباحث جدیدی بهش اضافه میشه.
کتاب سه قسمت داره.
بخش اول
از صفر شروع میکنه به گفتن مباحث و دانش پیشین خاصی نمیخواد. یکم ریاضی و یکم آمار و احتمالات
مباحثی مطرح میشه مثل اینکه پاداش چیه، هدف رو چطوری پیدا کنیم، آینده نگر بودن یا نبودن عامل و الگوریتمای ابتدایی یادگیری تقویتی توضیح داده میشن
بخش دوم
توی بخش اول همه اطلاعات توی جداول ذخیره میشه که عملا فقط برای مسائل کوچک مثل پیدا کردن مسیر توی یه جدول میشه ازشون استفاده کرد
برای استفاده از یادگیری تقویتی توی مسائل بزرگتر (مثل بازیهای آتاری) نیازه که مقادیر با پارامترهایی تخمین زده بشن. برای تخمین زدن این مقادیر باید از روشهای هوش مصنوعی (برای مثال، شبکه عصبی) استفاده کرد. پس فصل دوم مثل اول ساده نیست و داشتن دانشی متوسط از هوش مصنوعی و روشهاش لازمه
بخش سوم
توی قسمت سوم کتاب هم کاربرد ها و چالشهای پیش رو برای این نوع یادگیری رو نویسنده توضیح داده
مزیت این روش نسبت به یادگیری ماشین اینه که نیاز به دانش و اطلاعات زیادی از محیط نیست و عامل طی تعامل با محیط ،فقط با پاداش و تنبیه میفهمه که چه کاری خوبه و چه کاری بد
برای یادگیری بازیهای کامپیوتری هم از یادگیری تقویتی استفاده میشه
در کل به همه دانشجوهای هوش مصنوعی پیشنهاد میکنم این کتاب رو بخونن
یادگیری تقویتی نوعی یادگیری بر پایه پاداش و تنبیه ئه، شبیه به روشی که انسان یاد میگیره
یعنی کاری که براش سود داره رو بیشتر انجام میده و کاری که بهش آسیب میرسونه رو کمتر انجام میده
ویژگی خوب این کتاب فراگیر بودنشه. در هر ویرایش مباحث جدیدی بهش اضافه میشه.
کتاب سه قسمت داره.
بخش اول
از صفر شروع میکنه به گفتن مباحث و دانش پیشین خاصی نمیخواد. یکم ریاضی و یکم آمار و احتمالات
مباحثی مطرح میشه مثل اینکه پاداش چیه، هدف رو چطوری پیدا کنیم، آینده نگر بودن یا نبودن عامل و الگوریتمای ابتدایی یادگیری تقویتی توضیح داده میشن
بخش دوم
توی بخش اول همه اطلاعات توی جداول ذخیره میشه که عملا فقط برای مسائل کوچک مثل پیدا کردن مسیر توی یه جدول میشه ازشون استفاده کرد
برای استفاده از یادگیری تقویتی توی مسائل بزرگتر (مثل بازیهای آتاری) نیازه که مقادیر با پارامترهایی تخمین زده بشن. برای تخمین زدن این مقادیر باید از روشهای هوش مصنوعی (برای مثال، شبکه عصبی) استفاده کرد. پس فصل دوم مثل اول ساده نیست و داشتن دانشی متوسط از هوش مصنوعی و روشهاش لازمه
بخش سوم
توی قسمت سوم کتاب هم کاربرد ها و چالشهای پیش رو برای این نوع یادگیری رو نویسنده توضیح داده
مزیت این روش نسبت به یادگیری ماشین اینه که نیاز به دانش و اطلاعات زیادی از محیط نیست و عامل طی تعامل با محیط ،فقط با پاداش و تنبیه میفهمه که چه کاری خوبه و چه کاری بد
برای یادگیری بازیهای کامپیوتری هم از یادگیری تقویتی استفاده میشه
در کل به همه دانشجوهای هوش مصنوعی پیشنهاد میکنم این کتاب رو بخونن
July 18, 2023
I came across this book as the textbook for Reinforcement Learning Specialization on Coursera. However, I continued to read it for fun afterwards.
It serves as an excellent introduction to reinforcement learning (RL) providing great insight into not only the techniques of RL, but also the fundamental motivations and underlying ideas behind RL. It explains concepts simple and jargon free manner that is easy to understand. While at times it is wordy, I was sufficiently interested to continue reading regardless.
I found particularly interesting the last section which elaborates on the connections of RL with psychology and neuroscience, as well as, applications, case studies, and frontiers. I would recommend this book to anyone interested in getting into RL.
To Read: 2.8, 2.9, 5.8, 5.9, 6.7, 6.8, 7, 8.4-8.10, 9.5.1, 9.5.2, 9.5.5, 9.6-9.12, 12
It serves as an excellent introduction to reinforcement learning (RL) providing great insight into not only the techniques of RL, but also the fundamental motivations and underlying ideas behind RL. It explains concepts simple and jargon free manner that is easy to understand. While at times it is wordy, I was sufficiently interested to continue reading regardless.
I found particularly interesting the last section which elaborates on the connections of RL with psychology and neuroscience, as well as, applications, case studies, and frontiers. I would recommend this book to anyone interested in getting into RL.
To Read: 2.8, 2.9, 5.8, 5.9, 6.7, 6.8, 7, 8.4-8.10, 9.5.1, 9.5.2, 9.5.5, 9.6-9.12, 12
December 16, 2020
Введение в теорию обучения с подкреплением. Истоки, принципы, теория, методы, практические примеры. Читал второе издание, там уже есть и AlphaGo и DOTA. Книга все же о теории, не руководство
January 4, 2012
For me, this is one of the best books on AI. Even though the material is not that simple, everything is clearly explained and the book is comprehensible even for people who are not familiar with the concepts. Even though this is an older book, it is still the best I've seen on the topic.
December 27, 2018
Really good textbook.
I was surprised about how well we understand much of RL. Coming from ML this was a welcome novelty.
Although, I would have like to see a few more of the proofs and for there to be exercises.
I was surprised about how well we understand much of RL. Coming from ML this was a welcome novelty.
Although, I would have like to see a few more of the proofs and for there to be exercises.
Shelved as 'to-read-in-part'
November 10, 2021El curso Microsoft DAT257x Reinforcement Learning Explained recorre el libro cover to cover.
August 7, 2022
This is a wonderful textbook that's surprisingly available for free. I used this while taking the machine learning and reinforcement learning courses for OMSCS.
January 6, 2018
The book I spent my Christmas holidays with was Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto. The authors are considered the founding fathers of the field. And the book is an often-referred textbook and part of the basic reading list for AI researchers. Given my own interest and fledgling attempts in the area (I trained my first models in 2017), I thought worthwhile to spend some time learning some basics.
Reinforcement learning is one of the hottest fields in programming. But what does it mean specifically? Basically it is learning what to do - how to map situations to actions - so as to maximize a numerical reward signal. The computer is not told which actions to take, as in most forms of machine learning, but instead must discover which actions yield the most reward by trying them. In the most interesting and challenging cases, actions may affect not only the immediate reward, but also the next situation and, through that, all subsequent rewards.
The truth is you don’t have to read books like this to do some very basic AI work in 2018. If you have coding experience and some grasp of statistics and logics, you can skip to Youtube videos and openly available free courses with open-sourced code examples. You can use Tensorflow on your home computer or cloud deals by Google, Amazon or Microsoft to do the training, etc.
Another important thing to understand is you won’t learn programming or machine learning just by reading books. You’ve got to get your hands dirty. You have to do actual coding. That’s how human learning works ;).
But reading this book will certainly help. The book does require some grasp of math, logics, statistics, set theory and probability. But you can learn along the way.
To approach reinforcement learning, the best way is to first understand the problem it tries to resolve and only then study the algorithms which attempt that in one way or another. The authors explain that the reinforcement learning agent and its environment interact over a sequence of discrete time steps. The specification of their interface defines a particular task: the actions are the choices made by the agent; the states are basis for making the choices; and the rewards are the basis for evaluating the choices. Everything inside the agent is completely known and controllable by the agent; everything outside is incompletely controllable but may or may not be completely known. A policy is a stochastic rule by which the agent selects actions as a function of states. The agent's objective is to maximize the amount of reward it receives over time.
The return is the function of future rewards that the agent seeks to maximize. It has several different definitions depending upon whether one is interested in total reward or discounted reward. The first is appropriate for episodic tasks, in which the agent environment interaction breaks naturally into episodes; the second is appropriate for continual tasks, in which the interaction does not naturally break into episodes but continues without limit.
An environment satisfies the Markov property if its state compactly summarizes the past without degrading the ability to predict the future. This is rarely exactly true, but often nearly so; the state signal should be chosen or constructed so that the Markov property approximately holds. If the Markov property does hold, then the environment is called a Markov decision process (MDP). A finite MDP is an MDP with finite state and action sets. Most of the current theory of reinforcement learning is restricted to finite MDPs, but the methods and ideas apply more generally. A policy's value function assigns to each state the expected return from that state given that the agent uses the policy. The optimal value function assigns to each state the largest expected return achievable by any policy, write the authors.
After dealing with the reinforcement learning problem and some history of the field in Part I, Sutton and Barto analyze a variety of methods to deal with a variety of tasks for machine learning. You will read about dynamic programming, Monte Carlo methods, temporal difference learning (Sutton himself has contributed a lot to this approach).
All of the reinforcement learning methods the authors explore in this book have three key ideas in common. First, the objective of all of them is the estimation of value functions. Second, all operate by backing up values along actual or possible state trajectories. Third, all follow the general strategy of generalized policy iteration (GPI), meaning that they maintain an approximate value function and an approximate policy, and they continually try to improve each on the basis of the other. Interesting insight is that these approaches can be combined quite efficiently.
Part III offers a number of case studies where reinforcement learning was applied.
Although the book would have benefited greatly if it included the analysis of deep reinforcement learning techniques yielding fantastic results over the past few years, the book is a great source to learn from.
Reinforcement learning is one of the hottest fields in programming. But what does it mean specifically? Basically it is learning what to do - how to map situations to actions - so as to maximize a numerical reward signal. The computer is not told which actions to take, as in most forms of machine learning, but instead must discover which actions yield the most reward by trying them. In the most interesting and challenging cases, actions may affect not only the immediate reward, but also the next situation and, through that, all subsequent rewards.
The truth is you don’t have to read books like this to do some very basic AI work in 2018. If you have coding experience and some grasp of statistics and logics, you can skip to Youtube videos and openly available free courses with open-sourced code examples. You can use Tensorflow on your home computer or cloud deals by Google, Amazon or Microsoft to do the training, etc.
Another important thing to understand is you won’t learn programming or machine learning just by reading books. You’ve got to get your hands dirty. You have to do actual coding. That’s how human learning works ;).
But reading this book will certainly help. The book does require some grasp of math, logics, statistics, set theory and probability. But you can learn along the way.
To approach reinforcement learning, the best way is to first understand the problem it tries to resolve and only then study the algorithms which attempt that in one way or another. The authors explain that the reinforcement learning agent and its environment interact over a sequence of discrete time steps. The specification of their interface defines a particular task: the actions are the choices made by the agent; the states are basis for making the choices; and the rewards are the basis for evaluating the choices. Everything inside the agent is completely known and controllable by the agent; everything outside is incompletely controllable but may or may not be completely known. A policy is a stochastic rule by which the agent selects actions as a function of states. The agent's objective is to maximize the amount of reward it receives over time.
The return is the function of future rewards that the agent seeks to maximize. It has several different definitions depending upon whether one is interested in total reward or discounted reward. The first is appropriate for episodic tasks, in which the agent environment interaction breaks naturally into episodes; the second is appropriate for continual tasks, in which the interaction does not naturally break into episodes but continues without limit.
An environment satisfies the Markov property if its state compactly summarizes the past without degrading the ability to predict the future. This is rarely exactly true, but often nearly so; the state signal should be chosen or constructed so that the Markov property approximately holds. If the Markov property does hold, then the environment is called a Markov decision process (MDP). A finite MDP is an MDP with finite state and action sets. Most of the current theory of reinforcement learning is restricted to finite MDPs, but the methods and ideas apply more generally. A policy's value function assigns to each state the expected return from that state given that the agent uses the policy. The optimal value function assigns to each state the largest expected return achievable by any policy, write the authors.
After dealing with the reinforcement learning problem and some history of the field in Part I, Sutton and Barto analyze a variety of methods to deal with a variety of tasks for machine learning. You will read about dynamic programming, Monte Carlo methods, temporal difference learning (Sutton himself has contributed a lot to this approach).
All of the reinforcement learning methods the authors explore in this book have three key ideas in common. First, the objective of all of them is the estimation of value functions. Second, all operate by backing up values along actual or possible state trajectories. Third, all follow the general strategy of generalized policy iteration (GPI), meaning that they maintain an approximate value function and an approximate policy, and they continually try to improve each on the basis of the other. Interesting insight is that these approaches can be combined quite efficiently.
Part III offers a number of case studies where reinforcement learning was applied.
Although the book would have benefited greatly if it included the analysis of deep reinforcement learning techniques yielding fantastic results over the past few years, the book is a great source to learn from.
October 10, 2022
I was initially intimidated by this book but it turned out to be intuitive and not focused on formal math too much. Good examples, good "scientific storytelling". I didn't read the whole book, just the first third
February 4, 2017
Great book explaining the basic concepts of reinforcement learning. Parts I and II are very well explained. Part III I didn't like much but still quite informative, seems to be oriented for future research.
March 9, 2019
Concise introduction to the field that is fueling the development of autonomous agents.
July 5, 2021
Very well written. Maybe not the best book if you want to quickly jump into Deep RL and applying it to your problem.
November 28, 2021
This book is amazing! It goes through the main ideas of reinforcement learning algorithms, starting from the very basic, and explaining with detail each improvement, finishing explaining general learning algorithms with eligibility traces. When I started I thought it was going to be like most math textbooks that define some concepts and then derive mathematically algorithms, but it is written as a person who is explaining the important points regarding the algorithms, not spending time in math. But no details are missed, as from the explanation, you can grasp clear ideas and then easily interpret the math notation, required mostly for exercises. First chapters are dull, in the sense that explain things that look obvious, but in reality, it is defining the basis for the rest of the book, as it is explained as extensions to the same basic idea. The end chapters look into psychology and neurological details that may be related with learning algorithms, and research frontiers to motivate further research. The book is much more dense than what it looks from a less than 500 pages book, and each page contains a lot of information, so it is not a fast read. Be warned!
December 8, 2021
Just like the first edition, this is a very readable book (for the most part). It's a textbook but you could go cover to cover. Sutton writes lucid prose and the organization is great. This book provides a very well structured introduction to the different reinforcement learning diciplines. I thought the extra chapters on psychology, neuroscience were a fantastic edition. My only complaint is that when the book relies so heavily on prose, instead of math, diagrams, etc. it can be confusing when the author gets lazy with the text. This happens at the end of section I (Chapter 8) where the writing becomes repetitive, vague, and tangential. It also happens in some other chapters in section II. It felt like the authors were running out of steam in these chapters. But at the end of the day, what other book are you going to read? This is the OG for RL.
April 4, 2020
A really excellent textbook serving as an introduction to the field of Reinforcement Learning.
Does a really great job in a lot of ways:
i) very clear descriptions of almost all concepts
ii) a very systematic description that allows you to build a framework for understanding the field, permitting you to fit new bits of information into that framework
iii) Shows current work, extensions and touches on some recent exciting developments
Not so good:
i) trys very hard to be light on the maths, to the extent that it can limit understanding (especially on eligibility traces)
ii) Can seem to waffle on a little at times
But overall really truly excellent - a great guide to the uninitiated who are armed with the requisite background.
Does a really great job in a lot of ways:
i) very clear descriptions of almost all concepts
ii) a very systematic description that allows you to build a framework for understanding the field, permitting you to fit new bits of information into that framework
iii) Shows current work, extensions and touches on some recent exciting developments
Not so good:
i) trys very hard to be light on the maths, to the extent that it can limit understanding (especially on eligibility traces)
ii) Can seem to waffle on a little at times
But overall really truly excellent - a great guide to the uninitiated who are armed with the requisite background.
March 4, 2018
A little dated, but in terms of learning the basics without a whole lot of digging, this is probably the best book out there. If you are thinking about getting into RL, I would recommend reading this first, then maybe Decision Making Under Uncertainty, reading some papers, reading the white paper on OpenAI's gym, and then messing around with gym. Sutton gives some excellent resources for understanding the history of RL and the maths behind it all, and if you have the time, it's worth reading all the way through the book. The chapters are pretty well laid out in terms of knowledge building.
February 10, 2018
I only had enough knowledge to follow this book up until about chapter 10. Even so, so far one of my favourite books on machine learning. Clear, well-described problems within well-structured chapters, which build on each other in a logical way. Some folks have a working directory of the most illustrative problems from the book here: https://github.com/ShangtongZhang/rei...
September 9, 2020
Dense and informative. It would be helpful if there was a greater focus on building basic intuition with descriptive figures before diving into the technical details and heavy math. Like many textbooks the exercises felt like enormous jumps from the material and a better guide into them would be nice. Overall, though the book does give a nice tour-de-force of all the learning strategies developed to date and explores nuances of their behavior.
January 31, 2022
I haven't finished this book, but it was really helpful when doing some courses from the Reinforcement Learning specialization in Coursera from the University of Alberta a few years ago. The courses are based on the book and the instructors were students of Sutton and Barto themselves, and the book expands things further and it is an indispensable learning companion. The best thing is that the digital version of the book is free!
December 23, 2017
Since its arrival it has been considered the bible for reinforcement learning. Sutton and Barto explain everything very well. I recommend this book to everyone who wants to start in the field of reinforcement learning. I do have to say that the first edition is missing some new developments, but a second edition is on the way (free pdf can be found online).
January 14, 2019
"Also related to TD learning are Holland's (1975, 1976) early ideas about consistency among value predictions. These influenced one of the authors (Barto), who was a graduate student from 1970 to 1975 at the University of Michigan, where Holland was teaching. Holland's ideas led to a number of TD-related systems..."
January 1, 2020
worth re-reading.
great illustration on fundamental conceptual ideas.
needs some time to internalize all the methods and tricks about RL.
once you really got the idea, reinforcement learning becomes very intuitive.
yep, that's the most sensible way to build an automatic learning/optimizing robot.
great illustration on fundamental conceptual ideas.
needs some time to internalize all the methods and tricks about RL.
once you really got the idea, reinforcement learning becomes very intuitive.
yep, that's the most sensible way to build an automatic learning/optimizing robot.
Read
November 8, 2020This is the book I read while following my RL course in IISc.
A good supplemental material to this book would be David Silver's course on the same topic on YouTube.
This book is succinct and provides the required intuitions.
Along with the Math and the Algorithms the initial chapters provide the breakthroughs in Behavioral Psychology which led to progress in RL.
A good supplemental material to this book would be David Silver's course on the same topic on YouTube.
This book is succinct and provides the required intuitions.
Along with the Math and the Algorithms the initial chapters provide the breakthroughs in Behavioral Psychology which led to progress in RL.
March 14, 2024
A great book on Reinforcement Learning. At times it could be simplified and some more examples could be added to give the reader an easier understanding of the concepts. Overall, the book explains really well the problem of Reinforcement Learning, and many many different approches and different techniques on solving it.
July 8, 2018
Read the 2017 draft in parts. Provides rigorous descriptions of RL algorithms in a readable manner. One of my favorite books on Machine Learning.
I recommend anyone interested in learning RL to start with this and then moving into papers.
I recommend anyone interested in learning RL to start with this and then moving into papers.
December 21, 2018
Excellent introduction to reinforcement learning. Well written, and does a good job of walking the reader through the algorithms and building a depth and breadth of knowledge. A lot of interesting examples included as well.
Displaying 1 - 30 of 75 reviews