**ISBNs:** 1-886529-26-4 (Vol. I, 3rd Edition), 1-886529-44-2
(Vol. II, 4th Edition), 1-886529-08-6 (Two-Volume Set, i.e., Vol. I, 3rd ed. and Vol. II, 4th edition)

**Vol. I, 3RD EDITION, 2005**, 558 pages,
hardcover

**Vol. II, 4TH EDITION: APPROXIMATE DYNAMIC PROGRAMMING 2012**, 712
pages, hardcover

Vol. I (ISBN10: 1-886529-26-4 or ISBN13: 978-1-886529-26-7): $89.00,

Vol. II (ISBN10: 1-886529-44-2 or ISBN13: 978-1-886529-44-1): $89.00,

Two-volume set, latest editions (ISBN10: 1-886529-08-6 or ISBN13: 978-1-886529-08-3): $134.50

**The TWO-VOLUME SET consists of the LATEST EDITIONS OF VOL. I AND VOL. II, ** i.e., Vol. I, 3rd ed. and Vol. II, 4th ed.

Contents, Preface, Videos (12-hours) from Youtube. Ordering, Home

The leading and most up-to-date textbook on the far-ranging algorithmic methododogy of Dynamic Programming, which can be used for optimal control, Markovian decision problems, planning and sequential decision making under uncertainty, and discrete/combinatorial optimization. The treatment focuses on basic unifying themes, and conceptual foundations. It illustrates the versatility, power, and generality of the method with many examples and applications from engineering, operations research, and other fields. It also addresses extensively the practical application of the methodology, possibly through the use of approximations, and provides an extensive treatment of the far-reaching methodology of Neuro-Dynamic Programming/Reinforcement Learning.

The first volume is oriented towards modeling, conceptualization, and finite-horizon problems, but also includes a substantive introduction to infinite horizon problems that is suitable for classroom use. The second volume is oriented towards mathematical analysis and computation, treats infinite horizon problems extensively, and provides an up-to-date account of approximate large-scale dynamic programming and reinforcement learning. The text contains many illustrations, worked-out examples, and exercises.

This extensive work, aside from its focus on the mainstream dynamic programming and optimal control topics, relates to our Abstract Dynamic Programming (Athena Scientific, 2013), a synthesis of classical research on the foundations of dynamic programming with modern approximate dynamic programming theory, and the new class of semicontractive models, Stochastic Optimal Control: The Discrete-Time Case (Athena Scientific, 1996), which deals with the mathematical foundations of the subject, Neuro-Dynamic Programming (Athena Scientific, 1996), which develops the fundamental theory for approximation methods in dynamic programming, and Introduction to Probability (2nd Edition, Athena Scientific, 2008), which provides the prerequisite probabilistic background.

** New features of the 4th edition of Vol. II (see the Contents and Preface for
details):**

Contains a substantial amount of new material, as well as a reorganization of old material. The length has increased by more than 60% from the third edition, and most of the old material has been restructured and/or revised. Volume II now numbers more than 700 pages and is larger in size than Vol. I. It can arguably be viewed as a new book!

A major expansion of the discussion of approximate DP (neuro-dynamic programming), which allows the practical application of dynamic programming to large and complex problems. Approximate DP has become the central focal point of this volume.

Extensive new material, the outgrowth of research conducted in the six years since the previous edition, has been included.

The first account of the emerging methodology of Monte Carlo linear algebra, which extends the approximate DP methodology to broadly applicable problems involving large-scale regression and systems of linear equations.

Expansion of the theory and use of contraction mappings in infinite state space problems and in neuro-dynamic programming.

" This is an excellent textbook on dynamic programming written by a master expositor. Between this and the first volume, there is an amazing diversity of ideas presented in a unified and accessible manner. This new edition offers an expanded treatment of approximate dynamic programming, synthesizing a substantial and growing research literature on the topic.
"

* Benjamin Van Roy, at Amazon.com.
*

** New features of the 3rd edition of Vol. I (see the Preface for
details):**

includes new material on aggregation methods, model predictive control, multiobjective DP, and constrained DP.

gives an expanded coverage of minimax control methods for estimation and control of systems with a non-probabilistic (set membership) description of uncertainty.

provides textbook accounts of recent original research on approximate DP, limited lookahead policies, rollout algorithms, and model predictive control.

includes a substantial number of new exercises, detailed solutions of many of which are posted on the internet (see below).

"In conclusion, the new edition represents a major upgrade of this well-established book. The coverage is significantly expanded, refined, and brought up-to-date. This is the only book presenting many of the research developments of the last 10 years in approximate DP/neuro-dynamic programming/reinforcement learning (the monographs by Bertsekas and Tsitsiklis, and by Sutton and Barto, were published in 1996 and 1998, respectively). The book is a rigorous yet highly readable and comprehensive source on all aspects relevant to DP: applications, algorithms, mathematical aspects, approximations, as well as recent research. It should be viewed as the principal DP textbook and reference work at present.
With its rich mixture of theory and applications, its many examples and exercises, its unified treatment of the subject, and its polished presentation style, it is eminently suited for classroom use or self-study."

* Panos Pardalos, in
Optimization Methods & Software Journal, 2007.
*

** Review of Vol. I, 3rd Edition:**

"In addition to being very well written and organized, the material has several special features
that make the book unique in the class of introductory textbooks on dynamic programming. For
instance, it presents both deterministic and stochastic control problems, in both discrete- and
continuous-time, and it also presents the Pontryagin minimum principle for deterministic systems
together with several extensions. It contains problems with perfect and imperfect information,
as well as minimax control methods (also known as worst-case control problems or games against
nature). I also has a full chapter on suboptimal control and many related techniques, such as
open-loop feedback controls, limited lookahead policies, rollout algorithms, and model
predictive control, to name a few. ... In conclusion the book is highly recommendable for an
introductory course on dynamic programming and its applications."

* Onesimo Hernandez Lerma, in
Mathematic Reviews, Issue 2006g.
*

**Among
its special features, the book:**

provides a unifying framework for sequential decision making

treats simultaneously deterministic and stochastic control problems popular in modern control theory and Markovian decision popular in operations research

develops the theory of deterministic optimal control problems including the Pontryagin Minimum Principle

introduces recent suboptimal control and simulation-based approximation techniques (neuro-dynamic programming), which allow the practical application of dynamic programming to complex problems that involve the dual curse of large dimension and lack of an accurate mathematical model

provides a comprehensive treatment of infinite horizon problems in the second volume, and an introductory treatment in the first volume

"In conclusion, this book is an excellent source of reference ... The
main strengths of the book are the clarity of the
exposition, the quality and variety of the examples, and its coverage
of the most recent advances."

* Thomas W.
Archibald, in IMA Jnl. of Mathematics Applied in Business & Industry*

"Here is a tour-de-force in the field."

* David K. Smith, in
Jnl. of Operational Research Society*

"By its comprehensive coverage, very good material
organization, readability of the exposition, included
theoretical results, and its challenging examples and
exercises, the reviewed book is highly recommended
for a graduate course in dynamic programming or for
self-study. It is a valuable reference for control theorists,
mathematicians, and all those who use systems and control theory in their
work. Students will for sure find the approach very readable, clear, and
concise. Misprints are extremely few."

* Vasile Sima, in SIAM Review*

"In this two-volume work Bertsekas caters equally effectively to
theoreticians who care for proof of such concepts as the
existence and the nature of optimal policies and to
practitioners interested in the modeling and the quantitative and
numerical solution aspects of stochastic dynamic programming."

*Michael Caramanis, in Interfaces*

"The textbook by Bertsekas is excellent, both as a reference for the
course and for general
knowledge. It is well written, clear and helpful"

Student evaluation guide for the Dynamic Programming and Stochastic
Control course at the
Massachusetts Institute of Technology

The author is McAfee Professor of Engineering at the Massachusetts Institute of Technology and a member of the prestigious US National Academy of Engineering. He is the recipient of the 2001 A. R. Raggazini ACC education award and the 2009 INFORMS expository writing award. He has been teaching the material included in this book in introductory graduate courses for nearly forty years.

The material listed below can be freely downloaded, reproduced, and distributed.

- Videos from Youtube, for a 6-lecture 12-hour short course on Approximate Dynamic Programming, Tsinghua University, Beijing, China, 2014
- Prof. Bertsekas' Lecture slides for a 6-lecture short course on Approximate Dynamic Programming, Tsinghua University, Beijing, China, 2014.
- Prof. Bertsekas' Course Lecture Slides, 2004
- Prof. Bertsekas' Course Lecture Slides, 2012
- Theoretical problem solutions, Volumes 1 and 2
- Course Material at Open Courseware at MIT
- Prof. Bertsekas' Research Papers on Dynamic and Neuro-Dynamic Programming
- Prof. Bertsekas' Ph.D. Thesis at MIT, 1971, Control of Uncertain Systems with a Set-Membership Description of the Uncertainty, which contains material supplementary to Sections 4.6 and 6.5 of Vol. 1
- Errata

[Return to Athena Scientific Homepage]