From Dynamic Programming to HJB to Pontrygin Minimum Principle

Chapter 3 of the 3rd edition of Dimitri Betsekas's book "Dynamic Programming and Optimal Control" is on continuous-time optimal control. Betsekas used Dynamic Programming "the principle of optimality" developed by Bellman to formulate an iterative cost-to-go function for control problems. The basic philosophy is as follows: the cost-to-go function at step K is related to the current state and the control decision and the cost-to-go function for step K+1. Applying Taylor's expansion and organizing the equation, we then have the Hamiltonian-Jacobi-Bellman (HJB) equation.

With HJB, we can solve a linear quadratic regulator problem. The cost-to-go function has to be guessed. In the LQR problem, the guess is simple x'K(t)x. Once this is guessed, we can proceed and find the optimal control policy u* that can minimize HJB. Further the condition of K(t) can be identified as the continuous-time Riccati Equation. Let this problem be infinite horizon, the continuous-time Riccati equation evolves to the Algebraic Riccati Equation (ARE).

Here is a question. With HJB, can we solve the parking problem? The problem is like this: you want to drive your car to the origin at the minimum time. How should you drive given the limit of acceleration? Intuitively, one can answer, we should accelerate the car to the maximum for the first half distance. Then we decelerator the car to the maximum.

Using HJB, we have to guess the cost-to-go function. And this is difficult to do. In Betrsekas's book, he gave an example to show the difficulty. One has to first guess the control policy, then derive the cost-to-go function. BTW, this function has different expression for variety of conditions. It is not the continuous function we are comfortable with.

To be able to solve a bang-bang control problem, we have to go further to the Pontrygin Minimum Principle. This principle comes from HJB by this argument: if the optimal state trajectory x* is given, our life will be easier. and indeed, we can have a very concise expression: Hamiltonian function H(x, u, p) = g(x,u) + p*f(x,u) which bears similarity as a Lagrangian function L = obj(x) + lambda*h(x). The difference: f(x,u) is the ODE function, h(x) is the equality constraint; Also integration of g(x,u) is a component of the objective function.

Hamiltonian function greatly simplifies our life. p(t) is given by adjoint equations and these can be derived based on HJB's differentiation with respect to time and state. The optimization problem converts to find control policy u* over a horizon that will minimize the Hamiltonian function. And you do not need to guess the cost-to-go function. Everything is straightforward. With that, a bang-bang control problem can be solved with ease.

As a control engineer, we tend to remember the friendly Hamiltonian but forget the Dynamic Programming philosophy. Basic philosophy is important to understand - we will need it to extend such philosophy to problems with different format.

As an end note, in fact, we can simply make the integration in the objective function as a sum. Further, the ODE equation can be converted to discrete equations. In addition, each step of the state can be expressed just by control policy and initial state. We can formulate a Lagrangian and solve an unconstrained optimization problem. This objective function has nothing to do with x_k (state at each step). It is only related to control policy. Solving requires much more computing time than that required by Dynamic Programming formulation. Refer Stephen Boyd's lecture slides Linear Dynamic Systems.