Course Schedule
Final Exam (Assigned: June 10, 2019; Due: June 13, 2019 Pacific Time 5 PM via email to instructor)
Week 10:
Lecture 20 (06/04): Solving the determinstic HJB: method of characteristics, analytical guess, Hopf-Lax representation formula; stochastic HJB: problem setting, result, derivation outline, Hamiltonian, relation with Backward Kolmogorov operator. [Lecture 20 notes]
Week 9:
Lecture 19 (05/31): State estimation for linear Gaussian systems; Kalman filter in discrete time; LQG problem: DP equation and its solution; controller-estimator architecture; certainty equivalence; optimal estimator for linear non-Gaussian systems; deterministic DP in continuous time: value function, Bellman's principle of optimality, and HJB PDE; connections between the deterministic HJB and the PMP: optimized Hamiltonian and costates. [Lecture 19 notes]
Lecture 18 (05/30): State estimation for uncontrolled Markov chain; the same for controlled Markov chain; optimal control of a Markov chain with partial observation; intro to separation principle; conditional mean and MMSE estimator; brief review of basic probability and Gaussian random vectors; MMSE estimator when the state and observation are jointly Gaussian. [Lecture 18 notes]
Lecture 17 (05/28): Contraction mapping for the Bellman operator in infinite horizon discounted MDP with countable states; optimality of stationary policy; value iteration; policy iteration; monotonicity of the Bellaman operator; linear programming solution; intro to POMDP; state estimation for a Markov chain. [Lecture 17 notes]
Week 8:
Lecture 16 (05/23): DP equations in discrete time; a worked out example for deterministic DP recursion; a worked out example for stochastic DP recursion; infinite horizon problems: total cost and total discounted cost; finite horizon discounted cost; infinite horizon discounted cost: system of nonlinear functional equations; contraction mapping theorem. [Lecture 16 notes]
Lecture 15 (05/21): DP in discrete-time: introductory example; Bellman's principle of optimality; backward recursion; feedback/law/policy ≠ actions/controls; stochastic DP; history-dependent policies: randomized and non-randomized; Markov policy; stochastic DP equations. [Lecture 15 notes]
Week 7:
Lecture 14 (05/16): Singular controls in nonlinear systems; geometric meaning of switching function in minimum time control affine nonlinear systems with bounded control; iterated Lie brackets and their use in proving the existence/non-existence of singular arcs; Fuller's problem: bang-bang optimal control with Zeno behavior; comparative summary of Maximum Principle and Dynamic Programming: theory, problem setting, and computation. [Lecture 14 notes]
Lecture 13 (05/14): Bang-Bang control for minimum time steering of LTI systems with bounded control; Bang-Off-Bang control: dead-zone-function, double integrator steering with integral of the absolute value of control cost and bounded control; mixed state and control inequality constraints; pure state inequality constraints. [Lecture 13 notes]
Week 6:
Lecture 12 (05/09): Interior point state equality constraints; control inequality constraints: general case, special case of Bang-Bang control; Example: minimum time steering of double integrator to origin with control inequality constrsaint, Bang-Bang structure and state feedback synthesis via switching curves. [Lecture 12 notes]
Lecture 11 (05/07): No lecture; [Midterm with solution]
Week 5:
Lecture 10 (05/02): Infinite horizon LQR: continuous and discrete-time, propositions and background info, solving CARE and DARE in MATLAB; Handling additional constraints in the OCP: integral equality constraints; equality constraints on functions of control and time; equality constraints on functions of state, control and time; equality constraints on functions of state and time but control not appearing explicitly. [Lecture 10 notes]
Lecture 9 (04/30): Solution of the 2PBVP for continuous-time finite horizon LQR with terminal cost: Riccati matrix ODE, Hamiltonian matrix and Bernoulli substitution, sufficiency; LQR with cross weights: Popov matrix; finite horizon continuous-time LQR for tracking; optimal cost for finite horizon LQR; properties of Riccati ODE; discrete time OCP and necessary conditions; discrete-time finite horizon LQR with terminal cost: solution of the 2PBVP, Riccati matrix recursion on the cone. [Lecture 9 notes]
Week 4:
Lecture 8 (04/25): Open-loop and feedback optimal control: timetable vs. policy, implication for instrumentation; minimum energy state transfer in LTV system: solution; regulation vs. tracking problem; LQR: problem formulation; finite horizon LQR with final time fixed: terminal cost and dispensing controllability, 2PBVP. [Lecture 8 notes]
Lecture 7 (04/23): Thrust angle programming: solution; minimum energy state transfer in LTV system: problem formulation; background on the State Transition Matrix (STM): definition, LTV solution, STM properties; background on the controllability Gramian: definition, equivalent statements for LTV controllability, Lyapunov matrix ODE; controllability Gramian in the LTI case, controllability in the LTI case. [Lecture 7 notes] [Remarks on Transversality Condition]
Week 3:
Lecture 6 (04/18): OCP template: Bolza form; Lagrange form and Mayer form and their equivalence; first order necessary conditions for optimality: state and costate ODEs, Pontryagin's Maxim principle (PMP), transversality condition; total derivative of the Hamiltonian along the optimal solution; Examples: shortest planar path is straight line, temperature control in a room (fixed and free terminal state), thrust angle programming (intercepting moving target in minimum time): problem formulation. [Lecture 6 notes]
Lecture 5 (04/16): Newtonian mechanics from CoV, Hamiltonian: via convex conugate and via Beltrami identity, Hamilton's canonical equations, EL equation for deriving equations of motion: example of ball in rotating tube, holonomic pointwise equality constraint: converting the constrained CoV problem to unconstrained CoV problem, pendulum example, from CoV to OCP. [Lecture 5 notes]
Week 2:
Lecture 4 (04/11): Example: minimal surface problem (shape of co-axial soap film); functional gradient and gradient descent interpretation of the EL equation; integral equality constraints: augmented Lagrangian and example of isoperimetric (Dido's) problem; pointwise equality constraints: Lagrange multiplier is a function; multi-doF EL equation: general form; Newtonian mechanics as special case of multi-doF EL equation. [Lecture 4 notes]
Lecture 3 (04/09): Beltrami identity; EL equation for Lagrangians with higher order derivatives; Hilbert's theorem and its corollary; worked out examples: straight line is shortest curve, cycloid is fastest curve (Brachistochrone with constant g), hypocycloid is fastest curve (Brachistochrone with variable g inside Earth), Laplace equation, linear and nonlinear Poisson equations. [Lecture 3 notes]
Week 1
Lecture 2 (04/04): Necessary conditions of optimality for CoV: Euler-Lagrange equation; Comparison with OPT necessary conditions; Two preparatory results to derive EL equation: (i) a version of Divergence Theorem, (ii) Vanishing Lemma; Statement and proof of EL equation for the unconstrained CoV problem; Sufficient conditions for the existence of minimum: joint convexity of the Lagrangian; Sufficient conditions for the uniqueness of minimum: strcit joint convexity of the Lagrangian; examples and counterexamples for sufficient condition. [Lecture 2 notes]
Lecture 1 (04/02): Course logistics; overview of optimization (OPT), calculus of variations (CoV) and optimal control problems (OCP); finite vs. infinite dimensioanl optimization; single agent vs. multi-agent decision making: OCP vs. differential game; template mathematical formulation for OPT, CoV and OCP; examples: minimum length curve, minimum time curve, isoperimetric problem; stories: Brachistochrone and Dido problem. [Lecture 1 notes]