Estimation and control using sampling‐based Bayesian reinforcement learning

• Published in: IET cyber-physical systems Vol. 5; no. 1; pp. 127 - 135
• Main Authors: Slade, Patrick, Sunberg, Zachary N., Kochenderfer, Mykel J.
• Format: Journal Article
• Language: English
• Published: Southampton The Institution of Engineering and Technology 01-03-2020; John Wiley & Sons, Inc; Wiley
• Subjects: Adaptive control; Algorithms; Approximation; autonomous control systems; Bayes methods; Bayes‐adaptive Markov decision process; belief networks; certainty equivalent model predictive control; changing dynamics; control objectives; Control tasks; Decision making; decision theory; discrete time simulations; Discrete time systems; discrete-time nonlinear systems; Dynamic programming; information gathering actions; input-constraints; Kalman filters; Linear systems; Markov processes; Methods; mobile robots; modelling errors; Monte Carlo methods; Monte Carlo simulation; Monte Carlo tree search parameters; Noise control; Noise prediction; offline optimisation method; optimal actions; optimisation; Parameter uncertainty; path planning; Performance prediction; Planning; pose dynamics; Predictive control; probabilistic bounding heuristic; Probability; process noise; Process parameters; QMDP solution; real-world autonomous systems; recursive feasibility guarantees; Recursive functions; Robots; Robust control; sampling-based bayesian reinforcement learning; Search methods; Statistical analysis; stochastic systems; Systems stability; tree searching; uncertain systems; unscented Kalman filter
• ISSN: 2398-3396; 2398-3396
• DOI: 10.1049/iet-cps.2019.0045

Real‐world autonomous systems operate under uncertainty about both their pose and dynamics. Autonomous control systems must simultaneously perform estimation and control tasks to maintain robustness to changing dynamics or modelling errors. However, information gathering actions often conflict with optimal actions for reaching control objectives, requiring a trade‐off between exploration and exploitation. The specific problem setting considered here is for discrete‐time non‐linear systems, with process noise, input‐constraints, and parameter uncertainty. This study frames this problem as a Bayes‐adaptive Markov decision process and solves it online using Monte Carlo tree search with an unscented Kalman filter to account for process noise and parameter uncertainty. This method is compared with certainty equivalent model predictive control and a tree search method that approximates the QMDP solution, providing insight into when information gathering is useful. Discrete time simulations characterise performance over a range of process noise and bounds on unknown parameters. An offline optimisation method is used to select the Monte Carlo tree search parameters without hand‐tuning. In lieu of recursive feasibility guarantees, a probabilistic bounding heuristic is offered that increases the probability of keeping the state within a desired region.