Reinforcement Learning of Structured Stabilizing Control for Linear Systems With Unknown State Matrix

• Published in: IEEE transactions on automatic control Vol. 68; no. 3; pp. 1746 - 1752
• Main Authors: Mukherjee, Sayak, Vu, Thanh Long
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-03-2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation models; Algorithms; Automatic control; Closed loops; Computational modeling; Continuous time systems; Control systems; Cyber-physical systems; Distributed control; Dynamical systems; ENGINEERING; Feedback control; Heuristic algorithms; Linear quadratic regulator; linear quadratic regulator (LQR); Linear systems; Mathematical analysis; Mathematical models; Multiagent systems; Optimal control; reinforcement learning; reinforcement learning (RL); stability guarantee; structured learning; Trajectory; Trajectory measurement
• ISSN: 0018-9286; 1558-2523
• DOI: 10.1109/TAC.2022.3155384

This article delves into designing stabilizing feedback control gains for continuous-time linear systems with unknown state matrix, in which the control gain is subjected to a structural constraint. We bring forth the ideas from reinforcement learning (RL) in conjunction with sufficient stability and performance guarantees in order to design these structured gains using the trajectory measurements of states and controls. We first formulate a model-based linear quadratic regulator (LQR) framework to compute the structured control gain. Subsequently, we transform this model-based LQR formulation into a data-driven RL algorithm to remove the need for knowing the system state matrix. Theoretical guarantees are provided for the stability of the closed-loop system and the convergence of the structured RL (SRL) algorithm. A remarkable application of the proposed SRL framework is in designing distributed static feedback control, which is necessary for automatic control of many large-scale cyber-physical systems. As such, we validate our theoretical results with numerical simulations on a multiagent networked linear time-invariant dynamical system.