Discounted fully probabilistic design of decision rules

• Published in: Information sciences Vol. 690; p. 121578
• Main Authors: Kárný, Miroslav, Molnárová, Soňa
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01-02-2025
• Subjects: Closed loop; Design principles; Discounting; Kullback-Leibler's divergence; Probabilistic techniques; Probabilistic techniques; Discounting; Closed loop; Kullback-Leibler's divergence; Design principles
• ISSN: 0020-0255
• DOI: 10.1016/j.ins.2024.121578

Axiomatic fully probabilistic design (FPD) of optimal decision rules strictly extends the decision making (DM) theory represented by Markov decision processes (MDP). This means that any MDP task can be approximated by an explicitly found FPD task whereas many FPD tasks have no MDP equivalent. MDP and FPD model the closed loop — the coupling of an agent and its environment — via a joint probability density (pd) relating the involved random variables, referred to as behaviour. Unlike MDP, FPD quantifies agent's aims and constraints by an ideal pd. The ideal pd is high on the desired behaviours, small on undesired behaviours and zero on forbidden ones. FPD selects the optimal decision rules as the minimiser of Kullback-Leibler's divergence of the closed-loop-modelling pd to its ideal twin. The proximity measure choice follows from the FPD axiomatics.
MDP minimises the expected total loss, which is usually the sum of discounted partial losses. The discounting reflects the decreasing importance of future losses. It also diminishes the influence of errors caused by:
▪ the imperfection of the employed environment model;
▪ roughly-expressed aims;
▪ the approximate learning and decision-rules design.
The established FPD cannot currently account for these important features. The paper elaborates the missing discounted version of FPD. This non-trivial filling of the gap in FPD also employs an extension of dynamic programming, which is of an independent interest.
•We provide the discounted fully probabilistic design (FPD) of decision rules that extends Markov decision processes.•The non-trivial filling of this gap in FPD is the main contribution.•We extend the standard dynamic programming to additive losses with summands dependent on the optimised decision rules.•The solution opens a way to the data-based choice of the discounting rate.•An experiment indicates that the solution counteracts the adverse impact of bad modelling on the decision quality.