Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning

• Published in: Journal of modern power systems and clean energy Vol. 9; no. 3; pp. 534 - 544
• Main Authors: Yan Du, Fangxing Li, Helia Zandi, Yaosuo Xue
• Format: Journal Article
• Language: English
• Published: IEEE 01-01-2021
• Subjects: Bidding strategy; day-ahead electricity market; deep reinforcement learning; Markov game; multi-agent deterministic policy gradient (MADDPG); Nash equilibrium (NE)
• ISSN: 2196-5420
• DOI: 10.35833/MPCE.2020.000502

In this paper, a day-ahead electricity market bidding problem with multiple strategic generation company (GEN-CO) bidders is studied. The problem is formulated as a Markov game model, where GENCO bidders interact with each other to develop their optimal day-ahead bidding strategies. Considering unobservable information in the problem, a model-free and data-driven approach, known as multi-agent deep deterministic policy gradient (MADDPG), is applied for approximating the Nash equilibrium (NE) in the above Markov game. The MAD-DPG algorithm has the advantage of generalization due to the automatic feature extraction ability of the deep neural networks. The algorithm is tested on an IEEE 30-bus system with three competitive GENCO bidders in both an uncongested case and a congested case. Comparisons with a truthful bidding strategy and state-of-the-art deep reinforcement learning methods including deep $Q$ network and deep deterministic policy gradient (DDPG) demonstrate that the applied MADDPG algorithm can find a superior bidding strategy for all the market participants with increased profit gains. In addition, the comparison with a conventional-model-based method shows that the MADDPG algorithm has higher computational efficiency, which is feasible for real-world applications.