A Novel Reinforcement Learning Framework for Hybrid Microgrid Dispatch Strategy based on Markov Decision Processes

Akbari, Ehsan

doi:10.22075/mseee.2026.40860.1263

A Novel Reinforcement Learning Framework for Hybrid Microgrid Dispatch Strategy based on Markov Decision Processes

Document Type : Research Article

Author

Ehsan Akbari

Department of Electrical Engineering, Mazandaran University of Science and Technology, Babol, Iran.

10.22075/mseee.2026.40860.1263

Abstract

This paper introduces a novel Model-Free Markov Chain-based Reinforcement Learning (MFMC-RL) framework for the optimal energy management of hybrid microgrids. The core challenge in microgrid operation lies in managing the inherent uncertainties associated with intermittent renewable power generation, variable load demands, and complex battery system dynamics, particularly within a large-scale, non-linear environment. To overcome the computational burdens associated with centralized control strategies, we develop a unified, model-free learning approach. This system leverages the statistical power of Markov Chains to accurately model the stochastic nature of the environmental disturbances (load and renewable output). The proposed MFMC-RL framework achieves optimal hourly operational scheduling by learning the optimal control policy directly from system interactions, effectively maximizing overall microgrid profitability while minimizing operational costs and external grid dependency. The effectiveness and robustness of the developed learning-based control strategy are rigorously evaluated through extensive simulations using real-world data sourced from Iranian renewable energy installations and contemporary energy market pricing structures. The results indicate that the proposed MFMC-RL framework achieves up to 11.8% faster convergence and reduces operational costs by 5.2% compared to the DDPG method, while maintaining superior stability under high uncertainty.

Keywords

Hybrid Microgrids

Energy Management System (EMS)

Reinforcement Learning

Markov Decision Processes (MDP)

Subjects

Power System

[1] K. A. Moungos, D. G. Kyriakou, and F. D. Kanellos, “Towards the integration of interconnected microgrids to deregulated electricity markets,” Electric Power Systems Research, vol. 239, pp. 111264–111264, Nov. 2024.

[2] Q. Hassan, S. Algburi, A. Z. Sameen, H. M. Salman, and M. Jaszczur, “A Review of Hybrid Renewable Energy Systems: Solar and wind-powered solutions: Challenges, opportunities, and Policy Implications,” Results in Engineering, vol. 20, no. 1, p. 101621, Nov. 2023.

[3] M. A. Kamarposhti, I. Colak, H. Shokouhandeh, C. Iwendi, Sanjeevikumar Padmanaban, and S. S. Band, “Optimum operation management of microgrids with cost and environment pollution reduction approach considering uncertainty using multi‐objective NSGAII algorithm,” IET Renewable Power Generation, vol. 19, no. 1, Aug. 2022.

[4] A. Dong and S.-K. Lee, “The Study of an Improved Particle Swarm Optimization Algorithm Applied to Economic Dispatch in Microgrids,” Electronics, vol. 13, no. 20, pp. 4086–4086, Oct. 2024.

[5] M. Naguib, W. A. Omran, and H. E. A. Talaat, “Performance Enhancement of Distribution Systems via Distribution Network Reconfiguration and Distributed Generator Allocation Considering Uncertain Environment,” Journal of Modern Power Systems and Clean Energy, vol. 10, no. 3, pp. 647–655, May 2022.

[6] M. Alilou, B. Tousi, and H. Shayeghi, “Home energy management in a residential smart micro grid under stochastic penetration of solar panels and electric vehicles,” Solar Energy, vol. 212, pp. 6–18, Dec. 2020.

[7] M. Amini, A. Khorsandi, B. Vahidi, S. H. Hosseinian, and A. Malakmahmoudi, “Optimal sizing of battery energy storage in a microgrid considering capacity degradation and replacement year,” Electric Power Systems Research, vol. 195, p. 107170, Jun. 2021.

[8] A. Sui, F. Wei, X. Lin, C. Wu, Z. Wang, Z. Li, “Multi-energy-storage energy management with the robust method for distribution networks,” Electrical Power and Energy System, vol. 118, pp. 1-15, June 2020.

[9] M. Cao, Q. Xu, X. Qin, J. Cai, “Battery energy storage sizing based on a model predictive control strategy with operational constraints to smooth the wind power,” Electrical Power and Energy Systems, vol. 115, pp. 1-10, Feb. 2020.

[10] L. Lei, Y. Tan, G. Dahlenburg, W. Xiang, and K. Zheng, "Dynamic Energy Dispatch Based on Deep Reinforcement Learning in IoT-Driven Smart Isolated Microgrids," IEEE Internet of Things Journal, 2020.

[11] X. Xu, Y. Jia, Y. Xu, Z. Xu, S. Chai, and C. S. Lai, "A Multi-agent Reinforcement Learning-based Data-driven Method for Home Energy Management," IEEE Transactions on Smart Grid, 2020.

[12] Z. Hu, M. Zhu, P. Chen, and P. Liu, "On convergence rates of game theoretic reinforcement learning algorithms," Automatica, Vol. 104, pp. 90-101, 2019.

[13] K. Deshpande, P. Möhl, A. Hämmerle, G. Weichhart, H. Zörrer, and A. Pichler, "Energy Management Simulation with Multi-Agent Reinforcement Learning: An Approach to Achieve Reliability and Resilience," Energies, Vol. 15, No. 19, p. 7381, 2022.

[14] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529-533, 2015.

[15] T. P. Lillicrap et al., “Continuous control with deep reinforcement learning,” in International Conference on Learning Representations (ICLR), 2016.

[16] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in Neural Information Processing Systems, vol. 30, pp. 6379-6390, 2017.

[17] D. Cao, W. Hu, J. Zhao, G. Zhang, B. Zhang, Z. Chen, and F. Blaabjerg, “Multi-agent deep reinforcement learning for microgrid energy management in smart cities,” IEEE Transactions on Industrial Informatics, vol. 16, no. 11, pp. 7020-7031, 2020.

[18] J. R. Vazquez-Canteli and Z. Nagy, “Reinforcement learning for demand response: A review of algorithms and modeling techniques,” Applied Energy, vol. 235, pp. 1072-1089, 2019.

[19] Y. Ye, D. Qiu, M. Sun, D. Papadaskalopoulos, and G. Strbac, “Deep reinforcement learning for pricing-based demand response of smart buildings,” IEEE Transactions on Smart Grid, vol. 11, no. 1, pp. 710-721, 2020.

[20] H. Li, Z. Wan, and H. He, “Real-time energy management of a multi-energy microgrid with integrated deep reinforcement learning,” IEEE Transactions on Smart Grid, vol. 11, no. 5, pp. 4190-4201, 2020.

[21] J. Lei, Y. Yang, D. Zheng, Y. Li, and C. Chen, “Deep reinforcement learning-based energy management of a multi-energy microgrid,” Energy, vol. 228, p. 120610, 2021.

[22] M. Daneshvar, B. Mohammadi-Ivatloo, K. Zare, and S. Asadi, “Optimal stochastic energy management of multi-microgrids with high penetration of renewable energy sources,” International Journal of Electrical Power & Energy Systems, vol. 121, p. 105930, 2020.