Abstract--This paper introduces a novel Model-Free Markov Chain-based Reinforcement Learning (MFMC-RL) framework for the optimal energy management of hybrid microgrids. The core challenge in microgrid operation lies in managing the inherent uncertainties associated with intermittent renewable power generation, variable load demands, and complex battery system dynamics, particularly within a large-scale, non-linear environment. To overcome the computational burdens associated with centralized control strategies, we develop a unified, model-free learning approach. This system leverages the statistical power of Markov Chains to accurately model the stochastic nature of the environmental disturbances (load and renewable output). The proposed MFMC-RL framework achieves optimal hourly operational scheduling by learning the optimal control policy directly from system interactions, effectively maximizing overall microgrid profitability while minimizing operational costs and external grid dependency. The effectiveness and robustness of the developed learning-based control strategy are rigorously evaluated through extensive simulations using real-world data sourced from Iranian renewable energy installations and contemporary energy market pricing structures. The results indicate that the proposed MFMC-RL framework achieves up to 11.8% faster convergence and reduces operational costs by 5.2% compared to DDPG method, while maintaining superior stability under high uncertainty.