| 摘要: | 隨著再生能源發展，智慧電網也逐步成為趨勢。虛擬電廠（Virtual Power Plant, VPP）透過資通訊技術整合分散式能源（Distributed Energy Resources, DERs）、儲能系統（Energy Storage System, ESS）與可控負載，參與需量反應、餘電躉售等市場，已成為提升電網彈性與經濟效益的關鍵。然而，分散式能源具高度不確定性與動態性，傳統條件式控制策略難以有效應對，需引入具備學習能力與適應性的調度方法。
本研究提出一套基於強化學習（Reinforcement Learning, RL）之虛擬電廠用戶側控制策略，針對儲能系統充放電行為與可控負載卸載決策進行優化控制。方法採用深度Q學習（Deep Q-Network, DQN）與深度確定性策略梯度演算法（Deep Deterministic Policy Gradient, DDPG）進行比較，並建構具時間變動與經濟參數的不確定性模擬環境。七項輸入特徵包含時間資料、電池儲能狀態（State of Charge, SOC）與各項功率等，兩項輸出動作則涵蓋儲能功率與卸載功率。設計包含八項具經濟與操作意涵的獎勵函數，分別定義獲得的獎勵與總利潤、電池壽命、充放電與卸載動作行為等之間的關係。
為提升學習效率與樣本利用率，本研究於DDPG模型導入優先經驗回放機制（Prioritized Experience Replay, PER）與Ornstein-Uhlenbeck擾動噪聲（OU Noise）。PER依TD誤差排序經驗並結合重要性取樣（Importance Sampling, IS）修正學習偏差，加速策略收斂；OU Noise則提供連續動作空間中的平滑探索能力。
本研究共分為六種模擬情境，比較條件式控制策略（Rule-Based Control Strategy, Rule-Based）、DQN、DDPG在經濟性上的表現，包括時間電價造成的運轉成本，餘電躉售收益、卸載成本與契約容量超約罰鍰，並進一步以雨流計數法（Rainflow Counting Method, Rainflow）計算放電深度（Depth of Discharge, DoD）與次數，計算電池壽命損耗，驗證本方法於虛擬電廠控制應用中的潛力與成效，並於實際場域進行驗證。
;With the rise of renewable energy, Virtual Power Plant (VPP) has become a key for enhancing economic benefits and flexibility. VPP integrates distributed energy resources (DERs), storage systems and controllable demand through information and communication technologies, participating in markets such as demand response and surplus energy selling. However, due to the uncertainty and variability of DERs, traditional rule-based control methods are insufficient and need to apply adaptive strategies with learning capabilities.
This study proposes a reinforcement learning-based control strategy for demand-side VPP, focusing on the optimal coordination of battery storage and demand shedding decisions. Two algorithms—Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG)—are compared within a simulated environment reflecting time-varying and economic uncertainties. Input features include time, demand, utility, solar and SOC; output actions comprise charging/discharging and demand shedding power. 8 reward functions are designed to capture total profit, battery life, charge-discharge and demand shedding behavior.
To improve convergence and sample efficiency, DDPG incorporates Prioritized Experience Replay (PER) and Ornstein-Uhlenbeck (OU) noise. PER prioritizes samples based on TD error and uses importance sampling (IS) to correct bias, while OU noise enhances exploration in continuous action spaces.
The framework is tested across 6 scenarios, comparing Rule-Based, DQN, and DDPG performance in terms of economic benefits and battery life degradation measured by Rainflow method. Results demonstrate the effectiveness of reinforcement learning in VPP control. |
