Cooperative primary frequency modulation control method for distributed energy storage based on reinforcement learning-model predictive control

doi:10.19799/j.cnki.2095-4239.2025.0296

Abstract

Abstract:

To enhance the frequency characteristics of power grids and fully leverage the rapid response advantages of distributed energy storage systems (DESSs), a cooperative primary frequency control method based on reinforcement learning-model predictive control (RL-MPC) is proposed. First, a primary frequency control model incorporating DESSs is established based on frequency response characteristics, state of charge (SOC), and power control strategies. Then, a hierarchical mixed control architecture is designed: the upper layer employs a deep Q-network (DQN) to dynamically optimize the MPC weight matrix while sensing frequency deviation, rate of change, and SOC distribution entropy in real time. The lower layer utilizes distributed MPC to determine the output sequences of multi-node energy storage units and introduces a graph attention network (GAT) to achieve adaptive optimization of the communication topology. This approach reduces computational complexity in coordinated control and enhances the strategy's generalization capability. Finally, simulations conducted in Matlab/Simulink verify that the proposed method effectively improves the primary frequency response speed and control accuracy of DESSs, thereby strengthening overall power system stability.

Key words: distributed energy storage, frequency modulation, model predictive control, reinforcement learning, graph attention network

CLC Number:

TK 02

Qian MA, Liang XIAO, Bing CHENG, Qin GAO, Chunxiao LIU, Yihua ZHU, Chengxiang LI. Cooperative primary frequency modulation control method for distributed energy storage based on reinforcement learning-model predictive control[J]. Energy Storage Science and Technology, 2025, 14(8): 3138-3148.

Figures/Tables 21

Fig. 1

Table 1

Fig. 2

Fig. 3

Table 2

Fig. 4

Table 3

Fig. 5

Fig. 6

Table 4

Fig. 7

Fig. 8

Fig. 9

Table 5

Fig. 10

Fig. 11

Table 6

Table 7

Fig. 12

Fig. 13

Fig. 14

References 15

[1]	桂玥, 赵熙临. 综合储能协同控制方法在一次调频中的应用[J]. 湖北工业大学学报, 2025, 40(1): 25-31. DOI: 10.3969/j.issn.1003-4684.2025.01.005.
	GUI Y, ZHAO X L. Application of a comprehensive energy storage cooperative control method in primary frequency modulation[J]. Journal of Hubei University of Technology, 2025, 40(1): 25-31. DOI: 10.3969/j.issn.1003-4684.2025.01.005.
[2]	黎萌, 林章岁, 林毅, 等. 基于改进模型预测控制的分布式储能辅助调频控制方法[J]. 水利水电技术(中英文), 2023, 54(S2): 447-456. DOI: 10.13928/j.cnki.wrahe.2023.S2.071.
	LI M, LIN Z S, LIN Y, et al. Distributed energy storage assisted frequency regulation control method based onimproved model predictive control[J]. Water Resources and Hydropower Engineering, 2023, 54(S2): 447-456. DOI: 10.13928/j.cnki.wrahe. 2023.S2.071.
[3]	池志坤, 袁至, 李骥. 基于系统频率与SOC状态预测的储能一次调频控制策略[J/OL]. 高电压技术, 2025: 1-15. (2025-01-23). https://link.cnki.net/doi/10.13336/j.1003-6520.hve.20241421.
	CHI Z K, YUAN Z, LI J. Primary frequency regulation control strategy of energy storage based on prediction of system frequency and state of charge[J/OL]. High Voltage Engineering, 2025: 1-15. (2025-01-23). https://link.cnki.net/doi/10.13336/j. 1003-6520.hve.20241421.
[4]	贺悝, 郭罗权, 谭庄熙, 等. 考虑调频阶段需求的混合储能一次调频综合控制策略[J]. 太阳能学报, 2024, 45(9): 697-708. DOI: 10. 19912/j.0254-0096.tynxb.2023-1580.
	HE L, GUO L Q, TAN Z X, et al. Comprehensive control strategy of hybrid energy storage in primary frequency regulation considering demands of frequency regulation stages[J]. Acta Energiae Solaris Sinica, 2024, 45(9): 697-708. DOI: 10.19912/j.0254-0096.tynxb.2023-1580.
[5]	孙冉, 王建波, 马彦钊, 等. 基于强化学习的新能源场站储能一次调频自适应控制策略[J]. 储能科学与技术, 2024, 13(3): 858-869. DOI: 10.19799/j.cnki.2095-4239.2023.0658.
	SUN R, WANG J B, MA Y Z, et al. Adaptive control strategy for primary frequency regulation for new energy storage stations based on reinforcement learning[J]. Energy Storage Science and Technology, 2024, 13(3): 858-869. DOI: 10.19799/j.cnki.2095-4239.2023.0658.
[6]	刘小龙, 庞敬磊, 吕智嘉, 等. 考虑荷电状态的电池储能一次调频综合控制方法[J]. 中国测试, 2024, 50(10): 59-65.
	LIU X L, PANG J L, LÜ Z J, et al. A comprehensive control method for primary frequency modulation of battery energy storage based on state of charge[J]. China Measurement & Test, 2024, 50(10): 59-65.
[7]	肖家杰, 李培强, 毛志宇, 等. 基于双层协调控制的电池储能参与电网二次调频策略[J]. 电力自动化设备, 2024, 44(8): 9-17. DOI: 10. 16081/j.epae.202401004.
	XIAO J J, LI P Q, MAO Z Y, et al. Strategy for battery energy storage participating in secondary frequency regulation of power grid based on two-layer coordinated control[J]. Electric Power Automation Equipment, 2024, 44(8): 9-17. DOI: 10.16081/j.epae. 202401004.
[8]	刘传斌, 矫文书, 吴秋伟, 等. 基于模型预测控制的风储联合电场参与电网二次调频策略[J]. 上海交通大学学报, 2024, 58(1): 91-101. DOI: 10.16183/j.cnki.jsjtu.2022.217.
	LIU C B, JIAO W S, WU Q W, et al. Strategy of wind-storage combined system participating in power system secondary frequency regulation based on model predictive control[J]. Journal of Shanghai Jiao Tong University, 2024, 58(1): 91-101. DOI: 10.16183/j.cnki.jsjtu.2022.217.
[9]	蔡振华, 黎灿兵, 阳同光, 等. 考虑动态频率惯量特性的储能电池参与电网一次调频控制[J]. 上海交通大学学报, 2024, 58(12): 1946-1956. DOI: 10.16183/j.cnki.jsjtu.2023.257.
	CAI Z H, LI C B, YANG T G, et al. Participation of energy storage batteries in primary frequency control for power grid considering dynamic frequency inertia characteristics[J]. Journal of Shanghai Jiao Tong University, 2024, 58(12): 1946-1956. DOI: 10.16183/j.cnki.jsjtu.2023.257.
[10]	HONG F, WEI K C, JI W M, et al. A cross-entropy-based synergy method for capacity configuration and SOC management of flywheel energy storage in primary frequency regulation[J]. Energy, 2025, 316: 134498. DOI: 10.1016/j.energy.2025.134498.
[11]	王建波, 孙冉, 刘忠凯, 等. 面向储能辅助火电机组一次调频的深度强化学习控制策略[J]. 西安交通大学学报, 2024, 58(6): 186-192.
	WANG J B, SUN R, LIU Z K, et al. Deep reinforcement learning control strategy for primary frequency regulation of energy storage assisted thermal power units[J]. Journal of Xi'an Jiaotong University, 2024, 58(6): 186-192.
[12]	胡志勇, 黄志博, 王博, 等. 考虑多重死区影响的储能电站虚拟惯量协调优化策略[J/OL]. 储能科学与技术, 2025: 1-13. (2025-02-08). https://link.cnki.net/doi/10.19799/j.cnki.2095-4239.2024.1187.
	HU Z Y, HUANG Z B, WANG B, et al. Coordinated optimization strategy of virtual inertia of energy storage power station considering multiple dead zones[J/OL]. Energy Storage Science and Technology, 2025: 1-13. (2025-02-08). https://link.cnki.net/doi/10.19799/j.cnki.2095-4239.2024.1187.
[13]	ZHOU D, ZOU Z W, DAN Y Q, et al. An integrated strategy for hybrid energy storage systems to stabilize the frequency of the power grid through primary frequency regulation[J]. Energies, 2025, 18(2): 246. DOI: 10.3390/en18020246.
[14]	郑子萱, 张家琛, 陈韵竹, 等. 协同考虑调频指令冲突抑制与优化分配的储能集群分层调频控制策略[J/OL]. 中国电机工程学报, 2025: 1-13. (2025-01-24). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=ZGDC2025012200A&dbname=CJFD&dbcode=CJFQ.
	ZHENG Z X, ZHANG J C, CHEN Y Z, et al. Hierarchical frequency modulation control strategy of energy storage cluster considering conflict suppression and optimal allocation of frequency modulation instructions[J/OL]. Proceedings of the CSEE, 2025: 1-13. (2025-01-24). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=ZGDC 2025012200A&dbname=CJFD&dbcode=CJFQ.
[15]	严晓生, 刘仲稳, 赵建红, 等. 混合储能辅助火电机组一次调频及其容量配置[J]. 太阳能学报, 2024, 45(11): 647-654. DOI: 10.19912/j.0254-0096.tynxb.2023-1151.
	YAN X S, LIU Z W, ZHAO J H, et al. Primary frequency regulation and capacity configuration of hybrid energy storage auxiliary thermal power unit[J]. Acta Energiae Solaris Sinica, 2024, 45(11): 647-654. DOI: 10.19912/j.0254-0096.tynxb.2023-1151.

SOC运行区间	年均SOH衰减率/%	寿命终点时间/年
95%~100%	8.2	2.4
90%~95%	5.6	3.6
85%~90%	3.9	5.1
80%~85%	2.7	7.4

输入：节点特征矩阵 H, 初始全连接拓扑
输出：稀疏邻接矩阵 A_sparse
def GAT_Topology_Optimization(H):
#1.线性特征映射	Z=linear_transform(H, W)	# W为可学习权重
#2.计算注意力系数	A=fully_connected_graph()	# 初始全连接邻接矩阵
	for i in nodes:
	for j in neighbors(i)
	e_ij=leaky_relu(dot(a,concat(Z[i], Z[j])))
	A[i][j] = e_ij
#3.归一化并稀疏化	A_softmax=softmax(A, axis=1)	# 剔除权重<0.2的边
	A_sparse=threshold(A_softmax, thresh=0.2)
	return A_sparse

参数	数值	参数	数值
储能容量	1 MW/0.25 MWh	额定频率	50 Hz
K_G	24	F_HP	0.5
T_G	0.08	T_CH	0.3
T_RH	10	T_b	0.1
M	10	D	1

方法	最大频率波动	稳态频率偏差	回复稳态时间	SOC指标
方法1	0.1464	0.092	20	—
方法2	0.0576	0.053	8	0.412
方法3	0.0637	0.049	10	0.408
方法4	0.0548	0.049	6	0.415

方法	频率偏差	SOC
方法1	0.0554	—
方法2	0.0335	0.1292
方法3	0.0292	0.1946
方法4	0.0321	0.7529