基于强化学习-模型预测控制（RL-MPC）的分布式储能协同一次调频控制方法

doi:10.19799/j.cnki.2095-4239.2025.0296

储能科学与技术 ›› 2025, Vol. 14 ›› Issue (8): 3138-3148.doi: 10.19799/j.cnki.2095-4239.2025.0296

• 储能系统与工程 • 上一篇

基于强化学习-模型预测控制（RL-MPC）的分布式储能协同一次调频控制方法

马骞¹(), 肖亮¹, 程冰², 高琴¹, 刘春晓¹, 朱益华³, 李成翔³

^1.中国南方电网电力调度控制中心，广东广州 510530
^2.海南电网有限责任公司，海南海口 570203
^3.直流输电技术全国重点实验室(南方电网科学研究院有限责任公司)，广东广州 510663

收稿日期:2025-03-27 修回日期:2025-04-30 出版日期:2025-08-28 发布日期:2025-08-18
通讯作者: 马骞 E-mail:maqian@csg.cn
作者简介:马骞（1978—），男，博士，教授级高级工程师，研究方向为电网运行策划、新能源并网运行、系统运行风险防控、规划运行分析，E-mail：maqian@csg.cn。
基金资助:
南方电网有限责任公司科技项目(ZDKJXM20222007)

Cooperative primary frequency modulation control method for distributed energy storage based on reinforcement learning-model predictive control

Qian MA¹(), Liang XIAO¹, Bing CHENG², Qin GAO¹, Chunxiao LIU¹, Yihua ZHU³, Chengxiang LI³

^1.China Southern Power Grid Power Dispatching and Control Center, Guangzhou 510530, Guangdong, China
^2.Hainan Power Grid Company Limited, Haikou 570203, Hainan, China
^3.State Key Laboratory of HVDC (Electric Power Research Institute of China Southern Power Grid Company Limited), Guangzhou 510663, Guangdong, China

Received:2025-03-27 Revised:2025-04-30 Online:2025-08-28 Published:2025-08-18
Contact: Qian MA E-mail:maqian@csg.cn

摘要/Abstract

摘要：

为改善配电网频率特性，充分发挥分布式储能系统的快速响应优势，提出了一种基于强化学习-模型预测控制(reinforcement learning-model predictive control, RL-MPC)的分布式储能协同一次调频控制方法。首先根据分布式储能的频率响应特性、荷电状态(SOC)、功率控制策略，建立了含分布式储能并网的一次调频控制模型；然后通过构建分层混合控制架构，上层采用深度强化学习(deep reinforcement learning, DRL)动态优化MPC权重矩阵，实时感知频率偏差、变化率及储能荷电状态分布熵值，下层采用分布式MPC滚动求解多节点储能出力序列，并引入图注意力网络(graph attention network, GAT)实现通信拓扑自适应优化，降低分布式储能协同控制的计算复杂度，提升策略泛化能力；最后通过Matlab/Simulink仿真验证了所提方法能够有效提升分布式储能的一次调频响应速度和控制精度，增强电力系统的稳定性。

关键词: 分布式储能, 调频, 模型预测控制, 强化学习, 图注意力网络

Abstract:

To enhance the frequency characteristics of power grids and fully leverage the rapid response advantages of distributed energy storage systems (DESSs), a cooperative primary frequency control method based on reinforcement learning-model predictive control (RL-MPC) is proposed. First, a primary frequency control model incorporating DESSs is established based on frequency response characteristics, state of charge (SOC), and power control strategies. Then, a hierarchical mixed control architecture is designed: the upper layer employs a deep Q-network (DQN) to dynamically optimize the MPC weight matrix while sensing frequency deviation, rate of change, and SOC distribution entropy in real time. The lower layer utilizes distributed MPC to determine the output sequences of multi-node energy storage units and introduces a graph attention network (GAT) to achieve adaptive optimization of the communication topology. This approach reduces computational complexity in coordinated control and enhances the strategy's generalization capability. Finally, simulations conducted in Matlab/Simulink verify that the proposed method effectively improves the primary frequency response speed and control accuracy of DESSs, thereby strengthening overall power system stability.

Key words: distributed energy storage, frequency modulation, model predictive control, reinforcement learning, graph attention network

中图分类号:

TK 02

马骞, 肖亮, 程冰, 高琴, 刘春晓, 朱益华, 李成翔. 基于强化学习-模型预测控制（RL-MPC）的分布式储能协同一次调频控制方法[J]. 储能科学与技术, 2025, 14(8): 3138-3148.

Qian MA, Liang XIAO, Bing CHENG, Qin GAO, Chunxiao LIU, Yihua ZHU, Chengxiang LI. Cooperative primary frequency modulation control method for distributed energy storage based on reinforcement learning-model predictive control[J]. Energy Storage Science and Technology, 2025, 14(8): 3138-3148.

图/表 21

图1

表1

图2

图3

表2

图4

表3

图5

图6

表4

图7

图8

图9

表5

图10

图11

表6

表7

图12

图13

图14

参考文献 15

[1]	桂玥, 赵熙临. 综合储能协同控制方法在一次调频中的应用[J]. 湖北工业大学学报, 2025, 40(1): 25-31. DOI: 10.3969/j.issn.1003-4684.2025.01.005.
	GUI Y, ZHAO X L. Application of a comprehensive energy storage cooperative control method in primary frequency modulation[J]. Journal of Hubei University of Technology, 2025, 40(1): 25-31. DOI: 10.3969/j.issn.1003-4684.2025.01.005.
[2]	黎萌, 林章岁, 林毅, 等. 基于改进模型预测控制的分布式储能辅助调频控制方法[J]. 水利水电技术(中英文), 2023, 54(S2): 447-456. DOI: 10.13928/j.cnki.wrahe.2023.S2.071.
	LI M, LIN Z S, LIN Y, et al. Distributed energy storage assisted frequency regulation control method based onimproved model predictive control[J]. Water Resources and Hydropower Engineering, 2023, 54(S2): 447-456. DOI: 10.13928/j.cnki.wrahe. 2023.S2.071.
[3]	池志坤, 袁至, 李骥. 基于系统频率与SOC状态预测的储能一次调频控制策略[J/OL]. 高电压技术, 2025: 1-15. (2025-01-23). https://link.cnki.net/doi/10.13336/j.1003-6520.hve.20241421.
	CHI Z K, YUAN Z, LI J. Primary frequency regulation control strategy of energy storage based on prediction of system frequency and state of charge[J/OL]. High Voltage Engineering, 2025: 1-15. (2025-01-23). https://link.cnki.net/doi/10.13336/j. 1003-6520.hve.20241421.
[4]	贺悝, 郭罗权, 谭庄熙, 等. 考虑调频阶段需求的混合储能一次调频综合控制策略[J]. 太阳能学报, 2024, 45(9): 697-708. DOI: 10. 19912/j.0254-0096.tynxb.2023-1580.
	HE L, GUO L Q, TAN Z X, et al. Comprehensive control strategy of hybrid energy storage in primary frequency regulation considering demands of frequency regulation stages[J]. Acta Energiae Solaris Sinica, 2024, 45(9): 697-708. DOI: 10.19912/j.0254-0096.tynxb.2023-1580.
[5]	孙冉, 王建波, 马彦钊, 等. 基于强化学习的新能源场站储能一次调频自适应控制策略[J]. 储能科学与技术, 2024, 13(3): 858-869. DOI: 10.19799/j.cnki.2095-4239.2023.0658.
	SUN R, WANG J B, MA Y Z, et al. Adaptive control strategy for primary frequency regulation for new energy storage stations based on reinforcement learning[J]. Energy Storage Science and Technology, 2024, 13(3): 858-869. DOI: 10.19799/j.cnki.2095-4239.2023.0658.
[6]	刘小龙, 庞敬磊, 吕智嘉, 等. 考虑荷电状态的电池储能一次调频综合控制方法[J]. 中国测试, 2024, 50(10): 59-65.
	LIU X L, PANG J L, LÜ Z J, et al. A comprehensive control method for primary frequency modulation of battery energy storage based on state of charge[J]. China Measurement & Test, 2024, 50(10): 59-65.
[7]	肖家杰, 李培强, 毛志宇, 等. 基于双层协调控制的电池储能参与电网二次调频策略[J]. 电力自动化设备, 2024, 44(8): 9-17. DOI: 10. 16081/j.epae.202401004.
	XIAO J J, LI P Q, MAO Z Y, et al. Strategy for battery energy storage participating in secondary frequency regulation of power grid based on two-layer coordinated control[J]. Electric Power Automation Equipment, 2024, 44(8): 9-17. DOI: 10.16081/j.epae. 202401004.
[8]	刘传斌, 矫文书, 吴秋伟, 等. 基于模型预测控制的风储联合电场参与电网二次调频策略[J]. 上海交通大学学报, 2024, 58(1): 91-101. DOI: 10.16183/j.cnki.jsjtu.2022.217.
	LIU C B, JIAO W S, WU Q W, et al. Strategy of wind-storage combined system participating in power system secondary frequency regulation based on model predictive control[J]. Journal of Shanghai Jiao Tong University, 2024, 58(1): 91-101. DOI: 10.16183/j.cnki.jsjtu.2022.217.
[9]	蔡振华, 黎灿兵, 阳同光, 等. 考虑动态频率惯量特性的储能电池参与电网一次调频控制[J]. 上海交通大学学报, 2024, 58(12): 1946-1956. DOI: 10.16183/j.cnki.jsjtu.2023.257.
	CAI Z H, LI C B, YANG T G, et al. Participation of energy storage batteries in primary frequency control for power grid considering dynamic frequency inertia characteristics[J]. Journal of Shanghai Jiao Tong University, 2024, 58(12): 1946-1956. DOI: 10.16183/j.cnki.jsjtu.2023.257.
[10]	HONG F, WEI K C, JI W M, et al. A cross-entropy-based synergy method for capacity configuration and SOC management of flywheel energy storage in primary frequency regulation[J]. Energy, 2025, 316: 134498. DOI: 10.1016/j.energy.2025.134498.
[11]	王建波, 孙冉, 刘忠凯, 等. 面向储能辅助火电机组一次调频的深度强化学习控制策略[J]. 西安交通大学学报, 2024, 58(6): 186-192.
	WANG J B, SUN R, LIU Z K, et al. Deep reinforcement learning control strategy for primary frequency regulation of energy storage assisted thermal power units[J]. Journal of Xi'an Jiaotong University, 2024, 58(6): 186-192.
[12]	胡志勇, 黄志博, 王博, 等. 考虑多重死区影响的储能电站虚拟惯量协调优化策略[J/OL]. 储能科学与技术, 2025: 1-13. (2025-02-08). https://link.cnki.net/doi/10.19799/j.cnki.2095-4239.2024.1187.
	HU Z Y, HUANG Z B, WANG B, et al. Coordinated optimization strategy of virtual inertia of energy storage power station considering multiple dead zones[J/OL]. Energy Storage Science and Technology, 2025: 1-13. (2025-02-08). https://link.cnki.net/doi/10.19799/j.cnki.2095-4239.2024.1187.
[13]	ZHOU D, ZOU Z W, DAN Y Q, et al. An integrated strategy for hybrid energy storage systems to stabilize the frequency of the power grid through primary frequency regulation[J]. Energies, 2025, 18(2): 246. DOI: 10.3390/en18020246.
[14]	郑子萱, 张家琛, 陈韵竹, 等. 协同考虑调频指令冲突抑制与优化分配的储能集群分层调频控制策略[J/OL]. 中国电机工程学报, 2025: 1-13. (2025-01-24). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=ZGDC2025012200A&dbname=CJFD&dbcode=CJFQ.
	ZHENG Z X, ZHANG J C, CHEN Y Z, et al. Hierarchical frequency modulation control strategy of energy storage cluster considering conflict suppression and optimal allocation of frequency modulation instructions[J/OL]. Proceedings of the CSEE, 2025: 1-13. (2025-01-24). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=ZGDC 2025012200A&dbname=CJFD&dbcode=CJFQ.
[15]	严晓生, 刘仲稳, 赵建红, 等. 混合储能辅助火电机组一次调频及其容量配置[J]. 太阳能学报, 2024, 45(11): 647-654. DOI: 10.19912/j.0254-0096.tynxb.2023-1151.
	YAN X S, LIU Z W, ZHAO J H, et al. Primary frequency regulation and capacity configuration of hybrid energy storage auxiliary thermal power unit[J]. Acta Energiae Solaris Sinica, 2024, 45(11): 647-654. DOI: 10.19912/j.0254-0096.tynxb.2023-1151.

SOC运行区间	年均SOH衰减率/%	寿命终点时间/年
95%~100%	8.2	2.4
90%~95%	5.6	3.6
85%~90%	3.9	5.1
80%~85%	2.7	7.4

输入：节点特征矩阵 H, 初始全连接拓扑
输出：稀疏邻接矩阵 A_sparse
def GAT_Topology_Optimization(H):
#1.线性特征映射	Z=linear_transform(H, W)	# W为可学习权重
#2.计算注意力系数	A=fully_connected_graph()	# 初始全连接邻接矩阵
	for i in nodes:
	for j in neighbors(i)
	e_ij=leaky_relu(dot(a,concat(Z[i], Z[j])))
	A[i][j] = e_ij
#3.归一化并稀疏化	A_softmax=softmax(A, axis=1)	# 剔除权重<0.2的边
	A_sparse=threshold(A_softmax, thresh=0.2)
	return A_sparse

参数	数值	参数	数值
储能容量	1 MW/0.25 MWh	额定频率	50 Hz
K_G	24	F_HP	0.5
T_G	0.08	T_CH	0.3
T_RH	10	T_b	0.1
M	10	D	1

方法	最大频率波动	稳态频率偏差	回复稳态时间	SOC指标
方法1	0.1464	0.092	20	—
方法2	0.0576	0.053	8	0.412
方法3	0.0637	0.049	10	0.408
方法4	0.0548	0.049	6	0.415

方法	频率偏差	SOC
方法1	0.0554	—
方法2	0.0335	0.1292
方法3	0.0292	0.1946
方法4	0.0321	0.7529

基于强化学习-模型预测控制（RL-MPC）的分布式储能协同一次调频控制方法

Cooperative primary frequency modulation control method for distributed energy storage based on reinforcement learning-model predictive control

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 21

参考文献 15

相关文章 15

编辑推荐

Metrics

本文评价

方法	权重组合(α、β、γ)	最大频率波动/Hz	调节时间/s
方法1	1∶0.5∶0.5	0.189	5.33
方法2	1∶1∶0.5	0.149	4.89
方法3	1∶1∶1	0.127	3.61

[1]	孙召琴, 黎可, 杜杲娴, 胡晨, 牛萌, 朱真. 多端口能量路由器快速频率响应及稳定控制方法[J]. 储能科学与技术, 2025, 14(8): 2970-2982.
[2]	徐彩莹, 唐毓振, 李秋雨, 杨浩岳, 陈洋, 杨恒昭. 用于电力系统调频的超级电容储能系统[J]. 储能科学与技术, 2025, 14(8): 3078-3089.
[3]	魏路, 冷至益, 叶佳, 徐玉杰, 陈海生. 人工智能在飞轮储能中的应用[J]. 储能科学与技术, 2025, 14(8): 3019-3027.
[4]	王子滔, 李浩然. 考虑分布式储能功率支撑的配电网供电恢复频率控制方法[J]. 储能科学与技术, 2025, 14(7): 2738-2751.
[5]	陈勋. 基于深度强化学习的储能系统能量管理与优化调度策略[J]. 储能科学与技术, 2025, 14(6): 2439-2441.
[6]	魏江哲, 舒茂龙, 陈彦桥, 赵璐璐, 苏新凯, 刘牧阳. 考虑边际替代效应的火-储联合系统调频备用配置方法[J]. 储能科学与技术, 2025, 14(5): 2117-2129.
[7]	萨仁高娃, 邬超慧, 倪泽龙, 张悦, 姜新建, 田建宇. 基于强化学习的变参数PID的惯量飞轮有功控制策略[J]. 储能科学与技术, 2025, 14(5): 1982-1990.
[8]	王滔, 毛田, 周保荣, 赵文猛, 华浩. 基于储能荷电状态的虚拟同步机控制探讨[J]. 储能科学与技术, 2025, 14(5): 2032-2034.
[9]	刘晓悦, 陈炎, 孙小菲. 飞轮阵列参与电网一次调频双层控制策略[J]. 储能科学与技术, 2025, 14(4): 1536-1547.
[10]	董文琦, 张东晖, 曹一凡, 宁照轩, 姜新建, 李明, 史学伟. 新型惯量飞轮与高速飞轮参与电网惯性响应与一次调频的控制策略[J]. 储能科学与技术, 2025, 14(3): 1224-1233.
[11]	党少佳, 孙利强, 王深友, 田文涛, 胡鹏飞. 计及系统经济性与荷电状态均衡的多储能电站调频双层功率优化策略[J]. 储能科学与技术, 2025, 14(3): 1247-1257.
[12]	孙利强, 党少佳, 刘岗, 王深友, 胡鹏飞. 考虑充放电转换状态与功率约束的储能电站调频功率优化方法[J]. 储能科学与技术, 2025, 14(3): 1286-1298.
[13]	王雷, 闫瑞涛, 张凡, 闫娜, 岳芬, 傅旭, 刘梦晨, 杨韵彰. 基于现货电能量市场与一次调频市场联合优化模型的独立储能项目经济性分析[J]. 储能科学与技术, 2025, 14(2): 834-845.
[14]	庞娟, 孙金岭. 能源互联基础上分布式储能系统的应用及经济效益探讨[J]. 储能科学与技术, 2025, 14(2): 868-870.
[15]	田野, 王珊珊, 姚旭, 刘佳鑫, 韩晓冬. 配电网通信与分布式储能技术协同应用的意义与发展[J]. 储能科学与技术, 2025, 14(1): 190-192.

场景	最大频率波动/Hz	调节时间/s
场景A	0.296	7.5
场景B	0.251	4.89
场景C	0.196	3.75