Adaptive control strategy for primary frequency regulation for new energy storage stations based on reinforcement learning

doi:10.19799/j.cnki.2095-4239.2023.0658

Abstract

Abstract:

Based on traditional virtual inertia and virtual sag control methods, and considering the characteristics of new energy output, we propose a new energy storage adaptive control strategy for primary frequency regulation using a reinforcement learning algorithm. Herein, reinforcement learning agent dynamically adjusts the output proportion of the energy storage participating in primary frequency regulation through the virtual inertia control method, responding to real-time fluctuations in system frequency and frequency difference change rate. Subsequently, energy storage primary frequency regulation adaptive controller calculates the output proportion of the virtual sag control method, thereby obtaining the total primary frequency regulation output command for the energy storage. Considering the inherent randomness of active power output in new energy stations, reinforcement learning agent learns from the output disturbances, which are influenced by wind speed disturbances through the wind turbine model in the new energy station. The proposed control strategy maximizes the distinct advantages of virtual inertia and virtual sag control methods before and after frequency regulation. It achieves an optimal combination of the two control methods, unlocking the full potential of energy storage in participating in primary frequency regulation. To validate the effectiveness of the proposed strategy, we constructed a regional power grid frequency response model in Matlab/Simulink. This model simulated extreme working conditions and actual operational scenarios of the new energy station. The simulation results confirm that the strategy can appropriately adjust the proportion of virtual inertial output and virtual sagging output during the frequency modulation process. This adjustment reduces the operation depth of battery energy storage, effectively mitigates frequency fluctuation caused by variations in new energy output to the power grid, and enhances overall frequency quality.

Key words: battery energy storage, primary frequency regulation, new energy station, reinforcement learning, adaptive control

CLC Number:

TK 02

Ran SUN, Jianbo WANG, Yanzhao MA, Xiaoke ZHANG, Huaizhong HU. Adaptive control strategy for primary frequency regulation for new energy storage stations based on reinforcement learning[J]. Energy Storage Science and Technology, 2024, 13(3): 858-869.

Figures/Tables 22

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Table 1

Table 2

Regional power grid frequency response model simulation parameters"

参数名称/单位	取值	参数名称/单位	取值
H/s	5	$K g$	20
D/p.u.	1	$M b$	20
$T g$ /s	0.08	$K b$	15
$T C H$ /s	0.3	$T b e s s$ /s	0.1
$T R H$ /s	10

Table 2

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Table 3

Frequency regulation index under sudden change of new energy power output"

扰动大小	方法名称	$∆ f m a x$ /Hz	$∆ o m a x$ /(Hz/s)	$∆ f s$ /Hz
0.02 p.u.	本工作方法	-0.076	0.0995	-0.006
	对比方法	-0.083	0.0995	-0.006
	直接切换法	-0.120	0.1017	-0.006

Table 3

Fig. 10

Fig. 11

Fig. 12

Table 4

Frequency regulation index of condition one"

方法	本工作方法	对比方法	直接切换法
$f R M S E$	3.3342	3.7782	4.4078
$t d$ /%	52.13	57.27	58.90

Table 4

Fig. 13

Fig. 14

Fig. 15

Fig. 16

Fig. 17

Table 5

Frequency regulation index of condition two"

项目	本工作方法	对比方法	直接切换法
$f R M S E$	3.6931	4.1560	4.4822
$t d$ /%	77.40	78.95	79.75

Table 5

References 20

1	舒印彪, 赵勇, 赵良, 等. "双碳"目标下我国能源电力低碳转型路径[J]. 中国电机工程学报, 2023, 43(5): 1663-1672.
	SHU Y B, ZHAO Y, ZHAO L, et al. Study on low carbon energy transition path toward carbon peak and carbon neutrality[J]. Proceedings of the CSEE, 2023, 43(5): 1663-1672.
2	韩泽雷, 鞠平, 秦川, 等. 面向新型电力系统的频率安全研究综述与展望[J]. 电力自动化设备, 2023, 43(9): 112-124.
	HAN Z L, JU P, QIN C, et al. Review and prospect of research on frequency security of new power system[J]. Electric Power Automation Equipment, 2023, 43(9): 112-124.
3	叶林, 王凯丰, 赖业宁, 等. 低惯量下电力系统频率特性分析及电池储能调频控制策略综述[J]. 电网技术, 2023, 47(2): 446-464.
	YE L, WANG K F, LAI Y N, et al. Review of frequency characteristics analysis and battery energy storage frequency regulation control strategies in power system under low inertia level[J]. Power System Technology, 2023, 47(2): 446-464.
4	黄际元, 李欣然, 曹一家, 等. 考虑储能参与快速调频动作时机与深度的容量配置方法[J]. 电工技术学报, 2015, 30(12): 454-464.
	HUANG J Y, LI X R, CAO Y J, et al. Capacity allocation of energy storage system considering its action moment and output depth in rapid frequency regulation[J]. Transactions of China Electrotechnical Society, 2015, 30(12): 454-464.
5	李军徽, 侯涛, 穆钢, 等. 基于权重因子和荷电状态恢复的储能系统参与一次调频策略[J]. 电力系统自动化, 2020, 44(19): 63-72.
	LI J H, HOU T, MU G, et al. Primary frequency regulation strategy with energy storage system based on weight factors and state of charge recovery[J]. Automation of Electric Power Systems, 2020, 44(19): 63-72.
6	李欣然, 崔曦文, 黄际元, 等. 电池储能电源参与电网一次调频的自适应控制策略[J]. 电工技术学报, 2019, 34(18): 3897-3908.
	LI X R, CUI X W, HUANG J Y, et al. The self-adaption control strategy of energy storage batteries participating in the primary frequency regulation[J]. Transactions of China Electrotechnical Society, 2019, 34(18): 3897-3908.
7	王育飞, 杨铭诚, 薛花, 等. 计及SOC的电池储能系统一次调频自适应综合控制策略[J]. 电力自动化设备, 2021, 41(10): 192-198, 219.
	WANG Y F, YANG M C, XUE H, et al. Self-adaptive integrated control strategy of battery energy storage system considering SOC for primary frequency regulation[J]. Electric Power Automation Equipment, 2021, 41(10): 192-198, 219.
8	吴启帆, 宋新立, 张静冉, 等. 电池储能参与电网一次调频的自适应综合控制策略研究[J]. 电网技术, 2020, 44(10): 3829-3836.
	WU Q F, SONG X L, ZHANG J R, et al. Study on self-adaptation comprehensive strategy of battery energy storage in primary frequency regulation of power grid[J]. Power System Technology, 2020, 44(10): 3829-3836.
9	王佳宁, 杨仁海, 姚张浩, 等. 基于深度强化学习的有源中点钳位逆变器效率优化设计[J]. 电子与信息学报, 2023, 45(9): 3311-3320.
	WANG J N, YANG R H, YAO Z H, et al. Efficiency optimized design of active neutral point clamped inverter based on deep reinforcement learning[J]. Journal of Electronics & Information Technology, 2023, 45(9): 3311-3320.
10	WANG Y Y, LIU S Q, CHU M Z, et al. Primary frequency control of renewable energy based on deep reinforcement learning[C]//2021 China Automation Congress (CAC). Beijing, China. IEEE, 2021: 5183-5188.
11	SHI Z Y, XIE X Z, LU H B, et al. Deep reinforcement learning-based multidimensional resource management for energy harvesting cognitive NOMA communications[J]. IEEE Transactions on Communications, 2022, 70(5): 3110-3125.
12	谭庄熙. 电池储能参与电网调频的综合协同控制方法研究[D]. 长沙: 湖南大学, 2020.
	TAN Z X. Research on integrated collaborative control strategy of battery energy storage system participating in frequency regulation[D].Changsha: Hunan University, 2020.
13	KUNDUR P. 电力系统稳定与控制[M]. 北京: 中国电力出版社, 2002.
	KUNDUR P. Power system stability and control[M]. Beijing: China Electric Power Press, 2002.
14	LIU H, HU Z C, SONG Y H, et al. Decentralized vehicle-to-grid control for primary frequency regulation considering charging demands[J]. IEEE Transactions on Power Systems, 2013, 28(3): 3480-3489.
15	TANG Y, DAI J F, NING J, et al. An extended system frequency response model considering wind power participation in frequency regulation[J]. Energies, 2017, 10(11): 1797.
16	孙雪薇. 双馈风电机组惯量和一次频率调节与场站储能协调控制方法研究[D]. 保定: 华北电力大学, 2021.
	SUN X W. Research on coordinated control method of inertia and primary frequency regulation of doubly-fed wind turbine and energy storage in station[D]. Baoding: North China Electric Power University, 2021.
17	马智慧, 李欣然, 谭庄熙, 等. 考虑储能调频死区的一次调频控制方法[J]. 电工技术学报, 2019, 34(10): 2102-2115.
	MA Z H, LI X R, TAN Z X, et al. Integrated control of primary frequency regulation considering dead band of energy storage[J]. Transactions of China Electrotechnical Society, 2019, 34(10): 2102-2115.
18	徐飘. 电池储能参与电网调频综合自适应控制策略研究[D]. 长沙: 湖南大学, 2019.
	XU P. Research on battery energy storage participating in integrated adaptive control strategy of frequency regulation in power system[D]. Changsha: Hunan University, 2019.
19	黄武枫. 风电场风速概率分布及其拟合模型研究[D]. 南宁: 广西大学, 2021.
	HUANG W F. Research on probability distribution of wind farm and its fitting models[D]. Nanning: Guangxi University, 2021.
20	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[J]. ArXiv e-Prints, 2015: doi:10.48550/arXiv. 1509.02971.

参数名称/单位	取值
控制间隔/s	0.1
训练步数T	25000
最大训练回合数N	250
经验回放池容量	10⁶
批量采样数量	128
目标网络软更新参数	0.001

[1]	Bin LI, Jilei YE, Yu ZHANG, Shanshan SHI, Haojing WANG, Lili LIU, Mingzhe LI. Microgrid-coordinated control strategy with distributed new energy and electro-mechanical hybrid energy storage [J]. Energy Storage Science and Technology, 2023, 12(5): 1510-1515.
[2]	Zihao LIU, Xuesong ZHANG, Da LIN, Liqing SUN, Zhengyang LI, Rui XIONG. Joint energy and power state estimation method for energy storage battery based on extended Kalman filter [J]. Energy Storage Science and Technology, 2023, 12(3): 913-922.
[3]	Song CI, Congjia ZHANG, Baochang LIU, Yanglin ZHOU. Dynamic reconfigurable battery energy storage technology： Principle and application [J]. Energy Storage Science and Technology, 2023, 12(11): 3445-3455.
[4]	Ziya XING, Wei LIU, Juncheng GENG, Xinghua ZHOU, Yue XIA. Line-loss minimization strategy of a distribution network with battery energy storage systems based on heuristic algorithm [J]. Energy Storage Science and Technology, 2023, 12(11): 3406-3413.
[5]	Yu CAO, Tong JIANG, Chi LIU, Yong YANG, Wenfei LIU, Wenying LIU. Electrochemical energy storage participation in primary frequency regulation control strategy considering frequency characteristics and energy storage battery state [J]. Energy Storage Science and Technology, 2023, 12(10): 3120-3130.
[6]	Mingfei LI, Mumin RAO, Wanmei SUN, Shuxin CUI, Wei CHEN. Analysis method based on porous medium modeling for thermal management system of large capacity battery energy storage [J]. Energy Storage Science and Technology, 2022, 11(8): 2526-2536.
[7]	Yu SHI, Zhong ZHANG, Jingying YANG, Wei QIAN, Hao LI, Xiang ZHAO, Xintong YANG. Opportunity cost modelling and market strategy of energy storage participating in the AGC market [J]. Energy Storage Science and Technology, 2022, 11(7): 2366-2373.
[8]	Xinlei CAI, Kai DONG, Zijie MENG, Zhenfan YU, Boxiao WANG, Yang YU. AGC command tracking control strategy for battery energy storage power station based on optimized dynamic grouping technology [J]. Energy Storage Science and Technology, 2022, 11(5): 1475-1481.
[9]	Hao LI, Chang LIU, Bo MIAO, Jing ZHANG. Coordinative optimal dispatch of multi-park integrated energy system considering complementary cooling， heating and power and energy storage systems [J]. Energy Storage Science and Technology, 2022, 11(5): 1482-1491.
[10]	Feng TIAN, Zhijiang CHENG, Handi YANG, Tianxiang YANG. Fault-tolerant control strategy for modular multi-level hybrid converter battery energy storage system [J]. Energy Storage Science and Technology, 2022, 11(5): 1583-1591.
[11]	Fulin FAN, Junhui LI, CAMPOS-GAONA David, Gangui YAN. Energy storage-friendly frequency response service markets： The UK perspective [J]. Energy Storage Science and Technology, 2022, 11(4): 1278-1288.
[12]	Zhen YAO, Rui WANG, Xue YANG, Qi ZHANG, Qinghua LIU, Baoguo WANG, Ping MIAO. Current situations and prospects of zinc-iron flow battery [J]. Energy Storage Science and Technology, 2022, 11(1): 78-88.
[13]	Huan ZHU, Guojing LIU, Xing ZHANG, Fen YUE, Zhenhua YU. Policy and economic comparison of natural gas power generation and battery energy storage in peak regulation [J]. Energy Storage Science and Technology, 2021, 10(6): 2392-2402.
[14]	Tianao ZHANG, Hao LIU, Yongchong CHEN, Qingsong WANG, Shuxing ZHANG, Qiquan ZENG. Intrinsic safety of energy storage in a high-capacity battery [J]. Energy Storage Science and Technology, 2021, 10(6): 2293-2302.
[15]	Zeng'ang JIA, Zhibin LING, Xuguang LI. Thermal characteristics of lithium-ion battery with sinusoidal charge and discharge pulsating current [J]. Energy Storage Science and Technology, 2021, 10(6): 2260-2268.