Differential Variable Speed Limit Control Strategy Based on Reinforcement Learning

BAI Ruyu; JIAO Pengpeng; CHEN Yue; ZHANG Yao

doi:10.3963/j.jssn.1674-4861.2024.01.012

Volume 42 Issue 1

Feb. 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Transport Information and Safety > 2024 > 42(1): 105-114

BAI Ruyu, JIAO Pengpeng, CHEN Yue, ZHANG Yao. Differential Variable Speed Limit Control Strategy Based on Reinforcement Learning[J]. Journal of Transport Information and Safety, 2024, 42(1): 105-114. doi: 10.3963/j.jssn.1674-4861.2024.01.012

Citation:

BAI Ruyu, JIAO Pengpeng, CHEN Yue, ZHANG Yao. Differential Variable Speed Limit Control Strategy Based on Reinforcement Learning[J]. Journal of Transport Information and Safety, 2024, 42(1): 105-114. doi: 10.3963/j.jssn.1674-4861.2024.01.012

Citation:

PDF( 3713 KB)

Differential Variable Speed Limit Control Strategy Based on Reinforcement Learning

doi: 10.3963/j.jssn.1674-4861.2024.01.012

Beijing Key Laboratory of General Aviation Technology, Beijing University of Civil Engineering and Architecture, Beijing 100044, China

Received Date: 2023-08-01
Available Online: 2024-05-31

Abstract

Abstract

In addressing the challenges posed by variable traffic conditions within highway merging lanes impacted by merging vehicles, a reinforcement learning (RL) model is developed for differential variable speed limit (DVSL) control. Due to the difficulty of solving the DVSL control problem with high-dimensional action space, this paper optimizes the action space by using the speed limit change value, determines the state space as well as the reward function considering multiple factors; in the solution process, it is improved by using the Prioritized Experience Replay (PER) technique in order to improve the training efficiency and model performance; and at the same time, it proposes an inter-lane safety detection mechanism to assist the PER-DDQN to unfold the training and ensure the implementability of the lane-level variable velocity limit model. Furthermore, the merging area is simulated with SUMO to examine the performance of the DVSL controller. The results reveal that, compared with the no-control scenario, the proposed method yields a 41.88% reduction in overall travel time and a 5.65% increase in average speed. In the merging zone, a notable 66.91% reduction in travel time and a 43.42% increase in average speed are achieved. And the RL based DVSL control strategy effectively minimizes congestion time for each lane due to smoother speed changes. Furthermore, when evaluating the impact of varying penetration scenarios on the proposed method, the RL based DVSL control strategy outperforms the no-control scenario particularly when the penetration of connected-automated vehicles (CAVs) is below 60%. In scenarios with 20%, 40%, and 60% penetration rates, the average travel time is reduced by 41.88%, 13.38%, and 7.46%, with corresponding average speed improvements of 6.08%, 2.36%, and 1.61%, respectively. However, at penetration rate of 80% or higher, there is no significant improvement in the DVSL control strategy due to the improvement of CAVs to the traffic flow.
- intelligent traffic,
- differential variable speed limit control,
- control strategy,
- reinforcement learning,
- high-way merging area,
- mixed traffic flow

FullText(HTML)

References(30)

References

[1]	MENG X H, ZHANG Z Z, SHI Y Y. Research on traffic safety on freeway merging sections based on TTC and PET[C]. 2014 Trans Tech Publications, Switzerland: Applied Mechanics and Materials, 2014.
[2]	ALONSO B, PÓRTILLA Á I, MUSOLINO G, et al. Network Fundamental diagram(NFD)and traffic signal control: first empirical evidences from the city of Santander[J]. Transportation Research Procedia, 2017, 27: 27-34. doi: 10.1016/j.trpro.2017.12.112
[3]	李海舰, 刘中华, 陈开群, 等. 高速公路风险防控设施应用研究综述与展望[J]. 交通运输工程与信息学报, 2024, 22(1): 54-78. https://www.cnki.com.cn/Article/CJFDTOTAL-JTGC202401004.htm LI H J, LIU Z H, CHEN K Q, et al. An overview and prospect of the application of risk prevention and control facilities on high-ways[J]. Journal of Transportation Engineering and Information, 2024, 22(1): 54-78. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JTGC202401004.htm
[4]	SADAT M, CELIKOGLU H B. Simulation-based variable speed limit systems modelling: an overview and a case study on Istanbul freeways[J]. Transportation Research Procedia, 2017, 22: 607-614. doi: 10.1016/j.trpro.2017.03.051
[5]	HADIUZZAMAN M, QIU T Z. Cell transmission model based variable speed limit control for freeways[J]. Canadian Journal of Civil Engineering, 2013, 40(1): 46-56. doi: 10.1139/cjce-2012-0101
[6]	KHONDAKER B, KATTAN L. Variable speed limit: A micro-scopic analysis in a connected vehicle environment[J]. Transportation Research Part C: Emerging Technologies, 2015, 58: 146-159. doi: 10.1016/j.trc.2015.07.014
[7]	PIAO J, MCDONALD M. Safety impacts of variable speed limits-a simulation study[C]. 2008 11th International IEEE Conference on Intelligent Transportation Systems, Beijing, China: IEEE, 2008.
[8]	SORIGUERA F, TORNÉ J M, ROSAS D. Assessment of dynamic speed limit management on metropolitan freeways[J]. Journal of Intelligent Transportation Systems, 2013, 17(1): 78-90. doi: 10.1080/15472450.2012.719455
[9]	KIANFAR J, EDARA P, SUN C. Operational analysis of a free-way variable speed limit system in St. Louis, Missouri[J]. Journal of Intelligent Transportation Systems, 2015, 19(4): 385-398. doi: 10.1080/15472450.2014.989718
[10]	HEGYI A, DE SCHUTTER B, HELLENDOORN J. Optimal coordination of variable speed limits to suppress shock waves[J]. IEEE Transactions on Intelligent Transportation Systems, 2005, 6(1): 102-112. doi: 10.1109/TITS.2004.842408
[11]	HAN Y, HEGYI A, ZHANG L, et al. A new reinforcement learning-based variable speed limit control approach to improvetraffic efficiency against freeway jam waves[J]. Transportation Research Part C: Emerging Technologies, 2022, 144: 103900. doi: 10.1016/j.trc.2022.103900
[12]	韩雨, 郭延永, 张乐, 等. 消除高速公路运动波的可变限速控制方法[J]. 中国公路学报, 2022, 35(1): 151-158. doi: 10.3969/j.issn.1001-7372.2022.01.013 HAN Y, GUO Y Y, ZHANG L, et al. An optimal variable speed limit control approach against freeway jam waves[J]. China Journal of Highway and Transport, 2022, 35(1): 151-158. (in Chinese) doi: 10.3969/j.issn.1001-7372.2022.01.013
[13]	段荟, 刘攀, 李志斌, 等. 基于强化学习的汇流瓶颈区可变限速策略研究[J]. 交通运输系统工程与信息, 2015, 15(1): 55-61. https://www.cnki.com.cn/Article/CJFDTOTAL-YSXT201501011.htm DUAN H, LIU P, LI Z B, et al. Variable speed limit control at freeway merge bottlenecks based on reinforcement learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2015, 15(1): 55-61. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-YSXT201501011.htm
[14]	GREGURIĆ M, KUŠIĆ K, VRBANIĆ F, et al. Variable speed limit control based on deep reinforcement learning: A possible implementation[C]. 62nd International Symposium ELMAR-2020, Zadar, Croatia: IEEE, 2020.
[15]	韩磊, 张轮, 郭为安. 混合交通流环境下基于改进强化学习的可变限速控制策略[J]. 交通运输系统工程与信息, 2023, 23(3): 110-122. https://www.cnki.com.cn/Article/CJFDTOTAL-YSXT202303013.htm HAN L, ZHANG L, GUO W A. Variable speed limit control based on improved dueling double deep Q Network under mixed traffic environment[J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23(3): 110-122. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-YSXT202303013.htm
[16]	GREGURIĆ M, KUŠIĆ K, IVANJKO E. Impact of deep reinforcement learning on variable speed limit strategies in connected vehicles environments[J]. Engineering Applications of Artificial Intelligence, 2022, 112: 104850. doi: 10.1016/j.engappai.2022.104850
[17]	WU Y K, TAN H C, QIN L Q, et al. Differential variable speed limits control for freeway recurrent bottlenecks via deep actor-critic algorithm[J]. Transportation Research Part C: Emerging Technologies, 2020, 117: 102649. doi: 10.1016/j.trc.2020.102649
[18]	HAN L, ZHANG L, GUO W. Optimal differential variable speed limit control in a connected and autonomous vehicle environment for freeway off-ramp bottlenecks[J]. Journal of Transportation Engineering, Part A: Systems, 2023, 149(4): 04023009. doi: 10.1061/JTEPBS.TEENG-7456
[19]	李松, 张开碧, 李永福, 等. 理想诱导环境下的网联车与网联自动驾驶车混合交通流建模研究[J]. 交通运输工程与信息学报, 2023, 21(3): 31-58. https://www.cnki.com.cn/Article/CJFDTOTAL-JTGC202303003.htm LI S, ZHANG K B, LI Y F, et al. Modeling a mixed traffic flow of connected vehicles and connected autonomous vehicles in an ideal induction environment[J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 21(3): 31-58. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JTGC202303003.htm
[20]	秦严严. 智能网联环境下异质交通流特性分析方法研究[D]. 南京: 东南大学, 2019. QIN Y Y. Study on analytical method of heterogeneous traffic flow characteristics under connected and autonomous environment[D]. Nanjing: Southeast University, 2019. (in Chinese)
[21]	GUÉRIAU M, DUSPARIC I. Quantifying the impact of connected and autonomous vehicles on traffic efficiency and safety in mixed traffic[C]. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems, Rhodes, Greece: IEEE, 2020.
[22]	中华人民共和国交通运输部. 公路限速标志设计规范: JTG/T 3381-02—2020[S]. 北京: 交通运输部公路科学研究院, 2020. Ministry of Transport of the People's Republic of China. Design specifications for highway speed limit signs: JTG/T 3381-02—2020[S]. Beijing: Research Institute of Highway Ministry of Transport, 2020. (in Chinese)
[23]	XIAO Z, GUO X, GUO X, et al. Impact of cooperative adaptive cruise control on a multilane highway under a differentiated per-lane speed limit policy[J]. Transportation Research Record, 2021, 2675(10): 353-366. doi: 10.1177/03611981211011475
[24]	谢雨梅. 高速公路车道级可变限速控制策略优化研究[D]. 成都: 西南交通大学, 2022. XIE Y M. Study on optimization of lane-level variable speed limit control strategy on highway[D]. Chengdu: Southwest Jiaotong University, 2022. (in Chinese)
[25]	吴文静, 战勇斌, 杨丽丽, 等. 考虑安全间距的合流区可变限速协调控制方法[J]. 吉林大学学报(工学版), 2022, 52 (6): 1315-1323. https://www.cnki.com.cn/Article/CJFDTOTAL-JLGY202206009.htm WU W J, ZHAN Y B, YANG L L, et al. Coordinated control method of variable speed limit in on-ramp area considering safety distance[J]. Journal of Jilin University (Engineering and Technology Edition), 2022, 52(6): 1315-1323. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-JLGY202206009.htm
[26]	GUO Y, XU H, ZHANG Y, et al. Integrated variable speed limits and lane-changing control for freeway lane-drop bottle-necks[J]. IEEE Access, 2020(8)8: 54710-54721.
[27]	TREIBER M, HENNECKE A, HELBING D. Congested traffic states in empirical observations and microscopic simulations[J]. Physical Review E, 2000, 62(2): 1805-1824.
[28]	交通运输部公路局, 中交第一公路勘察设计研究院有限公司. 公路工程技术标准: JTG BO1—2014[S]. 北京: 人民交通出版社股份有限公司, 2014. Ministry of Transport of the People's Republic of China, CCCC First Highway Consultants Co. Ltd. Technical standard of highway engineering[S]. Beijing: China Communications Press Co., Ltd. 2014. (in Chinese)
[29]	MILANÉS V, SHLADOVER S E. Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data[J]. Transportation Research Part C: Emerging Technologies, 2014, 48: 285-300.
[30]	MILANÉS V, SHLADOVER S E, SPRING J, et al. Cooperative adaptive cruise control in real traffic situations[J]. IEEE Transactions on Intelligent Transportation Systems, 2013, 15(1): 296-305.

Relative Articles

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(11) / Tables(2)

Get Citation

PDF

XML

Article Metrics

Article views (219) PDF downloads(25)

Differential Variable Speed Limit Control Strategy Based on Reinforcement Learning

doi: 10.3963/j.jssn.1674-4861.2024.01.012

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Proportional views

Related

Differential Variable Speed Limit Control Strategy Based on Reinforcement Learning

doi: 10.3963/j.jssn.1674-4861.2024.01.012

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Proportional views

Related

Export File

Citation

Format

Content