Development of an intelligent jet engine controller using a model-based deep deterministic policy gradient technique
DOI:
https://doi.org/10.15282/jmes.19.3.2025.4.0842Keywords:
artificial intelligence , aero engines, DDPG algorithm, Deep Reinforcement LearningAbstract
The rapid advancement of artificial intelligence has the potential to significantly enhance the aerospace industry, particularly through the development of intelligent engine control systems. This study seeks to tackle the challenges of controlling complex, nonlinear aero-engines by applying Deep Reinforcement Learning techniques. Specifically, the Deep Deterministic Policy Gradient (DDPG) algorithm within an actor-critic framework to design an adaptive controller for the nonlinear thermodynamic model of the J85 jet engine are employed. The proposed method is evaluated against traditional PI controllers under various operating conditions, including different altitudes, Mach numbers, and humidity levels. Simulation results reveal that the DDPG-based controller outperforms PI control by achieving faster response times, 1.75 seconds (7.18%) faster during acceleration and 0.55 seconds (1.24%) during deceleration in standard conditions, and 1.09 seconds (4.79%) and 3.44 seconds (7.13%), respectively, under altered conditions. Moreover, the DDPG controller reduces turbine inlet temperature by up to 44.97% in the first case and 38.21% in the second case, and decreases surge margin by 54.83% and 56.18%, respectively, ensuring safer operation within limits. These findings demonstrate the DDPG algorithm's potential for substantial engine control performance and safety improvements. The study underscores the transformative potential of AI-driven control systems in aerospace applications, paving the way for more intelligent and adaptable engine management solutions.
References
[1] D. E. Miller, M. Rossi, “Simultaneous stabilization with near optimal LQR performance,” IEEE Transactions on Automatic Control, vol. 46, no. 10, pp. 1543-1555, 2001.
[2] K. Zhu, J. Zhao, G. M. Dimirovski, “H∞ tracking control for switched LPV systems with an application to aero-engines,” IEEE/CAA Journal of Automatica Sinica, vol. 5, no. 3, pp. 699-705, 2016.
[3] L. Gou, Z. Liu, D. Fan, H. Zheng, “Aeroengine robust gain-scheduling control based on performance degradation,” IEEE Access, vol. 8, pp. 104857-104869, 2020.
[4] D. Navarro-Tapia, A. Marcos, S. Bennani, “The VEGA launcher atmospheric control problem: A case for linear parameter‐varying synthesis,” Journal of the Franklin Institute, vol. 359, no. 2, pp. 899-927, 2022.
[5] M. Montazeri-Gh, A. Rasti, A. Jafari, M. Ehteshami, “Design and implementation of MPC for turbofan engine control system,” Aerospace Science and Technology, vol. 92, pp. 99-113, 2019.
[6] S. Liu, J. Bai, Q. Wang, W. Wang, “Tracking controller design for aero-engine based on improved multi-power reaching law of sliding mode control,” International Journal of Aeronautical and Space Sciences, vol. 20, no. 3, pp. 722-731, 2019.
[7] P. Razzaghi, A. Tabrizian, W. Guo, S. Chen, A. Taye, E. Thompson, et al., “A survey on reinforcement learning in aviation applications,” Engineering Applications of Artificial Intelligence, vol. 136, p. 108911, 2024.
[8] R. S. Sutton, A. G. Barto. Reinforcement Learning: An Introduction. 1st Ed. Cambridge: MIT Press Cambridge, 1998.
[9] J. Clifton, E. Laber, “Q-learning: Theory and applications,” Annual Review of Statistics and Its Application, vol. 7, no. 1, pp. 279-301, 2020.
[10] H. Jiang, R. Gui, Z. Chen, L. Wu, J. Dang, J. Zhou, “An improved sarsa () reinforcement learning algorithm for wireless communication systems,” IEEE Access, vol. 7, pp. 115418-115427, 2019.
[11] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529-533, 2015.
[12] H. Hasselt, “Double Q-learning,” Advances in Neural Information Processing Systems, vol. 23, pp. 1-9, 2010.
[13] V. Konda, J. Tsitsiklis, “Actor-critic algorithms,” Advances in Neural Information Processing Systems, vol. 12, pp. 1-7, 1999.
[14] Y. Yuan, D. Zhou, “Reinforcement learning for dual-control aircraft six-degree-of-freedom attitude control with system uncertainty,” Aerospace, vol. 11, no. 4, p. 281, 2024.
[15] R. Liu, F. Nageotte, P. Zanne, M. de Mathelin, B. Dresp-Langley, “Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review,” Robotics, vol. 10, no. 1, p. 22, 2021.
[16] M. Chowdhury, S. Keshmiri, “Interchangeable reinforcement-learning flight controller for fixed-wing UASs,” IEEE Transactions on Aerospace and Electronic Systems, vol. 60, no. 2, pp. 2305-2318, 2024.
[17] Z.-C. Qiu, Y. Yang, X.-M. Zhang, “Reinforcement learning vibration control of a multi-flexible beam coupling system,” Aerospace Science and Technology, vol. 129, p. 107801, 2022.
[18] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, et al., “Continuous control with deep reinforcement learning,” arXiv, pp. 1-14, 2016.
[19] S. Fujimoto, H. Hoof, D. Meger, “Addressing function approximation error in actor-critic methods,” in International Conference on Machine Learning, pp. 1587-1596, 2018.
[20] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International Conference on Machine Learning, pp. 1861-1870, 2018.
[21] J. Fang, Q. Zheng, W. Liu, H. Zhang, “Optimization control with multi-constraint of aeroengine acceleration process based on reinforcement learning,” in Third International Conference on Artificial Intelligence and Computer Engineering (ICAICE 2022), vol. 12610, pp. 1374-1383, 2023.
[22] Z. Wei, Z. Zhao, Z. Zhou, J. Ren, Y. Tang, R. Yan, “A deep reinforcement learning-driven multi-objective optimization and its applications on aero-engine maintenance strategy,” Journal of Manufacturing Systems, vol. 74, pp. 316-328, 2024.
[23] P. Li, W. Tang, J. Zhu, J. Dong, “Intelligent optimization method of rotor speed control policy for turbofan engine based on soft actor-critic,” in International Conference on Advanced Unmanned Aerial Systems, pp. 549-559, 2024.
[24] J. Yao, R. Li, Y. Guo, H. Wu, G. Cui, Z. Zhou, “Online performance seeking control of aero-engine based on stabilization-optimization dual loop,” Aerospace Science and Technology, vol. 158, p. 109927, 2025.
[25] Z. Wei, Z. Zhao, Z. Zhou, R. Yan, “Collaborative-sequential optimization for aero-engine maintenance based on multi-agent reinforcement learning,” Expert Systems with Applications, vol. 247, p. 123358, 2024.
[26] H. Wu, S. Zhong, M. Zhao, X. Fu, Y. Zhang, S. Fu, “Continual contrastive reinforcement learning: Towards stronger agent for environment-aware fault diagnosis of aero-engines through long-term optimization under highly imbalance scenarios,” Advanced Engineering Informatics, vol. 65, p. 103297, 2025.
[27] R. Schirru, D. Q. Vu, S. Razakarivony, S. Thépaut, A. Bauny, “Adaptive kalman filter by reinforcement learning for monitoring aircraft engines'performance against abrupt events,” in Turbo Expo, pp. 1-12, 2025.
[28] J. Xu, Y. Wang, Z. Wang, X. Wang, Y. Zhao, “Transient gas path fault diagnosis of aero-engine based on domain adaptive offline reinforcement learning,” Aerospace Science and Technology, vol. 155, p. 109701, 2024.
[29] H. Zhang, S. Wei, G. Xu, “Steady state controller design for aero-engine based on reinforcement learning NNs,” in 29th Chinese Control And Decision Conference (CCDC), pp. 2168-2173, 2017.
[30] Q. Zheng, C. Jin, Z. Hu, H. Zhang, “A study of aero-engine control method based on deep reinforcement learning,” IEEE Access, vol. 7, pp. 55285-55289, 2019.
[31] K. Miao, X. Wang, M. Zhu, S. Yang, X. Pei, Z. Jiang, “Transient controller design based on reinforcement learning for a turbofan engine with actuator dynamics,” Symmetry, vol. 14, no. 4, p. 684, 2022.
[32] R.-R. Qian, Y. Feng, M. Jiang, L. Liu, “Design and realization of intelligent aero-engine DDPG controller,” in Journal of Physics: Conference Series, vol. 2195, no. 1, p. 012056, 2022.
[33] B. Tao, L.-Y. Yang, D.-S. Wu, S.-L. Li, Z.-X. Huang, X.-S. Sun, “Deep reinforcement learning-based optimal control of variable cycle engine performance,” in International Conference on Advanced Robotics and Mechatronics (ICARM), pp. 1002-1005, 2022.
[34] Y. Zhu, M. Pan, W. Zhou, J. Huang, “Intelligent direct thrust control for multivariable turbofan engine based on reinforcement and deep learning methods,” Aerospace Science and Technology, vol. 131, p. 107972, 2022.
[35] J. Zhu, W. Tang, J. Dong, P. Li, “A virtual reinforcement learning method for aero-engine intelligent control,” in 8th International Conference on Automation, Control and Robotics Engineering (CACRE), pp. 138-143, 2023.
[36] J. Zhu, W. Tang, J. Dong, “Design of intelligent controller for aero-engine based on TD3 algorithm,” Information Technology and Control, vol. 52, no. 4, pp. 1010-1024, 2023.
[37] W. Gao, M. Pan, W. Zhou, F. Lu, J.-Q. Huang, “Aero-engine modeling and control method with model-based deep reinforcement learning,” Aerospace, vol. 10, no. 3, p. 209, 2023.
[38] X. Zhang, L. Zhonglin, J. Runmin, T. Zhang, “Deep reinforcement learning based active surge control for aeroengine compressors,” Chinese Journal of Aeronautics, vol. 37, no. 7, pp. 418-438, 2024.
[39] Y. Zhu, M. Pan, W. Zhou, J. Huang, “Self-evolution direct thrust control for turbofan engine individuals based on reinforcement learning methods,” Aerospace Science and Technology, vol. 144, p. 108734, 2024.
[40] H. Chen, S. Liu, W. Lyu, R. Dai, “Data-driven optimal thrust tracking control for a turbofan engine system with unknown dynamics based on adaptive dynamic programming,” Aerospace Systems, pp. 1-10, 2025.
[41] O. Altuntas, “Designation of environmental impacts and damages of turbojet engine: A case study with GE-J85,” Atmosphere, vol. 5, no. 2, pp. 307-323, 2014.
[42] A. Fawke, “Digital computer simulation of gas turbine dynamic behaviour,” PhD Thesis, University of Bristol, 1970.
[43] J. Kurzke, “Preparing compressor maps for gas turbine performance modeling: Smooth C 9,” User’s Manual, 2002.
[44] J. Kurzke, “Preparing turbine maps for gas turbine performance modeling: Smooth T 9,” User's Manual, 2003.
[45] S. Yarlagadda, “Performance analysis of J85 turbojet engine matching thrust with reduced inlet pressure to the compressor,” Master Thesis, University of Toledo, 2010.
[46] P. P. Walsh, P. Fletcher. Gas Turbine Performance. New Jersey: John Wiley & Sons, 2004.
[47] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, M. Riedmiller, “Deterministic policy gradient algorithms,” in International Conference on Machine Learning, pp. 387-395, 2014.
[48] R. Dong, J. Du, Y. Liu, A. A. Heidari, H. Chen, “An enhanced deep deterministic policy gradient algorithm for intelligent control of robotic arms,” Frontiers in Neuroinformatics, vol. 17, p. 1096053, 2023.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 The Author(s)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
