Deep Reinforcement learning (applied to CFD)
TD3 (off-policy) & PPO (on-policy) - Validation for DRL algorithm
(a) No control — the sinusoidal initial condition breaks down by the viscous term
(b) TD3-Control — deterministic-policy family
(c) PPO-Control — stochastic-policy family
Figure 01. DRL validation on the Burgers equation — (a) no control, (b) TD3, (c) PPO
Description: Machine learning is applied with CFD as a domain. Here, the Burgers equation controlled by reinforcement learning is shown above.
(Burgers equation is the one-dimensional Navier-Stokes equation - here it is solved by spectral method).
(b) and (c) are the results of training the agent to maintain the initial state (sin waveform); both algorithms hold the waveform successfully.
Turbulence control for drag reduction through deep reinforcement learning
Background: This research aims to reduce the skin friction that develops near the wall as a fluid flows. It is known that successfully controlling this frictional resistance in the shipbuilding and aviation industries could save roughly $38 billion per year, so the topic carries significant engineering and social value.
Method: The Navier-Stokes equations were solved directly by numerical methods to build the flow simulation, which was then coupled with a reinforcement-learning algorithm implemented directly in Tensorflow. The flow was modeled as a simplified channel flow, and the walls were equipped with numerically modeled blowing/suction actuators controlled by reinforcement learning. TD3 (Twin Delayed Deep Deterministic Policy Gradient) was applied as the control algorithm to reduce the drag of the fully developed flow.
Limitations of prior approaches & result: The classical Opposition Control technique requires sensors inside the flow field when applied to real experiments. Supervised learning based on Opposition Control was proposed to compensate, but it could not overcome the performance limit and was in fact even less efficient. In this research a new strategy was derived through reinforcement learning, overcoming these constraints: about 25% drag reduction, compared with the average 20% achieved by supervised-learning-based control.
(a) Actor-Critic network structure
Figure 02. Actor-Critic network structure inserted in the TD3 algorithm
(b) No control shear field
(c) Control of shear field
Figure 03. Shear fields — (b) no control, (c) DRL control reduces the high shear stress (RED to BLUE)
Hardest challenge - training stability: Using DDPG alone led to low training stability and frequent performance collapse. Switching from DDPG to TD3 and applying Double Critic Network (prevents Q-value overestimation), Target Policy Smoothing (relaxes the target policy's excessive sensitivity), and Delayed Policy Update (updates the policy only after the critic has stabilized) greatly improved both stability and performance. In addition, tuning the discount factor of the Q-function led to a new drag-reduction strategy distinct from classical Opposition Control.
Physical interpretation: Reinforcement learning helps discover phenomena that are hard for humans to find, but the physical interpretation must ultimately be done by humans. Based on the control strategy derived by the agent, a general differential equation describing the drag reduction was derived and its effect was partially verified quantitatively.
Outcome: This work was submitted as a graduate thesis and published in the international journal Physical Review Fluids (2023).