  Developing the collision-free flocking behavior for a dynamic squad of
fixed-wing UAVs is still a challenge due to kinematic complexity and
environmental uncertainty. In this paper, we deal with the decentralized
leader-follower flocking control problem through deep reinforcement learning
(DRL). Specifically, we formulate a decentralized DRL-based decision making
framework from the perspective of every follower, where a collision avoidance
mechanism is integrated into the flocking controller. Then, we propose a novel
reinforcement learning algorithm CACER-II for training a shared control policy
for all the followers. Besides, we design a plug-n-play embedding module based
on convolutional neural networks and the attention mechanism. As a result, the
variable-length system state can be encoded into a fixed-length embedding
vector, which makes the learned DRL policies independent with the number or the
order of followers. Finally, numerical simulation results demonstrate the
effectiveness of the proposed method, and the learned policies can be directly
transferred to semiphysical simulation without any parameter finetuning.
