  Integrating unmanned aerial vehicle (UAV) into the existing cellular networks
that are delicately designed for terrestrial transmissions faces lots of
challenges, in which one of the most striking concerns is how to adopt UAV into
the cellular networks with less (or even without) adverse effects to ground
users. In this paper, a cellular-connected UAV network is considered, in which
multiple UAVs receive messages from terrestrial base stations (BSs) in the
down-link, while BSs are serving ground users in their cells. Besides, the
line-of-sight (LoS) wireless links are more likely to be established in
ground-to-air (G2A) transmission scenarios. On one hand, UAVs may potentially
get access to more BSs. On the other hand, more co-channel interferences could
be involved. To enhance wireless transmission quality between UAVs and BSs
while protecting the ground users from being interfered by the G2A
communications, a joint time-frequency resource block (RB) and beamforming
optimization problem is proposed and investigated in this paper. Specifically,
with given flying trajectory, the ergodic outage duration (EOD) of UAV is
minimized with the aid of RB resource allocation and beamforming design.
Unfortunately, the proposed optimization problem is hard to be solved via
standard optimization techniques, if not impossible. To crack this nut, a deep
reinforcement learning (DRL) solution is proposed, where deep double duelling Q
network (D3QN) and deep deterministic policy gradient (DDPG) are invoked to
deal with RB allocation in discrete action domain and beamforming design in
continuous action regime, respectively. The hybrid D3QN-DDPG solution is
applied to solve the outer Markov decision process (MDP) and the inner MDP
interactively so that it can achieve the sub-optimal result for the considered
optimization problem.
