The global market for commercial unmanned aerial vehicles (UAVs), also known as drones, has grown significantly over Q. Wu the last decade and is projected to skyrocket to 45.8 billion dollars in 2025 from 19.3 billion dollars in 2020 [1]. The major driving factors behind such a dramatic market size growth are the steadily decreasing manufacturing costs and the increasing number of applications in a broad range of civilian and commercial sectors, including surveillance and monitoring, aerial imaging, precision agriculture, smart logistics, law enforcement, disaster response, and prehospital emergency care. Particularly, as announced by the Federal Aviation Administration during its press conference on the "Drone Integration Pilot Program" in Washington on November 8, 2017 [2]: "The need for us to integrate unmanned aircraft into the National Airspace System (NAS) continues to be a national priority. After the hurricanes, drones became a literal lifeline. They gave us an operational window that was a game changer at every level." This national program was launched to further explore the expanded use of drones, including beyond-visual-line-of-sight (BVLOS) flights, nighttime operation, and flights over people. Later, during the Notre Dame Cathedral fire in 2019 [3] , two UAVs equipped with high-resolution thermal imaging cameras were dispatched in BVLOS environments to help firefighters gauge the scene through the billowing smoke and effectively position firehoses to combat the blaze in real-time. Very recently, UAVs have also been applied worldwide to combat the spread of COVID-19, e.g., to facilitate communication/broadcast information, disinfect outbreak-affected areas, deliver critical supplies, and measure body temperatures [4] . In practice, depending on the specific use case as well as the corresponding requirements, UAVs can be classified into different categories, such as fixedwing versus rotary-wing UAVs, tethered versus untethered UAVs, etc. [5] .
Wireless communication is an essential technology to unlock the full potential of UAVs in numerous applications and has thus received unprecedented attention recently [5] - [7] . As shown in Fig. 1 , UAV communication technologies have evolved from the early direct link to the most recent 5G technologies in the course of the past twenty years, with significantly enhanced communication rate. Although technologies such as direct link, WiFi, and satellite communications are still useful in some remote scenarios where cellular services are unavailable, it is believed that exploiting the thriving 5Gand-beyond cellular networks to support UAV communications is the most promising and cost-effective approach, especially when the number of UAVs grows dramatically. On the one hand, to guarantee safe and efficient flight operations of multiple UAVs, it is of paramount importance to provide secure and ultra-reliable communication links between the UAVs and the line-of-sight (LoS) dominated air-ground channels inherent to UAVs at high altitudes lead to smaller path loss due to less severe shadowing and multi-path fading. As a result, leveraging spectrum sharing to simultaneously support both UAVs and ground users in mMTC scenarios will generate significant ground-air interference to UAVs in the downlink, while in the uplink, UAVs will cause stronger interference to a large number of ground users even if they are distributed in the network [11] . Moreover, high UAV mobility generally results in more frequent handovers and time-varying wireless backhaul links between UAVs and ground BSs/users, which poses practical challenges in URLLC scenarios. As such, it is imperative to develop new techniques and/or new designs, e.g., efficient handover management and resource allocation, to tackle the above issues to guarantee seamless service provisioning in the 3D space.
On the other hand, the emerging 3D wireless networks also introduce new design opportunities to better serve users in a cost-effective manner [13] - [16] . For example, by leveraging mobility to increase the operating altitude or bypass obstacles, UAV-BSs/users are able to intelligently reposition themselves to avoid signal blockage to ground nodes caused by e.g., high-rise buildings. This is practically appealing for millimeter wave (mmWave) communications that are expected to provide eMBB services. Furthermore, for IoT/sensor networks, a hybrid network architecture composed of high altitude UAV-BSs/relays and their ground counterparts is able to achieve wider ground coverage. In addition, UAV-enabled mobile data collectors can move close to IoT devices to collect data, such that their transmit energy is minimized. This is particularly beneficial in mMTC scenarios where connectivity and energy consumption are critical factors [17] . Moreover, non-trivial tradeoffs may exist in leveraging these advantages [14] , [18] - [20] . For instance, although exploiting the high mobility of a UAV access point/data sink via trajectory design could improve the communication throughput, this comes at the cost of exceedingly long user access delays [18] , [20] , which thus may not be appropriate for delay-sensitive URLLC applications. Furthermore, to reap the promised benefits of UAV-assisted cellular communications, there are still many important problems that need to be addressed, including more accurate airground channel modelling, traffic-adaptive UAV deployment and/or trajectory design, energy consumption modelling and energy-efficient design, and high-speed and reliable backhauling. secured backhaul from the ground, thus enabling unlimited endurance with high communication performance. There have also been several pilot projects launched in recent years to advance UAV research and UAV field tests, as also shown in Table I . In particular, in the ABSOLUTE project, UAVs are leveraged as aerial BSs for terrestrial and satellite communications to enhance the ground network capacity and extend the signal coverage, especially for public safety applications in emergency situations. Besides, the 5G!Drones project was funded by the European Commission in 2019 to drive UAV vertical application trials by leveraging advanced 5G features, and involves 20 industrial and academic partners from 8 European countries, covering verticals, commercial network operators, networking industry, research centers, and universities. Specifically, this project exploits network slicing as the key component to simultaneously run the three types of UAV services (eMBB, URLLC, and mMTC) on the same 5G infrastructure, demonstrating that each UAV application runs independently and does not affect the performance of other UAV applications, while providing different 5G services.
Besides, the third-generation partnership project (3GPP) has made significant efforts to ensure that cellular networks will meet the connectivity demands of UAVs [41] - [43] , as shown in Table II . In 2017, 3GPP approved a study item on enhanced LTE support for UAVs, where the main objective was to identify key challenges for using current LTE networks with downtilted BS antennas to provide connectivity to UAVs. To further meet the requirements of connectivity, identification, and tracking of UAVs, 3GPP recently considered the application of 5G networks in Release 17 [43] . Several other standardization bodies and working groups have also devoted substantial efforts to develop different UAV specifications [5] , including the International Telecommunication Union Telecommunication (ITU-T) standardization sector, the European Telecommunications Standards Institute (ETSI), and the IEEE Drones Working Group (DWG). Furthermore, to foster the research and innovation surrounding the study, design, and development of aerial communications, there has been an emerging technology initiative within the IEEE Communication Society (ComSoc) focusing on aerial users and networks [44] . One of its target applications is public safety, i.e., using aerial communications to deliver additional cellular coverage, for example during emergency situations, or to help first responders by providing advanced services. Moreover, two standards working groups jointly sponsored by IEEE ComSoc and IEEE Vehicular Technology Society (VTS) focus on aerial communications and networking standards [45] . 
It is worth noting that a number of papers in the literature provide overviews or surveys on UAV communication research [5] - [7] , [10] , [13] , [46] - [55] . Unlike these works, this paper explicitly focuses on identifying the new major challenges posed by the eMBB, URLLC, and mMTC use cases to UAV communication in 3D space, and highlights the most promising solutions to tackle them. Moreover, sensing and AI technologies are considered to further enhance the capabilities of future UAV cellular networks, but were not covered in previous overview and survey papers. To facilitate and inspire future research, various promising directions for further investigation are also highlighted.
Next, we provide an overview on the aforementioned five research areas respectively from Section II to Section VI. Specifically, in Section II, we highlight how emerging technologies, such as intelligent reflecting surface (IRS), can be leveraged to support eMBB applications in 3D space and also discuss the potential of exploiting massive multipleinput multiple-output (M-MIMO) and mmWave. In Section III, we provide an overview on short-packet communications (SPC) to guarantee URLLC in UAV networks. In Section IV, we consider non-orthogonal multiple access (NOMA), energy harvesting, and energy efficient designs for the support of the massive connectivity. In Section V, we focus on radio-based sensing in wireless networks with UAVs and introduce a joint UAV communication and sensing approach. In Section VI, we review machine learning methods particularly for UAV trajectory and communications design, discuss the computation offloading design for UAVs with mobile edge computing (MEC), and consider UAV-based distributed edge machine learning. In each section, directions for future research are also provided. Finally, the paper is concluded in Section VII.
eMBB is a natural evolution of 4G/LTE networks and will provide higher data rates and system capacity than current mobile broadband services, while guaranteeing moderate reliability, e.g., packet error rates (PERs) on the order of 10 −3 [9] . In particular, 5G cellular networks are expected to support a peak data rate of 10 Gbits/s for eMBB applications, which in principle is sufficient for supporting highrate UAV communication applications such as real-time video streaming and data relaying. On the other hand, UAV-enabled aerial platforms provide new degrees of freedom to further enhance the communication throughput or satisfy throughput requirements in a cost-effective manner. For terrestrial cellular networks, a variety of viable solutions have been proposed to meet the stringent eMBB service requirements [56] , [57] , namely, enhance spectral efficiency by deploying massive active transmit/receive antennas and/or passive reflecting elements such as M-MIMO and IRS; exploit unused new spectrum such as mmWave communications; reduce the transmitter-receiver distance and improve frequency reuse such as ultra-dense networks (UDNs) and device-to-device (D2D) communications. In this section, we focus on the exploitation of first two paradigms in 3D UAV cellular networks, with a particular focus on the newly emerging cost-effective IRS.
M-MIMO, as a key enabling technology in the current 5G standard, is promising for supporting cellular-connected UAV communications [6] , [58] - [62] . Equipped with fulldimensional large arrays, ground BSs can perform fine-grained 3D beamforming to mitigate interference among high-altitude UAVs and low-altitude terrestrial users, and thus, achieve much higher network throughput. However, the beamforming gain of M-MIMO critically depends on the availability and accuracy of channel state information (CSI) at the ground BSs, whereas cellular-connected UAVs introduce new challenges. First, UAVs can cause severe pilot contamination to a large number of ground BSs due to their strong LoS channels, which cannot be resolved by existing pilot decontamination techniques designed solely for terrestrial users. Second, the UAVs' high mobility in 3D space renders efficient beam tracking for them a more challenging task than for terrestrial users and can incur excessive pilot overhead [63] , [64] . Third, practical implementations may adopt the hybrid beamforming based M-MIMO architecture to support a large group of coordinated UAVs or a UAV swarm. This will further complicate the pilot contamination and beam tracking problems. To better address the inter-cell air-ground interference to achieve enhanced network throughput, a more ambitious approach referred to as cell-free M-MIMO, which combines ideas from distributed/network MIMO and coordinated multipoint transmission, was proposed recently, where massive antennas distributed over a large geographical area are connected with a central processing unit (CPU) [65] . In this case, both UAVs and ground users are surrounded by multiple BS antennas, rather than the conventional case where each BS is surrounded by multiple users [65] . Due to the LoS-dominant air-ground channels, such a user-centric architecture provides more degrees of freedom to exploit the macro diversity offered by the many distributed BSs. Nevertheless, key issues that remain to be solved include efficient centralized and distributed power control, low-complexity fronthaul/backhaul provisioning, and network scalability with respect to UAV swarms [66] - [69] .
Another promising approach to support rate-demanding eMBB services in 3D space is to leverage the enormous chunks of new spectrum available in the mmWave bands [70] , [71] . Several inherent limitations of mmWave communications, such as high signal attenuation and high vulnerability to blockage, can be alleviated by exploiting the UAV mobility. For example, UAVs acting either as aerial platforms or users can not only fly towards ground nodes to reduce the propagation loss, but can also intelligently adjust their trajectories to bypass surrounding obstacles to increase the probability of LoS paths. However, similar to M-MIMO, mmWave communication requires the use of a large number of antennas and the channel coherence time is shortened due to the high UAV mobility and shorter wavelength signals. As such, fast channel variations will result in practice, which renders effective dynamic beam training and tracking techniques imperative. Some existing works proposed the use of movement prediction filters such as Kalman filters to track the time-varying UAV-ground channels [72] - [74] . Besides, a fast beam searching algorithm was developed in [75] to track the UAV-to-ground channels based on a predetermined hierarchical codebook. Other important topics for future research include low-complexity spectrum management [76] , and high-speed and reliable backhaul design.
Despite the promising advantages of M-MIMO and mmWave communications, their required high complexity and hardware cost as well as increased energy consumption are still crucial issues faced in practical implementation [56] , [57] . As an alternative, IRS has recently emerged as a new and costeffective solution to improve the received power and suppress air-ground interference in 3D space [77] - [80] . An IRS is composed of a large number of passive reflecting elements, each of which is able to reflect the impinging electromagnetic wave with a tunable reflection coefficient (including an amplitude and a phase shift) [81] - [84] . By smartly coordinating the reflections of all elements, an IRS is able to reconfigure the wireless channel with the desired signals added coherently and interference cancelled at designated receivers, thus significantly enhancing the communication throughput without the need for deploying new active BSs or relays. Furthermore, IRSs possess appealing advantages in practice such as light weight, and thus can be easily mounted on walls and even the surface of high-speed moving vehicles to support numerous applications [85] - [90] . As such, IRS has been considered as a disruptive technology for transforming our current "dumb" radio environment into an intelligent one, which potentially benefits a wide range of vertical industries such as transportation, manufacturing, and smart cities. Recently, IRS has also been recognized as a promising technology for the future sixth-generation (6G) ecosystem [79] , [91], [92] and studied in various system setups [93] - [101] . Generally speaking, as shown in Fig. 2 , IRS can be either deployed on the ground to assist UAV communications or attached to UAVs to assist terrestrial communications [102] , as elaborated below.
A. IRS for UAV Communications 1) Terrestrial IRS Assisted UAV Communications: For data dissemination and collection in IoT networks, a single UAV may need to fly sequentially approaching each IoT device/cluster for high data rate transmission. However, this not only compromises the user access delay but also increases the UAV propulsion energy consumption [15] , [103] , [104] . To overcome these drawbacks, one possible solution is to dispatch multiple UAVs, however this requires sophisticated UAV-UAV coordination and thus increases the operational cost as well as the signalling overhead. As an alternative, a properly deployed IRS can help resolve this issue. As shown in Fig. 2 (a), the UAV only needs to directly cover a subset of the IoT devices and the significant path loss between the UAV and other devices can be effectively compensated by leveraging the deployment of IRS. This is particularly useful for uplink transmissions due to the limited energy supply of IoT devices, which helps reconcile the energy trade-off in ground-to-UAV communications [19] . On the other hand, as mentioned previously, the ground BS antennas are typically downtilted in current cellular networks, i.e., their main lobes point towards the ground for optimizing the coverage for ground users, and UAVs flying above the BSs are only supported through the side lobes. Indeed, recent 3GPP studies have confirmed that UAVs receive weak signals from existing terrestrial BSs and hence supporting aerial users will require further research and development [41] , [42] . Fortunately, by leveraging the signals reflected by IRSs via 3D passive beamforming to serve the UAVs in the sky, the communication links between ground nodes and UAVs can be greatly improved, which helps eliminate the need for significantly modifying the configurations of existing ground BS antennas.
Motivated by the above advantages of IRS, there have been a number of works in this research direction [105] - [110] . For instance, assuming that the UAV-ground link is blocked, the authors in [105] maximized the average achievable data rate in IRS-assisted UAV-ground communication systems by jointly optimizing the UAV's trajectory and the IRS phase shifts. It was shown that rather than flying towards the ground user, the UAV should fly towards the IRS to significantly improve the data rates by exploiting the reflections of the large IRS aperture. Yet, this study only focused on the case of a single user being served by a single-antenna UAV and the proposed solution was not applicable to more general multi-user systems. To address this issue, resource allocation, including user scheduling, power allocation, UAV trajectory, and beamforming was optimized in [106] to minimize the total power consumption of an IRS-assisted multi-user multiantenna UAV communication system, subject to a given required minimum achievable rate of each ground user. It was shown that deploying an IRS can shorten the UAV flying trajectory without compromising the QoS, as compared to the case without IRS. This work was then further extended to multi-IRS scenarios [107] , [108] . In particular, the overall weighted data rate taking into account the geographical fairness of all the users was maximized in [108] to jointly optimize the UAV's trajectory and the phase shifts of the reflecting elements of the IRSs. Instead of using conventional convex optimization methods, the authors in [108] proposed a low-complexity deep Q-network (DQN)-based solution by discretizing the trajectory, which is useful for practical systems with discrete phase-shift control. Then, they further proposed a deep deterministic policy gradient (DDPG)-based solution to facilitate continuous phase-shift control.
In [109] , the potential of IRS to enhance cellular communications for UAVs was investigated, where the received signal power gain was analyzed as a function of the UAV height and various IRS parameters including its size, altitude, and distance from the BS. In particular, it was shown that for a UAV hovering at 50 meters (m) above the ground, a 21 dB received power gain can be achieved with a properly deployed IRS comprising 100 passive elements based on the 3GPP ground-to-air channel models. Besides, due to the downtilted antenna pattern of the BSs, the optimal IRS deployment altitude was shown to decrease as the IRS-BS distance increases so as to effectively reflect the signals coming from the BS. Also, in [110] , an IRS was deployed to assist a UAV relaying system where the average capacity, outage probability, and average bit-error rate (BER) were analyzed. It was found that if the IRS is deployed between the source and the UAV relay, the optimal position of the UAV for maximizing the outage performance moves closer to the destination, which is different from the case without the IRS where the UAV relay needs to be deployed in the middle between source and destination.
2) UAV IRS Assisted Terrestrial Communications: In practice, small lightweight UAVs usually can support only a limited payload, and may not be able to carry bulky RF transceivers in BSs/relays. Besides, such active nodes generally incur high energy consumption, which further aggravates the energy consumption of UAVs and limits endurance. On the other hand, in terrestrial wireless networks, the deployment of IRS is highly restricted by the availability of the existing infrastructure such as building facades, lampposts, and advertising boards. As such, unlike the case in Fig. 2 (a), it may not be possible to find proper deployment locations for installing an IRS to establish the desired (virtual) LoS links between the transceivers. Compared to terrestrial IRS, a UAV-based IRS (UIRS) is more likely to have strong LoS links with the ground nodes due to the relatively higher altitude and mobility of UAVs, thus reducing the probability of a blockage between them [111] . This is particularly important for high frequency mmWave and THz communication systems that are very sensitive to blockage [112] . For example, as shown in Fig. 2 (b) , since an IRS only reflects the signals to its front halfspace, deploying an IRS on the building facade only helps to enhance the communication data rate of user 1 and the lack of coverage behind the building remains still unsolved. Although it is possible to connect user 2 to the BS via multihop IRS reflections if multiple IRSs are deployed, this requires properly located buildings in the surrounding environment as well as additional multi-IRS coordination and signal processing complexity. In this case, deploying a UAV equipped with IRS is practically appealing [102] . More importantly, UAVs with high mobility are able to adapt their positions dynamically according to the changes in the communication environment and thus can maintain persistent (virtual) LoS links between transceivers.
To provide some insights on UIRS deployment, in Fig. 2 (b), we focus on the UIRS-aided communication for user 2 and the BS-user distance is given by D m. The BS-UIRS horizontal distance is given by d m and the UIRS's altitude is H m. We optimize the horizontal location of the UIRS (indicated by d) to maximize the achievable rate/signal-to-noise ratio (SNR) of user 2. For simplicity, LoS channels are assumed for both the BS-UIRS and UIRS-user links, while the BS-user direct link is assumed to be severely blocked and thus ignored. Then, the receive SNR at user 2 assuming optimal IRS passive beamforming is given by [83] , [102] 
where P denotes the BS transmit power, β 0 denotes the channel power gain at a reference distance of 1 m, N denotes the number of IRS reflecting elements, and σ 2 denotes the receiver noise power. By setting the first-order derivative of the denominator with respect to d to zero, the optimal UIRS placement for maximizing the receive SNR can be easily derived as [83] , [102] 
(2)
It is interesting to note that depending on the values of H and D, the optimal deployment location changes from the midpoint to locations with equal distances to the transmitter (BS) and the receiver (user 2), as also verified in Fig. 3 for P = 20 dBm, β 0 = −30 dB, N = 250, σ 2 = −100 dBm, and D = 400 m. Considering that D in practice is expected to be considerably larger than H to obtain effective reflection from the UIRS, the UIRS should be deployed close to the transceivers, which is quite different from the deployment strategy for active relays. Nevertheless, practical UIRS deployment also needs to consider other factors such as LoS versus non-LoS links, channel rank and condition number, and the use of a single large UIRS or multiple smaller cooperative UIRSs [79] , [81] , [113] . However, there has been only limited work in this line of research so far [111] , [112] , [114] - [116] . In [111] , the minimum SNR within a given rectangular service area (e.g., a hot spot in a cellular network) was maximized by jointly optimizing aerial IRS placement, the IRS phase shifts, and the transmit beamforming at the BS. To gain useful insights, the authors first derived the optimal UIRS placement and phase shifts for the special case of single-location SNR maximization, where the optimal horizontal location was shown to depend only on the ratio between the UIRS height and the source-destination distance. As for the general case of area coverage enhancement, an efficient two-step method was proposed by decoupling the phase-shift optimization from the UIRS placement design, where a 3D beam broadening and flattening technique was proposed to obtain 3D beam patterns matching the service area size on the ground [102] . For mobile users in practical systems, the dynamic selfpositioning of UIRS is required, which motivated the work in [112] . Specifically, to maximize the downlink mmWave communication capacity of a mobile outdoor user, a reinforcement learning (RL) approach based on Q-learning and neural networks was proposed in [112] to enable efficient deployment and phase-shift optimization for a self-sustained UIRS. The main advantage of this approach lies in that it requires no prior knowledge of the dynamic environment, but learns the characteristics of the environment during the service process of the UIRS, based on data measurements and feedback received during each communication stage. The results showed that significant gains can be achieved by using such a UIRS as compared to a static IRS in terms of the average data rate as well as the LoS probability. Yet, to simplify the system design, it was assumed in [112] that the BS only transmits, and thus, the UIRS only reflects signals during the hovering state.
Despite these existing works, the study of IRS-assisted eMBB air-ground communications is still in a very early stage and many interesting and important problems need to be addressed. First, the IRS reflections could be very sensitive to UAV jittering and user location uncertainty [117] , which may cause significant performance degradation for UAVs and/or ground users due to the reliance on LoS channels. Particularly, the current joint UAV trajectory and IRS phase-shift designs utilized trajectory discretization [105] - [108] where the UAV is considered to be at a fixed sample location during each interval. However, in reality, the UAV is continually moving, which can easily render the IRS passive beamforming ineffective due to the signal misalignment arising from UAV movement. As such, the development of robust algorithms for tuning the phase shifts at the IRS is still a practically challenging task. Joint reflection amplitude and phase-shift control at the IRS could be a possible approach to alleviate airground interference, which, however, increases the hardware cost and design complexity. Hence, how to strike an optimal performance-complexity/cost tradeoff when considering practical hardware imperfections such as discrete and/or coupled reflection amplitudes and phase shifts [118] - [121] remains an open problem. In addition, practical low-complexity channel estimation/tracking methods are required for acquiring the UIRS-user channels along their 3D trajectories, which is more challenging than the case of terrestrial IRSs that are at fixed locations [79] .
Furthermore, for a given network with both ground users and UAVs, how to jointly optimize the resource allocation and the deployment of terrestrial IRSs and UIRSs including their locations and densities is another important and interesting topic to pursue. Finally, most existing studies focus on the utilization of IRSs to enhance the primary end-toend communication by exploiting passive beamforming [105] - [112] , [114] - [116] . However, in practice, UIRSs also need to deliver their own information, including control signals to acknowledge their current status, environmental parameters (such as temperature, humidity and pressure) obtained by UAV sensors, etc., to the BS. Rather than equipping each UIRS with a dedicated transmitter, which is not cost-effective and also requires extra power consumption, a more appealing approach is to modulate the IRS information onto the reflected signals to achieve passive information transfer, which leads to the new paradigm of symbiotic communication [122] - [124] that aims to simultaneously transmit IRS information and enhance the communication quality of the primary link. 
To fully embrace the era of the Internet-of-Everything (IoE), a well-developed wireless infrastructure focusing on low latency and high reliability is necessary. As a result, starting from the 5G wireless systems, the concept of URLLC has been introduced as a key performance indicator [127] , [128] . Unlike classical high data rate oriented multimedia streaming services, in which high rate data flows from a source to a sink, typical URLLC services focus on conveying sensing information, control commands, and feedback information in short packets which need to be delivered reliably in an extremely short period. Table III shows some emerging URLLC applications and requirements. It can be seen that these applications impose heterogeneous requirements on both the reliability and latency. In particular, for the Internet-of-Drones (IoD), the need for rapidly conveying control signals in highly dynamic UAV communication scenarios imposes strict requirements on both latency and reliability. Unfortunately, despite the development of advanced wireless communication techniques, e.g., massive MIMO [129] - [132] , mmWave transmission [133] - [135] , and full-duplex communication [136] - [138] , it is believed that a single wireless link between a user and a terrestrial BS cannot satisfy URLLC requirements. As a result, different kinds of diversities are necessary to offer the flexibility to trade more bandwidth for achieving shorter delays and higher reliability [139] .
Recently, UAV-based communications have been widely recognized as one of the disruptive and enabling technologies to address this issue. Indeed, aerial nodes usually enjoy unobstructed LoS air-to-ground channels [140] . Thus, conventional terrestrial communication systems with blocked communication links may deploy UAVs as aerial relays to substantially improve system data rate and/or coverage to realize URLLC by exploiting spatial diversity. Besides, in the case of natural disasters and disease outbreaks, it may not be possible to guarantee URLLC via standalone conventional terrestrial communication networks. In these cases, UAVs can be employed as ad-hoc aerial BSs to offer temporary URLLC links. In fact, by exploiting the high maneuverability of UAVs, fast, highly flexible, and cost-effective deployment of communication infrastructure can be ubiquitously established, especially in temporary hot spots, disaster areas, complex terrains, and rural areas. On the other hand, UAVs can also serve as a physical carrier for user equipment connecting to existing networks requiring URLLC services, e.g., to report sensing information. In both cases, physical communication of UAVs can be divided into payload delivery (PD) and control and nonpayload communication (CNPC) [6] . Specifically, for CNPC links, short-packet control information is required to be conveyed from a ground BS to UAVs or exchanged between UAVs with high reliability and low latency [141] . As for PD links, UAVs can be adopted to provide certain types of data services requiring not only extremely high data rates but also ultra-low latency and high reliability, e.g. conveying real-time video and haptic information in the Tactile Internet. However, existing wireless communication protocols and techniques were designed mainly for terrestrial communications and do not fully capture the hybrid characteristics of air-to-ground and air-toair communications. More importantly, applying conventional techniques/protocols to UAV-based communication systems may introduce an exceedingly long delay and raises serious safety issues for controlling aerial nodes, which remains a major obstacle for realizing efficient UAV communications. Next, we show how to leverage short-packet communication to meet the requirement of URLLC applications.
To guarantee URLLC in UAV-based communication systems, various approaches have been proposed in the literature and 5G standardization. For instance, by exploiting the excessive spatial degrees of freedom offered by massive MIMO, in theory, the reliability of UAV-based communications in the ground-to-air links can be theoretically improved to a certain extent [142] . However, the deployment of a large number of antennas at UAVs is challenging due to the aforementioned energy and size limitations of practical UAVs. On the other hand, the actual communication latency, T L , is comprised of several components which can be summarized as follows:
where T ttt denotes the time-to-transmit latency, which is the time needed to transmit a packet; T prop is the signal propagation delay from the transmitter to the receiver; T proc is the time consumed for encoding, decoding, synchronization, and channel estimation; T retex is the time required for retransmission (if any); and T sig is the time incurred by signaling exchanges such as connection request, scheduling grant, channel training, feedback, and queuing delay. In response to the harsh maximum delay allowance specified in Table III , different methods for reducing the overall communication latency have been proposed in the literature [143] by minimizing the different components in (3). For instance, if the communication link is shadowed, one can deploy a UAV as a wireless relay to establish an additional reliable end-to-end communication link which can help to reduce the time spent on retransmission, i.e., T retex . Also, non-coherent transmission and detection were advocated in [144] . In particular, for these schemes, acquiring the knowledge of instantaneous channel conditions is not necessary, which reduces both T proc and T sig needed for channel training and estimation at the expense of some performance loss in signal detection. Besides, a new short slot structure exploiting the concept of mini-time slots was proposed for 5G [145] to realize URLLC in practice via shortening T ttt in (3).
On the other hand, to further improve system performance, channel capacity-achieving error-correcting codes have been proposed and adopted in 5G communication systems, e.g., polar codes and low-density parity-check (LDPC) codes. However, due to the requirement of short packets with low latency, the blocklength of the coded messages is short which in turn jeopardizes the communication reliability. More importantly, the Shannon capacity, which assumes infinite blocklength codes, is no longer achievable and cannot be used to accurately characterize the system performance. Although one can perform proper resource allocation to reduce the performance degradation by taking into account the impact of short blocklength codes, most existing resource allocation algorithms for UAV communications, e.g., [146] - [149] , were designed based on Shannon capacity as the performance metric, thereby assuming implicitly infinite block length transmission. Besides, these algorithms do not offer sufficient flexibility to strike a balance between delay (codelength) and reliability.
Recently, driven by the need for URLLC, the study of SPC has received significant attention in the literature [150] - [153] . These works have captured the relationship between the achievable rate, decoding error probability, and packet length, which facilitates the design of effective resource allocation. In the following, we summarize the key techniques for resource allocation design taking into account the impact of short packets in UAV communications.
1) Performance Metric for SPC: Unlike conventional systems, where Shannon's capacity theorem can be used to characterize performance and the probability of erroneous decisions becomes negligible as the coded packet length is sufficiently long, short packets are adopted in URLLC systems to achieve low latency, which makes decoding errors unavoidable. As an alternative, for evaluating the performance of SPC, the notion of normal approximation for finite blocklength codes was developed in [154] . For ease of illustration, we focus on a simple UAV-to-ground BS communication channel. Suppose that the BS transmits a message to the UAV over a quasi-static flat fading channel with a given packet length, N B = W T , where W is the bandwidth and T denotes the transmission duration. The achievable rate 1 (bit/s/Hz) for the given finite blocklength can be accurately approximated as follows [154] :
where C = log 2 (1 + γ) denotes the Shannon channel capacity that can be achieved for infinite packet lengths, γ is the received SNR which is a function of the UAV's trajectory, and is the required decoding packet error probability. Variable V is the channel dispersion and Q −1 (·) is the inverse of the Gaus-
For the complex-valued additive white Gaussian noise (AWGN) channel, the channel dispersion is given by
As can be observed from (4), the subtrahend accounts for the penalty caused by the finite blocklength. In particular, for a sufficiently long blocklength, the normal approximation approaches the conventional Shannon capacity. As a result, the normal approximation in (4) serves as a useful tool for optimizing the allocation of the system resources to strike a balance between the blocklength (delay), achievable rate, and reliability. In the following, we focus on an illustrative example to highlight how URLLC can be facilitated in UAV communication systems via resource allocation.
2) Example for Resource Allocation for URLLC: Suppose there is a ground BS communicating with a ground user. Due to heavy shadowing in the direct channel, as shown in Fig. 4 , a UAV adopting the decode-and-forward protocol is dispatched to serve as a wireless relay to facilitate reliable end-to-end communication resulting in a two-hop relaying system [156] , [157] . The 3D coordinates of the BS, the user, and the UAV are (0, 0, 0), (a, b, 0), and (x, y, H), respectively, with H being the UAV's fixed altitude. For ease of presentation, we denote the channel power gain from the BS to the UAV and that from the UAV to the user by h 1 and h 2 , respectively. Also, the transmit powers of the BS and the UAV are given by P BS and P UAV , respectively. It is well known that for a sufficiently high altitude, the free-space channel model can accurately capture the channel characteristic. Thus, h 1 and h 2 are modelled as
respectively, where β 0 is the channel power gain at a given reference distance. As a result, the received SNR at the UAV and the user can be expressed as
respectively, where σ 2 UAV and σ 2 User denote the power of the AWGN at the UAV and the user, respectively. As can be observed from (7), the received SNRs of the two hops depend on the position of the UAV. Then, by applying the normal approximation in (4), the achievable rates of the first hop and the second hop are given by
respectively, where m 1 and m 2 are the blocklengths adopted at the BS and the UAV, respectively. Furthermore, the channel dispersions of the first hop and the second hop are given
respectively. In practice, it is common to maximize the total achievable rate of the user by jointly optimizing the position of the UAV relay and the blocklengths in the two hops, taking into account the end-to-end communication delay requirement. Thus, the proposed resource allocation policy is determined by solving the following optimization problem:
where T Block in (10) denotes the time needed for conveying one unit of blocklength and constraint C1 guarantees that the maximum transmission delay does not exceed D max . Constraint C2 enforces that the blocklengths are non-negative integers. Finally, constraint C3 ensures that the position of the UAV relay is within the serving cell with a radius of r max . Problem (10) is a non-convex optimization problem. The non-convexity is caused by the combinatorial constraint in C2 and the non-convex normal approximation appearing in the objective function. However, by analyzing the structure of problem (10), it can be solved optimally via a combination of the penalty method and monotonic optimization [158] at the expense of a high computational complexity. As an alternative, one can apply approximations to simplify the problem at hand. For instance, a high SNR approximation is commonly adopted. In particular, in the high SNR regime, i.e., γ → ∞, the channel dispersion can be approximated by
such that it becomes a constant. Then, successive convex approximation and the penalty method can be employed to obtain a suboptimal solution to (10) with polynomial time computational complexity. In Fig. 5 , we show the achievable data rate of the considered UAV-assisted relaying system versus the position of the UAV for different BS transmit powers. The maximum delay is 1 ms while each mini-time slot is T Block = 0.01 ms with 200 kHz of bandwidth. In other words, there are a maximum of 100 blocks for transmission. For simplicity, the channel-tonoise ratio (CNR) at the reference distance of 1 meter is set as β0 σ 2 User = β0 σ 2 UAV = 60 dB and the required decoding packet error probability is = 0.1%. The achievable rate for the finite blocklength case is obtained by optimizing m 1 and m 2 in (10). For comparison, we also consider a performance benchmark for which the delay constraint C1 in (10) is dropped and an infinite blocklength is assumed for transmission. As can be observed from Fig. 5 , there is a performance gap between the cases of finite blocklength and infinite blocklength, as the former has an insufficient number of coded information blocks for sufficiently averaging out the impact of Gaussian noise to approach the system capacity. This gap is generally inevitable in URLLC as the end-to-end delay constraint is stringent. On the other hand, the optimal UAV position maximizing the endto-end achievable rate depends heavily on the power budgets of both the UAV and the ground BS. In fact, the optimal UAV position attempts to balance the achievable rate of the two hops in the system. In particular, when the power budget at the BS is smaller than that of the UAV, e.g. P BS = 13 dBm, P UAV = 23 dBm, the optimal UAV position is closer to the BS than to the user. Although the power budget of the BS is limited, the optimal strategy reduces the path loss between the BS and the UAV for establishing an efficient first communication hop. Besides, the higher UAV transmit power can be exploited to compensate for the longer communication distance between the UAV and the ground user to balance the achievable rate of the two hops. In contrast, when the power budget of the BS is larger than that of the UAV, e.g., P BS = 23 dBm and P UAV = 13 dBm, the optimal UAV position moves towards the user, allowing the system to again fully exploit the power budgets of both the BS and the UAV to maximize the system performance.
A critical issue in realizing URLLC in UAV systems is the limited spectrum available for simultaneous ground-to-ground, ground-to-air, and air-to-air communications. Unlike terrestrial communication systems where ground BSs are connected to a data hub via high-speed fixed-line backhaul links, e.g., optical fibres, the implementation of ground-to-air and airto-ground links usually relies on dedicated wireless communication channels. In particular, UAV-based communication systems have a stringent demand for system resources in both the ground-to-air and air-to-ground links due to the required support of high data rate backhauling and exchange of timecritical UAV control signals. In this context, the allocation of dedicated orthogonal radio resources to ground-to-air links is commonly considered in the literature. Yet, the spectral efficiency of orthogonal resource allocation is low. In fact, the wireless spectrum is scarce and is already congested by existing communication systems. Furthermore, in order to guarantee URLLC, multiple UAVs should be deployed simultaneously which puts a significant burden on the need for available spectrum. Hence, conventional multiple access schemes based on orthogonal spectrum partition would quickly exhaust the available resources even for moderate numbers of UAVs and ground users. More importantly, this would introduce a long delay in the provisioning of URLLC and raises serious safety issues in controlling aerial nodes. Thus, efficient multiple access schemes based on non-orthogonal spectrum utilizations have to be developed for enabling UAVbased URLLC systems.
Besides, existing algorithms for URLLC in UAV-based communication systems have been developed based on the assumption of perfect hardware. Yet, various types of hardware imperfections exist in practical UAV systems and must be accounted for [161] . For example, due to the finite precision of electronic circuits and the imperfection in manufacturing mechanical components, the control of the UAV's trajectory and positioning may not perform as expected [117] . As for the hardware modules for wireless communication, phenomena such as power amplifier non-linearity, non-linear phase noise, frequency and phase offsets, in-phase and quadrature (I/Q) imbalance, and quantization noise jointly degrade the decoding capability of the receivers. Hence, it is still unclear whether it will be possible to guarantee URLLC in UAV-based communication systems. Thus, a thorough performance analysis taking into account the impact of hardware impairments is needed.
The connection density of MTC networks will be around 10 million devices per km 2 by 2030 [162] . Currently, for fast and cheap implementation, machine-type devices (MTD) access wireless networks via low-cost commercial technologies, e.g., Zigbee, bluetooth, and WiFi. Besides, the application of long range radio (LoRa) and cellular networks to the IoE has been proposed for enabling massive access. Indeed, it is beneficial and economical to deploy the IoE using existing networks. However, traditional communication networks are designed and used for human-to-human (H2H) communications to serve a relatively small number of users, compared to the large number of devices in IoE, cf. Table IV . The fundamental challenges in the implementation of MTC are massive connectivity, ultra-high reliability and low latency requirements, and energy-efficient transmission. As URLLC has been discussed in Section III, in this section, we tackle the challenges in MMTC from the perspectives of massive access and energyefficient transmission.
Unlike traditional communication networks which focus on the performance of the downlink for a small number of users, MTC has a much higher demand on system resources in the uplink due to the associated massive uplink connectivity [163] . For instance, security cameras can be installed on UAVs for capturing images of specific areas for security surveillance. The captured images are periodically conveyed to a ground BS for further data analysis. In this context, allocating dedicated radio resources to MTD orthogonally has been proposed in the literature, e.g., [17] , due to the simple design. However, for MTC, since the number of MTDs is large, orthogonal multiple access (OMA) would quickly exhaust the valuable radio resources and introduce exceedingly long delays. Furthermore, some commonly adopted random access protocols for H2H communications such as Carrier Sense Multiple Access (CSMA) and ALOHA are strictly suboptimal for MTC, since they suffer from congestion and overloading in the presence of a huge number of MTDs. Therefore, efficient multiple access mechanisms such as grant-free random access and NOMA [164] - [166] , are necessary to enable the massive connections required for MTC communications. On the other hand, MTDs for IoE are usually powered by batteries with limited capacities in practice. Although the lifetime of the traditional H2H communication networks can be extended by replacing or recharging the batteries, this may be inconvenient, dangerous (e.g. in a toxic environment), and costly in MTC networks due to the large number of MTDs. Generally speaking, the energy consumption challenge to achieve green and self-sustainable MTC networks could be addressed by a proper combination of energy harvesting and/or energy-efficient designs which has drawn significant interest from both academia and industry [56] , [57] , [130] , [167] - [169] .
The emergence of UAVs as a viable solution to deliver communication services in the past decades has introduced a paradigm shift in MTC. Conventionally, long-distance communication between ground MTDs and their home BSs creates a system performance bottleneck which results in a strict limit on the lifetime of the communication network. In fact, by exploiting the flexility and high-mobility of UAVs, one can deploy a UAV as a sink node to collect data directly from ground MTDs. For example, for the case of a large sensor network deployed in a rural area with limited cellular coverage, a UAV can serve as a sink node to collect information data from the sensors [170] . In particular, by exploiting the LoS communication channels from the UAV to the ground sensors, the sensors can transmit with low power which can help extend the lifetime of the sensing network. Furthermore, by taking into account the periodic and sparse transmission characteristics of sensors, UAV semi-persistent scheduling can be performed to reduce control plane traffic between the UAV and the BS. On the other hand, as an MTD itself, a UAV can help relay the data from the ground MTDs so as to improve the energy efficiency of the whole communication system. However, despite the various practical advantages of deploying UAVs, the performance of UAV-enabled MTC is still limited by the small onboard battery equipped at UAVs due to the size, weight, and power (SWaP) constraints of UAVs. Hence, practical methods for enabling sustainable MTC via UAV communications are needed.
NOMA is a key enabling technology for future mMTC applications due to its capability of allowing simultaneous transmissions of multiple devices in the same resource block [128] , [159] , [171] . Specifically, NOMA leverages superposition coding (SC) at the transmitters and successive interference cancellation (SIC) at the receivers to achieve efficient access as well as to partially mitigate co-channel interference. It has been shown that NOMA is particularly effective for cases where the users experience substantially different channel conditions [171] .
1) NOMA for Cellular-connected UAVs: Due to its advantages, NOMA is practically appealing for application in cellular networks with co-existing UAV and ground users. Particularly, NOMA allows UAVs/a UAV swarm to reuse the resource blocks that are already occupied by ground users, thus improving the number of aerial users that can be supported even for a high ground user density, as compared to the non-scalable OMA scheme. On the one hand, more reliable air-to-ground communications can be established due to the existence of LoS links, as compared to the non-LoS terrestrial channels between the ground users and BSs. Besides, the strong LoS air-ground channels allow the UAVs to be visible to a large number of ground BSs over a wide area. As a result, the UAVs can be potentially served by more BSs simultaneously than the ground users, thus achieving a higher macro-diversity gain in terms of user association. On the other hand, using NOMA in such unique air-ground channel environments also brings challenges since the UAVs' uplink transmissions will greatly degrade the received ground users' signals at a large number of ground BSs, considering the high frequency reuse factors of current multi-cell systems. Furthermore, ensuring that each individual BS can cancel the UAV's uplink interference via signal decoding may severely limit the performance gain of NOMA over OMA, since the UAV's achievable rate will be then limited by the BS with the worst channel conditions. To address this issue, a new decodeand-forward (DF) based cooperative NOMA scheme, was proposed in [172] , which exploits interference cancellation among collaborative adjacent BSs with backhaul links, e.g., the existing X2 interface in LTE [173] . Specifically, some BSs with better channel conditions are selected to decode the UAV's signals first, and then forward the decoded signals to their backhaul-connected BSs for interference cancellation. It was shown that the proposed scheme achieves higher data rates than both OMA and non-cooperative schemes, especially when the ground traffic is congested. To further improve the performance, a quantize-and-forward (QF) based cooperative interference cancellation approach was proposed in [174] where the adjacent BSs only quantize the received UAV signals without decoding them.
In contrast, in the downlink, UAV receivers suffer from strong co-channel interference from a large number of ground BSs. However, the interference mitigation techniques proposed for uplink transmission are not applicable in the downlink because the roles of the UAVs have changed from interference sources to interference victims. One straightforward approach is to increase the transmit power of all ground BS. However, this provides only a marginal gain as the interference increases as well. Another possible practical approach is to leverage cooperative beamforming, where the available BSs that are not serving ground users in the UAV's occupied resource block, transmit collaboratively to the UAV to improve its received power to overcome the co-channel interference. However, as the ground user density increases, the number of such BSs drops quickly and the co-channel interference also increases, thus rendering this approach ineffective. To overcome this difficulty, a new cooperative beamforming scheme with interference transmission and cancellation (ITC) was proposed in [175] . Specifically, the signals for terrestrial users reusing the same resource block as the UAV are first forwarded by their home BSs to the BSs serving the UAV, and then transmitted along with the UAV's signals via cooperative beamforming. As a result, the desired signal power at the UAV receiver is improved and the terrestrial interference is also suppressed, without affecting the existing transmissions. This centralized implementation requires excessive backhaul transmissions among different BSs, which may be costly to implement. To lower the implementation complexity and signaling overhead, a distributed algorithm requiring only local information exchange among BSs was proposed based on the concept of divide-and-conquer in [175] . It was demonstrated that such a distributed design can still significantly improve the UAV's performance as compared to the conventional schemes without ITC, especially when the terrestrial user density is high. However, current research results only consider a limited number of UAVs and the support of massive UAVs or UAV swarm is still a challenging problem that needs further efforts [176] .
2) UAV-enabled NOMA: On the other hand, by proactively leveraging their high mobility, UAVs as aerial BSs inherently possess the ability to effectively exploit asymmetric channel conditions of different ground devices to realize the performance gains promised by NOMA. For instance, the authors in [177] studied UAV enabled NOMA transmissions to two quasi-static ground users. The capacity region was characterized by jointly optimizing the UAV's trajectory and transmit power/rate allocations over time, subject to the constrained maximum speed and transmit power of practical UAVs. A fundamental result is that regardless of the multiple access scheme, the optimal UAV trajectory follows a simple hoverfly-hover (HFH) policy. In particular, it was revealed that in the cases of infinitely high UAV speed and/or long flight duration, NOMA and time division multiple access (TDMA) achieve the same rate performance. However, NOMA generally outperforms OMA (including TDMA and frequency division multiple access (FDMA)), and the capacity gain achieved by NOMA over OMA decreases as the UAV maximum speed and/or flight duration increases. The comparison of two-user NOMA and OMA for UAV-assisted communication was also extended to other design objectives such as sum-rate [178] and outage probability [179] . In particular, [179] considered a more practical Rician air-ground channel setup where the UAV flies with a constant speed following a circular trajectory, with the objective of minimizing the outage probability. The condition under which NOMA outperforms TDMA was also derived in terms of the channel and UAV trajectory parameters. On the other hand, for a general setup with multiple users, proper user pairing with bandwidth allocation is also important to unlock the full potential of NOMA. In [180] , the UAV was assumed to pair one near user (cell-centered) with one far user (cell-edge). Then, the multi-user rate max-min optimization problem was formulated by jointly optimizing the bandwidth allocation, power allocation, UAV altitude, and antenna beamwidth. In subsequent works, the use of NOMA was extended to multiple-antenna and multi-UAV networks [181] - [184] .
Besides multiple access schemes, promising techniques to prolong the lifetime of MTDs have also been proposed including energy conservation schemes and wireless power transfer (WPT) techniques [185] - [190] . Particularly, WPT provides several competitive advantages over the other energy harvesting techniques such as its continuous availability and long service range. However, the performance of energy harvesting at the MTDs critically depends on the locations of the power sources (e.g., BS, WiFi, etc.). For MTDs that are close the power sources, a sufficient amount of energy can be harvested, conversely, for those which are far away from the power sources, energy harvesting may not be possible at all due to the high sensitivity of typical energy receivers [56] , [191] . Fortunately, UAVs with their high mobility provide a new promising solution to tackle these shortcoming of WPT. The first relevant work [192] studied a UAV-mounted energy transmitter broadcasting wireless energy to charge two MTDs. The results showed that when the distance between the two MTDs is very small, the Pareto boundary of the energy region can be achieved by fixing the UAV's horizontal position. In contrast, when the distance between the two MTDs is large, to achieve the Pareto boundary of the energy region, the UAV should fly and hover between the two MTDs. Subsequently, this work was extended to a more general setup with multiple MTDs in [193] where the authors studied two different objective functions, namely the maximization of the sum energy harvested by all the MTDs and the max-min energy harvested among MTDs. Different from the above works, which obtained locally optimal solutions, a globally optimal one-dimension UAV trajectory design was developed with the aim of maximizing the minimum energy received by either one of the MTDs [194] . In addition, UAV-aided wireless powered communication network (WPCN) is drawing considerably attention in the research community [6] . Specifically, receiver devices in WPCN first harvest energy from the signals sent by the UAV in the downlink, and then utilize the harvested energy to transmit information to the UAV in the uplink. Depending on the configuration of the UAVs, the UAV-aided WPCN can be classified into two categories. The first is the integrated UAV-aided WPCN, where the energy transmitter and information receiver are carried by one UAV [195] - [198] . The other is the separated UAV-aided WPCN, where the energy transmitter and information receiver are installed on two different UAVs [199] , [200] . Furthermore, UAV-enabled simultaneous wireless information and power transfer (SWIPT) was proposed recently, where the UAV is deployed as an aerial BS to simultaneously send information and energy to the receivers on the ground [201] - [203] .
On the other hand, different from terrestrial BSs which are connected to the power grid, UAVs are typically powered by on-board batteries with limited energy storage capacity, which thus drastically limits their endurance. To overcome the short flight time of UAV, several techniques have been proposed, which can be classified into two categories: external-powered UAV and mechanical dynamic optimization.
1) External-powered UAV: For external-powered UAVs, depending on the external sources used for powering the UAV, they can be further divided into two approaches, namely solarpowered and laser-powered UAVs. For solar-powered UAVs, a proper size solar panel is installed on the UAV, which first harvests solar energy and then converts it to electrical energy enabling the UAV to fly and communicate [204] - [206] . Solarpowered UAVs were experimentally verified in [204] , [205] . The results revealed that by carrying a solar panel, the UAV can continuously fly for more than 24 hours, which renders this approach appealing in practice. However, the amount of harvested solar energy depends heavily on the flight altitude, weather conditions, and temperature [207] , [208] . In particular, it was shown in [208] that there exists a tradeoff between the harvested solar energy and the quality-of-service (QoS) for the ground users. Specifically, if the UAV operates above the clouds, it can harvest more solar energy but incurs high path loss between the UAV and the ground users due to the high flight altitude. If the UAV flies below the clouds, less solar energy is available since the solar energy is mainly absorbed by the clouds, while the quality of the communication link between the UAV and the ground users is improved. Instead of exploiting solar energy, a photovoltaic receiver can be installed on the UAV to harvest power from a laser power beacon. Compared to solar power, the laser-beam power supply is more stable and can deliver much more energy to the receiver with a narrowly focused energy beam [209] , [210] .
2) Mechanical dynamic optimization: For mechanical dynamic optimization, the research was mainly focused on how to minimize the UAV's communication energy and propulsion energy consumption [138] , [211] - [218] . The UAV's communication energy consumption comprises the circuit power consumption as well as the transmit power consumption. Some relevant works optimized the deployment of the UAV to minimize the transmit power while achieving the different design objectives such as the users' QoS, communication coverage range, and delay [213] - [215] . However, accurately modelling the propulsion energy consumption is generally difficult since it crucially depends on many practical factors, including the UAV velocity, the UAV acceleration, etc. Accordingly, in [211] , a theoretical propulsion energy consumption model for fixed-wing UAVs was derived, which is given by
where c 1 and c 2 are constants which are related to the air density, drag coefficient, and wing area; g and m represent the gravitational acceleration and the aircraft's mass, respectively; v and a denote the UAV's velocity and acceleration vectors, respectively. It is observed from (12) that when the UAV velocity is equal to 0, the UAV's propulsion energy tends to infinity, which indicates that the UAV trajectory should be carefully designed to avoid excessive energy consumption and improve endurance. However, for communication service provisioning, it is desirable to deploy the UAV hovering above the ground users [15] , [216] . As a result, there is a tradeoff between minimizing the UAV's energy consumption and maximizing the communication throughput. As a result, a new metric, namely the energy efficiency defined as the ratio of the throughput and the UAV energy consumption, has been proposed and widely adopted to strike a balance between the above two important design goals [6] . Different from the notion of energy efficiency adopted in traditional cellular networks, where the energy consumption is related to the transceiver's circuit power consumption and transmit power, energy-efficient designs for UAVs depend also the UAV's velocity and acceleration, which makes the design problem much more challenging. A handful of works have considered energy-efficient UAV design for different applications such as UAV-assisted mobile edge computing and UAVassisted backscatter communication [104] , [138] , [219] , [220] . However, the current research results only focus on a single UAV or few UAVs, energy-efficient design for a large number of collaborative UAVs is still a key challenge due to the excessive data and control information exchange between them needed for efficient cooperation.
For many practical data collection applications in mMTC networks, a large amount of devices may be distributed in a wide area while each of them only has small bursts of data to transmit. In this case, having UAVs getting close to each of them to collect data may lead to high propulsion energy consumption. This thus leads to an energy consumption tradeoff between UAVs and ground devices [19] . To address this issue, inter-UAV cooperation among UAV swarms is a promising solution where a UAV cluster head can be dynamically selected and other UAVs first collect data in a small area and then transmit to the UAV cluster for further processing.
Besides, for the case of large-scale deployment of UAVs in heterogenous applications, the communication activities of UAVs to the same ground BS can be highly be random, which may call for random access protocols. Although the grant-based random access protocol is simpler in practical implementation, while as the number of UAVs grows quickly, it would result in a high probability of access failures due to collision as well as high signaling overhead arising from handshakes. As a consequence, grant-free random access or unsourced random access protocols with properly designed UAV grouping methods are another important aspect to achieve efficient UAV swarm communications [159] . Moreover, security and privacy are other important issues in safeguarding wireless communication with massive UAV and ground devices [55] , [138] , [221] - [225] .
Besides the requirement of high-performance wireless communications, the ability to support effective and efficient sensing is also essential for realizing the vision of integrating UAVs into 5G-and-beyond networks. Commercial UAVs nowadays are already equipped with a multitude of sensors of various types, such as inertial measurement unit (IMU), accelerometers, tilt sensors, and current sensors. Such embedded sensors provide important real-time information for ensuring safe UAV operation, such as the UAV's position and orientation estimates, direction and flight path maintaining, and power consumption management.
On the other hand, for future wireless networks, where UAVs of large-scale deployment will be seamlessly integrated into terrestrial communication systems, sensing merely relying on those on-board embedded sensors will be insufficient. Instead, a combination of both UAV embedded sensing and infrastructure-based sensing is needed to achieve high sensing performance, in terms of response time, sensing range, coverage, reliability, accuracy, and efficiency. In this section, we focus on radio-based sensing, where the detection and parameter estimation of the targets of interest are based on the radio signals echoed/scattered by the targets. Compared with alternative sensing technologies such as acoustic-based, vision-based, and light-based sensing, radio sensing is less vulnerable to poor environmental conditions (such as noisy background or dark areas) and can typically support a larger sensing range. Besides, the fact that both radio sensing and wireless communications utilize radio signals to accomplish their tasks makes it necessary to study both systems jointly.
As illustrated in Fig. 6 , UAV sensing can be classified into two main paradigms, namely sensing for UAV and UAV for sensing. In the former case, sensing technologies are utilized to support safe UAV flight and low-altitude airspace monitoring and traffic management. By contrast, for the paradigm of UAV for sensing, dedicated UAVs are dispatched as aerial flying platforms to provide sensing support from the sky.
Two typical use case scenarios of sensing for UAV are sense-and-avoid (SAA) and adversary UAV detection, tracking, and classification.
SAA plays an indispensable role to ensure safe UAV flight, especially for autonomous or semi-autonomous UAVs [226] . Different from manned aircrafts, UAVs do not have a pilot onboard, and thus they heavily rely on their own sensed information for swift response for collision and/or obstacle avoidance. Note that even for UAVs under real-time remote control, SAA is important for avoiding catastrophic events, considering the long round-trip radio propagation delays and the extra response time of ground pilots. To achieve fast response even in highly dynamic environments, SAA typically relies on the sensed information of on-board sensors, though ground based SAA (GBSAA) is also possible. Note that besides sensing for collision/obstacle avoidance, in many applications such as UAV-enabled fertilizer or pesticide spraying, sensing may also be necessary for constant-altitude maintenance so as to ensure uniform spraying [227] , which is a challenging task in tough terrains. While many contemporary commercial UAVs are already enabled with SAA capabilities, most of them are vision-or light-based 2 , which makers them vulnerable to poor environmental conditions.
Another important use case of sensing for UAV arises from the imperative need for the detection, tracking, and classification of potentially illegitimate and hazardous UAVs. Besides their benign usage, UAVs could also be misused, either intentionally or unintentionally, which may jeopardize public safety and/or threaten privacy. Under such circumstances, UAVs are usually non-cooperative or even deceptive, which renders applying active detection and localization methods infeasible [228] , but rather requires reliance on radar sensing based on passively echoed/scaterred signals. While radar sensing for aircraft detection has long been used in military, its application for UAVs is facing new challenges. For instance, different from manned aircrafts, UAVs usually have much smaller radar cross section (RCS), which makes them more difficult to detect. Besides, rotary-wing UAVs are capable of flying at low speed or hovering, which makes them difficult to separate from stationary clutter background or other natural flying objects such as birds.
Radar sensing for UAV networks has received significant research attention recently. For example, the key techniques of detection, tracking, and interdiction of amateur drones are discussed in [229] . Beyond UAV detection, UAV classification is also important for safety and privacy protection in the future internet-of-drones (IoD) era. For example, in [230] , the authors studied the classification of micro-drones into loaded or unloaded vehicles using multi-static radar. In [231] , the authors studied the problem of classifying UAVs versus other flying objects such as birds and manned aircrafts, by extracting the features of the tracks generated by radar, where a track is defined as a series of plots associated with each target observed in consecutive radar scans.
Another promising paradigm of UAV sensing is to utilize UAVs as aerial nodes to provide wireless sensing support from the sky, which we refer to as UAV for sensing. Compared with conventional ground sensing, UAV-based sensing has several advantages. First, thanks to its elevated altitude and reduced signal blockage, UAV-based sensing typically has a wider field of view (FoV) compared to ground sensors. Besides, the highly controllable 3D UAV mobility makes it possible to flexibly deploy UAV sensors to hard-to-reach areas, such as poisonous or hazardous locations. Furthermore, the high mobility of UAVs also introduces a new design degree of freedom (DoF) for sensing performance optimization, via 3D sensor trajectory optimization. This is particularly appealing for target tracking, where the UAV locations can be dynamically adjusted to best track the target. Therefore, UAV-based sensing has a wide range of potential applications, such as law enforcement, precision agriculture, 3D environment map construction, search and rescue, and military operations.
Due to the aforementioned advantages, UAV-based sensing has received growing interest recently. For instance, various applications of UAV-based sensor platforms were outlined in [227] , together with a discussion of experiments with an imaging radar. In [232] , the authors surveyed UAV-based remote sensing for field-based crop phenotyping. In [233] , UAV-based synthetic aperture radar (SAR) was used to enhance the radar aperture for high angular resolution. Besides, multiple flying UAVs were utilized in [234] to localize moving RF sources, and the UAV trajectory was optimized to minimize a bound on the achievable localization error. UAV-aided air quality sensing was discussed in [235] . In [236] , the authors proposed a dynamic and reconfigurable aerial radar network composed of UAVs to detect and track other unauthorized/malicious UAVs. It was demonstrated that by exploiting the new DoFs offered by UAV trajectory optimization, the dynamic aerial radar network is able to improve the tracking performance over a conventional terrestrial radar network with fixed deployment.
For a basic UAV-based radar system with N T transmit antennas and N R receive antennas, the input-output relationship can be modelled as
where K denotes the total number of targets, y(t) ∈ C N R ×1 and x(t) ∈ C N T ×1 represent the received and transmitted signal waves, respectively, α k is the complex-valued coefficient from the transmitter to the receiver associated with the kth target, θ k and φ k are the elevation and azimuth angles of the kth target, respectively, with the superscripts (·) (R) and (·) (T ) denoting the angle-of-arrival (AoA) and angle-ofdeparture (AoD), respectively. Note that different from ground sensing where the azimuth angle is usually of primary interest, for UAV-based sensing, both the azimuth and elevation angles are important due to the elevated UAV position. Furthermore, a(·) and b(·) denote the receive and transmit array responses, respectively, τ k is the radio propagation delay for signals reflected/scattered by the kth target, and n(t) ∈ C N R ×1 denotes the receiver noise. Note that in (13), we have only included the signal components reflected/scattered by the K targets of interest. This may be a valid model for UAV sensing in wide-open areas or after proper clutter subtraction, when the signal components originating from background clutter have been properly eliminated. The radar performance is critically affected by the radar waveform x(t), which is usually designed to have good autocorrelation properties. One popular waveform is the linear frequency-modulated continuous waveform (FMCW) [237] , where multiple chirp signals are transmitted with the frequency linearly increasing with time. Different from wireless communication systems, the transmitted radar waveform x(t) is known at both the transmitter and receiver. Therefore, based on the knowledge of the transmitted and received waveforms x(t) and y(t), the two fundamental problems in radar sensing are:
Radar detection: Detect the presence/absence of target k, i.e., α k = 0 for hypothesis H 0 and α k = 0 for hypothesis H 1 . In this case, the sensing performance can be measured in terms of the probability of correct detection/misdetection and the probability of false alarm.
Radar estimation: If target k is present, estimate the key parameters θ
k , and τ k , based on which the target state, such as location and moving velocity, can be further determined. Typical performance metrics for radar estimation include the accuracy or mean squared error (MSE) of the estimation. For unbiased estimators, the classic Cramer-Rao lower bound (CRLB) provides the theoretical performance bound on the variance of the estimate [238] . In [239] , the authors proposed a new performance metric for radar estimation that is similar to communication rate, called radar estimation rate. Specifically, the target is treated as a passive node with some entropy about its own state that unwillingly communicates with the radar receiver, and radar estimation rate corresponds to how much additional information is gained about the target's state after the radar estimation process. Other sensible radar performance metrics include the number of resolvable targets, response time, and time and range resolution.
Note that (13) is applicable for both bi-static and monostatic radars. In the former, the radar transmitter and receiver are geographically separated, whereas for the latter, they are collocated. For mono-static radar, (13) can be further simplified since we have N T = N R , θ
k , and b(·) = a(·). Besides, τ k corresponds to the round-trip delay between the radar transmitter/receiver and the kth target, which is related to the target distance r k and radial velocity v k as τ k = 2(r k +v k t) c , where c is the speed of light. The technologies for radar sensing in general have been advanced tremendously during the past decades. In particular, it has been shown that MIMO radar is a powerful approach to enhance the radar performance. Specifically, for an N R × N T MIMO radar with orthogonal waveforms transmitted from the N T transmit antennas, we may achieve the same effective angular resolution as a virtual phased array radar with N T N R elements [240] - [242] . This thus lays a strong foundation for cellular sensing to achieve super resolution with massive MIMO in 5G-and-beyond networks. Besides, full-dimensional MIMO radar also provides angular resolution in both the azimuth and elevation domains, which is essential for UAV sensing.
Another promising technology for UAV sensing is mmWave sensing, which offers two promising advantages. First, compared to sub-6 GHz systems, physical objects are electrically larger in mmWave systems because of their shorter wavelength. Thus, small objects like micro-UAVs that might be invisible for microwave systems become more visible and more easily detectable for mmWave systems [243] . To illustrate this fact, the authors in [243] used the radar crosssection (RCS) as a performance measure to compare the delectability of small objects with different frequencies. It was shown that a small drone illuminated by mmWave radar (60 GHz) has a RCS about 30 dB higher than that by a 2 GHz microwave radar. Second, mmWave systems usually have a large bandwidth, which leads to high time and range resolution. In particular, it is known that the radar range resolution ∆ is given by ∆ = c/(2B), where c is the speed of light and B is the system bandwidth. This thus makes mmWave systems a promising technology for high-performance radar detection and estimation. Therefore, mmWave UAV detection has received significant research interest, as reviewed in [243] .
Traditionally, wireless communication and radar sensing were considered to be two independent systems that are fully separated in spectrum. With the continuous spectrum expansion of cellular communication systems (from sub-6 GHz to mmWave or even TeraHertz) and the need for developing more intelligent wireless networks, there has been a significant research interest recently in joint communication and sensing (JCAS) [21] - [24] . On the one hand, JCAS facilitates the design of a highly flexible and efficient systems utilizing the spectrum allocated for both purposes, which thus offers a new solution approach for resolving the spectrum scarcity. On the other hand, JCAS enables the sharing of wireless infrastructure and RF hardware, so as to build compact and light-weight wireless equipment with both communication and sensing capabilities. Note that this feature is particularly appealing for UAV networks, considering the severe SWaP and endurance constraints of UAVs, together with the significant role that radar sensing would play towards the integration of UAV into 5G-and-beyond cellular networks.
The preliminary level of JCAS is communication-radar coexistence [244] , where communication and radar sensing are still designed as two separate systems with their own respective waveforms and transmitter/receiver designs, but their mutual interference is properly controlled via resource allocation. This usually corresponds to the scenario when radar and communication share neither the transmitter nor the receiver. For example, a low-altitude UAV that continuously broadcasts radar waveforms for SAA needs to take into account the potential interference caused to the communication between nearby BSs and ground users, and vice versa.
A more advanced approach for JCAS is known as dual function radar-communications (DFRC) [245] , where both the radar and communication functions are implemented in the same device. DFRC is ideally suited for application in UAV networks. For example, a DFRC-enabled cellular BS may simultaneously communicate with the ground users it serves and monitor the low-altitude airspace to detect adversary UAVs. DFRC can be loosely classified as radarcentric, communication-centric, and integrated design. For radar-centric DFRC, the transmitted waveform x(t) in (13) is radar-oriented, such as the traditional FMCW, but with additional information-bearing symbols embedded for communication, e.g., via phase or frequency modulation of the FMCW pulse [246] . For MIMO radar with transmit beamforming, the information-bearing symbols may also be embedded into the radar beam pattern, e.g., by amplitude modulation of the radar side-lobe levels [247] . While powerefficient and simple to implement, such radar-centric DFRC can only support very low data rates. On the other hand, for communication-centric DFRC, the system is primarily designed for wireless communications, with the additional radar functionality. For example, there has been significant interest in utilizing the standard orthogonal frequency-division multiplexing (OFDM) communication waveforms for radar sensing [248] - [250] . By optimizing the various OFDM parameters, such as the sub-carrier spacing and power allocation, different trade-offs between communication and radar sensing performance can be achieved. However, for high mobility scenarios like UAV networks, the sensing performance (such as the unambiguous range) with OFDM waveforms is vulnerable to sub-carrier misalignment [251] . Lastly, for integrated radar-communication design, the DFRC waveform might be derived from the scratch without restriction to radar-or communication-centric designs, which makes further performance improvements possible. For example, in [252] , the weighted combination of the radar and communication waveforms was used as the transmitted waveform of a DFRC MIMO transmitter, and the resulting transmit covariance matrix was optimized to preserve the desired radar beampattern, while guaranteeing a minimum required SINR requirement at each communication user. Beamforming for JCAS with analog or hybrid analog/digital antenna architectures was studied in [253] and [21] , respecitvely.
The research on sensing in UAV networks is still in its infancy, with many new opportunities and challenges ahead. For example, due to the severe SWaP constraint, more compact, light-weight, and energy-efficient sensing devices are required for UAV-based sensing than for conventional ground-based sensing. Besides, the limited endurance of UAVs makes it quite challenging to build an aerial sensing network merely relying on UAVs that is available all day. In this regard, future sensing networks are expected to be highly heterogeneous, and may utilize complementary ground, aerial, and even space sensing platforms to maximize sensing performance, which deserves in-depth investigations. On the other hand, while JCAS has been studied extensively for terrestrial systems, e.g., for automotive applications, its application in UAV networks is still in an early stage. In particular, with elevated UAV positions, the coverage requirement for both communication and sensing extends from the traditional 2D plane to the 3D airspace, for which cost-effective networking architectures and signal transmission/reception techniques need to be developed. Besides, when UAVs are utilized as aerial communication and sensing platforms, the new DoFs offered by UAV trajectory design should be exploited for improving the sensing and communication performance. To resolve the SWaP problem of UAVs, energy-efficient designs for JCAS in UAV networks, which take into account the UAV propulsion energy as well as the communication and sensing energy, deserve further investigation. Another promising research direction is to exploit the sensed information, such as the 3D radio propagation environment, to enhance UAV communications [254] .
In addition to supporting communication and sensing services, the integration of AI is expected to be another important feature of future cellular networks, towards the vision of network intelligence [25] . Generally speaking, the interplay between AI and wireless networks has enabled two emerging paradigms, namely, AI-empowered wireless communications (see, e.g., [25] , [26] ) and edge intelligence (see e.g., [255] , [256] ). In the former paradigm, AI and machine learning techniques are utilized as a new data-driven mathematical tool (in contrast to conventional model-driven methods) for optimization of wireless systems to enhance the communication performance. In the latter paradigm, AI and mobile edge computing (MEC) capabilities [257] , [258] are incorporated into BSs and APs at the network edge, for enabling various intelligent applications (such as autonomous driving and industrial automation) with extensive communication and computation requirements [255] , [256] . As compared to the conventional cloud and on-device intelligence, edge intelligence can significantly reduce the end-to-end latency and minimize the traffic loads to the core network.
Following the above two paradigms, AI is expected to also play an important role for 5G-and-beyond UAV communication networks in two aspects. On the one hand, for highly dynamic UAV-enabled 3D networks, AI and machine learning can serve as promising alternative mathematical tools to solve, e.g., joint UAV trajectory and communication design problems, where the conventional model-driven optimization may not work well due to the difficulty in obtaining accurate network state information. On the other hand, the integration of UAVs in edge intelligence not only facilitates emerging applications (such as drone VR and drone swarms), but also introduces new challenges in effectively handling computationintensive and latency-critical AI tasks from the sky. This calls for the joint design of the UAVs' mobility/trajectory control together with the communication and computation resource allocation, which, however, may be particularly difficult as UAVs may act as aerial users or aerial edge servers or both.
In this section, we review machine learning methods for UAV trajectory and communications design, discuss computation offloading design for UAVs with MEC, and present distributed edge machine learning via UAVs, followed by some open research problems.
The joint design of the UAVs' movements over time (e.g., deployment locations and flight trajectories) and the communication resource allocation is crucial for optimizing the performance of 5G-and-beyond UAV communication networks. Conventionally, such joint design is implemented in an offline manner based on model-driven optimization approaches, where the network state information (such as the locations of the communication nodes, the CSI, and the service request information) is assumed to be (perfectly or partially) known prior to the optimization. In this case, the joint design can be formulated as a deterministic optimization problem that is generally solvable via convex and non-convex optimization techniques. Such offline designs, however, may not work well in practice, due to the time-and spatial-varying traffic demands, the user mobility, and the complicated channel propagation environments introduced by the UAVs. To tackle this challenge, researchers have shifted from conventional model-driven approaches to alternative data-driven approaches by exploiting emerging machine learning techniques.
In general, machine learning can be classified into three categories, namely supervised learning, unsupervised learning, and RL. RL is particularly useful for the joint UAV movement and communication design. To be specific, RL optimizes the actions of one or more agents in an environment to maximize the cumulative reward over a certain time horizon [259] . Supposing that UAVs are the agents of interest, then RL can be efficiently utilized in UAV communication networks for rapidly adapting to the dynamic environment [260] , by properly choosing the UAVs' actions (e.g., deployment/trajectory design and resource allocations) and the reward functions (e.g., communication rate). In the literature, there are two lines of research that exploit RL for optimizing the UAVs' operation in UAV-assisted communication systems [260] - [267] and cellular-connected UAVs [268] , [269] , when the UAVs serve as BSs and users, respectively, as elaborated in the following.
1) UAVs as BSs: First, consider the case where one single UAV serves as an aerial BS for UAV-assisted wireless communications and needs to simultaneously serve multiple users on the ground. In this case, the UAV needs to find the optimal flight trajectory maximizing the performance, while taking into account the wireless channel conditions of all users. However, as obstacles (e.g., buildings or trees) are non-uniformly distributed over space, the UAV may possess LoS and non-LoS (NLoS) links with different ground users at different locations. In practice, as the obstacles' locations may not be known prior to the optimization, the LoS/NLoS property of the wireless channels becomes uncertain, thus making it infeasible to formulate the UAV trajectory design as an explicit optimization problem. Although prior works assumed probabilistic LoS channel models and accordingly used average channel power gains in their problem formulations [6] , such designs would lead to highly compromised solutions. In contrast, RL is an efficient model-free method, which can learn the environment on-the-fly based on the past experiences, and automatically generate optimized solutions. For instance, the authors in [261] assumed that one UAV communicates with multiple ground users over orthogonal channels, where the UAV aims to maximize the communication sum rate for all users by optimizing the trajectory. By applying Q-learning (a model-free RL technique), the UAV can autonomously learn its optimal trajectory for sum-rate maximization, without explicitly having the environment information. Furthermore, the authors in [262] studied the sum-rate maximization over a finite duration in a UAV-enabled uplink NOMA system for data collection, where the ground users may move dynamically over time. By assuming that the UAV only causally knows the users' (moving) locations and CSI, a Q-learning-based RL approach was proposed to enable the UAV to optimize its flying path in an online manner, where expert knowledge of well-established wireless channel models was exploited to initialize the Q-table values.
Besides the model-free design without any LoS/NLoS information in [261] , [262] , environment maps [263] , [270] , which contain the LoS/NLoS channel information over space, can be further exploited for more efficient trajectory design. For example, the authors in [263] considered the UAV-enabled data collection over a finite flight duration subject to obstacle avoidance constraints, for which a new end-to-end RL approach was proposed to efficiently control the UAV trajectory for enhancing the communication performance. For this setup, a double deep Q-network (DDQN) with combined experience replay was trained to learn the UAV control policy. By exploiting a multi-layer map of the environment, the DDQN is applicable for general scenarios when the system and channel parameters (such as the number of ground users, their locations, and the maximum flight duration) can vary.
Next, when multiple UAVs are deployed in a network for serving multiple users on the ground, it becomes important for the UAVs to jointly design their trajectories and communications to collaboratively enhance the network coverage and throughput performance. For instance, in [264] , a threestep machine learning based approach was proposed to maximize the satisfaction of the users, where the cell partition, the 3D deployments of multiple UAVs, and their dynamic movements are sequentially optimized via K-means and two Q-learning-based algorithms, respectively. Furthermore, decentralized deep RL based trajectory designs were developed to ensure coverage and user fairness while maximizing the energy efficiency of UAVs [265] and to minimize the ageof-information (AoI) when the UAVs execute sensing tasks through cooperative sensing and transmission [267] . Moreover, the authors in [266] considered the scenario of UAV-enabled caching, where a distributed algorithm based on the machine learning framework of liquid state machine (LSM) was developed to optimize the content caching and resource allocation decisions at the UAVs.
2) UAVs as Users: In cellular-connected UAV scenarios, UAVs act as aerial users in cellular networks to perform tasks like packet delivery and aerial inspection. In this scenario, how to maintain the cellular connection while addressing the task is the main concern. Normally, conventional designs optimize the UAV trajectories based on distance-dependent channel models under a fixed path loss exponent. These works, however, overlook the complicated air-ground channel propagation environments, and thus may lead to unexpected communication outage at the cellular BSs in coverage holes where the wireless channels of the ground BSs are blocked by obstacles such as high buildings. To tackle this challenge, using machine learning based designs for refining the UAV trajectories are emerging to maximize the connectivity probability during a task. For instance, in the case with one single UAV, the authors in [268] proposed a deep RL-based solution approach, namely a dueling double deep Q-network (dueling DDQN), to design the navigation/trajectory for optimally balancing between the UAV mission completion time versus the expected duration of communication outage. More specifically, this design exploits the dual use of the UAV's signal measurements not only for training the DQN, but also for creating a radio map to predict the outage probabilities at different locations, thus leading to an interesting framework termed simultaneous navigation and radio mapping (SNARM) that greatly improves the learning performance.
Besides the signal blockage due to obstacles in the communication path, the co-channel interference among nearby BSs is another issue that may affect the system performance. Recall that the signal path loss for UAV-ground channels is altitudedependent, i.e., when the UAV stays at a higher altitude, the LoS probability of the UAV-ground channels increases. Exploiting such properties, deep Q-learning was used in [269] for optimizing the height of the UAV for maximization of the spectral efficiency, via properly balancing between the desirable signal and harmful interference.
To enable edge intelligence applications with both UAVs and AI integrated, a large volume of data are to be generated at distributed edge devices (including both UAVs and conventional smart sensors and smart phones), which need to be properly processed via sophisticated AI training and inference algorithms. However, the implementation of these AI tasks is generally data-and computation-intensive, which cannot be handled locally by wireless devices themselves. As such, computation task offloading is an appealing solution to handle AI tasks, which allows the UAV edge servers to offload their AI tasks to MEC servers with high computation capabilities, and then download the computation results after MEC execution. Fig. 7(a) shows the scenario when a UAV acts as an aerial user or edge device in cellular networks, which has certain computation tasks to be executed via offloading to BSs on the ground. Besides edge devices, UAVs may also carry MEC servers to support the AI implementation of on-ground devices. As shown in Fig. 7(b) , the UAV can help the widely distributed devices on the ground to execute their computation-intensive AI tasks, especially in emergency situations. Under both scenarios, the joint UAV trajectory and communication/computation design is crucial. For facilitate such joint design, in this line of research the AI tasks are normally modeled as general computation tasks with certain data and computation requirements, as detailed in the following.
1) UAVs as Edge Devices: First, as shown in Fig. 7(a) , assume that a UAV user needs to fly from an initial to a final location (e.g., for packet delivery) and has certain computation tasks to be executed during the mission (e.g., processing of sensed data in real time for autonomous flight). During the flight, the UAV needs to offload its computation tasks to ground BSs/MEC servers, and download the computation results from the corresponding BSs after their remote execution. In this case, the UAV's trajectory design becomes a more complicated problem due to the distributed computation constraints at the BSs. For instance, a UAV may prefer to partition the computation tasks and offload them to different BSs to exploit their cooperative computation gain; and accordingly, the trajectory must be carefully design in order to meet the offloading-execution-downloading requirements of different BSs. It should be further noted that the UAV can only download the computation results from the same BS to which the tasks were offloaded, thus resulting in the coupling between (uplink) offloading and (downlink) downloading. This makes the trajectory design even more difficult, as the UAV may need to fly back and forth during offloading and downloading.
There have been various attempts to address the aforementioned problems. For example, the authors in [271] proposed a TDMA-based protocol for computation offloading, where the UAVs can offload their tasks to different BSs for parallel execution in orthogonal time slots, in order to fully exploit the computation resources available at the distributed BSs. In particular, the UAV's mission completion time is minimized by jointly optimizing the UAV trajectory and the computation offloading scheduling, subject to the UAV's flight constraints, and the ground BSs' (GBSs') computation capacity constraints. For illustration, Fig. 8 shows the optimized UAV trajectories for computation offloading for different computation loads (measured in computation task input bits L [271] ). There are four GBSs and the UAV's initial and final coordinates are (0,0) and (1000 m, 1000 m), respectively. It is observed that when the computation load is small (i.e., L = 100 Mbits), the UAV flies in a straight line from the initial to the final location. When L increases to 200 Mbits, the flight trajectory deviates from the straight line for more efficient offloading. For L = 600 Mbits, the UAV flies back and forth between different GBSs in order to exploit multiple GBSs' distributed computation resources more efficiently via time sharing. This phenomenon is significantly different from the UAV trajectory design for communications, where the UAV may successively visit different ground nodes, instead of flying back and forth for computation offloading because of the inherent distributed computation capacity constraints.
2) UAVs as Edge Servers: In another line of research, the UAVs can be utilized as MEC servers in the sky to remotely execute the computation tasks of ground devices, as shown in Fig. 7(b) , which may correspond to scenarios when the ground infrastructure is damaged or not available. As the UAV's limited communication and computation resources are shared among multiple ground devices, the UAV should carefully design the joint communication and computation scheduling, together with the trajectory design, to maximize the computation performance of ground devices in a fair and efficient manner [272] - [274] . Furthermore, the single-UAV-assisted computation offloading in Fig. 7 (b) has been extended to various other setups. For instance, the authors in [275] presented a UAV-relayassisted MEC system by integrating UAVs into the cooperative MEC design [276] , where the UAV not only acts as the MEC server for remote task execution, but also serves as a relay to help ground devices offload tasks to another GBS. In this system, the UAV exploits the joint communication and computation cooperation for computation performance enhancement. Moreover , the authors in [277] , [278] proposed UAV-enabled MEC wireless-powered systems by integrating the UAV with wireless powered MEC [258] , where the UAV serves as energy transmitter, communication transceiver, and MEC server. In this system, the UAV trajectory is optimized jointly with the energy/communication/computation resource allocations for maximization of the computation performance of the ground devices, subject to a new type of wireless energy harvesting constraint as the energy consumption for communication and computation at each device cannot exceed the energy harvested wirelessly from the UAV. In addition, security concerns during offloading from ground devices to the UAV were addressed in [279] , where physical layer security is employed to combat against potential eavesdropping from ground attackers.
There have also been a handful of works investigating UAVassisted computation offloading to multiple UAV-mounted MEC servers [280] , [281] . In this case, the association between the ground devices and the UAVs is an important new issue to be considered, together with UAV trajectory and resource allocation design [280] . Different from conventional designs focusing on communication only, the user association in the MEC context needs to properly balance both the communication and computation loads, thus making it more difficult. Furthermore, the self-interest is another issue for multi-UAV networks, especially when the UAVs may belong to different service providers [281] . To address this issue, coalition formation for UAVs can be investigated by using game theory and RL.
Moreover, researchers also investigated UAV swarm applications with stringent computation requirements. Towards this end, one efficient solution is to employ high-altitude UAVs with larger size as edge servers to support the swarm UAVs' computation [282] , while another solution is to use the edge and cloud infrastructures on the ground to support the swarm UAVs' extensive computation tasks [283] .
Besides computation offloading towards a central MEC server, edge devices such as UAVs can also collaboratively perform AI tasks using their locally distributed data and computation capabilities. This technique is generally referred to as distributed edge learning. Among other approaches, federated edge learning is particularly important due to its advantages in preserving data security and privacy [256] , where a group of edge devices (UAVs of interest) jointly use their distributed data to train common AI or machine learning models without data sharing. As illustrated in Fig. 9 (a) and Fig. 9 (b), the edge server can either be a BS on the ground (e.g., as with a cellularconnected UAV) or another UAV in the sky (e.g., as in a UAV swarm). Federated edge learning is implemented in an iterative manner: in each iteration, the UAVs first update their local AI models, and then aggregate at the edge server to update global AI models. The above process requires the UAVs and edge server to frequently exchange their AI model parameters, and iterate between communication and computation operations. Because of this fact and the UAVs' 3D mobility, how to jointly optimize their multiple UAVs' trajectories together with the communication and computation scheduling over time is the key issue to be tackled.
While federated edge learning is currently a very hot topic in both the wireless communications and machine learning societies, the research on UAV-enabled edge learning is still in its infancy. For instance, wireless resource allocation for enabling efficient UAV-enabled federated edge learning was studied in [284] . Federated learning for UAV swarms was investigated in [285] . Furthermore, using UAVs as edge servers for federated learning was proposed in [286] , and the support of Internet of vehicles was considered in [287] .
D. Future Research 1) Cloud-Edge-Device Collaboration with UAVs: As UAVs may perform as both aerial users and service (communication/computation/AI) providers, future networks need to be aerial-ground integrated, where computation resources are distributed everywhere across heterogeneous aerial and ground nodes with distinct communication and computation capabilities (see, e.g., [288] ). How to match the time-and spatial-varying communication/computation/AI demands with distributed communication/computation/data supplies in such highly dynamic 3D networks is a challenging task. 2) Over-the-Air Computation From Sky: Over-the-air computation (AirComp) is an emerging technique to enable functional computation over wireless channels by exploiting wireless signal superposition property of multiple access channels [289] - [293] . AirComp leads to a fully-integratedcommunication-computation design, and thus can have abundant applications in distributed sensing, computation, federated learning [294] , [295] , and UAV swarms. When UAVs are involved in AirComp as either edge devices or servers, how to exploit their mobility for optimizing the AirComp performance is an interesting problem that is uncharted yet.
In this paper, we provided a comprehensive overview on the integration of UAVs into 5G-and-beyond networks, which is expected to drive UAV verticals and cellular networks to a win-win situation by jointly exploiting their high altitude and flexible 3D movement together with the most advanced wireless technologies. In addition to reviewing the state-of-theart research from both the academic and industrial viewpoints, we focused on the most promising solutions of addressing the challenges in communication, sensing, and AI, and also exploring new opportunities in system design and implementation of UAV cellular networks. Promising directions and open problems in 3D wireless networks with UAVs were also highlighted for future research. As the research in this frontier paradigm is new and remains largely unexplored, it is hoped that this paper will serve as a timely and useful resource for researchers to push forward.