. A worker dressed in a protective suit is collecting goods from our Hercules logistic autonomous vehicle. There is no person-to-person contact during the process of goods transportation. This photo was taken in Shenzhen, Guangdong, China, February 2020 during the outbreak of COVID-19.
Since December 2019, the coronavirus disease 2019 (COVID-19) has spread rapidly across China. As at the date of writing this article, the disease has been globally reported in 100 countries, infected over 100,000 people and caused over 3,000 deaths. Avoiding person-to-person transmission is an effective approach to control and prevent the epidemic. However, many daily activities, such as logistics transporting goods in our daily life, inevitably involve person-to-person contact. To achieve contact-less goods transportation, using an autonomous logistic vehicle has become the preferred choice. This article presents Hercules, an autonomous logistic vehicle used for contact-less goods transportation during the outbreak of COVID-19. The vehicle is designed with autonomous navigation capability. We provide details on the hardware and software, as well as the algorithms to achieve autonomous navigation including perception, planning and control. This paper is accompanied by a demonstration video and a dataset, which are available here 1 .
I. HARDWARE SYSTEM The hardware system of our vehicle mainly consists of a Drive-by-Wire (DBW) chassis, an Industrial Personal Computer (IPC), sensors, and accessories. Fig. 2(a) shows the * Equal Contribution 1 https://sites.google.com/view/contact-less-transportation sensors on the 3-D model of our vehicle. Fig. 2 (b) shows the top view of the modules in the chassis. The main modules of the chassis are those related to motion control: Motor Control Unit (MCU), Electric Power Steering (EPS), Electro-Hydraulic Brake (EHB) and Electronic Parking Brake (EPB). MCU supports both the speed-control and torque-control, with the input commands from the Vehicle Control Unit (VCU). EPS controls the vehicle steering. The combination of MCU and EPS controls the vehicle longitudinal and lateral motions. EHB controls the brake, and EPB controls the parking brake. The Body Control Module (BCM) controls various electronic accessories in the vehicle body, such as light and horn. VCU is the core of the DBW chassis, which is responsible for coordinating each module. It keeps communicating with the IPC, performing parameter checking and sending commands to other modules. In addition, VCU responds to critical safety signals, such as the stopping signal from the emergency button. There are some sensors in the chassis, such as a bumper, encoder and Tire Pressure Monitoring System (TPMS). The bumper and TPMS are both safety-critical sensors. Specifically, the bumper is used to avoid collisions. The encoder is a low-cost and effective sensor for odometry which measures the rotation of wheel precisely.
There are two batteries in our vehicle. A starter battery and a 72 V removable lithium-ion battery with around 9 KWh electricity, which can support the maximum 80 Km running distance. The lithium-ion battery powers the chassis, IPC, sensors and accessories. It has a built-in Battery Management System (BMS) to monitor and manage the battery. The removable design allows the vehicle to operate at 24 hours a day without stopping for a recharge. An On-Board Charger (OBC) with a Direct Current Direct Current (DCDC) converter takes about 5 hours to fully recharge the battery. The detailed specifications of our vehicle are displayed in Tab. I.
The IPC contains an Intel i7-8700 CPU with 6 cores and 12 threads, 32 GB memory, and a 1050Ti NVIDIA Graphics card. It is able to run deep learning-based algorithms. The accessories mainly consist of a 4G/5G Data Transfer Unit (DTU), Human Machine Interface (HMI), LED display, remote controller and cargo box. The DTU allows the IPC to connect to our cloud management platform via the Internet. The cargo box is replaceable. Fig. 3 shows the software architecture of our autonomous vehicle. It can be generally divided into two parts: software system running on the vehicle, and software system running on the cloud.
There are three main computing platforms on the vehicle: the IPC, Electronic Control Unit (ECU) and BCM. The IPC is used to run algorithms for autonomous navigation. The ECU is used to ensure the safety of the vehicle by energy management and safety diagnosis. It has three main functions: the united diagnostic services, driving torque coordination control, and the distribution and brake energy recovery system. The united diagnostic services are a series of applications checking and monitoring the mechanical and control systems. The driving torque coordination control is an application that dynamically distributes motor torque according to the vehicle and road status. The brake energy recovery is an application aimed at saving power and improving the travel distance. These applications run on the Real-Time Operating System (RTOS), which satisfies the real-time requirements.
The BCM connects the IPC and ECU. It also runs on the RTOS, which meets the real-time requirements. It detects the communication between the nodes of the CAN network by heartbeat protocols. When major nodes of this network experience an outage or crash, the BCM stops transmitting high-level CAN signals from the IPC to the VCU and waits for human interventions.
The sensors on the vehicle are synchronized by a 1 Hz Pulse per Second (PPS) signal from the external GNSS receiver. The IPC receives data from the sensors and uses them at different frequencies. For example, the state estimation module updates at 100 Hz, providing localization results for the system. The Lidar object detection module runs at 10 Hz according to the refresh rate of the Lidar. All the modules are developed on the Robot Operating System (ROS) to facilitate data communication.
The software platforms on the cloud mainly include the map server, the scheduling server and the log and simulation server. The map server stores the pre-built map for the traversed environments. The scheduling server performs the task allocations and collects the status of every registered running vehicle. It also plays the role of accessing the map data for routing, transmitting sensor data into the map server, recording the key information into the log server, and replaying the data recorded for a good trace-back. The log and simulation server runs the end-to-end simulator Carla 2 and 51Sim-One 3 . The clock synchronization between the platforms on the vehicle and cloud is manipulated based on the network time through the Network Time Protocol (NTP).
Perception serves as the fundamental component of autonomous navigation. It provides necessary information for planning and control. This section describes some of the key perception technologies used in our vehicle. 
The 3-D object detection aims to recognize and classify objects, as well as estimate their poses with respect to a specific coordinate system. We use multiple Lidars for object detection. To exploit the information from multiple Lidars, it is important to calibrate them. To this end, we propose a marker-based approach [1] for automatic calibration without any additional sensors and human intervention. We assume that three linearly independent planar surfaces forming a wall corner shape are provided as the calibration targets, ensuring that the geometric constraints are sufficient to calibrate each pair of Lidars. After matching the corresponding planar surfaces, our method can successfully recover the unknown extrinsic parameters with two steps: a closed-form solution for initialization based on the Kabsch algorithm [2] and a plane-to-plane iterative closest point (ICP) [3] for refinement.
The overview of our 3-D object detection is shown in Fig. 4 . The inputs to our approach are multiple point clouds captured by different Lidars. We adopt an early-fusion scheme to fuse the data from multiple calibrated Lidars at the input stage. With the assumption that the Lidars are synchronized, we transform the raw point clouds captured by all the Lidars into the base frame, and then feed the fused point clouds into the 3-D object detector. The final output is a series of 3-D bounding boxes. Each bounding box b ∈ B is parameterized as [c, x, y, z, w, l, h, γ], where c is the class of a bounding box, and [x, y, z] denotes the bottom center of a box. We employ VoxelNet [4] as the 3D object detector, which consists of three main components: a Feature Learning Network (FLN) for feature extraction; Middle Layers (ML) for feature embedding with sparse convolution; and a Region Proposal Network (RPN) for box prediction and regression. Fig. 5 . The schematic diagram of our 3-D point-cloud mapping system. After the initialization, the system will estimate the states and refine the global map and Lidar poses in the odometry and mapping sub-modules, respectively.
The FLN encodes the sparse points into a dense feature representation. In [4] , the authors first voxelize the input point clouds and then apply a bunch of Voxel Feature Encoding layers (VFELayers). The VFELayer is composed of multi-layer perceptrons and a max-pooling layer. In the max-pooling layer, the point-wise features inside a voxel are aggregated into voxel-wise features, which will be forwarded to the ML for feature embedding.
2) Middle Layers: The ML converts the 3-D dense grid to a 2-D Bird's-Eye-View (BEV) representation. To process the grids, the ML is implemented with several 3-D convolution layers, where the 3-D convolutional operations, BN layer, and ReLU layer are applied sequentially. The convolution kernel size is set as [d, 1, 1], since we denote the dense grids in the order of z, x, y. As a result, these layers can aggregate voxelwise features within a progressively expanding receptive field along the z-axis, as well as enrich the shape description.
In [4] , the ML costs almost 50% of the computation of the entire network due to the dense 3-D convolutional operations. The non-empty voxels usually occupy ≤ 1% of all grids. However, in the process of dense convolution, calculations are required on each voxel. Inspired by [5] , we utilize the spatially sparse convolution [6] to replace the dense convolution layers in the basic ML. Hence, the inference time is boosted from approximately 250 ms to 56 ms even with a larger sampling resolution. Additionally, we add the DropBlock [7] to randomly deactivates patches of the feature map during training. This forces the network to search for more evidence to detect an object with several invisible parts.
3) Region Proposal Network: The RPN applies convolutional layers to predict the probability and regression map from the extracted features. Since the increasing invariance and large receptive fields of top-level nodes yield smooth responses and cause inaccurate localization, the RPN adopts skip-layers [8] to combine high-level semantic features with low-level spatial features [4] . Each entry of the probability map estimates the probability of the objectiveness, and one entry in the regression map regresses the pose and size of a bounding box. We jointly estimate (sin θ , cos θ ) instead of θ to avoid the angular singularity problem described in [9] .
To train the network, we adopt the classification and regression loss [10] to train the stage-2 network: Fig. 6 . The schematic diagram of our moving-object detection system. The white point cloud and the blue point cloud represent the detected objects in the current frame and the last frames, respectively.
where p i represents the posterior probability of objectiveness, and u i and u * i ∈ R 7 is the regression output and ground truth for the positive anchors. L cls is defined as the classification loss, f cls (·) denotes the focal loss [11] , and L reg is defined as the normalized regression loss where we use the SmoothL1 function to calculate the loss f reg (·).
The 3-D point-cloud mapping aims to build the 3-D map of the traversed environments. Fig. 5 shows the diagram of our mapping system. The inputs to the system are the raw data from the IMU and 3-D Lidar (i.e., accelerometer and gyroscope readings from the IMU and point clouds from the 3-D Lidar). The system starts with an adapted initialization procedure, followed by two major sub-modules: the Lidarinertial odometry and rotationally constrained mapping. Since the vehicle usually remains still at the beginning of mapping, we do not need to excite the IMU to initialize the module as described in [12] , which is more suitable for hand-held applications. With the stationary IMU readings, the initial orientation for the first body frame F B 0 can be obtained by aligning the average of the IMU accelerations to the opposite of the gravity direction in the world frame F W . The initial velocity v B 0 , and IMU biases b a 0 , b g 0 are set to zero. The state estimation problem in the Lidar-inertial odometry sub-module is aimed at optimally fusing Lidar and IMU measurements in a local window. The state X in our odometry contains the body pose [13] , Lidar features are extracted from each sweep. UsingT W L i , we can associate feature points to these in the local map and form the relative Lidar measurements according to [12] . With the pre-integration and relative Lidar measurements, we can write the state estimation problems in a fixed-size window of N as:
where r P (X) is the prior term from marginalizing out old states, r L (m, X) is the relative Lidar measurement, and r B (z β β +1 , X) is the pre-integration term. The optimized X can be obtained by solving (2), using the Levenberg-Marquart algorithm. After the optimization, the state for propagation and local map can be updated. In our odometry sub-module, the window-size is used to limit the number of parameters. Although the odometry sub-module can make the system align with gravity well, it lacks a larger global map to constrain the local map. Thus, after a long time, the system may drift by using only the local constraints.
To reduce the drift, we will refine the undistorted point cloud and Lidar pose to an incremental global map. Following the normal mapping module from [13] , we can write the global map problem as
By registering a point cloud to the global map, we can achieve a more consistent map than using only local window Lidarinertial odometry. However, if we only use the pose from the odometry as a prior to minimize 3, we would waste the property of the orientation of the Lidar-inertial odometry that is aware of the gravity direction. Therefore, we propose rotationally constrained mapping. At the optimization, we will apply a modified Jacobian J C θ θ θ z according to the confidence of the orientation estimation and update the Lidar pose as stated by [12] .
Moving objects are very common in traffic environments. Detection of moving objects is crucial for the motion planning of our autonomous vehicle in dynamic environments [14] . Based on the aforementioned object detection and odometry estimation, we propose an ego-motion compensated movingobject detection system. The schematic diagram is shown in Fig. 6 . The input to our moving-object detection system is consecutive point-cloud frames with object detection results including both moving and static ones. We first use the aforementioned Lidar-inertial odometry estimation algorithm to find the transformation between the two frames. Then, the last point-cloud frame is transformed to the current coordinate frame with the transformation matrix. We use object tracking to associate the bounding boxes across the two frames. The moving objects are detected by subtracting the corresponding bounding boxes from the warped and current point-cloud frames. Specifically, we set a threshold for the distance between the bounding boxes. Moving objects are detected with distances above the threshold. As shown in Fig. 6 , there are three detected objects: a cyclist and two cars. With ego-motion compensation and frame differencing, a moving cyclist and a moving vehicle are detected.
IV. PLANNING Planning enables autonomous vehicles to acquire future paths or motions towards the destination. The planning for autonomous driving is challenging, because traffic environments are usually with dynamic objects, bringing about risks and uncertainties [15] . Autonomous vehicles are required to interact with various road participants, including cars, motorcycles, bicycles, pedestrians, etc. The planner needs to meet the vital requirements of safety, and the kinematic and dynamic constraints of vehicles, as well as the traffic rules. To satisfy these requirements, our planning is hierarchically divided into four layers: route planning, behavioral planning, local path planning, and motion planning. Fig. 7 shows the four-layer planning process.
The route planning aims at finding the global path from the global map. For autonomous driving, the route planner typically plans a route given the road network. For structured environments with clear road maps, we can use path planning algorithms, such as A*, to find the route. However, in unstructured environments, such as an industrial estate or housing district, we collect the past drivable routes as the teaching process. Fig.8(a) shows the global route established inside an industrial estate. The red lines show the global path and the blue lines show the candidate path as an alternative.
Behavioral planning decides the manoeuvres for local navigation. It is a high-level representation of a sequence of vehicle motions. Typical manoeuvres are lane keeping and overtaking. This layer receives information from the global maps and finds the category of the local area to give specifications on path planning. For example, unstructured environments, like parking lots, have different requirements on planning. Given the road map and the localization of the ego-vehicle, features of the local area can be obtained. As shown in Fig. 7 , road information that indicates the road environment classification of the global path segments is helpful for behavioral planning. Furthermore, traffic information from traffic signs helps in making decisions. The road and traffic information together with the estimation of other moving agents allows the behavioral planner to follow or overtake the front car, or pull over the ego-vehicle.
The local path planning generates a geometric path from the starting pose to the goal pose for the vehicle to follow. The time complexity of this process increases with increased path length, so it is often limited to a local range to ensure realtime planning. The local path planner needs to tackle motion constraints of the vehicle to generate collision-free paths that conform to the lane boundaries and traffic rules. Fig. 7 shows the online local path planning for driving on standard roads. Here we plan the path in the Frenet coordinate system. With the global path as the reference path, it defines the lateral shift to the path and the distance travelled along the path from the start position. We drew multiple samples with different speeds and lateral offsets. Then a graph search method is adopted to search the path with the minimum cost. To define the cost of the coordinates of each curve, we take into consideration the quality of the curve, ending offset to the global path, and other factors (e.g., the potential trajectories of other agents).
Given the planned path, motion planning is the final layer which optimizes the trajectory with dynamic constraints from the vehicle, the requirements for comfort and energy consumption. The planned trajectory specifies the velocity and acceleration of the vehicle at different timestamps, so it is also called trajectory planning. Though the path planned in the Frenet frame contains speed information, the dynamic constraint of the vehicle is not yet considered. Besides this, the local planning process is time-consuming and has a low update rate, which is inadequate to handle dynamic obstacles and emergency cases. The motion planner optimizes the trajectory given the information of obstacles, the constraints from the vehicle, and the path from the local path planner. It outputs the final trajectories for the controller to follow at a much higher updating rate to ensure safety. Fig. 8(b) shows the planned trajectory as yellow curves. The planned path is shown as the green trace. The planned trajectory differs from the path in the first segment, which is critical for collision avoidance. The motion planner shows more strict requirements for safe distances to ensure safety.
The main task of vehicle control is to track the planned trajectory. In the past decade, many trajectory tracking controllers have been developed, among which the Model Predictive Controller (MPC) [16] is the most popular one. The schematic diagram of our controller is shown in Fig. 9 .
As we can see, there are two inputs to the trajectory tracking controller. One is the trajectory s(t), which includes the information (e.g., desired coordinates, curvatures, speed) from the motion planner, the other is the feedback information x (t) from the state estimator. Sometimes, sensor feedback from the chassis cannot be directly sent to the controller, or more feedback quantities are required by the controller, which is difficult to obtain from sensors. In such cases, a state feedback estimator is required but is not a must. In Fig. 9 , the output of the trajectory tracking controller u(t) is sent to the chassis after being processed by a lower controller. The lower controller can work for many purposes. For example, our autonomous vehicle can work in both the autonomous-driving mode and the parallel-driving model (i.e., the remote control mode). The trajectory tracking controller only functions in the autonomous-driving mode, which means that only in the autonomous-driving mode the lower controller takes as input u(t).
The unmanned-car control can be divided into the lateral control, which controls steer angles, and the longitudinal control, which controls the car speed. There are two types of MPCs in the area of the autonomous vehicle. One is kinematics-based while the other is dynamics-based. A kinematics-based MPC is a combined controller that integrates the lateral control and longitudinal control. Therefore, the longitudinal PID controller highlighted in the dashed box in Fig. 9 is not required. The vector of two control quantities u(t) (i.e., steer angle and speed), will be directly given by the MPC. However, the dynamics-based MPC is a standalone lateral controller of which the output is a control quantity of the steering angle. In such a case, a longitudinal PID controller that outputs the speed control quantity will be required. And the outputs of these two controllers constitute u(t).
No matter whether it is kinematic or dynamic, an MPC is essentially composed of three key parts: the predictive model, rolling optimization, and feedback correction, as shown in Fig. 9 . The main function of the predictive model is to predict the future output of the controlled plant based on the simplified plant model, historical information and future input. The goal of rolling optimization is to determine the final control quantities through the optimization of an objective function. Rolling optimization must be repeatedly carried out in real-time. Feedback information is used to correct the predictive results based on the car feedback information so as to process rolling optimization in the next period. The major difference between the kinematics-based MPC and dynamicsbased MPC is the simplified vehicle model. The vehicle kinematics model is totally geometry-based, which takes the vehicle as a two-wheeled bicycle model, namely the Ackerman model. It applies while the vehicle is operated at low speed. The dynamic-based MPC is much more accurate than the kinematic-based MPC, and is derived from force analysis. However, the model complexities increase as well, leading to a more complicated optimization process and reduced real-time performance.
This section describes the real tasks of contact-less goods transportation using our vehicle during the outbreak of COVID-19 in China. From Feb. 2, 2020 to Mar. 7, 2020, our vehicles were deployed in 3 different places (Zibo, Shandong; Suzhou, Jiangsu; Shenzhen, Guangdong) in China. The total running distances of each vehicle reached 1100 Km. The details of the transportation tasks are summarized in Tab. II. Selected demonstration photos during the tasks are displayed in Fig. 10 . Note that in case of failures during the autonomous navigation, such as unavoidable accidents and system errors, we build a parallel driving system for our vehicle to back up our navigation. The parallel driving system is a remote control system based on 4G/5G technology. The vehicle control is immediately taken over by a human driver in case of any failure. We expect less human intervention during our goods transportation tasks. The vehicle is evaluated by the number of occurrences of human interventions.
We first deploy our vehicle in Zibo, Shandong due to the urgent requirement of a local logistics company. The tasks are to deliver fresh vegetables from a supplier to quarantined residential areas. There are two routes for the tasks, both of which have good traffic conditions (e.g., very few cars running on the roads). As the tasks require the vehicle loaded with goods that approximately approach the maximum payload, such as 960 KG. To ensure safety, the vehicle speed is limited to 20 Km/h. There is no human intervention during the tasks.
The tasks in Suzhou are divided into two parts. One task is to deliver lunch meals from a restaurant to a company. The vehicle receives 100 boxes of meals from the restaurant, and delivers them to the building where the company is located. As shown in the route map, the vehicle needs to take two U-turns and one left turn. The second task is to disinfect a road in a residential district every night. We mount a sprayer on our cargo box so that the vehicle can be used to disinfect traversed areas. The traversed route is approximately 600 m long, but the environment is very crowded (e.g., vehicles and bicycles parked along the road). In these two tasks, few human interventions are involved to give way to other vehicles to avoid accidents or go through areas crowded with obstacles.
Similar to the tasks in Suzhou, the tasks in Shenzhen are to deliver lunch meals, but the routes in Shenzhen have heavy traffic. So human intervention is frequently involved. We classify the human interventions into 4 levels: Z1 to Z4, the definitions of which are shown in Tab. III. The Z1 level indicates the most serious case. Without human intervention, there will be an accident for Z1. The seriousness decreases from Z1 to Z4.
As shown in Tab. III, Z1 never occurred during the tasks in Shenzhen. There are 5 occurrences of Z2, which are caused by the initialization failure of the Lidar-inertial-based localization and the false detections of traffic lights at night. There are 25 occurrences of Z3. The main reason is that there are so many road users around the vehicle that it stops to avoid collisions. There are 2 occurrences of Z4, both of which happen on very narrow roads. On such roads, even experienced human drivers cannot go through quickly.
For object detection, we found that real-time performance deserves much more attention than accuracy in practice. The perception algorithms should be efficient since they are always the front-end of the entire autonomous navigation system. Therefore, we replaced the dense convolution layers with spatially sparse convolution [6] in our 3D object detection module. As a result, the inference time is boosted from 250 ms approximately to 56 ms.
For the point-cloud mapping, we found that our system was capable of dealing with the rapid motion of the vehicle and short-term point-cloud occlusions. Since most of the Lidars are mounted parallel to the ground and the vehicles always move along the ground, the typical ring structure of the point clouds makes the system difficult to observe in terms of translational movements vertical to the ground plane. Drift in this direction is inevitable during long-time operations. In practice, we learned that the GPS localization results signaled the potential loop, which led to a consistent larger-region map for Lidar-based localization. In very crowded dynamic traffic environments, the system could be degraded by the disturbances from moving objects. To tackle this issue, we use semantic segmentation to remove movable objects to get clear point-cloud data.
For the planning part, we found the four-layer hierarchical planning necessary and effective. The application environments of our vehicle are complex in the sense of road structures, traffic conditions, driving etiquette, etc. Layer-wise planning makes the system extensible for multiple environments and various requirements. Furthermore, it is essential to attach importance to the uncertainty from the perception modules. The uncertainty comes from the limited accuracy of the perception system as well as the time delay in processing. For the planning, we avoid hitting the critical conditions and leave safe distances for the planned trajectory.
For the control part, we found that the greatest advantage of using an MPC can be gained by adding multiple con-straints in the control process. When the vehicle operates at low speed, the kinematic constraints restrain the vehicle motion planning and control. But with the increment of speed, dynamic characteristics become more influential. As aforementioned, the dynamic-based MPC is much more accurate than the kinematic-based MPC since the predictive model is more accurate. However, we found that the complexmodel prediction is not the best option. With regards to low-and middle-speed operation environments, a simplified predictive model with multiple constraints would be sufficient. As is known, the introduction of multiple constraints in the optimization may narrow down the range of solutions and even lead to no solution, which may corrupt the program. In such a case, it would be wise to bring in the relaxing factor when designing the objective function that could avoid the nosolution situation. Another factor that needs to be considered carefully when applying the MPC is the determination of the prediction horizon. A long prediction horizon can improve the control accuracy to a certain degree, but would consume more computation power and reduce the real-time performance. Different controlled plants in different operation environments may have different requirements on the control frequency. With the guarantee of the control frequency, our strategy is to select a prediction horizon that is as long as possible.
The successful daily operations demonstrated that using our autonomous logistic vehicle could effectively avoid virus spread due to human contact. It effectively provides a physical wall between the recipient and sender in the process of goods transportation. For example, one task is to deliver meals to quarantined persons suspected. Our vehicle automatically drives to the infected area and notifies the persons to collect the food through short messages. There is no contact between the persons sending the food and the quarantined during the whole process, which successfully avoid virus spread through humanto-human contact. Currently, there is a huge demand for contact-less goods transportation in many infected areas. We believe that continuous operations could extensively improve our vehicle and speed up the maturity of our autonomousnavigation technology. 