A Semantics of Spatial Expressions for interacting with unmanned aerial vehicles (2022)

Abstract: This paper describes an investigation of establishing communication between a quadro- tor and a human by means of qualitative spatial relations using speech recognition. It is based on a system capable to receive, interpret, process, act, transmit and execute the commands given. This system is composed of a quadrotor equipped with a GPS, IMU sensors and radio communication, and a computer acting as a ground station, that is capable of understanding and interpreting the received commands and correctly provide answers according to an underlying qualitative reasoning formalism. Tests were performed, whose results show that the error rate was less than five percent for vertical and radial dimensions, otherwise, in horizontal dimension, we had an error rate of almost ten percent.

## introduction unmanned aerial vehicles (uav) have been gaining popularity in recent years due to their potential for novel applications @xcite . one of the most well-known types of uavs is the quadrotor, owing to their fair cost-benefit and a large number of off-the-shelf programming tools available for application development. some potential current and future activities involving uavs include, for instance, mapping large areas @xcite , recording movie scenes @xcite , and search and rescue missions @xcite . one of the challenges for achieving a large-scale use of uav applications, however, comes from the need to make more natural the way humans interact with such vehicles, especially for the nonspecialised public @xcite . this issue justified the development of an area of research known as human-robot interaction (hri). hri aims to develop strategies for facilitating the interaction with robotic agents in various situations, such as teaching children, rehabilitation, housework and many others. however, hri is still to be considered in the context of uav applications. nevertheless, the most direct way to achieve a high level of communication and understanding between robots and humans is the vocal commands usage to transmit and answer the commands between these agents. when two people want to talk to each other in everyday situations, they rarely use quantitative information, especially when talking about space and its relations @xcite . for example, when we say to a child to catch something at a table we do not tell the distance in meters or the relative altitude, we just give the basic qualitative information, like if it's close or far, under the table or on it. this observation motivated the present investigation, which aim is to develop new methods of human-robot communication using qualitative information. in general terms, this work aims to bridge the gap in the communication between a human and a quadrotor using speech recognition and a qualitative way of interpreting commands. ideas from qualitative spatial reasoning @xcite will be used to provide the basis for this communication. ## related work the research reported in this paper is related to qualitative spatial reasoning (qsr) @xcite , which is a subfield of knowledge representation in ai that aims at the formalisation of spatial knowledge and the development of reasoning methods about this knowledge. in hri, qsr ideas used a probabilistic model of interactions @xcite based on the qualitative trajectory calculus (qtc) @xcite . in that work, the robotic agent had to interpret the space around it while making decisions to avoid collisions and interacting with a human operator using qualitative information. more recently, @xcite presents a multi-robot localisation system based on qualitative spatial information where a sensory-deprived robot was guided to a goal location by other robots by passing high-level spatial commands. although no human interaction was considered in @xcite , the system presented would be suitable for achieving a human level of representing spatial concepts, as it has the right combination of qualitative representation with probabilistic localisation. another relevant work where qtc relations were used to enable autonomous agents to make decisions and predict actions from other agents by using just qualitative information, was presented in @xcite . communication using spatial expressions was also considered with the introduction of a new formalism about qualitative location, named qualitative ego-sphere @xcite . the parameters of this formalism were obtained from human trials, and the resulting model was applied to two distinct situations: the first involved the information exchange between two robotic agents, and the second involved the interaction between a robotic agent and a human. as we shall see further in this paper, the qualitative ego-sphere model was used as the basis for the research reported here; however, this idea was extended in the present work by assuming a flying robot as the robotic agent interacting with a human. much work has been done recently on deep learning for speech recognition using large language models, such as bert @xcite , gpt @xcite among others @xcite . although these models show great accuracy in actual language interactions, the semantics of their language constructs is unclear. in contrast, there is a growing interest in the development of formal semantics for spatial expressions, providing a rigorous account for verbal communication @xcite @xcite . this work presents a preliminary application of these ideas in the context of human-robot interaction. ## experimental setup the quadrotor used in this research was an arducopter with an apm 1-2560 board, an imu board, a radio receiver, two xbee's for the telemetry, and a gps. the dimension of the vehicle is 64x64x18 cm. to use the concept of ego-sphere applied to this quadrotor, we have to consider that the uav was the point of view v. the dimensions of the quadrotor were very important for this development, as well as its actuation area, in order to define the qualitative model. in this context, the region at was considered as a 1m radius centred at v, because it is the shortest distance to avoid a collision that can be perceived by the uav gps system. similarly, near and far were considered as 5m and 10m respectively. the vertical discretisations of the sphere, upper, lower, below and under, received the values 5m, 2m, -2m, -5m, respectively. a new parameter was added to the latter category, the same command, as it was necessary to command the robot to stay at its current location. the final category, the horizontal location, was divided equally on the trigonometric circle so that each command would have 45 between adjacent regions in the circle. the first step of this research was to control the uav, and for that we used a range of existing software, such as the ground control station (gcs), radio transmitters, mobile apps, and others. we chose a gcs to control our system, as it can be installed on any computer and can have a wide range of peripherals attached to the system. the software used was the ardupilot mission planner foot_0 , and it has all the functions and tools needed for controlling this kind of drone. this software has a very large range of applications, such as support for the autonomous mission, control of all the optional hardware for this model and visual control of the basic functionality needed to fly. using this software it was not difficult to find the appropriate function to have direct control from the computer. all the commands sent to the quadrotor were in a specific message type using mavlink @xcite . the meaning of the prefix mav is micro air vehicle, which is a common element of a large variety of uav applications. mavlink protocol is a library used in several programming languages that contain functions to translate and send messages between the vehicle and the control station, in this case, a computer. thus, this library was a tool needed for the gcs, bringing standard protocol and portability to our code, making it possible to use the same code on other gcs or other software that used the same protocol. the idea of code portability was the main reason to use this protocol. this usage also made it possible to send flying commands using xbee, a radio transmitter/receiver module integrated into the quadrotor attached to the base system. the first attempt at developing the interface between the control station and the quadrotor was to emulate radio signals from the computer and send them as normal commands by the radio transmitter. however, that was not a good approach, as those radio commands were very specific and did not have any type of support for autonomous flight. an alternative was to control the drone from the specific autonomous commands available in the mission planner, and using these commands implies using the full platform of the ground station and all the functionalities present in this software also. to accomplish this task, the flight plan tab of the mission planner was modified to work with direct commands and not a specific mission, as originally designed. for that, it was necessary to include equations and functions about latitude and longitude coordinates. equations 1 and 2 describe how this information was used to determine the future trajectory points. in the equations above: altitude commands were sent directly by the mavlink protocol, using the data from the imu board, which contains a barometer, an accelerometer and a gyroscope; however, distance and direction were sent by latitude and longitude. after receiving gps signals and calculating the future point, we created the functions to control the ego-sphere commands, such as left, upper and near. being able to control the drone directly over the control station, without the radio controller, allowed the implementation of the voice recognition system. we adopted the microsoft speech library from visual studio @xcite as the basis for the voice recognition system, as this library allowed the processing of voice commands directly from the control station and sending commands to the drone without using the onboard computer in the drone. the speech recognition module worked well with our functions, serving as the interface between human users and the ground station. the commands listed in the the voice recognition had great precision without background noise, especially because every word processed by the system was approximated by a previously defined word in the user-defined vocabulary. however, if a spurious sound is similar to one of these words, it can be misclassified as a valid command. to avoid such recognition problems, we configured the library with a confidence precision of 85%, which reduced ambiguity drastically. a grammar class was used as a reference to the voice recognition module so that the application could use the language constraints in the recognition, which increased the hit rate of recognising commands. four grammatical rules were defined containing different commands categories: the first contained all the basic commands (ok plane, start the engines, stop etc); the second had the horizontal dimension of ego-sphere (left, front, right etc); the third was defined with vertical dimensions of ego-sphere (upper, lower, under and below) and; the last had the radial distance defined by ego-sphere (at, near and far). for our system to change the grammar at the appropriate time, we needed to establish an order of commands. the order was to call the horizontal references first, then the vertical and finally the distance, all according to the ego-sphere definitions. this order is described at figure 4 . ## results a flying test was necessary to check if the recognition accuracy would satisfy the project goals and if the tests were consistent when flying with the radio by using direct commands. we found that the autonomous flight had certain issues, such as the stabilisation that was not precise and problems with the altitude holding. so, although the code was entirely developed for the physical platform, the evaluation of the system developed was conducted in a simulated engine, called flight gear @xcite . the usage of flight gear gave us a virtual ambient that emulates real flight, so every sensor data was received with precision and every command was sent with minimum delay compared with a real, non-simulated, flight. the first test was conducted considering the basic commands, whereby we observed that commands with similar sounds, such as arm and disarm were not possible to be used on this application, because the similarity between these two words generated ambiguity in their recognition. so the major portion of the commands had to be changed to other words, which did not generate any kind of ambiguity. the final version of the basic commands was listed on testing ego-sphere commands took longer than testing the basic commands, due to the complexity of the theory and the number of different words to be recognised. subsequently, we divided the ego-sphere into two identical parts considering its symmetry, passing by the centre in a vertical cut, thus dividing the left side from the right side. for ease, just the left side was used in the experimental evaluation. after dividing the sphere, four test sessions were executed with thirty complete commands in each one of them, approximately. one complete command was composed of three ego-sphere commands, one of each dimension. every command given to the system was analysed according to the theory described before. thus, to consider the command successful, we needed to analyse each one of the categories separately. to neutralise the influence of the different combinations of words, ev-ery session had the same list of commands executed in a different order, embracing a large range of possibilities. the tests were made on different days with different noise rates, with about 75db of noise, composed of background voices and ambient sounds. that was made to maintain a realistic noise rate, representing the behaviour that could exist in real situations. in total, 131 complete commands were tested. a compilation of the results obtained for each category is shown in ## conclusion in this research, we bridged the gap between qualitative communication in an hri setting using voice commands and the qualitative ego sphere model as a basis of space qualitative information. the results showed the necessity of increasing the precision of our system, but also that our objective of simplifying the interaction between humans and robots has been achieved. one of the contributions that can be related to this study is the accessibility improvement of nonspecialist users to complex systems, like uavs -unmanned aerial vehicles. using the approach presented in this paper, everyone able to pronounce the correct sequence of commands is capable of controlling the system successfully, and all the work with stabilisation will be the responsibility of the autonomous system itself. another important contribution was facilitating the location requests to the quadrotor using quantitative information. in this case, for instance, the vehicle can be requested to go near or far the objective, using qualitative expressions, without the need of receiving the precise distance and coordinates of the goal location. this can be an advantage in emergency situations, where the answer time may be critical. the lower error ratio obtained in the tests suggests the efficacy of the method investigated in this paper, but also brings atop the discussion about the equipment used on the system. with more precise instruments, such as using infrared sensors to filter the overall results, and some improvements on the code we can develop a system more consistent and achieve a higher level of communication between humans and a robot.