Knowledge extraction from aeronautical messages ( NOTAM s) with self-supervised language models for aircraft pilots (2022)

Abstract: During their pre-flight briefings, aircraft pilots must analyse a long list of NoTAMs (NOtice To AirMen) indicating potential hazards along the flight route, sometimes up to pages for long-haul flights. NOTAM free-text fields typically have a very special phrasing, with lots of acronyms and domain-specific vocabulary, which makes it differ significantly from standard English. In this paper, we pretrain language models derived from BERT on circa 1 million unlabeled NOTAMs and reuse the learnt representations on three downstream tasks valuable for pilots: criticality prediction, named entity recognition and translation into a structured language called Airlang. This self-supervised approach, where smaller amounts of labeled data are enough for task-specific fine-tuning, is well suited in the aeronautical context since expert annotations are expensive and time-consuming. We present evaluation scores across the tasks showing a high potential for an operational usability of such models (by pilots, airlines or service providers), which is a first to the best of our knowledge.

## introduction each upcoming flight requires a preparation phase for the crew. during this phase, the pilots check all the elements concerning the flight, being the meteorological conditions, fuel supply, or safety related notifications. pilots receive these notifications from the aviation authorities under the form of small text messages, called notice to airmen (notam). it represents a large number of messages to read and process before the flight, sometimes up to 100 pages for long-haul flights, which translates to a long analysis time. the messages are mostly, but not only, written in the english language. however, the phrasing being very special, with a lot of acronyms, technical words and without the usual grammar and syntax rules, the language differs from standard english. in this paper, we aim to apply the latest advances in natural language processing (nlp) to the aeronautical field leading to more autonomy and less overhead for the pilots. in particular, the use of language models like bert enables leveraging lots of unlabeled data for pretraining and fine-tuning on downstream tasks with limited amounts of labeled data. our main contribution is to present a knowledge extraction pipeline adapted to the aeronautical context. we introduce a language model trained from scratch on a large amount of raw notams, followed by three downstream tasks: criticality prediction, named entity recognition and translation into a structured language called airlang. this paper is organised as followed: in section 2, the notams are detailed both on the operational and the linguistic side, the problem is defined in section 3 with a focus on the three downstream tasks. in section 4, after a brief reminder of the state of the art, we present our approaches and the results of our experiments. ## notams in aeronautical context a notam is a message filled by aviation authorities to alert pilots about potential hazards along a flight route or at a location that could affect the flight. the message can inform about temporary disruptions (from a few hours to one year maximum) on aeronautical infrastructures (for example, closure or limited usage of runway or taxiway in a given airport), about inoperable radio navigational aids, military exercises with resulting airspace restrictions, temporary erections of obstacles near airfields (e.g. cranes), passage of flocks of birds through airspace, etc. an example of a notam message is shown in figure 1 ; more details about the fields can be found in appendix a. ## conclusion and perspectives in this work, we presented the use of modern selfsupervised language models (derived from bert) to extract knowledge from notam aeronautical messages. we showed that a single deep learning model pretrained on circa 1 million unlabelled notams can be efficiently reused on downstream tasks with dedicated fine-tuning. notam criticality prediction can support pilots during their pre-flight briefing by highlighting the most important messages. furthermore, named entity recognition can be applied to extract relevant parts of no-tams (e.g. closed runways/taxiways, specific conditions/exceptions...). finally, automatic translation to a domain-specific structured language (air-lang) used by pilot apps during flight, can support operational teams providing services to airlines. the evaluation scores on these tasks show a high potential for an operational usability of such models (by pilots, airlines or service providers), which is a first to the best of our knowledge. in the future, alternative nlp methods such as summarization for notams could be explored to continue reducing pilots' workload. while the use of deep learning networks (and pretrained language models) enabled increasing accuracy in a lot of nlp downstream tasks, they are known to be overconfident in their predictions. it is an issue in the aviation context given its safety-critical nature, where trust in systems' predictions is key. in that respect, uncertainty quantification methodssuch as conformal predictions (vovk v. and shafer, 2005)(angelopoulos and bates, 2021) -could give a reliable measure of confidence in model's outputs. the robustness of the model could also be assessed through adversarial attacks, as in @xcite . finally, formal methods could be used for verification and could pave the way to the certification of such deep learning models, required for any use on board.