DISCOSQA : A Knowledge Base Question Answering System for Space Debris based on Program Induction (2023)

Abstract: Space program agencies execute complex satellite operations that need to be supported by the technical knowledge contained in their extensive information systems. Knowledge Base (KB) databases are an effective way of storing and accessing such information to scale. In this work we present a system, developed for the European Space Agency, that can answer complex natural language queries, to support engineers in accessing the information contained in a KB that models the orbital space debris environment. Our system is based on a pipeline which first generates a program sketch from a natural language question, then specializes the sketch into a concrete query program with mentions of entities, attributes and relations, and finally executes the program against the database. This pipeline decomposition approach enables us to train the system by leveraging out-of-domain data and semi-synthetic data generated by GPT-3, thus reducing overfitting and shortcut learning even with limited amount of in-domain training data.

## introduction space debris are uncontrolled artificial objects in space that are left in orbit during either normal operations or due to malfunctions. collisions involving space debris can generate secondary debris which can cause more collisions, potentially leading to a runaway effect known as kessler syndrome @xcite , which in the worst-case scenario could make large ranges of orbits unusable for space operations for multiple generations. therefore, space agencies have established departments responsible for cataloging the space debris environment, which can be used for space traffic management, collision avoidance, re-entry ## dataset collection and use we used two sources of data for the study. the first one is the original dataset which first introduced the kopl-format for general-domain knowledgebase question answering (described in §3.1). the second one is for our particular domain with information about objects in space ( §3.2). we also describe in §3.3 how we further collected training data for fine-tuning a language model, and how we augmented this dataset with new question-answer pairs ( §3.4). ## methodology and model architecture we describe in §4.1 the problem that our model aims to solve. in §4.2, we describe what modification we applied to the methodology from @xcite . ## experiments we next describe our dataset ( §5.1), the way we trained our models ( §5.2) and our results ( §5.3). ## conclusion we developed a system for esa to address the challenge of answering complex natural language questions on their discos kb. the main obstacles included a lack of training data, the diversity and regular updates of the database entries, and the need for an economically feasible solution. the program transfer for complex kbqa methodology was selected for its potential to reduce the amount of required training samples through transfer learning and its capability to potentially gener-7 https://openai.com/blog/chatgpt alize for examples which were never seen during training. a data collection study was conducted with domain experts, which was then used to augment the data through leveraging the underlying ontology of the kb and prompting a large language model to generate fitting questions. the architecture was retrained with different domainspecific models and baselines to determine the benefits of using a domain-specific pre-trained encoder. although the results were mixed, the best performance was achieved by cosmicroberta, a pre-trained model on a space domain corpus. with an accuracy of over 90% of predicting the right entity on the validation set over the vast pool of candidate entities, the method demonstrates its strong ability to predict the correct input arguments for unseen examples. this is further demonstrated in the comparison with general-purpose models such as chatgpt or gpt-4, where our method achieved competitive results. therefore, this approach has the potential to be extended to other databases and query languages in the future, especially in scenarios where there are few to no training examples.