A Copy Mechanism for Handling Knowledge Base Elements in SPARQL Neural Machine Translation (2022)

Abstract: Neural Machine Translation (NMT) models from English to SPARQL are a promising development for SPARQL query generation. However, current architectures are unable to integrate the knowledge base (KB) schema and handle questions on knowledge resources, classes, and properties unseen during training, rendering them unusable outside the scope of topics covered in the training set. Inspired by the performance gains in natural language processing tasks, we propose to integrate a copy mechanism for neural SPARQL query generation as a way to tackle this issue. We illustrate our proposal by adding a copy layer and a dynamic knowledge base vocabulary to two Seq2Seq architectures (CNNs and Transformers). This layer makes the models copy KB elements directly from the questions, instead of generating them. We evaluate our approach on state-of-the-art datasets, including datasets referencing unknown KB elements and measure the accuracy of the copy-augmented architectures. Our results show a considerable increase in performance on all datasets compared to non-copy architectures.

## introduction the semantic web organizes concepts in optimized, machine-readable, knowledge bases (kb) (or knowledge graphs). still, as these knowledge bases are not immediately designed with a human user in mind, the sparql protocol and rdf query language (sparql) is hardly accessible to laypeople with little-to-no knowledge of programming languages. this creates a strong accessibility bias, as it prevents users from accessing sizeable amounts of information because of their lack of a specific skillset. one way to bypass any need for prior knowledge is by allowing the users to query kbs using natural language questions. figure 1 illustrates the task at hand. more specifically, using neural q: what is villa la mauresque ? select ?a where { dbr:villa_la_mauresque dbo:abstract ?a } a: the villa la mauresque is located in cap ferrat (alpes-maritimes) and was remodeled in 1927 ... machine translation (nmt) to translate natural language questions to sparql queries has proven to be an interesting avenue to solve this challenge, with bleu-score performances of more than 90% across multiple datasets @xcite . however, behind these high-performing architectures are models that rarely return the correct answer to a question about a topic they have never seen in training, even if the information is available in the kb. as a single wrong answer can negatively affect the user's trust in the model, this limitation becomes a critical downfall for an automatic sparql query generation model. the main goal of this paper is to propose a mechanism to effectively generate accurate sparql queries. in particular, we aim at handling out-of-vocabulary (oov) knowledge base elements at the schema level ( classes, properties) and the instance level. as such, we put forth the following research questions: our main contributions are as follows. each kb element has a unique uri, which is used to reference it in a sparql query and a label, which is their name in a natural language. if there is no label, we can generate one from the element's uri. ## results we trained and evaluated our implementation of the models using google colab gpus. we compare our results to those reported by @xcite , who train their model on hpc servers using the fairseq implementations of the cnns2s and transformer architectures. it is important to note that they report the peak performance while we report the average of three runs. this means that we expect slightly lower performances when reproducing their results. baseline architectures on original datasets. baseline architectures on tagged datasets. copy-augmented architectures. ## discussion in view of these results, it is clear that, given a working tagging mechanism, the use of a copyaugmented architecture is an excellent advantage for sparql nmt architectures as it allows them to handle kb elements not seen in training. furthermore, comparing the results with and without copy reported in there is however still room for improvement. some of the limitations of this research lie in the use of template-based entries. in its current state, our copy-augmented architecture depends on questions following specific templates. as shown by the results reported in we also need to improve the way oov datasets are generated to be able to get a representative accuracy metric that is not biased by false positives. to do so, we must ensure most -if not all -queries return a non-empty answer. finally, another limitation is that our copyaugmented models depend on tagged questions to reach their top performance. ## conclusion this paper determined that, coupled with a copyaugmented architecture, integrating the kb elements directly in the questions is sufficient for a sparl nmt model to handle oov kb elements and to obtain a significant increase in performance. these tagged datasets were used to train baseline and copy-augmented versions of the transformer and the convs2s architectures. using a copy layer, we report perfect performances on the monument dataset and the generated oov monument dataset. for lc-quad, we report an increase in bleu score of 20 points and an increase in answer accuracy of about 40 points. for dbnqa, our results show an increase in bleu score of 35 points on average, as well as an increase in answer accuracy of 40 points. our future work will involve the design of a neural tagging model and a joint tagging objective for our seq2seq models, as well as the comparison of our copy-augmented models with large pre-trained models and the use of these models as our encoders-decoders. notable models on which to test our methodology include t5 @xcite , bart @xcite and @xcite , as well as models that can generate code such as codex @xcite .