The Role of Output Vocabulary in T 2 T LM s for SPARQL Semantic Parsing (2023)

Abstract: In this work, we analyse the role of output vocabulary for text-to-text (T2T) models on the task of SPARQL semantic parsing. We perform experiments within the the context of knowledge graph question answering (KGQA), where the task is to convert questions in natural language to the SPARQL query language. We observe that the query vocabulary is distinct from human vocabulary. Language Models (LMs) are pre-dominantly trained for human language tasks, and hence, if the query vocabulary is replaced with a vocabulary more attuned to the LM tokenizer, the performance of models may improve. We carry out carefully selected vocabulary substitutions on the queries and find absolute gains in the range of 17% on the GrailQA dataset.

## introduction knowledge graph question answering (kgqa) is the task of finding answers to questions posed in natural language, using triples present in a kg. typically the following steps are followed in kgqa: 1) objects of interest in the natural language question are detected and linked to the kg in a step called entity linking. our focus in this work is the query building phase, henceforth referred to as kgqa semantic parsing. the motivation of our work stems from @xcite , where minor vocabulary substitutions to handle non-printable special characters for t5 @xcite produced better results on the task of sparql semantic parsing. in this work, we extend the idea and replace the entire sparql vocabulary with alternate vocabularies. as in @xcite , we replace certain special characters in the sparql vocabulary, such as { , } with textual identifiers, as t5 is known to have problems dealing with these special characters @xcite . we call this a masked query, and in this work, we test the ability of the models to generate this masked query, given the natural language question as input. a sample question, the original sparql query, and the corresponding masked query are as shown below (for the wikidata kg @xcite ) : is it true that an olympic-size swimming pool's operating temperature is equal to 22.4 ? ask where { wd:q2084454 wdt:p5066 ?obj filter(?@xmath0.4) } ask where ob ent0 rel0 ?obj filter ( ?@xmath1.4 ) cb in the era of pre-trained language models (lms) @xcite it is common practice to fine-tune models on custom downstream datasets. this requires supervised training which results in modification of weights of the models using some training algorithm. more recently, the technique of prompting of language models @xcite has been developed, which elicits the desired response from a lm through a task description and a few inputoutput examples. @xcite shows that such a strategy works better for larger models. it has however been observed that prompt design is brittle in behaviour and displays sensitivity to the exact phrase @xcite . a more recent innovation is that of prompt tuning @xcite , where the task-specific prompt is learnt on a smaller external neural network. the gradients are computed and flow through the lm, but leave the weights of the lm itself unchanged. instead, the weights of the prompt tuning network change and produce a custom and continuous prompt which produces the desirable response from the lm. a similar method is prefix tuning (li and liang, 2021), which is known to perform better for generation tasks @xcite . in this method, the original inputs and outputs are kept the same, but the input is pre-pended with a continuous prefix learnt in the external network. this prefix allows the model to understand the exact task to be performed by it. as primary contribution, in this work, we perform an analysis of how the complexity of output vocabularies affects the performance on the kgqa semantic parsing task for prefix and finetuned language models. code and data can be found at https://github.com/debayan/ sparql-vocab-substitution . ## related work a study of low-resource semantic parsing using prompt tuning was performed by @xcite on the top v2 @xcite and overnight @xcite datasets. prompt tuning, while not the same as prefix tuning, still keeps the lm weights frozen while the prompts are learnt on an external network. in their experiments, they perform a single kind of vocabulary substitution but find no noticeable performance improvements. no specific study is made of the change in performance with vocabularies of varying complexities, which is a task we undertake. another difference is that we perform experiments in the high-resource use case as opposed to low-resource. another work which is similar to ours is @xcite , where the authors experiment with prefix tuning on the task of semantic parsing, and find problems with non-standard vocabularies of logical forms. in their case, they work with the top v2 @xcite and pizza @xcite datasets. the keywords in those datasets consist of words joined by underscores (eg: in:get_reminder_data_time ), which poses a problem for the sub-word tokenizer of the transformer based models. they find that fine tuning a model on these datasets outperforms prefixtuning by a large margin. however, when they add the non-standard keywords to the tokenizer vocabulary and re-train the tokenizer to generate new embeddings for these keywords, fine tuning and prefix tuning perform at par. our work is different in a few respects: firstly, due to the specific research focus of our group, we experiment with a semantic parsing dataset for kgqa, namely grailqa @xcite . secondly, instead of retraining the tokenizer, we perform a simpler procedure of pre-processing the dataset by replacing the current vocabulary with a new vocabulary. we then train the models on this modified dataset, and as a post-processing step, substitute back the original vocabulary in place of the new vocabulary. ## prefix tuning prefix tuning prepends a set of tunable weights to every key-value pair in the transformer attention. the transformer attention is represented as follows: here h k and h v represent the key prefix and the value prefix respectively. following li and liang (2021) we model these prefixes using a two layer mlp as follows: ## models and experimental setup we carry out prefix-tuning and fine-tuning experiments with two versions of the t5 model: namely t5-small (60 million parameters) and t5-base (220 million parameters). questions are fed as input during training while masked sparql queries, as described in section 1, are provided as labels for supervision. for evaluation, we use the exact-match metric. a generated query is matched token by token, while ignoring white-spaces, to the gold query. the percentage of queries matched is reported. ## vocabulary the original vocabulary of the grailqa dataset consists of 48 words. the t5 tokenizer splits these words into 124 sub-words. this tokenizer specific vocabulary size (tsvs) is seen in the last column of we wish to see how a new output vocabulary affects performance, and as a result, we construct a set of special vocabularies and substitute them in-place of the original sparql vocabulary. with reference to the settings in ## datasets for our experiments, we require a dataset which contains a mapping of natural language questions to their corresponding logical forms and is large in size, since we test the high resource use-case. grailqa 4 is based on the freebase knowledge graph @xcite and consists of 64,331 questions designed to test three levels of generalisation, ie, i.i.d, compositional and zeroshot. for our purposes, we split the train set itself to three parts, since we are not interested in testing compositional generalisation aspects of the test set of this dataset. we are left with the following configuration: test: 8868, dev: 4434, train: 31035. (a) figure 2: fine-tuning accuracy drop is more gradual when compared to prefix tuning, and the performance of t5-small and t5-base are similar. @xmath2, @xmath3://dki-lab.github.io/grailqa/ ## analysis as seen in this finding is different from that of @xcite , who find their in-vocab setting performing no better overall. they attribute it to the substitutions possibly masking the meanings of the intents, for their given dataset. on the contrary, we find significant gains for grailqa. it must be noted however, that we perform high-resource prefix tuning while they perform low-resource prompt tuning, and hence results may differ. as seen in figure 1 , for the char settings, as the size of vocabulary increases, the prefix tuning accuracy drops. in the said figure, we define vocabulary compression ratio as the size of the new vocabulary divided by the size of the original vocabulary. apart from vocabulary size, the query length also matters. we dual-define vocabulary compression ratio as the size of query length after substitution of new vocabulary divided by size of original query length, and plot on the same graph. when compared to the fine-tuning plot (figure 2 ), prefix tuning has a steeper drop in accuracy, and the performance for t5-small and t5-base vary more significantly. it leads to the finding that finetuning is less sensitive to vocabulary changes, and the difference in model sizes between t5-small and t5-base also seems to matter less. in figures 1 and 2 , it can be seen that the original setting for the masked sparql vocabularies produce accuracies which are below the char family vocabulary curves. it suggests that vocabulary compression ratio alone is not a deciding factor in accuracy. if the vocabulary family changes from sparql to characters, there is an initial shift in accuracy, and after that the complexity of the character vocabulary further affects the accuracy. in ## error analysis we performed an error analysis on a sample of 100 randomly selected questions which produced an incorrect output. in the original setting, roughly 50% errors were due to the presence of non-printable characters in the query (eg: ^). we found that in the initial masked query, while we had replaced some non-printable characters in the pre-processing stage (eg: {, } ), we had not managed to replace the full set of non-printable characters. the original t5 paper mentions curly braces as one of the class of tokens that are not present in the pre-training corpus, however, a comprehensive list of the tokens that do not work with t5, or work with limited efficiency, is not available. in this scenario, it seems that a better approach is to replace the entire vocabulary with one that is entirely known to t5, for example, english words. when comparing errors made by original, that were fixed by dictionary and char1, we observed that roughly 30% of the cases were of variable placement, where the variable placeholders like ent0, rel0 were found to be in the wrong order in the output query in the original setting. rest of the corrections belonged to the category of syntax errors. this points to the finding that alternate vocabularies improve the ability of t5 to correctly produce logical forms from a semantic perspective. to analyse the effect of increasing complexity of vocabulary, we compare 100 randomly selected errors made by char8 with char2. in both these settings, no character is non-printable, and the only errors are either syntax errors, variable placement errors, structural errors or intent errors. out of the 100 questions, 90 were found to be correct in char2 setting. in the remaining 90 in the char8 setting, the highest proportion of errors belonged to syntax (where the query is malformed). the next most prominent class of errors belonged to variable placement, followed by structural errors (eg: two triples instead of three). the major takeaway from this analysis is that for char2 there were no syntax errors, while in char8 there are a significant number of such errors. ## conclusion in this work we carried out experiments with new output vocabularies, where we carefully substituted the original members of the vocabulary with the new ones. we found that when the original sparql vocabulary is replaced with words from an alternate vocabulary closer to the t5 tokenizer vocabulary, the model consistently perform better. as a contribution, we believe that our findings will enable researchers in the field of semantic parsing to deploy smaller models with a modified vocabulary and still find satisfactory performance. this would, in the longer term, lead to energy savings. as future work, we would like to explore the behaviour of the same models in more depth using attention maps. moreover, the significant shift in initial performance on changing vocabulary from original to char and dictionary demands further investigation. similarly, the relatively lower performance of the dictionary setting when compared to char1 setting, in spite of having lower tokenized vocabulary size (tsvs) needs to be investigated further. perhaps sub-words which are seen more frequently during pre-training task of the lm perform better when substituted into the semantic parsing output vocabulary.