Compositional Generalization in Multilingual Semantic Parsing over W ikidata (2022)

Abstract: Semantic parsing (SP) allows humans to leverage vast knowledge resources through natural interaction. However, parsers are mostly designed for and evaluated on English resources, such as CFQ (Keysers et al., 2020), the current standard benchmark based on English data generated from grammar rules and oriented towards Freebase, an outdated knowledge base. We propose a method for creating a multilingual, parallel dataset of question-query pairs, grounded in Wikidata. We introduce such a dataset, which we call Multilingual Compositional Wikidata Questions (MCWQ), and use it to analyze the compositional generalization of semantic parsers in Hebrew, Kannada, Chinese, and English. While within- language generalization is comparable across languages, experiments on zero-shot cross- lingual transfer demonstrate that cross-lingual compositional generalization fails, even with state-of-the-art pretrained multilingual encoders. Furthermore, our methodology, dataset, and results will facilitate future research on SP in more realistic and diverse settings than has been possible with existing resources.

## introduction semantic parsers grounded in knowledge bases (kbs) enable knowledge base question answering (kbqa) for complex questions. many semantic parsers are grounded in kbs such as freebase @xcite , dbpedia @xcite , and wikidata @xcite , and models can learn to answer questions about unseen entities and properties @xcite @xcite ). an important desired ability is compositional generalization-the ability to generalize to unseen combinations of known components @xcite . one of the most widely used datasets for measuring compositional generalization in kbqa is cfq (compositional freebase questions; @xcite , which was generated using grammar rules, and is based on freebase, an outdated and unmaintained english-only kb. while the need to expand language technology to many languages is widely acknowledged @xcite , the lack of a benchmark for compositional generalization in multilingual semantic parsing (sp) hinders kbqa in languages other than english. furthermore, progress in both sp and kb necessitates that benchmarks can be reused and adapted for future methods. wikidata is a multilingual kb, with entity and property labels in a multitude of languages. it has grown continuously over the years and is an important complement to wikipedia. much effort has been made to migrate freebase data to @xcite @xcite , but only in english. investigating compositional generalization in cross-lingual sp requires a multilingual dataset, a gap we address in this work. we leverage wikidata and cfq to create multilingual compositional wikidata questions (mcwq), a new multilingual dataset of compositional questions grounded in wikidata (see figure 1 for an example). beyond the original english, an indo-european language using the latin script, we create parallel datasets of questions in hebrew, kannada, and chinese, which use different scripts and belong to different language families: afroasiatic, dravidian, and sino-tibetan, respectively. our dataset includes questions in the four languages and their associated sparql queries. our contributions are: our code for generating the dataset and for the experiments, as well as the dataset itself and trained models, are publicly available on https:// github.com/coastalcph/seq2sparql . cfq uses the distribution-based compositionality assessment (dbca) method to generate multiple train-test splits with maximally divergent examples in terms of compounds, while maintaining a low divergence in terms of primitive elements (atoms). in these maximum compound divergence (mcd) splits, the test set is constrained to examples containing novel compounds, that is, new ways of composing the atoms seen during training. for measuring compositional generalizations, named entities in the questions are anonymized so that models cannot simply learn the relationship between entities and properties. cfq contains 239,357 english question-answer pairs, which encompass 49,320 question patterns and 34,921 sparql query patterns. by the time cfq was released, freebase had already been shut down. on that account, to our knowledge, there is no existing sp dataset targeting compositional generalization that is grounded in a currently usable kb, which contains up-to-date information. we therefore migrate the dataset to such a kb, namely, wikidata, in §3. moreover, only a few studies have evaluated semantic parsers' performance in a multilingual setting, due to the scarcity of multilingual kbqa datasets @xcite . no comparable benchmark exists for languages other than english, and it is therefore not clear whether results are generalizable to other languages. compositional generalization in typologically distant languages may pose completely different challenges, as these languages may have different ways to compose meaning @xcite . we create such a multilingual dataset in §4, leveraging the multilinguality of wikidata. ## migration to wikidata wikidata is widely accepted as the replacement for freebase. it is actively maintained and represents knowledge in a multitude of languages and domains, and also supports sparql. migrating freebase queries to wikidata, however, is not trivial, as there is no established full mapping between the kbs' properties and entities. an obvious alternative to migration would be a replication of the original cfq generation process but with wikidata as the kb. before delving into the details of the migration process, let us motivate the decision not to pursue that option: the grammar used to generate cfq was not made available to others by @xcite and is prohibitively too complex to reverse-engineer. our migration process, on the other hand, is general and can similarly be applied for migrating other datasets from freebase to wikidata. finally, many competitive models with specialized architecture have been developed for cfq @xcite @xcite . our migrated dataset is formally similar and facilitates their evaluation and the development of new methods. ## generating multilingual questions to create a typologically diverse dataset, starting from our english dataset (an indo-european language using the latin script), we use machine translation to three other languages from different families (afroasiatic, dravidian, and sino-tibetan), which use different scripts: hebrew, kannada, and chinese ( §4.1). for a comparison to machine translation and a more realistic evaluation with regard to compositional sp, we manually translate a subset of the test sets of the three mcd splits ( §4.2) and evaluate the machine translation quality ( §4.3). ## experiments while specialized architectures have achieved state-of-the-art results on cfq @xcite @xcite , these approaches are english-or freebase-specific. we therefore experiment with sequence-to-sequence (seq2seq) models, among which t5 @xcite has been shown to perform best on cfq @xcite . we evaluate these models for each language separately ( §5.1), and subsequently evaluate their cross-lingual compositional generalization ( §5.2). ## limitations mcwq is based on cfq, a rule-base generated dataset, and hence it has the inherited unnaturalness in question-query pairs of high complexity. secondly, we use machine translation to make mcwq multilingual. although this is the dominant approach for generating multilingual datasets @xcite and we have provided evidences that mcwq has reasonable translation accuracy and fluency with human evaluation and comparative experiments in §4.3 and §5.1, machine translation would nevertheless create substandard translation artifacts @xcite . one alternative is to write rules for template translation. the amount of work can possibly be reduced by refering to a recent work @xcite in which english rules are provided for syntactic dependency parsing on cfq's question fields. furthermore, the assumption that an english kb is a ''canonical'' conceptualization is unjustified, as speakers of other languages may know and care about other entities and relationships @xcite . therefore, future work must create multilingual sp datasets by sourcing questions from native speakers rather than translating them. ## conclusion the field of kbqa has been saturated with work on english, due to both the inherent challenges of translating datasets and the reliance on english-only dbs. in this work, we presented a mcwq mt5-base+rir information unit 1. model publicly available? yes 2. time to train final model 592 hours 3. time for all experiments 1315 hours 4. energy consumption 2209.2 kwh 5. location for computations denmark 6. energy mix at location 191 gco2eq/ kwh 7. co2eq for final model 189.96 kg 8. co2eq for all experiments 421.96 kg