Building a question-answering virtual assistant skill currently requires large annotated datasets, which are too expensiveto acquire for all but the largest companies . This paper proposes Schema2QA, an open-source toolkit that can build a Q&A skill from a database schema , requiring just a few manual annotations on each field. The key concept is to cover the space of possible compound queries on the database with millions of domain-specific questions synthesized with the help of a corpus of generic query templates. The synthesized data and a small paraphrase set are used to train a novel neural network based on the BERT pretrained model. We apply Schema2QA to two different  URL domains, restaurants and people, and show that the skills we built can answer complex queries accurately. Our skills achieve an overall accuracy between 74\% and 78 \% on crowdsourced questions for the domains. Any websites with URL metadata in these domains can use Schema2QA to build their website-specific skill automatically. We show that with transfer learning  from the restaurant to the hotel domain, we can achieve a 64\% accuracy on crowdsourced questions with no manual effort. On the restaurant domain, Schema2QA outperforms the best commercial assistant by 21\% on a set of diverse crowdsourced questions, while achieving equally good accuracy on questionscommonly asked to existing virtual assistants .
Building a question-answering agent currently requires large annotated datasets, which are prohibitively expensive . This paper proposes Schema2QA, an open-source toolkit that can generate a Q&A system from a database schema augmented with a few annotations for each field. The key concept is to cover the space of possible compound queries on the database with a large number of in-domain questions synthesized with the help of a corpus of generic query templates. The synthesized data and a small paraphrase set are used to train a novel neural network based on the BERT pretrained model. We use Schema2QA to generate Q A systems for five URL domains, restaurants , people, movies, books and music, and obtain an overall accuracy between 64\% and 75 \% on crowdsourced questions for these domains. Once annotations and paraphrases are obtained for a URL schema, no additional manual effort is needed to create a Q A agent for any website that uses the same schema. Furthermore, we demonstrate that learning can be transferred from the restaurant to the hotel domain, obtaining a 64\% accuracy on crowdsourced questions with no manual effort. Schema2QA achieves an accuracy of 60\% on popular restaurant questions that can be answered using URL. Its performance is comparable to Google Assistant, 7\% lower than Siri, and 15\% higher than Alexa. It outperforms all these assistants by at least 18\% on more complex, long-tail questions .
R
virtual assistant skill
agent
R
too expensiveto acquire for all but the largest companies
prohibitively expensive
R
build
generate
R
skill
system
R
, requiring just a few manual annotations on
augmented with a few annotations for
R
millions of domain-specific
a large number of in-domain
R
apply
use
R
two different
generate Q
A
A systems for five
R
and people, and show that the skills we built can answer complex queries accurately. Our skills achieve
, people, movies, books and music, and obtain
R
74\% and 78
64\% and 75
R
the domains. Any websites with URL metadata in these domains can use Schema2QA to build their website-specific skill automatically. We show that with transfer learning
these domains. Once annotations and paraphrases are obtained for a URL schema, no additional manual effort is needed to create a Q
A
A agent for any website that uses the same schema. Furthermore, we demonstrate that learning can be transferred
R
we can achieve
obtaining
R
On the restaurant domain, Schema2QA outperforms the best commercial assistant by 21\% on a set of diverse crowdsourced questions, while achieving equally good accuracy on questionscommonly asked to existing virtual assistants
Schema2QA achieves an accuracy of 60\% on popular restaurant questions that can be answered using URL. Its performance is comparable to Google Assistant, 7\% lower than Siri, and 15\% higher than Alexa. It outperforms all these assistants by at least 18\% on more complex, long-tail questions