As a promising paradigm , interactive semantic parsing has shown to improve both semantic parsing accuracy and user confidence in the results . In this paper , we propose a new , unified formulation of the interactive semantic parsing problem , where the goal is to design a modelbased intelligent agent . The agent maintains its own state as the current predicted semantic parse , decides whether and where human intervention is needed , and generates a clarification question in natural language . A key part of the agent is a world model : it takes a percept ( either an initial question or subsequent feedback from the user ) and transitions to a new state . We then propose a simple yet remarkably effective instantiation of our framework , demonstrated on two text-to-SQL datasets ( WikiSQL and Spider ) with different state-of-the-art base semantic parsers . Compared to an existing interactive semantic parsing approach that treats the base parser as a black box , our approach solicits less user feedback but yields higher run-time accuracy . 1 Natural language interfaces that allow users to query data and invoke services without programming have been identified as a key application of semantic parsing ( Berant et al . , 2013 ; Thomason et al . , 2015 ; Dong and Lapata , 2016 ; Zhong et al . , 2017 ; Campagna et al . , 2017 ; Su et al . , 2017 ) . However , existing semantic parsing technologies often fall short when deployed in practice , facing several challenges : ( 1 ) user utterances can be inherently ambiguous or vague , making it difficult to get the correct result in one shot , ( 2 ) the accuracy of state-of-the-art semantic parsers are still not high enough for real use , and ( 3 ) it is hard for users to validate the semantic parsing results , especially with mainstream neural network models that are known for the lack of interpretability . In response to these challenges , interactive semantic parsing has been proposed recently as a practical solution , which includes human users in the loop to resolve utterance ambiguity , boost system accuracy , and improve user confidence via human-machine collaboration ( Li and Jagadish , 2014 ; He et al . , 2016 ; Chaurasia and Mooney , 2017 ; Su et al . , 2018 ; Gur et al . , 2018 ; Yao et al . , 2019 ) . For example , Gur et al . ( 2018 ) built the DialSQL system to detect errors in a generated SQL query and request user selection on alternative options via dialogues . Similarly , Chaurasia and Mooney ( 2017 ) and Yao et al . ( 2019 ) enabled semantic parsers to ask users clarification questions while generating an If-Then program . Su et al . ( 2018 ) showed that users overwhelmingly preferred an interactive system over the noninteractive counterpart for natural language interfaces to web APIs . While these recent studies successfully demonstrated the value of interactive semantic parsing in practice , they are often bound to a certain type of formal language or dataset , and the designs are thus ad-hoc and not easily generalizable . For example , DialSQL only applies to SQL queries on the WikiSQL dataset ( Zhong et al . , 2017 ) , and it is non-trivial to extend it to other formal languages ( e.g. , λ-calculus ) or even just to more complex SQL queries beyond the templates used to construct the dataset . Aiming to develop a general principle for building interactive semantic parsing systems , in this work we propose model-based interactive semantic parsing ( MISP ) , where the goal is to design a model-based intelligent agent ( Russell and Norvig , 2009 ) that can interact with users to complete a semantic parsing task . Taking an utterance ( e.g. , a natural language question ) as input , the agent forms the semantic parse ( e.g. , a SQL query ) in steps , potentially soliciting user feedback in some steps to correct parsing errors . As illustrated in Figure 1 , a MISP agent maintains its state as the current semantic parse and , via an error detector , decides whether and where human intervention is needed ( the action ) . This action is performed by a question generator ( the actuator ) , which generates and presents to the user a humanunderstandable question . A core component of the agent is a world model ( Ha and Schmidhuber , 2018 ) ( hence model-based ) , which incorporates user feedback from the environment and transitions to a new agent state ( e.g. , an updated semantic parse ) . This process repeats until a terminal state is reached . Such a design endows a MISP agent with three crucial properties of interactive semantic parsing : ( 1 ) being introspective of the reasoning process and knowing when it may need human supervision , ( 2 ) being able to solicit user feedback in a human-friendly way , and ( 3 ) being able to incorporate user feedback ( through state transitions controlled by the world model ) . The MISP framework provides several advantages for designing an interactive semantic parser compared to the existing ad-hoc studies . For instance , the whole problem is conceptually reduced to building three key components ( i.e. , the world model , the error detector , and the actuator ) , and can be handled and improved separately . While each component may need to be tailored to the specific task , the general framework remains unchanged . In addition , the formulation of a modelbased intelligent agent can facilitate the application of other machine learning techniques like reinforcement learning . To better demonstrate the advantages of the MISP framework , we propose a simple yet re-markably effective instantiation for the text-to-SQL task . We show the effectiveness of the framework based on three base semantic parsers ( SQLNet , SQLova and SyntaxSQLNet ) and two datasets ( WikiSQL and Spider ) . We empirically verified that with a small amount of targeted , testtime user feedback , interactive semantic parsers improve the accuracy by 10 % to 15 % absolute . Compared to an existing interactive semantic parsing system , DialSQL ( Gur et al . , 2018 ) , our approach , despite its much simpler yet more general system design , achieves better parsing accuracy by asking only half as many questions . This work proposes a new and unified framework for the interactive semantic parsing task , named MISP , and instantiates it successfully on the textto-SQL task . We outline several future directions to further improve MISP-SQL and develop MISP systems for other semantic parsing tasks : Improving Agent Components . The flexibility of MISP allows improving on each agent compo-nent separately . Take the error detector for example . One can augment the probability-based error detector in MISP-SQL with probability calibration , which has been shown useful in aligning model confidence with its reliability ( Guo et al . , 2017 ) . One can also use learning-based approaches , such as a reinforced decision policy ( Yao et al . , 2019 ) , to increase the rate of identifying wrong and solvable predictions . Lifelong Learning for Semantic Parsing . Learning from user feedback is a promising solution for lifelong semantic parser improvement ( Iyer et al . , 2017 ; Padmakumar et al . , 2017 ; Labutov et al . , 2018 ) . However , this may lead to a non-stationary environment ( e.g. , changing state transition ) from the perspective of the agent , making its training ( e.g. , error detector learning ) unstable . In the context of dialog systems , Padmakumar et al . ( 2017 ) suggests that this effect can be mitigated by jointly updating the dialog policy and the semantic parser batchwisely . We leave exploring this aspect in our task to future work . Scaling Up . It is important for MISP agents to scale to larger backend data sources ( e.g. , knowledge bases like Freebase or Wikidata ) . To this end , one can improve MISP from at least three aspects : ( 1 ) using more intelligent interaction designs ( e.g. , free-form text as user feedback ) to speed up the hypothesis space searching globally , ( 2 ) strengthening the world model to nail down a smaller set of plausible hypotheses based on both the initial question and user feedback , and ( 3 ) training the agent to learn to improve the parsing accuracy while minimizing the number of required human interventions over time .

Challenge: Interactive semantic parsing systems can take user utterances to improve their outputs but are currently limited in languages or datasets due to their ad-hoc nature.
Approach: They propose to approach interactive semantic parsing with a mode-based agent interacting with users as needed to perform error-less parsing.
Outcome: The proposed simple agent parser outperforms existing baseline parsers while keeping the interactions with users in half.