FLIN : A Flexible Natural Language Interface for Web Navigation (2021)

Abstract: AI assistants can now carry out tasks for users by directly interacting with website UIs. Current semantic parsing and slot-filling techniques cannot flexibly adapt to many different websites without being constantly re-trained. We propose FLIN, a natural language interface for web navigation that maps user commands to concept-level actions (rather than low-level UI actions), thus being able to flexibly adapt to different websites and handle their transient nature. We frame this as a ranking problem: given a user command and a webpage, FLIN learns to score the most relevant navigation instruction (involving action and parameter values). To train and evaluate FLIN, we collect a dataset using nine popular websites from three domains. Our results show that FLIN was able to adapt to new websites in a given domain.

## introduction ai personal assistants, such as google assistant, can now interact directly with the ui of websites to carry out human tasks @xcite . users issue commands to the assistant, and this executes them by typing, selecting items, clicking buttons, and navigating to different pages in the website. such an approach is appealing as it can reduce the dependency on third-party apis and expand an assistant's capabilities. this paper focuses on a key component of such systems: a natural language (nl) interface capable of mapping user commands (e.g., "find an italian restaurant for 7pm") into navigation instructions that a web browser can execute. one way to implement such an nl interface is to map user commands directly into low-level ui actions (button clicks, text inputs, etc.). the ui elements appearing in a webpage are embedded by concatenating their dom attributes @xcite . then, a scoring function @xcite or a neural policy @xcite ) are * work done while interning at microsoft research. to achieve this goal, we take two steps. first, we conceptualize a new way of designing nl interfaces for web navigation. instead of mapping user commands into low-level ui actions, we map them into meaningful "concept-level" actions. conceptlevel actions are meant to express what a user perceives when glancing at a website ui. in the example shown in figure 1 , the homepage of opentable has a concept-level action "let's go" (where "let's go" is the label of a search button), which represents the concept of searching something which can be specified using various parameters (a date, a time, a number of people and a search term). intuitively, websites in a given domain (say, all restaurant websites) share semantically-similar conceptlevel actions and the semantics of a human task tend to be time invariant. hence, learning at the level of concept-level actions can lead to a more flexible nl interface. however, while concept-level actions vary less than raw ui elements, they still manifest with different representations and parameter sets across websites. searching a restaurant in opentable.com, for example, corresponds to an action "let's go" which supports up to four parameters; in yelp.com, the same action is called "search" and supports two parameters (search term and location). websites in flin semantic parser action extractor let's go {people: '2 people', time: '7:00 pm', location, restaurant, or cuisine: 'italian'} find an italian restaurant for me and my friend available at 7 pm user home page let's go {<time>, <date>, <people>, <location, restaurant, or @xmath0:30 pm, 7:00 pm, 7:30 pm, …} date {mar 26, 2020, mar 27, 2020, …} people {2 people, 3 people, …} top cuisines near chicago {<top cuisine near @xmath1, parameters and parameter values current webpage next webpage search results page action executor .. one domain may also have different action types (e.g., making a table reservation vs. ordering food). our second insight to tackle this problem is to leverage semantic parsing methods in a novel way. traditional semantic parsing methods @xcite @xcite @xcite deal with environments that have a fixed and known set of actions, hence cannot be directly applied. instead, we propose flin, a new semantic parsing approach where instead of learning how to map nl commands to executable logical forms (as in traditional semantic parsing), we leverage the semantics of the symbols (name of actions/parameters and parameter values) contained in the logical form (the navigation instruction) to learn how to match it against a given command. specifically, we model the semantic parsing task as a ranking problem. given an nl command c and the set of actions a available in the current webpage, flin scores the actions with respect to c. simultaneously, for each parameter p of an action, it extracts a phrase m from c that expresses a value of p, and then scores p's values with respect to m to find the best value assignment for p. each action with its associated list of parameter value assignments represents a candidate navigation instruction to be ranked. flin learns a net score for each instruction based on corresponding action and parameter value assignment scores, and outputs the highest-scored instruction as the predicted navigation instruction. to collect a dataset for training and testing flin, we built a simple rule-based action extractor tool that extracts concept-level actions along with their parameters (names and values, if available) from webpages. the implementation and evaluation of this tool is out of scope for this paper. foot_0 in a complete system, illustrated in figure 1 , we envision the action extractor to extract and pass the concept-level actions present in the current webpage to flin, which computes a candidate navigation instruction n to be executed by an action executor (e.g., a web automation tool such as selenium (2020) or ringer @xcite ). overall, we make the following contributions: (1) we conceptualize a new design approach for nl interfaces for web navigation based on conceptlevel actions; (2) we build a match-based semantic parser to map nl commands to navigation instructions; and (3) we collect a new dataset based on nine websites (from restaurant, hotel and shopping domains) and provide empirical results that verify the generalizability of our approach. code and dataset are available at https://github.com/microsoft/flin-nl2web . work on nl-guided web task execution includes learning from demonstrations @xcite , building reinforcement learning agents @xcite , training sequence to sequence models to map natural language commands into web apis @xcite , and generating task flows from apis @xcite . these techniques assume different problem settings (e.g., reward functions) or deal with low-level web actions or api calls. unlike flin, they do not generalize across websites. @xcite propose an embeddingbased matching model to map natural language commands to low-level ui actions such as hyperlinks, buttons, menus, etc. unlike flin, this work does not deal with predicting parameter values (i.e., actions are un-parametrized). models that jointly perform intent detection and slot filling @xcite @xcite are not applicable to our problem for three reasons. first, they are trained on a per-application basis using application-specific intent and slot labels, and thus cannot generalize across websites. second, they semantically label words in an utterance, but do not do value assignment, hence they cannot output executable navigation paths. third, they perform multi-class classification (i.e., they assume only one intent to be true for a user query) and have no notion of state (e.g., current webpage). they do not deal with intents with overlapping semantics which may occur across pages of the same website (e.g., in the example shown in figure 1 , the same user query may map to the action "let's go" in the opentable's home page or to the action "find a table " in the search results page). ## problem formulation let a @xmath2, a 2 , ..., a n } be the set of conceptlevel actions available in a webpage w. each action @xmath3, p 2 , ..., p k }. each parameter @xmath4.e., a set of values that can be assigned to parameter p), and can be either closed domain or open domain. for closed-domain parameters, the domain is bounded and consists of a finite set of values that p can take; the set is imposed by the website ui, such as the available colors and sizes for a product item or the available reservation times for a restaurant. for open-domain parameters, the domain is, in principle, unbounded, but, in practice, it consists of all words/phrases which can be extracted from an nl command c. with reference to figure 1 , the "let's go" (search) action has n @xmath5, "date", "people", "location, restaurant, or cuisine"}. the first three parameters are closed domain and the last one (the search term) is open domain. the action extractor module (figure 1 ) names actions and parameters after labels and texts appearing in the ui (or, if absent, using dom attributes); it also automatically scrapes values of closed-domain parameters (from drop-down menus or select lists). given the above setting, our goal is to map an nl command c issued in w into a navigation instruction n , consisting of a correct action name n a * corresponding to action a * ∈ a w and an associated list of @xmath6, given by {( ## the flin model the task of solving the above semantic parsing problem can be decomposed into two sub-tasks: (i) action recognition, i.e., recognizing the action @xmath7, and (ii) parameter recognition and value assignment, i.e., deciding whether a parameter of an action is expressed in c and, if so, assigning the value to that parameter. a parameter is expressed in c by a mention (word or phrase). for example, in figure 1 , "me and my friend" is a mention of parameter "people" in c and a correct parsing should map it to the domain value "2 people". thus, the second sub-task involves first find an italian restaurant for me and my friend available at 7pm. (c: command) shared layer bi-lstm rnn word embed. % 3 4 % 3 5 % 3 6 averaging % 7 % &3 action name vector mean parameter vector action name -parameter vector command vector parameter value match score parameter vectors bi-lstm rnn word embed. % 8 9: '()(.) 0 , <= (> , , + ? ) word-level similarity % @ a 9: averaging lexical similarity 0 , bcd (> , , + ? ) 0 , (> , , + ? ) bi-lstm rnn m e a n d char embed. lstm rnn char embed. lstm rnn char-level word repr. learning layer % 8 9: '()(.) 0 , .e"f (> , , + ? ) char-level similarity % @ a 9: parameter mention extraction ghijkli : me and my friend [cls] people [sep] find an italian .. at 7pm [sep] char-level word repr. learning extracting a mention of a given parameter from c, and then matching it against a set of domain values to find the correct value assignment. for an open-domain parameter, the extracted mention becomes the value of the parameter and no matching is needed, e.g. in figure 1 , the mention "italian" should be assigned to the parameter "location, restaurant, or cuisine". with reference to figure 2 , flin consists of four components, designed to solve the aforementioned two sub-tasks: (1) action scoring, which scores each available action with respect to the given command ( §4.1); (2) parameter mention extraction, which extracts the mention (phrase) from the command for a given parameter ( §4.2); (3) parameter value scoring, which scores a given mention with a closed-domain parameter value or rejects it if no domain values can be mapped to the mention ( §4.3); and (4) inference (not shown in the figure), which uses the scores of actions and parameter values to infer the action-parameter-value assignment with the highest score as the predicted navigation instruction ( §4.4). ## evaluation we evaluate flin on nine popular websites from three representative domains: (i) restaurants (r), (ii) hotels (h), and (iii) shopping (s). we collect labelled datasets for each website ( §5.1) and perform in-domain cross-website evaluation. specifically, we train one flin model for each domain using one website, and test on the other (two) websites in the same domain. ideally, a single model could be trained by using the training data of all three domains and applied to all test websites, but we opt for domain-specific training/evaluation to better analyze how flin leverages the semantic overlap of concept-level actions (that exists across in-domain websites) to generalize to new websites. we discard cross-domain evaluation because the semantics of actions and parameters do not significantly overlap across our three domains. ## conclusion to generalize to many websites, nl-guided web navigation assistants require an nl interface that can work with new website uis without being re-trained each time. to this end, we proposed flin, a matching-based semantic parsing approach that maps user commands to concept-level actions. while various optimizations are possible, flin adapted well to new websites and delivered good performance. we have used it in restaurant, shopping and hotels websites, but its design can apply to more domains.