Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction (2021)

Abstract: We investigate grounded language learning through real-world data, by modelling a teacher-learner dynamics through the natural interactions occurring between users and search engines; in particular, we explore the emergence of semantic generalization from unsupervised dense representations outside of synthetic environments. A grounding domain, a denotation function and a composition function are learned from user data only. We show how the resulting semantics for noun phrases exhibits compositional properties while being fully learnable without any explicit labelling. We benchmark our grounded semantics on compositionality and zero-shot inference tasks, and we show that it provides better results and better generalizations than SOTA non-grounded models, such as word2vec and BERT.

## introduction most sota models in nlp are only intratextual. models based on distributional semantics -such as standard and contextual word embeddings @xcite @xcite -learn representations of word meaning from patterns of co-occurrence in big corpora, with no reference to extra-linguistic entities. while successful in a range of cases, this approach does not take into consideration two fundamental facts about language. the first is that language is a referential device used to refer to extra-linguistic objects. scholarly work in psycholinguistics @xcite , formal semantics @xcite and philosophy of language @xcite show that (at least some aspects of) linguistic meaning can be represented as a sort of mapping between linguistic and extra-linguistic entities. the second is that language may be learned based on its usage and that learners draw part of their generalizations from the observation of teachers' behaviour @xcite . these ideas have been recently explored by work in grounded language learning, showing that allowing artificial agents to access human actions providing information on language meaning has several practical and scientific advantages @xcite . while most of the work in this area uses toy worlds and synthetic linguistic data, we explore grounded language learning offering an example in which unsupervised learning is combined with a language-independent grounding domain in a realworld scenario. in particular, we propose to use the interaction of users with a search engine as a setting for grounded language learning. in our setting, users produce search queries to find products on the web: queries and clicks on search results are used as a model for the teacher-learner dynamics. we summarize the contributions of our work as follows: in particular, we learn neural representations for our domain of objects leveraging prod2vec @xcite : crucially, building the grounding domain does not require any linguistic input and it is independently justified in the target domain @xcite . in this setting, lexical denotation can also be learned without explicit labelling, as we use the natural interactions between the users and the search engine to learn a noisy denotation for the lexicon @xcite . more specifically, we use deepsets @xcite constructed from user behavioural signals as the extra-linguistic reference of words. for in-stance, the denotation of the word "shoes" is constructed from the clicks produced by real users on products that are in fact shoes after having performed the query "shoes" in the search bar. albeit domain specific, the resulting language is significantly richer than languages from agent-based models of language acquisition @xcite , as it is based on 26k entities from the inventory of a real website. methodologically, our work draws inspiration from research at the intersection between artificial intelligence and cognitive sciences: as pointed out in recent papers @xcite , extra-textual elements are crucial in advancing our comprehension of language acquisition and the notion of "meaning". while synthetic environments are popular ways to replicate child-like abilities @xcite , our work calls the attention on real-world information retrieval systems as experimental settings: cooperative systems such as search engines offer new ways to study language grounding, in between the oversimplification of toy models and the daunting task of providing a general account of the semantics of a natural language. the chosen ir domain is rich enough to provide a wealth of data and possibly to see practical applications, whereas at the same time it is sufficiently self-contained to be realistically mastered without human supervision. ## methods following our informal exposition in section 1, we distinguish three components, which are learned separately in a sequence: learning a languageindependent grounding domain, learning noisy denotation from search logs and finally learning functional composition. while only the first model (prod2vec) is completely unsupervised, it is important to remember that the other learning procedures are only weakly supervised, as the labelling is obtained by exploiting an existing user-machine dynamics to provide noisy labels (i.e. no human labeling was necessary at any stage of the training process). learning a representation space. we train product representation to provide a "dense ontology" for the (small) world we want our language to describe. those representations are known in product search as product embeddings @xcite : prod2vec models are word2vec models in which words in a sentence are replaced by products in a shopping session. for this study, we pick cbow @xcite as our training algorithm, and select @xmath0, optimizing hyperparameters as recommended by @xcite ; similar to what happens with word2vec, related products (e.g. two pairs of sneakers) end up closer in the embedding space. in the overall picture, the product space just constitutes a grounding domain, and re-using tried and tested @xcite neural representations is an advantage of the proposed semantics. learning lexical denotation. we interpret clicks on products in the search result page, after a query is issued, as a noisy "pointing" signal (tagliabue and cohn-gordon, 2019), i.e., a map between text ("shoes") and the target domain (a portion of the product space). in other words, our approach can be seen as a neural generalization of model-theoretic semantics, where the extension of "shoes" is not a discrete set of objects, but a region in the grounding space. given a list of products clicked by shoppers after queries, we represent meaning through an order-invariant op-eration over product embeddings (average pooling weighted by empirical frequencies, similar to yu et al. ( 2020 )); following @xcite , we refer to this representation as a deepset. since words are now grounded in a dense domain, set-theoretic functions for nps @xcite need to be replaced with matrix composition, as we explain in the ensuing section. learning functional composition. our functional composition will come from the composition of deepset representations, where we want to learn a function f : deepset × @xmath1. we address functional composition by means of two models from the relevant literature @xcite : one, additive compositional model (adm), sums vectors together to build the final deepset representation. the second model is instead a matrix compositional model (mdm): given in input two deepsets (for example, one for "nike" and one for "shoes") the function we learn as the form m v + n u, where the interaction between the two vectors is mediated through the learning of two matrices, m and n . since the output of these processes is always a deepset, both models can be recursively composed, given the form of the function f . ## experiments data. we obtained catalog data, search logs and detailed behavioral data (anonymized product interactions) from a partnering online shop, shop x. shop x is a mid-size italian website in the sport apparel vertical foot_1 . browsing and search data are sampled from one season (to keep the underlying catalog consistent), resulting in a total of 26, 057 distinct product embeddings, trained on more than 700, 000 anonymous shopping sessions. to prepare the final dataset, we start from comparable literature @xcite and the analysis of linguistic and browsing behavior in shop x, and finally distill a set of np queries for our compositional setting. in particular, we build a rich, but tractable set by excluding queries that are too rare (<5 counts), queries with less than three different products clicked, and queries for which no existing product embedding is present. afterwards, we zoom into np-like constructions, by inspecting which features are frequently used in the query log (e.g. shoppers search for sport, not colors), and matching logs and nps to produce the final set. based on our experience with dozens of successful deployments in the space, nps constitute the vast majority of queries in product search: thus, even if our intent is mainly theoretical, we highlight that the chosen types overlap significantly with real-world frequencies in the relevant domain. due to the power-law distribution of queries, one-word queries are the majority of the dataset (60%); to compensate for sparsity we perform data augmentation for rare compositional queries (e.g. "nike running shoes"): after we send a query to the existing search engine to get a result set, we simulate @xmath2. the final dataset consists of 104 "activity + sortal" 4 queries -"running shoes" -; 818 "brand + sortal" queries -"nike shoes" -, and 47 "gender + sortal" queries -"women shoes"; our testing data consists of 521 "brand + activity + sortal" (bas) triples, 157 "gender + activity + sortal" (gas) triples, 406 "brand + gender + activity + sortal" (bgas) quadruples. 5 tasks and metrics. our evaluation metrics are meant to compare the real semantic representation of composed queries ("nike shoes") with the one predicted by the tested models: in the case of the proposed semantics, that means evaluating how it predicts the deepset representation of "nike shoes", given the representation of "shoes" and "nike". comparing target vs predicted representations is achieved by looking at the nearest neighbors of the predicted deepset, as intuitively complex queries behave as expected only if the two representations share many neighbors. for this reason, quantitative evaluation is performed using two well-known ranking metrics: ndcg and jac3 since the only objects users can click on are those returned by the search box, query representation may in theory be biased by the idiosyncrasies of the engine. in practice, we confirmed that the embedding quality is stable even when a sophisticated engine is replaced by simple boolean queries over tf-idf vectors, suggesting that any bias of this sort is likely to be very small and not important for the quality of the compositional semantics. ## conclusions and future work in the spirit of @xcite , we argued for grounding linguistic meaning in artificial systems through experience. in our implementation, all the important pieces -domain, denotation, composition -are learned from behavioral data. by grounding meaning in (a representation of) objects and their properties, the proposed noun phrase semantics can be learned "bottom-up" like distributional models, but can generalize to unseen examples, like traditional symbolic models: the implicit, dense structure of the domain (e.g. the relative position in the space of nike products and shoes) underpins the explicit, discrete structure of queries picking objects in that domain (e.g. "nike shoes") -in other words, compositionality is an emergent phenomenon. while encouraging, our results are still preliminary: first, we plan on extending our semantics, starting with boolean operators (e.g. "shoes not nike"); second, we plan to improve our representational capabilities, either through symbolic knowledge or more discerning embedding strategies; third, we wish to explore transformer-based architectures @xcite as an alternative way to produce set-like representations. we conceived our work as a testable application of a broader methodological stance, loosely following the agenda of the child-as-hacker (rule et al., 2020) and child-as-scientist @xcite programs. our "search-engine-as-a-child" metaphor may encourage the use of abundant real-world search logs to test computational hypotheses about language learning inspired by cognitive sciences @xcite .