ISA : An Intelligent Shopping Assistant (2020)

Abstract: Despite the growth of e-commerce, brick-and-mortar stores are still the preferred destinations for many people. In this paper, we present ISA, a mobile-based intelligent shopping assistant that is designed to improve shopping experience in physical stores. ISA assists users by leveraging advanced techniques in computer vision, speech processing, and natural language processing. An in-store user only needs to take a picture or scan the barcode of the product of interest, and then the user can talk to the assistant about the product. The assistant can also guide the user through the purchase process or recommend other similar products to the user. We take a data-driven approach in building the engines of ISA’s natural language processing component, and the engines achieve good performance.

## introduction shopping in physical stores is a popular option for many people. each week, a lot of people enter supermarkets in which they are immersed with many different product choices. in many shopping centers, customer service representatives (csrs) are employed to answer questions from customers about products. however, a customer may experience long waiting time for assistance if all csrs are busy interacting with other customers. therefore, automated solutions can increase customer satisfaction and retention. in this paper, we introduce a mobile-based intelligent shopping assistant, isa, which is based on advanced techniques in computer vision, speech processing, and natural language processing. a user just needs to take a picture or scan the barcode of the product of interest. after that, the user can ask isa a variety of questions such as product 1 the work was conducted while the first author interned at adobe research. figure 1 : isa assists users at physical stores features, specifications and return policies. the assistant can also guide the user through the purchase process or recommend other similar products. this work can be used as the first step in fully automating customer service in shopping centers. with isa, no csrs will be needed as customers can simply turn to their phones for assistance. we have developed a fully functional prototype of isa. the rest of the paper is organized as follows. section 2 introduces some related work. section 3 gives an overview of the design and implementation of the system. finally, section 4 concludes the paper and suggests future directions. ## related work the most closely related branches of work to ours are probably customer service chatbots for e-commerce websites. for example, superagent @xcite is a powerful chatbot that leverages large-scale and publicly available e-commerce data. the researchers demonstrate superagent as an add-on extension to mainstream web browsers. when a user visits a product page, superagent crawls the information of the product from multiple data sources within the page. after that, the user can ask superagent about the product. unlike superagent, isa is designed to assist users at physical stores (figure 1 ). in addition to natural language processing techniques, isa also needs to use techniques in computer vision and speech processing when interacting with the users. ## conclusion and future work in this paper, we present isa, a powerful intelligent shopping assistant. isa is designed to achieve the goal of improving shopping experience in physical stores by leveraging advanced techniques in computer vision, speech processing, and natural language processing. a user only needs to take a picture or scan the barcode of the product of interest, and then the user can ask isa a variety of questions about the product. the system can also guide the user through the purchase decision or recommend other similar products to the user. there are many fronts on which we will be exploring in the future. currently the product specification qa engine answers only questions regarding the specifications of a product. we will implement engines for addressing other kinds of questions. we will also extend isa to better support other languages and informal text @xcite @xcite . in addition, we will conduct a user study to evaluate our system in the future. finally, we wish to extend this work to other domains such as building an as-