Augmented reality (AR) and the Internet of Things (IoT) have received significant attention as the key technologies for making our future living spaces smarter, more responsive, and more interactive, thereby changing our everyday lives [1, 2] . AR is a type of interactive medium that provides a view of the real world augmented by, and/or spatially registered with, useful computer-generated information. It empowers users to understand the world and amplify their intelligence in solving problems and conducting various tasks [3, 4] . In other words, AR offers a convenient approach for users to visualize and interact with physical objects and their associated data. In addition, a spatially The current bare Internet of Things (IoT) infrastructure has recently been extended to include smarter and more effective user interactions. Individual or meaningful sets and groups of IoT objects can be imbued with data and/or content in a distributed manner and efficiently utilized by the client. This distribution makes it possible to scale and customize interaction techniques such as augmented reality (AR). This paper proposes an architecture for combining the AR interface with IoT for an improved shopping experience. The proposed architecture is distributed and therefore scalable such that any IoT product can be accessed on the spot locally without any server restriction and provide intuitive AR-based visualization and interaction for a flexible product trial in the showroom. We identify three key architectural components required to support such a seamless and scalable AR service and experience for IoT-ready products: (1) objectcentric data management and visualization, (2) mechanism for accessing, controlling, and interacting with the object, and (3) content exchange interoperability. We illustrate the possible scenarios of shopping in the future with the interactive and smart digital information combined with the analog, that is, the real world. A proof-of-concept implementation is presented as applied to such a "digital-analog" style of shopping. In addition, its usability is experimentally assessed as compared to using the conventional control interface. Our experimental study shows that the subjects clearly experience higher usability and greater satisfaction with AR-interactive shopping, thereby demonstrating the potential of the proposed approach. Jo and Kim Hum. Cent. Comput. Inf. Sci. (2019) Cent. Comput. Inf. Sci. (2019) 9:1 registered and visually augmented interface offers a direct and semi-tangible interface, and is thus easy to comprehend and highly useful, particularly for everyday and/or anywhere usage [5] . For example, Microsoft showcased a future AR service using a pair of mixed reality smart glasses, which directly visualize and make the object's functionality interact with datasets from physical objects and structures within the user's environment [6] . In addition, recent IoT, as an infrastructure for "everywhere" services, offers an efficient way of managing the necessary and massive amounts of associated data (for example, individual product information) in a distributed and object-centric fashion [7] . IoT refers to a network of everyday physical objects embedded with minimal computing elements for the sensing, collecting, and/or sharing of data, and even controlling the objects themselves, such as electronic products. Such an infrastructure has been touted as the basis for the future smart environments through an intuitive control and contextbased services [8] . These two seemingly unrelated concepts, AR and IoT, might have different objectives, but can be complementary to each other along with the potential advantages and expected synergies of integrating them [9] . AR provides an intuitive method for users to visualize and interact with IoT objects and their associated data. In particular, context-aware AR services are made possible by using and tapping into the more refined environment information made available by the IoT infrastructure [10] . In fact, it can also provide a natural environment for combining the convenience of interactive digital information (e.g., AR-enabled) to a more effective, humane, and tangible/physical analog objects/world. In the midst of everything going digital, analog is making a comeback in our daily lives with the recent popularity of printed books, vinyl records, and film-based photos. In the previous studies conducted in the IoT or AR field, many people suggested everywhere data management and intuitive visualization such as a server-based approach to ubiquitous AR services with everyday physical objects. However, because the object recognition process, equivalent to looking up the content directory, involves complicated feature matching, for a vast number of objects, expanding the AR services to large everyday spaces has been difficult to achieve. In the cloud services for providing an AR service, it was difficult to provide scalability to the IoT object [1] . Thus, many researchers have focused on AR applications to carry and share the useful information connected with physical objects, and an enhanced AR system allows a user to connect to objects. In line with such thinking, i.e., the idea of synergistic marriage of AR and IoT, this paper presents a new AR shopping framework and experience enabled by an extension of the IoT as a control and product trial interface; in addition, we demonstrate our proposal using an actual prototype system and validate our claims in terms of its improved usability and system performance. We illustrate possible scenarios of shopping in the future with the interactive and smart digital information combined with the analog real world and present that as a proof-of-concept enables to immediately obtain information about shopping items and correctly visualize the information based on the exact location of the item to an AR client. The proof-of-concept implementation is presented as applied to such a "digital-analog" style of shopping. In addition, its usability is assessed experimentally as compared to using the conventional control interface. We believe and claim that there are key components required to support such a seamless and scalable AR service and experience for IoT-ready products: (1) object-centric data management and visualization, (2) mechanism for accessing, controlling, and interacting with the object, and (3) content exchange interoperability. Figure 1 shows a possible system architecture highlighting the three aforementioned components (for a detailed explanation, refer to later sections). The AR client (a mobile or glass device) can instantly connect to an IoT product, receive relevant object-specific data, control information and associated AR datasets for the given targeted service (e.g., recognizing the object and visualizing product information) and thereby interact directly with the physical object to try it out in situ, called direct control and natural interaction [1] . Thus, in the situation of object-centric data management, the data and/or content can also be uploaded to the objects for adding and creating new IoT services and applications, and AR provides an ideal and natural infrastructure for "everywhere" interaction with physical objects. In the context of shopping, services might include interactively visualizing the usage instruction, negotiating for the price and delivery, and test-driving the product through the device control interface with seamless content operability. Note that advertisement, product control, and the AR tracking datasets shown in Fig. 1 are required to visualize with pre-built and pre-stored AR contents with respect to the object's functionalities by various styles of AR interaction. Additionally, object tracking is a fundamental problem in AR. The proposed IoT products can also be to easily apply recognition and tracking for AR. Besides generic data and service content, individual IoT products of interest in the vicinity of the AR client can communicate the information required to recognize and track itself, including the features, algorithm type, and even the physical condition (for example, lighting, distance, or other companion reference object). That is, the AR client is "guided" by the target object itself to localize and track it [1] . Note that, in this scheme, the number of candidates in the matching, i.e., only the candidates in the interaction space of the AR client or user, is relatively low. This, in turn, makes it feasible to use a collective algorithmic method and reduce the number of features, templates, and models in the matching process, further lowering the data requirement. Figure 2 shows a more detailed scheme of AR-based interaction to support direct control of the shopping objects. AR-based interaction is expected to be much more intuitive, direct, and helpful, e.g., the GUI-based hand-held remote devices or conventional switchable interfaces (e.g., in turning on/off one of the displayed lamps in Fig. 2 ). For example, for a shopping customer to test products on the ceiling, if possible, one will usually need some help from the salesperson (who is not often readily available). Even if there exists a GUI-based app for such a purpose, there is the nuisance to download it and become familiar with the interface, which would be prohibitive for the millions of different products. Our proposed scheme eliminates many of such mental and physical obstacles with automatic discovery and connection and object-specific information exchange on the spot through a unified and standardized IoT framework. For instance, the AR service client interacts directly with the IoT object of interest in the immediate shopping area, and upon connection, immediately receives context-relevant AR shopping datasets (for tracking or customized service content, among other uses) [1] . Depending on the context, appropriate and available services, such as a simple product information display, appliance control, and an instruction manual, are shown, presented in a proper form (for example, through a mobile GUI, AR glasses, mobile AR, voice, spatially registered AR, or a simple overlay), and interacted with. Thus, we demonstrate our proposal using an actual prototype system for possible shopping scenarios presented as applied to such (2019) 9:1 a "digital-analog" style of shopping with the interactive and smart digital information in the future and validate our claims in terms of its improved usability and system performance with the proof-of-concept implementation. The rest of this paper is organized as follows. First, we provide a review of related research and requirements of our proposed IoT + AR architecture in Section II. In Sections III and IV, we discuss futuristic use-case scenarios and a detailed data flow of IoT + AR for shopping situations. Section V presents the actual implementation and Section VI presents the validation usability experiment. Finally, in Section VII, we summarize our study and conclude the paper with a discussion and directions for future work. The review of related research focuses on three key components and requirements in our proposed IoT + AR architecture, namely the current state-of-the-art on AR/IoT data management, previous approaches to interaction with IoT objects including the few cases of using AR, and standard content representation or system interoperability protocols. AR services commonly need to manage generic data and service contents for their constituent objects or augmentation targets, which are physical everyday objects. Herein, we review the current approaches for representing such physical object data for AR use (for example, the architecture and data-handling). Previous AR systems were implemented as a single application with all of the content and assets embedded in it, using programming libraries [11, 12] . As such, the augmented content of an object tended to be simple (for example, to simply demonstrate the augmentation capability) and unorganized. GPS-equipped mobile and smartphones have allowed for location-based geographical and AR services to be developed, for example, providing guidelines for commercial points of interest and tourism [13, 14] . Such a service has necessitated the separation of content (and its format specifications) and the underlying player to support the notion of "everywhere" content and service, as well as a unified management of content on the server. HTML [15], KML [16], and ARML [17] are markup languages for such purpose. For example, Wikitude proposed the augmented reality markup language (ARML) for location-based services [14] . ARML allows defining geographical points or landmarks of interest and associating GPS coordinates and simple augmentation content (for example, text, logos, and images). Several other content representation methods exist for AR services, but they require either a specific application or content type (for example, video-based [18], AR on-line manual [19] , and AR guide [20] ) or complicated scripting without sufficient abstraction. However, a standard interoperable content format for representing various comprehensive forms of AR services is yet to be proposed. In addition, in the near future, billions of physical objects can automatically communicate with computers and interconnect with one another for collective intelligent services [21] . In this context, scalable objects' naming and addressing and specifying standard . This allows a physical object to be referenced and associated with its virtual counterpart as the directly matched result. In our case, the client AR system receives "feature datasets" and "contents" information for each shopping item from the discovered IoT device in the user proximity in the standard format (e.g., front images of shopping items, product origin, and price) to recognize, visualize, and interact with the item [7] . In addition, information exchanged between the AR client and the shopping item is contained in the IoT device rather than retrieved from an external server. We assume that the future IoT object will have this information (feature datasets for AR tracking, generic contents, UI control structure) as a standard format. We can envision that different IoT objects may contain different AR information depending on its characteristics, for example, functionalities, process interfaces, and operating manuals.
Jo and Kim REF describe a scenario which uses an Internet of Thinks (IoT) and Augmented Reality (AR) service on a smart pad based on a step-by-step how-to-use guideline.
IoT + AR: pervasive and augmented environments for “Digi-log” shopping experience
Human-centric Computing and Information Sciences
Human-centric Computing and Information Sciences
Computer Science