An On-device Deep-Learning Approach for Attribute Extraction from Heterogeneous Unstructured Text (2021)

Abstract: Mobile devices, with their rapidly growing usage, have turned into rich sources of user information, holding critical insights for betterment of user experience and personalization. Creating, receiving and storing important information in the form of unstructured text has become a part and parcel of daily routine of users. From purchase deliveries in Short Message Service (SMS) or Notifications, to event booking details in Calendar applications, mobile devices serve as a portal for understanding user interests, behaviours and activities through information extraction. In this paper, we address the challenge of on-device extraction of user information from unstructured data in natural language from heterogeneous sources like messages, notification, calendar etc. The issue of privacy concern is effectively eliminated by the on-device nature of the proposed solution. Our proposed solution consists of 3 components – A Na ̈ıve-Bayes based classifier for domain identification, a Dual Character andWord based Bidirectional Long Short Term Memory (Bi-LSTM) and Conditional Random Field (CRF) model for attribute extraction and a rule-based Entity Linker. Our solution achieved a 93.29% F1 score on five domains (shopping, travel, event, service and personal). Since on-device deployment has memory and latency constraints, we ensure minimal model size and optimal inference latency. To demonstrate the efficacy of our approach, we have experimented on CoNLL- 2003 dataset and achieved comparable performance to existing benchmark results.

## introduction with an estimated 3.5 billion active users or about 80% of all mobile subscribers, short message service (sms) was the most widely used communication application in the past few years 1 . even with 1 https://en.wikipedia.org/wiki/sms the advent of social media and messenger applications, communication and information storage in digitised form are vastly prevalent via sms, notifications, calendar invites and mail. some examples we readily see are casual conversations with a friend over sms, online shopping related notifications and event booking details, just to name a few. according to 2020 annual report by ctia foot_0 , there were 2.1 trillion text messages exchanged worldwide, an increase of 52 billion messages since 2019. according to sms marketing statistics for 2020/2021 reported by financesonline foot_1 , 98% of sms are opened compared to only 20% of emails and 95% of the read sms are responded to within 3 minutes of delivery. moreover, sms is still the most powerful marketing tool for businesses with 75% of customers preferring receiving offers via sms. the ctr for text messages is much higher (9.18%), compared to other marketing channels such as google adwords (1.91%) and facebook (0.90%). apart from sms and notifications, researchers have investigated other potential data sources like calendar, email, user utterances and communication logs for extracting information. in fact, many establishments are making use of the personal knowledge extracted from different sources on smartphones to provide better service. for example, google extracts and summarizes travel, event and accommodation reservation information from emails foot_2 . however, most of the published literature is focussed on singular sources of information and/or certain domains of interest like bio-medical, events, etc. in this paper, we address the challenge of ondevice extraction of user information from unstructured data in natural language from heterogeneous sources, which include sms, notifications, calendar etc. our proposed system offers a unique method for efficient on-device functionalities. we achieve 93.29% f1 score on five domains (shopping, travel, event, service and personal). the system is implemented as a service on the device. the information that is obtained from these data sources with a preliminary analysis and the highlevel inferences sought from it, are summarized in a key consideration is the constant ongoing conflict between the service provider's desire to track the consumer and the consumer's concern for the privacy of their data. this issue is effectively addressed by the on-device nature of the proposed system, letting the user enjoy its benefits conveniently as the data processing is limited to local environment. some of the features afforded by this new dimension of user's data, which enables personalized device intelligence, are as follows: convenient grouping or reordering of sms/ notification/ calendar data according to user preferences • assist in better planning of activities, for instance, booking airport cab with prior knowledge of user's travel plans • recommender services based on understanding of user's shopping behaviour or preferred types of events (like concerts, sports matches, photography, art etc.) 2 related work digital communication devices continue to offer a growing variety of personalized services to enhance user experience. this is facilitated by increased access and extraction of user information available in both structured and unstructured forms. structured data, generally consisting of text entered in template fashion or in any pre-defined format (like date, zip code etc.), can be conveniently processed whereas unstructured text (like short message service (sms), notifications, calendar events etc.) poses multiple interesting challenges. firstly, apart from emanating from heterogeneous sources on the device, unstructured data on mobile devices does not always conform to grammatical correctness, rendering it difficult for most of the existing natural language processing (nlp) techniques better suited for formal grammar. secondly, most of the advanced information extraction techniques demand server-based deployment, raising privacy concerns of user data storage on cloud. there is limited exploration on on-device information extraction from unstructured text, befitting its memory and latency constraint requirements. thirdly, owing to the special nature of data in consideration, there is a lack of standardised benchmark datasets. most of the previous works have shown a significant amount of research effort being directed to collection, curation and pre-processing of short-text corpus. after data procurement comes the daunting task of annotation based on the identified guidelines of entities and attributes relevant to each domain of interest. there is some pre-existing work on domain classification and information extraction from email, sms, notifications and social media text. most of the previous works handling sms are primarily focused on spam-filtering or certain rudimentary levels of classification. almeida et al. (2011) and cormack et al. (2007), limit the task to binary classification of sms as spam or non-spam. dewi et al. (2017), explore the possibility of multi-class classification of messages into 4 categories with limited data instances. they achieve best results with logistic regression. in comparison, we categorize messages into 6 classes and perform further information extraction. with regard to information extraction from short texts like sms and notifications, traditionally various approaches have been investigated including use of pos taggers, regular expressions, hidden markov models (hmm), logistic regression, specific syntactic parsers or a combination of the above. @xcite investigate the extraction of named entities related to events or activities from chinese smses in handsets, using hidden markov models (hmm). although their method achieves a lower f-score on a small sms corpus of 1,000 messages, the authors significantly reduce the memory consumption. @xcite implement logistic regression to recognize name, date, location and time entities from messages. their reported f-scores for names and locations reaches 88 on an individual word basis, but they do not report on computational or memory resources required of their approach or exact corpus size. @xcite , exploit the syntactic structure in messages and used pattern matching for extraction. since pattern matching is not robust to variations in data, @xcite complement pattern matching with a logistic regression based classifier. recent works on sms and notifications involves the use of deep learning models for information extraction. @xcite implement a hybrid hierarchical lstm-cnn architecture for sms classification and then use class specific entity parsers based on pattern matching. @xcite use the insight that notifications are formatted using templates. templates are extracted using longest common subsequence mining and then clustered using dbscan algorithm. template semantic rules are then generated using a bi-lstm network. we believe ours is the first work that provides a generalized deep learning architecture for information extraction from multiple unstructured data sources. we also categorize inputs into multiple domains and link the attributes to pre-existing entities in the database. our system pipeline has been designed to cater to multiple applications such as customization services, recommender systems, knowledge base population, etc. requiring the holistic understanding of users. ## proposed methodology the proposed information extraction pipeline consists of 3 major components -domain classifier, attribute extractor and entity linker. a pictorial representation of the pipeline is depicted in fig. 1 . ## dataset for data collection we sent out an organization wide broadcast seeking voluntary participation from users with diverse demographics . for this purpose, the users were required to install an application developed by the team. the application masked the user's private information such as name, contact number, financial details etc. and allowed the user the option to filter messages before sharing. we collected ∼ 90k (90, 811) messages, ∼ 55k (54990) notification and ∼ 1k calendar dataset domains message notification calendar training set shopping 5715 621 0 travel 2982 353 77 event 2609 467 217 service 2508 571 0 personal 3865 5309 235 test set all five 2000 500 500 attribute relevant domains count id shopping, travel, event, service 3652 status shopping, travel, event, service 8751 vendor shopping, travel, event, service 3961 product shopping 2910 start date travel, event 2749 end date shopping, travel, event, service 2164 start time travel, event, service 2913 end time travel, event, service 1458 travel mode travel 1835 travel class travel 840 pnr travel 1553 vehicle number travel 1829 source location travel 2117 destination travel, event 2402 event name event 639 event type event 131 service type service 143 amount shopping, travel 1203 relationship personal 41 occasion personal 76 ## conclusion in this paper, we have proposed a unique on-device pipeline to extract relevant information from unstructured sources such as messages, notification, calendar entries etc. by making on-device extraction very efficient, we eliminate the issue of user privacy while providing a platform for an enhanced personalised experience. since the relevant domains are majorly non-overlapping, a naïve bayes classifier gives sufficiently good performance. for attribute extraction, we propose a dual word and character based bi-lstm + crf model, which achieves best results on our self-curated test set as well as conll-2003 test set. the feasibility of such a system was claimed through an on-device implementation using the proposed approach. the applications of such an on-device system can be envisioned across various personalization and recommendation services while maintaining user privacy.