LITE : I ntent-based Task Representation Learning Using Weak Supervision (2022)

Abstract: Users write to-dos as personal notes to themselves, about things they need to complete, remember or organize. To-do texts are usually short and under-specified, which poses a challenge for current text representation models. Yet, understanding and representing their meaning is the first step towards providing intelligent assistance for to-do management. We address this problem by proposing a neural multi-task learning framework, LITE, which extracts representations of English to-do tasks with a multi-head attention mechanism on top of a pre-trained text encoder. To adapt representation models to to-do texts, we collect weak-supervision labels from semantically rich external resources (e.g., dynamic commonsense knowledge bases), following the principle that to-do tasks with similar intents have similar labels. We then train the model on multiple generative/predictive training objectives jointly. We evaluate our representation model on four downstream tasks and show that our approach consistently improves performance over baseline models, achieving error reduction of up to 38.7%.

## introduction task management tools are widely used to organize tasks and keep track of progress in work and daily lives. examples include microsoft to-do, todoist, trello, and digital assistants such as amazon alexa and google assistant. machine learning techniques can automate various aspects of task management such as task creation @xcite , organization @xcite , prioritization, and decomposition of complex tasks @xcite . the goal of this work is to develop a single, general-purpose encoding system that converts todo task texts into real-valued vector representa- groceries milk eggs carrots today pay credit card call mom pick up chris at 4 description list name intelligent assistance task detection, organization, recommendation, … user-generated to-do texts figure 1: our aim is to encode user-generated to-do texts (list names and descriptions) into vector representations so that machine learning systems can provide various kinds of intelligent assistance. tions (fig. 1 ). using one encoding system for multiple task functions (task detection, organization, recommendation, etc.) as opposed to having multiple dedicated encoders saves the computational costs of updating models regularly and encoding texts from millions of users. representation learning has been extensively studied in natural language processing @xcite . adapting models pre-trained on massive amounts of raw texts to a target domain or a task has become common practice @xcite , with many publicly available pre-trained models such as bert @xcite , gpt-2 @xcite , and sentence encoders @xcite . leveraging word context is one of the key strengths of these pre-trained models. however, to-do texts exhibit unique characteristics that make this context-based modeling less effective ( §2). our analysis on a dataset of 6.5 million entries shows that to-do texts are short and often lack an action verb. while similar to web search queries, they are not written to be understood by a search engine but instead are personal notes to the users themselves and assume rich personal context. on the other hand, some task management applications allow users to organize their to-dos under userdefined lists, which, our analysis shows, can sometimes convey important information about their meaning (e.g., a "grocery" list vs. a "today" list). our hypothesis is that we can effectively fine-tune contextualized representation models for under-specified texts using multiple weaklysupervised prediction/generation tasks that focus on knowledge about to-do tasks. we induce supervision signals semi-automatically from existing resources so that to-do tasks that have similar intents share similar target labels. to this end, we propose lite, foot_0 a framework for training todo task representation models using the following auxiliary tasks: (1) autocompletion of to-do descriptions, (2) pre-action and goal generation based on comet @xcite , and (3) action attribute predictions based on framenet @xcite . we implement lite on top of existing pre-trained language models and evaluate its performance through downstream tasks on two proprietary and two publicly available datasets @xcite : urgent and important to-do detection, actionable to-do classification, co-located and co-timed to-do pair detection, and intent detection. overall, we make the following contributions: (1) a neural multi-task learning framework to fine-tune embeddings of to-do texts based on intents. 2 (2) a methodology to collect weak supervision signals from various resources without costly manual annotations. (3) an empirical comparison of contextual embeddings models on real todo texts, where lite outperforms various baseline models including bert, roberta, sentence-bert/roberta, achieving error reduction of 4.8-38.7%. ## method: multi-task learning (mtl) we propose a multi-task learning (mtl) framework to represent to-do descriptions along with their list names (fig. 3 ). our model first encodes text using off-the-shelf encoders ( §3.1). the token representations along with information about their types are merged by an intent extractor with multihead attention ( §3.2). we train the encoder and extractor on three auxiliary tasks ( §3.3,3.4). ## experiments our aim is to obtain a single, general-purpose representation model that is effective on various downstream applications. we run lite on top of bert base , bert large , and roberta and evaluate its performance. ## related work to-do management: intelligent systems can assist users with task management in many ways @xcite . to-do tasks can be inferred automatically from emails @xcite . systems can detect types of to-do items and suggest relevant applications or resources to users @xcite @xcite . once to-do tasks have been created, a system can help users manage the completion progress, e.g., by sending reminders @xcite . complex tasks can be decomposed automatically into more manageable sub-steps @xcite . in all these use cases, a common step is to represent the input language as computational vector representations, but none of the existing studies has produced general-purpose representations of to-do tasks. short-text representations: multiple nlp areas involve very short texts with some unique characteristics. several methods have been @xcite . tweets pose the added challenge of containing many non-standard colloquial expressions and contain non-language text like urls. still, @xcite present a similar finding to ours: massively pre-trained encoders do not always perform well. search queries are also short, with an average of three terms @xcite . unlike to-dos, information such as click logs @xcite can be used as an indicator of user intent. another key difference is that search queries are written with the goal of having a machine interpret them. multi-task learning: multi-task learning improves the performance of pre-trained language models in various nlp tasks @xcite @xcite . the common perception in the research community is that auxiliary training tasks are effective when they are similar to the target domain/task @xcite . however, there are few relevant tasks and datasets for the to-do domain. our study is the first work to propose a time-and cost-efficient way to harvest weak-supervision for mtl in that domain. ## conclusion we discussed how to produce general-purpose representations of short and under-specified to-do texts for performing various kinds of intelligent task assistance with a single encoder. our method, lite, uses a multi-head attention mechanism with token type embeddings on top of an off-the-shelf contextual text encoder to effectively induce semantic information from the combination of to-do descriptions and list names. the model is trained using three auxiliary tasks: autocompletion, pre-action and goal generation, and action arguments prediction. we applied lite to bert base , bert large , and roberta and compared them with various baseline models on four downstream tasks. lite consistently outperformed the baselines, demonstrating the effectiveness of our method.