Lin: Unsupervised Extraction of Tasks from Textual Communication (2020)

Abstract: Commitments and requests are a hallmark of collaborative communication, especially in team settings. Identifying specific tasks being committed to or request from emails and chat messages can enable important downstream tasks, such as producing todo lists, reminders, and calendar entries. State-of-the-art approaches for task identification rely on large annotated datasets, which are not always available, especially for domain-specific tasks. Accordingly, we propose Lin, an unsupervised approach of identifying tasks that leverages dependency parsing and VerbNet. Our evaluations show that Lin yields comparable or more accurate results than supervised models on domains with large training sets, and maintains its excellent performance on unseen domains.

## introduction in organizational settings where team members interact with each other, tasks are constantly exchanged through communications. for example, a team leader may request a team member to accomplish a task via email. team members may chat with each other to make commitments about how tasks are assigned within a team. efficient team collaborations, including creating to-do lists and meetings, presume that the tasks are clear. how can natural language processing (nlp) support such practices? we define a task as a verb phrase that specifies a single action to be carried out. one or more tasks could arise in a message from emails or chats, from a sender to a receiver. we consider the root verb of such a verb phrase as its main verb. therefore, we assume identifying main verbs is essential to identifying tasks. consider the following sentences indicating commitments or requests:(1) i will send the qa later today and (2) please reschedule the meeting to next week. each sentence contains only one verb, which can easily be identified as a main verb. however, in some cases, main verbs could be difficult to identify. for example, consider the following sentences containing multiple verbs, and thus, the main verb of a task (in bold) is not obvious (other verbs are underlined): (1) let me know what i need to do to be ready; (2) go ahead and start working on this; (3) before you arrange a meeting, we should think about a few things.; and (4) i would like to meet to discuss or appeal to greg. several contributions exist on identifying tasks from emails and chats. @xcite provide supervised machine learning classifiers for detecting whether a sentence includes a task. lampert et al. @xcite provide a binary classifier to detect requests. @xcite propose binary classifiers to detect the operational forms of business tasks such as commitments of different types. @xcite categorize commitments into three types-request information, schedule meeting, and promise action-and train a deep learning model to identify them. @xcite identify actions such as reply-yesno, reply-ack, and investigate, using a reparametrized long short-term memory (lstm) network. @xcite apply a sequence-to-sequence model to generate to-do lists based on commitments expressed in emails. there are two important limitations to the existing contributions. first, the majority of them are limited to detecting whether a sentence contains a task (binary classification) without identifying the specific task. second, existing studies leverage supervised approaches to identify tasks, which may not be easily generalized to domain-specific task detection where manually annotated datasets are not available. manual annotation of tasks can be cumbersome, especially for domain-specific datasets. to this end, we propose liṅ, an unsupervised approach to identify specific tasks from sentences. to extract specific tasks, liṅ identifies the main verb present in a sentence by jointly modeling the syntactic and semantic information in it. we have evaluated liṅ on an email dataset and a chat dataset. liṅ achieves an f1-score of 80% for the email dataset and 89% for the chat dataset. the results show a significant improvement over the state-of-the-art supervised approaches. ## method liṅ identifies tasks by jointly modeling the syntactic and semantic information in sentences. we observe that tasks can be identified based on the combination of thematic role of the performer of a task, tenses, and other syntactic and semantic features. to extract syntactic features, we consider dependency parsing @xcite . dependency parsing provides diverse relations for verbs, which helps us define more syntactic rules for verbs appearing in different parts of the sentence. to extract semantic features, we consider verbnet @xcite , a structured lexicon focused on verbs. ## experimental setup to evaluate liṅ, we consider two different datasets, an email dataset and a chat dataset. the email dataset comprises of 1,000 emails with a total of 6,418 sentences extracted from the enron corpus @xcite . we extracted 14,132 verb phrases from these sentences, of which 1,910 are labeled as tasks. the labeling was performed by two independent annotators, who achieved an inter-rater agreement (cohen's kappa) of 0.76, indicating a substantial agreement. they resolved their disagreements by discussion to produce the final dataset. the chat dataset comprises of 114 dialogues with a total of 300 sentences extracted from a task-oriented chatbot dialogue dataset @xcite . the same two annotators annotated the dataset with near perfect agreement (cohen's @xmath0.93). we evaluate liṅ as well as its syntax and semantics modules as separate models. for baselines, we adopt the universal sentence encoder @xcite with an svm classifier, pretrained @xcite models. for bert, we used the pretrained 12layer uncased version and fine-tuned it for our labeled dataset. since we want the models to find the exact verbs of tasks, a simple binary classification at sentence level would not be a good baseline. to avoid the sparsity of output classes, we separate the distinct verb phrases (vps) from sentences using dependency parsing and then train these models for binary classification of all vps (whether the current vp represents a task or not). to make sure that each vp contained only one main verb, we constructed vps by including only the immediate descendants of a verb in a parse tree. as a context, we provided the whole sentence along with current vp as inputs. we split the dataset using five-fold cross validation such that 80% was used for training and 20% for testing. the code and dataset can be found at https://github.com/parth27/lin . ## evaluation we adopt accuracy, precision, recall, and f1 score as our metrics. we posit that f1 score is the most reliable metric here since both the datasets are imbalanced and include a preponderance of negatives. ## conclusion identifying the main verbs of tasks in sentences of email or chat can facilitate important downstream tasks. our unsupervised approach achieves comparable or more accurate results than supervised baselines in domains with available training data. evaluation on a chat dataset shows that our unsupervised approach can extend well to unseen domains, which can save time and effort of manual annotations.