SLATE : A Sequence Labeling Approach for Task Extraction from Free-form Inked Content (2022)

Abstract: We present SLATE, a sequence labeling approach for extracting tasks from free-form content such as digitally handwritten (or “inked”) notes on a virtual whiteboard. Our approach allows us to create a single, low-latency model to simultaneously perform sentence segmentation and classification of these sentences into task/non-task sentences. SLATE greatly outperforms a baseline two-model (sentence segmentation followed by classification model) approach, achieving a task F1 score of 84.4%, a sentence segmentation (boundary similarity) score of 88.4% and three times lower latency compared to the baseline. Furthermore, we provide insights into tackling challenges of performing NLP on the inking domain. We release both our code and dataset for this novel task.

## introduction the shift to remote and hybrid working styles due to covid-19 has led to a large increase in virtual meetings. it has become increasingly important for participants to express themselves and brainstorm as naturally and effortlessly as possible, leading to an opportunity to extract entities from the large amounts of content created in these meetings. a natural entity of interest is a task created by a participant during the meeting which can be assigned to an individual to complete afterwards. while past works have investigated extracting tasks from typed content such as emails @xcite , there has been less focus on task extraction from more free-form content such as digitally handwritten (or inked) content on a virtual whiteboard or spoken content in a meeting. extracting tasks from freeform content is challenging as it is often not as wellstructured (e.g., poor grammar, inconsistent/lack of punctuation, typos, etc.) furthermore, since this content first needs to be converted to text (e.g., through a handwriting recognition model for ink or asr for speech), downstream nlp models must be robust to errors made by the recognition models. moreover, as we discuss in section 2, past approaches for task extraction from typed content assume text to already be segmented into sentences and focus on building a separate sentence-level task classification model. we cannot make this assumption for our scenario since automatic sentence segmentation models trained on typed content are unlikely to generalize well to the inconsistent capitalization and punctuation in free-form content @xcite . furthermore, separating classification and sentence segmentation creates a latency-challenge: classifying each sentence separately can cause the latency on cpu to scale linearly with the number of sentences, making real-time extraction challenging. in our work, we address these challenges with our proposed approach slate -a sequence labeling approach to task extraction from free-form content. particularly, we apply this approach to inked content, such as that in fig. 1 , and call this application as ink-slate. our contributions are: past work on analyzing digitally inked content has dominantly fallen into either the category of handwriting recognition @xcite or document layout analysis -grouping words into document lines @xcite , grouping document lines into blocks of spatially related text @xcite , or detecting indentation @xcite . our work differs from these past works as it analyzes the semantics of the inked content rather than the layout. furthermore, in an ink-analysis pipeline our work would sit downstream to handwriting recognition and layout analysis rather than replace them; we use, as input to our task extraction system, both recognized text and layout information extracted from the inked document. to our knowledge, our work is the first to tackle both sentence segmentation and task extraction for inked content. past work on task extraction has mainly been applied to typed content such as emails @xcite . other than our different domain of inked content, our work also differs in approach from these past works. particularly, these works assume input text to be already segmented into sentences and focus only on building a classification model to classify these sentences into tasks/non-tasks. thus, implicitly the task extraction systems in these works rely on a two-model approach: a sentence segmentation model to produce sentence candidates, followed by a classification model to classify sentences as tasks/non-tasks or different sub-categories of tasks. our approach, on the other hand, uses a single model to simultaneously perform both sentence segmentation and classification for inked content. sequence labeling is an nlp approach that predicts a label for each token within a sequence, rather than a label for the whole sequence. applications of sequence labeling have traditionally included named entity recognition (ner), part-ofspeech (pos) tagging, text segmentation, etc. in our work, we reframe the typically two-stage problem of task sentence extraction from documents as a single-stage sequence labeling problem. since our approach uses a single, shared model to both segment and classify text, it can be thought of as a form of multi-task learning. multi-task learning has shown to be data-efficient and less prone to overfitting to any single task @xcite . most previous works on sequence labeling use bi-lstm and crf layers in their models @xcite . we instead use a roberta architecture, following the recent success of fine-tuning pretrained transformer lms for data-constrained nlp. @xcite . ## results ## conclusion we have presented slate, a single-model, sequence labeling approach for extracting tasks from free-form content. it overcomes ink domain challenges via our custom ink dataset and ink-document layout information. our flagship configuration, slate-nti, is a single, low-latency model trained for both accurate sentence segmentation and task sentence classification on inked content. ming ye, herry sutanto, sashi raghupathy, chengyang li, and michael shilman. 2005. grouping text lines in freeform handwritten notes. in eighth international conference on document analysis and recognition (icdar'05), pages 367-371. ieee. ming ye and paul viola. 2004. learning to parse hierarchical lists and outlines using conditional random fields. in ninth international workshop on frontiers in handwriting recognition, pages 154-159. ieee. ming ye, paul viola, sashi raghupathy, herry sutanto, and chengyang li. 2007. learning to group text lines and regions in freeform handwritten notes. in ninth international conference on document analysis and recognition (icdar 2007), volume 1, pages 28-32. ieee.