DART : A Lightweight Quality-Suggestive Data-to-Text Annotation Tool (2020)

Abstract: We present a lightweight annotation tool, the Data AnnotatoR Tool (DART), for the general task of labeling structured data with textual descriptions. The tool is implemented as an interactive application that reduces human efforts in annotating large quantities of structured data, e.g. in the format of a table or tree structure. By using a backend sequence-to-sequence model, our system iteratively analyzes the annotated labels in order to better sample unlabeled data. In a simulation experiment performed on annotating large quantities of structured data, DART has been shown to reduce the total number of annotations needed with active learning and automatically suggesting relevant labels.

## introduction neural data-to-text generation has been the subject of much research in recent years @xcite . traditionally, the task takes as input structured data which comes in the form of tables with attribute and value pairs, and generates free-form, human-readable text. unlabeled data data sampler data raw text data text expert scorer update confidence score [ __dg_inform__ [ __arg_eattype_pub__ eattype_pub ] [ __arg_name__ name ] [ __arg_near__ near ] ] name[blue spice] eattype[coffee shop] area[city centre] __dg_inform__ __arg_eattype_pub__ eattype_pub __arg_name__ name __arg_near__ near name blue spice eattype coffee shop area city centre linearizer data sampler uncertainty scorer unlabeled data ( data, raw text ) confidence score expert ( data, text ) quality estimator send to label ## annotation framework dart is a desktop application built with pyqt5 foot_0 . it is compiled into a single executable with pyinstaller foot_1 , a tool that supports both mac os and windows environments. it contains an intuitive interface as described in section 3. annotation experts interact with dart in the following way: (1) a file containing unlabeled data is uploaded. (2) the system samples some data instances from the file, with a selection strategy based on signals from the sequence-to-sequence uncertainty scorer (section 2.1) and performed with the data sampler (section 2.2). (3) experts then annotate the provided data by correcting the suggested labels (available after the first iteration of (1)-( 2 )). (4) during the process of annotation, the labeled corpus quality is indicated by the annotation quality estimators (section 2.3) for experts to determine if the process were to be terminated. we discuss each component in more detail below. ## experiments data. we use two different types of structured data: (a) attribute-value pairs as used in the crowdsourced e2e dataset @xcite , and (b) the graph-structured data as defined in @xcite on the weather domain. to simulate the annotation process, we employ the given training, development and test sets of each datasets for annotation tool evaluation, with the test set kept fixed. this amounts to roughly 42k samples for e2e and 32k for the weather training sets. ## conclusions while a wide range of annotation tools for nlp tasks exists, most of these tools are targeted at non-textual labels. dart is designed to enable the ease of annotation where the labels are textual descriptions and the inputs are structured data. this is the initial version of the tool, and we hope to extend it to include a web-based version and to expand its functionality in the following ways: (1) support different types of encoders, and (2) improve upon the data sampling process.