FAST : F ast A nnotation tool for S mar T devices (2021)

Abstract: Working with a wide range of annotators with the same attributes is crucial, as in real-world applications. Although such application cases often use crowd-sourcing mechanisms to gather a variety of annotators, most real-world users use mobile devices. In this paper, we propose “FAST,” an annotation tool for application tasks that focuses on the user experience of mobile devices, which has not yet been focused on thus far. We designed FAST as a web application for use on any device with a flexible interface that can be customized to fit various tasks. In our experiments, we conducted crowd-sourced annotation for a sentiment analysis task with several annotators and evaluated annotation metrics such as speed, quality, and ease of use from the tool’s logs and user surveys. Based on the results of our experiments, we conclude that our system can annotate faster than existing methods while maintaining the annotation quality.

## introduction in the annotation of application tasks, it is important to work with a wide range of annotators as in real-world situations such as in the evaluation of the outputs of natural language generation (nlg) systems or sentiment analysis for user reviews. for instance, when evaluating the outputs of an nlg system for textual-ad creatives, the annotators, often called workers in a crowd-sourcing context, are usually required to annotate whether the generated text is fluent or not 1 . although such applications often use crowdsourcing to gather a wide variety of annotators, statistics have shown that a large percentage of real-world users currently use mobile devices such as smartphones @xcite . the application task itself, such as sentiment annotation and fluency assessment of the nlg system outputs, is simple. however, it is important to improve the operation interface and evaluate its contribution in detail. this is because a large amount of data is annotated by annotators with various operation proficiency levels. in addition to the level of proficiency, mobile devices are more likely than desktop devices to be used on the go, during spare time, and in parallel with other tasks @xcite . annotation using mobile devices is expected to collect more data as the time available for work is increased and the effort is reduced. we propose a novel annotation tool called "fast" foot_0 , and its contributions are summarized below. in this section, we review the work related to the issues addressed by fast. first, we note that fast is not aimed at performing complete linguistic annotations; rather it focuses on applied tasks such as quality assessment of nlg systems. more specifically, fast is designed for the annotation of individual tokens and sentences and not for the annotation of relations between tokens and sentences. in addition, we expect the annotators to be users with a wide range of attributes, such as those employed in crowd-sourcing. examples of tools that support extensive and detailed linguistic annotation are brat @xcite , mae (rim, 2016), inception @xcite , and anafora @xcite . recently, there have also been open-source software (oss) tools such as doccano @xcite that are suitable for applied tasks such as multi-label classification. doccano is an oss tool with mobile support and is currently one of the most popular annotation tools, with more than 4,000 stars on github. this tool is focused on industrial targets such as sentiment analysis done by general users, rather than inter-entity structural annotation done by experts e.g. dependency structure annotations or coreference annotations. in this respect, doccano and fast have been developed based on the same motivation, and the target tasks are also similar. although doccano supports mobile devices, it essentially has the same ui structure as its desktop version and thus implementing extensible ui system on the top of it requires ef-forts. therefore, we developed fast from scratch and introduced user interfaces dedicated to mobile devices and mechanisms for the easy custom annotation of interface elements which are required for simulating end-users' environment as close as possible. ## results and discussion the experiment results are presented in ## conclusion in this study, we proposed and developed fast, an annotation tool for application tasks that emphasize the impact of ui/ux on mobile devices, which is a crucial topic, although not investigated thus far. fast is a web application designed for use on any device, including mobiles and pcs. this web application is highly customizable in that the issuers can create views that are optimized for their tasks utilizing two types of ui: card ui and multilabel ui. we compared the devices and uis, as well as an existing tool with fast, in an experiment involving a sentiment analysis task; we also evaluated their efficiency, quality, and usability. the results showed that the mobile operation of fast provides annotators with a more userfriendly experience while maintaining the efficiency and quality of the pc. furthermore, in comparison with the existing tools, fast was able to reduce the annotation time by 43% and an improvement in work efficiency was also confirmed. in the future, we plan to conduct quantitative evaluations using additional detailed indicators such as the trajectory of user operations, and task load metrics measured using nasa tlx. moreover, we aim to increase the number of supported task types, enhance the management functions, and support on-premise environments to strengthen its usefulness as a general-purpose annotation tool.