Litescale: A Lightweight Tool for Best-worst Scaling Annotation (2021)

Abstract: Best-worst Scaling (BWS) is a methodology for annotation based on comparing and ranking instances, rather than classifying or scoring individual instances. Studies have shown the efficacy of this methodology applied to NLP tasks in terms of a higher quality of the datasets produced by following it. In this system demonstration paper, we present Litescale, a free software library to create and manage BWS annotation tasks. Litescale computes the tuples to annotate, manages the users and the annotation process, and creates the final gold standard. The functionalities of Litescale can be accessed programmatically through a Python module, or via two alternative user interfaces, a textual console-based one and a graphical Web-based one. We further developed and deployed a fully online version of Litescale complete with multi-user support.

## introduction annotation is a cornerstone of computational linguistics and natural language processing (nlp). much of modern nlp is based on machine learning, and in particular on supervised machine learning. as a consequence, large amounts of manually annotated data are constantly in high demand, and the quality of the annotation directly influence the predictive capability of models trained on them. the annotation of natural language resources comes in many forms of varying complexity and levels of abstraction. however, it is possible to categorize most of the approaches in three main families. categorial annotation is perhaps the most common approach, whereas each instance of a dataset is associated with a label from a fixed set of options. annotation can also consist of scalar values, that is, numeric values on a predetermined scale. finally, ranking is a type of annotation where multiple instances are put in a certain order by the annotator, who therefore does not make judgments on instances in isolation but rather on groups of them. recent literature highlights the advantages of the ranking strategy, in terms of the quality of the annotation produced with it, in particular when dealing with subjective aspects of natural language @xcite . the best-worst scaling model (bws) is a ranking-based annotation process developed by @xcite . bws has been proved to be beneficial to the quality of the resulting gold standard data for subjective-related tasks such as emotion detection @xcite and hate speech detection @xcite . the recent work of de bruyne, luna and de clercq, orphée and hoste, veronique (2021) further proved the validity of bws as a method of annotation for sentiment-related tasks, dominance in particular. several general-purpose annotation tools have been proposed in the literature. among the most popular, webanno @xcite and brat @xcite both provide webbased interfaces and the possibility to implement rich annotation schemas, including annotation at the word and span level, and links between annotations. a number of online services are also available to perform linguistic annotation, especially in a crowdsourced fashion, such as amazon mechanical turk foot_0 , appen foot_1 , or lighttag foot_2 . all these systems, however, allow the user to devise schemas that associate labels or scores to individual units of text, whether words, spans, sentences or others. in other words, they natively give the possibility of implementing categorial or ranking annotations. to our knowledge, there is no publicly available system that implements a general-purpose ranking annotation methodology, in particular of the best-worst scaling flavour. ## best-worst scaling annotation in best-worst scaling (bws) annotation, the annotator is shown a tuple consisting of a fixed number of instances, and a phenomenon to annotate. the annotator is then asked to select the instance from the tuple which expresses the phenomenon to the maximum extent, and the one that expresses it to the least extent. for example, in a sentiment polarity annotation task, the annotator may be asked to select the sentences conveying the most positive and the most negative sentiment. starting from the set of instances to annotate, tuples must be created such that: the size of the tuples and the number of tuples in which each instance appears are parameters to be determined at the time of the creation of the tuples. the ratio between the values of these two parameters determines the number of tuples with respect to the number of input instances. in particular, if the two parameters are equal, then the number of tuples will be approximately the same as the instances to annotate. since the number of tuples where any individual instance appears equals the size of the tuples (four items), the number of final questions for the annotators is roughly equivalent to the number of items to be annotated. once the annotation is complete, a score is computed for each instance as the difference between the number of times the instance was selected as "best" and the number of times it was selected as "worst". the score can easily be normalized to fit a [-1, 1] or [0, 1] interval, depending on the needs of the downstream application. since the tuples are computed once, at the beginning of the annotation process, every annotator is presented the same set of instances. it is therefore possible to compute standard measures of inter-annotator agreement such as fleiss' kappa or krippendorf's alpha @xcite . ## litescale in this section, we introduce litescale, a free and open source software package that implements the best-worst scaling annotation framework. litescale is composed of a core python library implementing the core functions, and two alternative interfaces, a console-based textual one and a web-based graphical one. the user can create annotation projects by providing a tab-separated text file, and obtain a similarly formatted file at the end of the annotation. a diagram depicting the information flow and the modules of litescale is in figure 1 . ## multi-user online platform despite being a standalone application, litescale is natively multi-user, because the annotations produces by different users are represented as different json files. these files can reside on the same machine, or different machines if multiple installation of litescale took place. in the latter case, it is sufficient to copy the annotation files of a project from one installation to another to generate a gold standard comprising all the annotations. this process, however can be tedious and error-prone. moreover, the version of litescale presented so far needs to be executed as a series of python scripts, therefore requiring at least a minimal amount of skill in such technology from the annotators. to overcome these issues, and provide an even easier and more accessible user experience, we developed a fully online, multi-user version of litescale, described in this section. the online version of litescale offers the same functionalities as the standalone version described in the previous section, with a user interface very similar to the web-based interface in section 3.3. the front-end was however implemented from scratch, while retaining the core library for the basic functions, although with some modifications. litescale online is a web application implemented with the flask foot_5 python framework. more precisely, the software is designed in a modular way. a restful http api is provided in order to expose the core library functionalities, and a web application provides the user interface by connecting to the api. figure 8 shows the modular architecture of this version of litescale. the restful api is intended as a layer of abstraction to facilitate the future development of tools that access the core library functions remotely, or application that provide alternative interfaces, such as mobile applications. the restful api is implemented with the flask-restful extension of the flask framework foot_6 . the http verbs and their semantics are summarized in in order to make litescale online ready to scale up the number of users and projects, the core library has been modified. in particular, the inner representation and persistence of instances, projects, tuples, and annotations, does not rely on json files but rather on a sql database. the workflow in the online version of litescale is substantially the same as the standalone version, including the interface to create new projects (figure 9 ) and the annotation interface itself. however, additional functionalities were implemented to account for the online multi-user environment. the users can sign up with a valid email address and set up proper log in credentials after receiving a confirmation by email. the user authentication features are managed by the werkzeug foot_7 python library, including the storage of encrypted passwords in the database. moreover, the association of users to projects is also managed through the web application. by default, when a project is created, the user who created it assumes the role of owner of that project. the owner of a project can invite other users to join their projects, by indicating a valid email address (figure 10 ). the recipients of such invitation will receive an automated email notification, inviting them to sign up to litescale if they are not already registered. ## evaluation we conducted a pilot test in order to evaluate the efficacy of litescale in supporting the best-worst scaling annotation of datasets for nlp tasks. we extracted a sample of short hotel reviews in english from the list of 1,000 hotels and their reviews provided by datafiniti's business database foot_8 . we selected the 20 shortest reviews for each of the five ratings (1 to 5 stars) for a total of 100 instances, shuffled then randomly and created an annotation task in litescale. three annotators were asked to perform the annotation and record the times spent annotating each 10 annotations (pairs of best/worst judgments). we computed the pearson correlation between the scores resulting from litescale and the original ratings associated to the reviews, obtain-ing a score of 0.77, indicating strong correlation with the ground truth. the agreement between the annotators is relatively high, considering the subjectivity of the sentiment polarity annotation task. interestingly, the three annotators agreed more on the worst judgments (two annotators agreed 91% of the times, all three agreed 66% of the times) than on the best ones (two annotators agreed 88% of the times, all three agreed 58% of the times). the annotation took about 43 minutes on average. however, the average time spent annotating tends to decrease consistently as the annotation task progresses, as shown in figure 11 . this is not surprising, considering the nature of the bws annotation, where the same instance is shown multiple times, and therefore the annotator gains familiarity with the instances over time. ## conclusion and future work we introduced litescale, a flexible software tool to create and manage linguistic annotation tasks based on the best-worst scaling methodology. litescale is easy to use, free and open source 10 , and ships with different user interfaces. moreover, a multi-user online version of litescale is also presented 11 , which runs entirely in a web browser and provides all the functionalities of the original software. while the bws methodology has been thoroughly tested in the literature, this particular tool has not been systematically tested for its usability (but it is being used at the moment for the creation of several language resources). although all the main features are implemented, there is always room to add extra functionalities and improving the existing ones. a short video describing the main functions of litescale is abailable on youtube: https:// youtu.be/sozwdmh2ah0