Original: RelRoll: A Relative Elicitation Mechanism for Scoring Annotation with A Case Study on Speech Emotion
Revised: RelRoll: A Relative Elicitation Mechanism for Scoring Annotation with A Case Study on Speech Emotion

Original: [Figure]
Revised: [Figure]

Original: Finally additional
Revised: 1) Expert annotators first give reliable absolute labels to training data . The system trains a network using the dataset. 3)

Original:  A 
Revised: The network provides hints (highlights) for non-expert annotators. 4) Non-expert annotators give relative labels to new data. Finally, the network transforms relative labels to absolute labels and additional labeled data is acquired. A BSTRACT

Original: It is challenging for non-experts to give reliable labels in scoring an- notation tasks because of the lack of domain knowledge and the high variability.
Revised: It is challenging for non-experts to give reliable labels in scoring an- notation tasks because of the lack of domain knowledge and the high variability.

Original: We present RelRoll , a mechanism allowing non-experts to give reliable ratings, with a case study on speech emotion.
Revised: We present RelRoll , a mechanism allowing non-experts to give reliable ratings, with a case study on speech emotion.

Original: It includes two main features: a relative labeling interface highlighting emotion-changing sentences and an approach to estimating absolute labels from relative labels.
Revised: It includes two main features: a relative labeling interface highlighting emotion-changing sentences and an approach to estimating absolute labels from relative labels.

Revised: Highlighting can help non-experts focus on sentences needing their actions.

Original: Given new sentences, we utilize a network to predict emotion-changing sentences and highlight them on the interface, which is trained on training set labeled by experts’ absolute annotations.
Revised: Given new sentences, we utilize a network to predict emotion-changing sentences and highlight them on the interface, which is trained on training set labeled by experts’ absolute annotations.

Original: Non-experts will give relative labels to new sentences through our interface.
Revised: Non-experts will give relative labels to new sentences through our interface.

Original: We estimate  absolute labels of new sentences by combining the network output and relative labels.
Revised: We estimate the absolute labels of new sentences by combining the network output and relative labels.

Original: We ran a user study to compare the proposed interface to the commonly used absolute interface.
Revised: We ran a user study to compare the proposed interface to the commonly used absolute interface.

Original: Results showed that RelRoll could improve the annotation agreement and the absolute label annotation accu- racy.
Revised: Results showed that RelRoll could improve the annotation agreement and the absolute label annotation accuracy.

Original: We discussed the user experience enhancement on helpfulness, intuitiveness and easiness.
Revised: We discussed the user experience enhancement on helpfulness, intuitiveness, and easiness.

Original: Interaction design process and methods—User interface

Original: Information systems—Information retrieval—Retrieval tasks and goals—Sentiment analysis

Original: 1 I NTRODUCTION
Revised: 1 I NTRODUCTION

Original: Difficult scoring annotation is ubiquitous (image, audio, motion, etc.)
Revised: Scoring annotation is ubiquitous (image, audio, motion, etc.)

Original: but challenging, such as image quality assessment (IQA), speech emotion annotation, rating physical mobility score(PMS) of clinical videos, etc.
Revised: but challenging, such as image quality assessment, speech emotion anno- tation, rating physical mobility score of clinical videos, etc.

Original: Ideally, these tasks would be conducted in a laboratory and trained experts with specific domain knowledge are recruited to finish the task under supervision.
Revised: Ideally, these tasks would be conducted in a laboratory and trained experts with specific domain knowledge are recruited to finish the task un- der supervision.

Original: However, it is always time-consuming, costly, and limited in the number of annotators.
Revised: However, it is always time-consuming, costly, and limited in the number of annotators.

Original: To enable more efficient and more cost-effective annotation collection, it is a desirable option to include non-expert annotators  in the loop.
Revised: To enable more efficient and more cost-effective annotation collection, it is a desirable option to include non-expert annotators (e.g., crowd workers) in the loop.

Original: Here we define a non-expert annotator as other works, a person (1) has no prior annotation experience in  specific domain and (2) has no sufficient domain knowledge [9,21].
Revised: Here we define a non-expert annotator as other works, a person (1) has no prior annotation experience in a specific domain and (2) has no sufficient domain knowledge [9,21].

Original: For example, it is not feasible to have a non-expert annotator to understand the full range of emotion scale and give reliable annotations [1].
Revised: For example, it is not feasible to have a crowd worker who has never learned emotions and has no annotation experience to understand the full range of emotion scales and give reliable annotations [1].

Original: We focus on  speech emotion annotation task in this study, which is important to Speech Emotion Recognition (SER) as an example of difficult scoring annotation tasks.
Revised: We focus on the speech emotion annotation task in this study, which is important to Speech Emotion Recognition (SER) as an example of difficult scoring annotation tasks.

Original: However, we expect our techniques to be extensible to other tasks of scoring sequences of data.
Revised: However, we expect our techniques to be extensible to other tasks of scoring sequences of data.

Original: Most prior research uses a dimensional description model [39] to label emotions with numbers, which describes emotions along several orthogonal descriptive axes, such as pleasure (from unhap- piness to happiness), arousal (from sleepiness to excitement), etc.
Revised: Most prior research uses a dimensional description model [40] to label emotions with numbers, which describes emotions along several orthogonal descriptive axes, such as pleasure (from unhap- piness to happiness), arousal (from sleepiness to excitement), etc.

Original: The advantage of using the dimensional description model is that its dynamic characteristics correspond to the intrinsic property of emotions, continuously changing [12], which allows a smoother transition between emotional states.
Revised: The advantage of using the dimensional description model is that its dynamic characteristics correspond to the intrinsic property of emotions, continuously changing [12], which allows the smooth transition between emotional states.

Original: Due to a lack of expert annotators, while non-expert annotators are possible in other domains (e.g., labeling common objects in images), they would struggle to label emotions in speech because (1) it is infeasible for non-experts to learn the full range of relationships between absolute values and emotions and (2) the emotion annotation task is highly variable, which may introduce variations among non-experts.
Revised: Due to a lack of expert annotators, while non-expert annotators are possible in other domains (e.g., labeling common objects in images), they would struggle to label emotions in speech because (1) it is infeasible for non-experts to learn the full range of relationships between absolute values and emotions and (2) the emotion annotation task is highly variable, which may introduce variations among non-experts.

Original: In this study, we propose RelRoll , a mechanism allowing non- expert annotators to give reliable scoring labels through relative labels.
Revised: In this study, we propose RelRoll , a mechanism allowing non- expert annotators to give reliable scoring labels through relative labels.

Original: Here we define absolute labels as continuous numerical labels of even intervals (e.g., “1”, “2”, “3”, etc.), and discrete relative labels as level-changing labels of uneven intervals (e.g., “higher”or “lower”).
Revised: Here we define absolute labels as continuous numerical labels of even intervals (e.g., “1”, “2”, “3”, etc.), and discrete relative labels as level-changing labels of uneven intervals (e.g., “higher”or “lower”).

Original: Instead of learning the full range relationships between numerical values and emotions, it is easier for non-experts to tell how much the emotion changes roughly.
Revised: Instead of learning the full range relationships between numerical values and emotions, it is easier for non-experts to tell how much the emotion changes roughly.

Revised: For example, it is easier to tell the change in emotion of a sentence relative to the previous one (e.g., higher arousal) than to give its absolute arousal value (e.g., 4).

Original: The concept of RelRoll is shown in Fig.
Revised: The concept of RelRoll is shown in Fig.

Original: First, we train a neural network on training set annotated by ex- perts through absolute labels, which predicts the absolute emotion (pleasure and arousal) labels.
Revised: First, we train a neural network on training set annotated by ex- perts through absolute labels, which predicts the absolute emotion (pleasure and arousal) labels.

Original: We utilize the network to predict dif- ferences between contiguous sentences in a dialogue from test set and set a threshold to determine which sentences will be highlighted on the interface.
Revised: We utilize the network to predict differ- ences between contiguous sentences in a dialogue from test set and set a threshold to determine which sentences will be highlighted on the interface.

Original: Highlighted sentences are supposed to have emotional changes, which we assume annotators shall focus on during the labeling process.
Revised: Highlighted sentences are supposed to have emotional changes, which we assume annotators shall focus on during the labeling process.

Original: Non-expert annotators are asked to give relative labels to sentences compared to their previous sentences while listening to the dialogue.
Revised: Non-expert annotators are asked to give relative labels to sentences compared to their previous sentences while listen- ing to the dialogue.

Original: We compute the latent emotion values of those sentences from relative labels by Bradley-Terry Model [4].
Revised: We compute the latent emotion values of those sentences from relative labels by Bradley-Terry Model [4].

Original: We then estimate the absolute labels of those sentences by the affine transformation the latent emotion values to the distribution predicted by the network.
Revised: We then estimate the absolute labels of those sentences by the affine transfor- mation of the latent emotion values to the distribution predicted by the network.

Original: We ran a user study to compare the proposed relative labeling interface to the traditional absolute (non-relative) labeling interface in a speech emotion annotation task.
Revised: We ran a user study to compare the proposed relative labeling interface to the traditional absolute (non-relative) labeling interface in a speech emotion annotation task.

Original: Results demonstrated that  annotation agreement of the proposed interface with highlights was higher than both the relative interface without highlights and the absolute interface.
Revised: Results demonstrated that the annotation agreement of the proposed interface with highlights was higher than both the relative interface without highlights and the absolute interface.

Original: In addition, it showed that our interface could improve the annotation accuracy.
Revised: In addition, it showed that our interface could improve the annotation accuracy.

Original: Furthermore, participants responded with their preference of our proposed interface over the traditional interface, on easiness , intuitiveness , and helpfulness in questionnaires.
Revised: Furthermore, participants responded with their preference for our proposed interface over the traditional interface, on easiness , intuitiveness , and helpfulness in questionnaires.

Original: Their responses proved that RelRoll could lower the learning curve of annotation.
Revised: Their responses proved that RelRoll could lower the learning curve of annotation.

Original: Our contribution is mainly three-fold:
Revised: Our contribution is mainly three-fold:

Original: • RelRoll , an approach to estimating absolute labels from relative labels given by non-expert annotators, which can lower the cost of collecting labels for large datasets.
Revised: • RelRoll , an approach to estimating absolute labels from relative labels given by non-expert annotators, which can lower the cost of collecting labels for large datasets.

Original: • a relative labeling interface design for non-expert annotators in scoring annotation tasks, which can improve annotation agreement, labeling accuracy and user experience.
Revised: • a relative labeling interface design for non-expert annotators in scoring annotation tasks, which can improve annotation agreement, labeling accuracy, and user experience.

Original: • takeaways from a36 people user study to elicit absolute la- bels from relative labels in a speech emotion annotation task, validating the usability of RelRoll . 2 R ELATED W ORK
Revised: • takeaways from a user study eliciting absolute labels from relative labels in a speech emotion annotation task, validating the usability of RelRoll . 2 R ELATED W ORK

Original: In this section, we explain the background and describe related work.
Revised: In this section, we explain the background and describe related work.

Original: We start with labeling tasks (scoring annotation and speech emotion) and then introduce labeling mechanisms (absolute and relative labeling).
Revised: We start with labeling tasks (scoring annotation and speech emotion) and then introduce labeling mechanisms (absolute and relative labeling).

Original: 2.1 Scoring Annotation and Speech Emotion
Revised: 2.1 Scoring Annotation and Speech Emotion

Original: Many scoring annotation tasks are of high variability, most of which are done in laboratory and controlled conditions.
Revised: Many scoring annotation tasks are of high variability, most of which are done in laboratory and controlled conditions.

Original: Speech emotion annotation usually involves multiple annotators to account for vari- ations in emotion perception across individuals [7].
Revised: Speech emotion annotation usually involves multiple annotators to account for vari- ations in emotion perception across individuals [7].

Original: Image quality assessment is designed for evaluating the human perceptual quality of images, which needs subjective annotations [20].
Revised: Image quality assessment is designed for evaluating the human perceptual quality of images, which needs subjective annotations [20].

Original: In addition,  rating is also commonly used in clinical diagnosis of physical mo- bility [33], where experts rate the severity of patients’ mobility dysfunctions and recognize their need for supervision or assistance.
Revised: In addition, the rating is also commonly used in clinical diagnosis of physical mobility [34], where experts rate the severity of patients’ mobility dysfunctions and recognize their need for supervision or assistance.

Original: In speech emotion annotation, there are two principal models to describe emotions, the categorical description model and the dimensional description model.
Revised: In speech emotion annotation, there are two principal models to describe emotions, the categorical description model and the dimensional description model.

Original: The categorical description model [16] is more common, in which emotions are described around some basic emotion categories, such as happy, angry, sad, etc.
Revised: The categorical description model [16] is more common, in which emotions are described around some basic emotion categories, such as happy, angry, sad, etc.

Original: The other commonly used dimensional model is developed by Mehrabian and Russel [34], using pleasure, arousal, and dominance to represent all emotions.
Revised: The other commonly used dimensional model is developed by Mehrabian and Russel [35], using pleasure, arousal, and dominance to represent all emotions.

Original: Pleasure and arousal are termed as “core affect”, while dominance is related to the relationship between people.
Revised: Pleasure and arousal are termed as “core affect”, while dominance is related to the relationship between people.

Original: We only focus on pleasure and arousal in this work.
Revised: We only focus on pleasure and arousal in this work.

Original: Semi-natural or natural datasets are the most suitable for dimensional emotion analysis [19], such as IEMOCAP [7], MSP- PODCAST [32].
Revised: Semi-natural or natural datasets are the most suitable for di- mensional emotion analysis [19], such as IEMOCAP [7], MSPPODCAST [33].

Original: In these datasets, emotional speech is collected from talk shows, interviews, panel discussions, dialogues,  plays within a context.
Revised: In these datasets, emotional speech is collected from talk shows, interviews, panel discussions, dialogues, and plays within a context.

Original: Sentences are collected from a continuous speech, which inspired our design of a continuum labeling interface.
Revised: Sentences are collected from a continuous speech, which inspired our design of a continuum labeling interface.

Original: In this work, we use the improvised sessions (80 dialogues) from IEMOCAP following the suggestions from [45], as they are more similar to natural speech and more emotion-balanced [8].
Revised: In this work, we use the improvised sessions (80 dialogues) from IEMO- CAP following the suggestions from [47], as they are more similar to natural speech and more emotion-balanced [8].

Original: 2.2 Absolute and Relative Labeling
Revised: 2.2 Absolute and Relative Labeling

Original: Labeling tasks are commonly done through absolute labeling, e.g., giving a numerical or semantical label to an instance.
Revised: Labeling tasks are commonly done through absolute labeling, e.g., giving a numerical or semantical label to an instance.

Original: It has wide applications in video, image, audio, text, 3D model, and other data formats [9,18,22,28,37,43].
Revised: It has wide applications in video, image, audio, text, 3D model, and other data formats [9,18,22,29,38,45].

Original: However, it is not always the most reliable method.
Revised: However, it is not always the most reliable method.

Original: When there is a lack of commonly accepted scale, absolute labeling will introduce disagreement and inconsistency to the result [6].
Revised: When there is a lack of commonly accepted scale, absolute labeling will introduce disagreement and inconsistency to the result [6].

Original: Especially when non-experts are involved, the disagreement and inconsistency will invalidate the labels.
Revised: Especially when non-experts are involved, the disagreement and inconsistency will invalidate the labels.

Original: As for classification annotation, many techniques including structured la- beling [10,26], explanatory debugging [27], etc., tried to improve the consistency and agreement among crowdsourcing workers.
Revised: As for classification annotation, many techniques including structured la- beling [10,27], explanatory debugging [28], etc., tried to improve the consistency and agreement among crowdsourcing workers.

Original: An alternative method to absolute labeling is relative labeling.
Revised: An alternative method to absolute labeling is relative labeling.

Original: Relative labeling means that annotators will compare one instance to another, through a preference or ranking expression, such as “better than”, “worse than”, “equal”, etc.
Revised: Relative labeling means that annotators will compare one instance to another, through a preference or ranking expression, such as “better than”, “worse than”, “equal”, etc.

Original: This method focuses on the local relationship instead of the global scale, which makes the task more intuitive and easier.
Revised: This method focuses on the local relationship instead of the global scale, which makes the task more intuitive and easier.

Original: Relative labeling is reliable and accurate in many tasks when variability is involved in the task, for example, emotion labeling, age labeling, quality labeling, and so on [15,23,30,31,36].
Revised: Relative labeling is reliable and accurate in many tasks when variability is involved in the task, for example, emotion labeling, age labeling, quality labeling, and so on [15,24,31,32,37].

Original: However, collecting relative labels is costly as a pairwise comparison will square the number of labeling times (i.e. N pairwise comparisons per instance, N 2 labels needed in total).
Revised: However, collecting relative labels is costly as a pairwise comparison will square the number of labeling times (i.e. N pairwise comparisons per instance, N 2 labels needed in total).

Original: Many researchers try to improve this challenge through interaction support.
Revised: Many researchers try to improve this challenge through interaction support.

Original: SorTable [40] allows users to make sets of relative labels and uses the Bayesian method to estimate the complete ranking.
Revised: SorTable [41] allows users to make sets of relative labels and uses the Bayesian method to estimate the complete ranking.

Original: Goldilocks [11] displays previously annotated items as anchors and allows users to give the upper and the lower bounds of a new instance by referring to already-labeled instances.
Revised: Goldilocks [11] displays previously annotated items as anchors and allows users to give the upper and the lower bounds of a new instance by referring to already-labeled instances.

Original: Both Absolute labeling and relative labeling can be integrated with a continuum interface to realize fast labeling. 
Revised: Annotation process can be integrated with techniques to realize more efficient labeling. OLALA [42] only selects the ambiguous predicted portions for humans to annotate through active learning. Felix et al. [18] proposed an unsupervised machine learning method to allow users to discover and create label categories in an evolution- ary fashion.

Original: Ranjay et al.
Revised: Ranjay et al.

Original: [25] proposed a technique for producing rapid absolute labels for a running stream of images.
Revised: [26] proposed a technique for producing rapid absolute labels for a running stream of images.

Original: SCRIBE [29] introduced an approach to collecting complete transcripts of videos by merging partial inputs from groups of non-expert captionists in real time.
Revised: SCRIBE [30] introduced an approach to collecting complete transcripts of videos by merging partial inputs from groups of non-expert captionists in real time.

Revised: SCRIBE can be effective for obtaining relatively accurate answers from a group of non-experts, but it is infeasible for only one non-expert in most cases.

Original: In the emotion labeling task, Self-Assessment Manikin (SAM) [3] is the most commonly used interface, which can be categorized as an absolute labeling interface.
Revised: In the emotion labeling task, Self-Assessment Manikin (SAM) [3] is the most commonly used interface, which can be categorized as an absolute labeling interface.

Original: Two-dimension slider with color representation [13], facial expression suggestion [5], and mobile integration [44] have also been introduced to assist users with la- beling emotions.
Revised: Two-dimension slider with color representation [13], facial expression suggestion [5], and mobile integration [46] have also been introduced to assist users with la- beling emotions.

Original: However, none of the research tried to include non-expert annotators because uncertainty may be introduced.
Revised: However, none of the research tried to include non-expert annotators because uncertainty may be introduced.

Original: To allow non-expert annotators to give reliable labels, we select relative labeling as our approach.
Revised: To allow non-expert annotators to give reliable labels, we select relative labeling as our approach.

Original: To balance the uncertainty and cost, we integrate the Bradley-Terry Model [4] with a continuum labeling interface, which allows us to obtain N 2 pairwise difference labels in N time.
Revised: To balance the uncertainty and cost, we integrate the Bradley-Terry Model [4] with a continuum labeling interface, which allows us to obtain N 2 pairwise difference labels in N time.

Original: 3 R EL R OLL I NTERFACE
Revised: I NTERFACE

Original: In speech emotion annotation tasks, non-experts may be faced with a steep learning curve to learn the relationship between speech emotions and numerical values through tutorials and practice.
Revised: In speech emotion annotation tasks, non-experts may be faced with a steep learning curve to learn the relationship between speech emotions and numerical values through tutorials and practice.

Original: We need to reduce the difficulty of labeling tasks to include non-experts in the loop.
Revised: We need to reduce the difficulty of labeling tasks to include non-experts in the loop.

Original: Another challenge is the low agreement among annotators due to the high variations in emotion understanding, which also exists among expert annotators.
Revised: Another challenge is the low agreement among annotators due to the high variations in emotion understanding, which also exists among expert annotators.

Original: Thus it is also important to improve the agreement among the annotators to ensure the labeling quality.
Revised: Thus it is also important to improve the agreement among the annotators to ensure the labeling quality.

Original: In this section, we explain the details of designing the interface to elicit relative labels from non-experts, with two key design goals, (1) to lower the learning curve and (2) to reduce the variability.
Revised: In this section, we explain the details of designing the interface to elicit relative labels from non-experts, with two key design goals, (1) to lower the learning curve and (2) to reduce the variability.

Original: We com- pared the proposed interface to the baseline interface with design details.
Revised: We com- pared the proposed interface to the baseline interface with design details.

Original: 3.1 Baseline Interface
Revised: 3.1 Baseline Interface

Original: The most commonly used labeling interface in speech emotion an- notation tasks is the absolute labeling interface.
Revised: The most commonly used labeling interface in speech emotion an- notation tasks is the absolute labeling interface.

Original: Fig.
Revised: Fig.

Original: 2 shows an example of an absolute labeling interface for emotional speech.
Revised: 2 shows an example of an absolute labeling interface for emotional speech.

Original: We created this interface by modifying the prior work, SAM [3].
Revised: We created this interface by modifying the prior work, SAM [3].

Original: It resembles the commonly used speech emotion labeling interface, which also serves as the baseline interface in our user study.
Revised: It resembles the commonly used speech emotion labeling interface, which also serves as the baseline interface in our user study.

Original: It has three main components, the brief description of the dialogue content, the labeling component, and the transcript.
Revised: It has three main components, the brief description of the dialogue content, the labeling component, and the transcript.

Original: Users have to listen to a sentence (one blue block in Fig. 2) and give their label to the sentence by a slider.
Revised: Users have to listen to a sentence (one blue block in Fig. 2) and give their label to the sentence by a slider.

Original: As in this example, the numerical scale is 1, 2, 3, 4, and 5, which is used in the data collection process of IEMOCAP [7].
Revised: As in this example, the numerical scale is 1, 2, 3, 4, and 5, which is used in the data collection process of IEMOCAP [7].

Original: This type of interface does not support lowering  learning curve and reducing variability.
Revised: This type of interface does not support lowering the learning curve and reducing variability.

Original: For example, it is difficult to assign an absolute numeric value to pleasure for a non-expert, as they do not understand the relations between those numbers and pleasure.
Revised: For example, it is difficult to assign an absolute numeric value to pleasure for a non-expert, as they do not understand the relationship between those numbers and pleasure.

Original: We add  following two features (not present in the original SAM) to the baseline to facilitate fast labeling.
Revised: We add the following two features (not present in the original SAM) to the baseline to facilitate fast labeling.

Original: Non-stop playing with sentence-level splits.
Revised: Non-stop playing with sentence-level splits.

Original: The labeling component also serves as an audio streaming player of the dialogue.
Revised: The labeling component also serves as an audio streaming player of the dialogue.

Original: In each task, users are required to focus on one emotional dimension (pleasure or arousal) of one speaker in the dialogue.
Revised: In each task, users are required to focus on one emotional dimension (pleasure or arousal) of one speaker in the dialogue.

Original: During labeling, the playing head indicates which sentence is playing.
Revised: During labeling, the playing head indicates which sentence is playing.

Original: We choose the non-stop playing for fast labeling, which requires users to give labels before one sentence ends.
Revised: We choose the non-stop playing for fast labeling, which requires users to give labels before one sentence ends.

Original: Users can modify their labels by rewinding to previous sentences if needed.
Revised: Users can modify their labels by rewinding to previous sentences if needed.

Original: Keyboard interaction.
Revised: Keyboard interaction.

Original: Users operate labeling and audio playing through keyboard interactions, up arrow and down arrow to move the thumb on the slider, left arrow and right arrow for the rewinding and fast-forwarding of the audio.
Revised: Users operate labeling and audio playing through keyboard interactions, up arrow and down arrow to move the thumb on the slider, left arrow and right arrow for the rewinding and fast-forwarding of the audio.

Original: We choose the keyboard interaction instead of the mouse interaction, as we believe it is easier to operate.
Revised: We choose the keyboard interaction instead of the mouse interaction, as we believe it is easier to operate.

Original: In this example, users are  to 
Revised: In this example, users are listening to the 3rd sentence and giving a label, 3.

Original: 3.2 Proposed Interface
Revised: 3.2 Proposed Interface

Original: Our proposed interface of RelRoll is shown in Fig.
Revised: Our proposed interface of RelRoll is shown in Fig.

Original: It has  same components as in the baseline interface.
Revised: It has the same components as in the baseline interface.

Original: It also shares non-stop playing with sentence-level splits and keyboard interaction as the baseline interface.
Revised: It also shares nonstop playing with sentence-level splits and keyboard interaction as the baseline interface.

Original: Three main different features are described below.
Revised: Three main different features are described below.

Original: Relative labeling.
Revised: Relative labeling.

Original: We use a five-level scale of relative labels, “much higher”: ↑↑ , “higher”: ↑ , “no difference”: —, “lower”: ↓ , and “ much lower”: ↓↓ .
Revised: We use a five-level scale of relative labels, “much higher”: ↑↑ , “higher”: ↑ , “no difference”: —, “lower”: ↓ , and “ much lower”: ↓↓ .

Original: They allow users to respond how much they perceive a sentence’s pleasure or arousal is different from its previous one’s.
Revised: They allow users to respond how much they perceive a sentence’s pleasure or arousal is different from its previous one’s.

Original: We choose five as the number of levels because it is readily comprehensible and enables people to express their perception of emotion changes.
Revised: We choose five as the number of levels because it is readily comprehensible and enables people to express their perception of emotion changes.

Original: Users give relative labels by pressing up arrow and down arrow on the keyboard.
Revised: Users give relative labels by pressing up arrow and down arrow on the keyboard.

Original: If no action is taken when a sentence playing ends, there will be a dash — for “no difference” shown.
Revised: If no action is taken when a sentence playing ends, there will be a dash — for “no difference” shown.

Original: Highlight suggestions.
Revised: Highlight suggestions.

Original: To further assist non-expert annotators, we highlight sentences that may need them to take actions.
Revised: To further assist non-expert annotators, we highlight sentences that may need them to take action.

Original: We train a neural network to predict which sentence has a different emotion as compared to its previous one.
Revised: We train a neural network to predict which sentence has a different emotion as compared to its previous one.

Original: If the network predicts a difference, there will be a light yellow highlight above the blue sentence block.
Revised: If the network predicts a difference, there will be a light yellow highlight above the blue sentence block.

Original: We believe highlights can makethe labeling easier for non-experts, as highlights will reduce their mental effort.
Revised: We believe highlights can make labeling easier for non-experts, as highlights will reduce their mental effort.

Original: Transcript reference.
Revised: Transcript reference.

Original: Another feature implementing the idea of relativity is the transcript component.
Revised: Another feature implementing the idea of relativity is the transcript component.

Original: Transcripts of all speakers are shown below the labeling component, where the last sentences are also shown to users for reference.
Revised: Transcripts of all speakers are shown below the labeling component, where the last sentences are also shown to users for reference.

Original: We believe that transcripts of last sentences can remind users how last sentences sound.
Revised: We believe that transcripts of last sentences can remind users how last sentences sound.

Original: It may help them make decisions on labeling emotion changes between current sentences and previous sentences.
Revised: It may help them make decisions on labeling emotion changes between current sentences and previous sentences.

Revised: In this example, users are listening to the 5th sentence and giving a relative label, ↓ .

Original: 4 I MPLEMENTATION
Revised: 4 I MPLEMENTATION

Original: Overview of the mechanism is shown in Fig.
Revised: An overview of the mechanism is shown in Fig.

Original: We use an existing dataset (IEMOCAP) with expert labels as training set , and then train the speech emotion prediction network (Sect. 4.1).
Revised: We use an existing dataset (IEMOCAP) with expert labels as training set , and then train the speech emotion prediction network (Sect. 4.1).

Original: The network is used for highlighting sentences for non-expert annotation and calculating the distribution of the sentences’ emotion.
Revised: The network is used for highlighting sentences for non-expert annotation and calculating the distribution of the sentences’ emotion.

Original: The relative labels from non-experts are quantified to absolute labels using a
Revised: The relative labels from non-experts are quantified to absolute labels using a

Original: Bradley-Terry Model (Sect. 4.2) and affine transformation of the distribution (Sect. 4.3).
Revised: Bradley-Terry Model (Sect. 4.2) and affine transformation of the distribution (Sect. 4.3).

Original: 4.1 Network
Revised: 4.1 Network

Revised: The SER neural network in the mechanism is designed for two purposes: (1) to predict emotion-changing sentences (highlighted sentences in Fig.

Revised: 3) on the interface and (2) to predict the emotion value distribution of new sentences of a speaker from a conversation.

Revised: In this section, we describe features, architecture, and how we predict highlighting.

Original: Features.
Revised: Features.

Original: We extract frame-by-frame low-level descriptors (LLDs) such as fundamental frequency and Mel-frequency cepstral coefficients (MFCCs) by a commonly used audio processing tool, Opens- mile [17].
Revised: We extract frame-by-frame low-level descriptors (LLDs) such as fundamental frequency and Mel-frequency cepstral coefficients (MFCCs) by a commonly used audio processing tool, Opensmile [17].

Original: For each sentence, we extract global statistics of the LLDs, such as arithmetic mean (e.g., mean of the fundamental frequency).
Revised: For each sentence, we extract global statistics of the LLDs, such as arithmetic mean (e.g., mean of the fundamen- tal frequency).

Original: These global statistics are referred to as high-level functionals (HLF).
Revised: These global statistics are referred to as high-level functionals (HLF).

Original: This method generates 6,373 features for each sentence.
Revised: This method generates 6,373 features for each sentence.

Original: Network.
Revised: Network architecture.

Original: To predict emotion difference between a pair of sen- tences, we build a simple SER neural network similar to the work by Parthasarathy et al.
Revised: To predict emotion difference between a pair of sentences, we build a simple SER neural network similar to the work by Parthasarathy et al.

Original: The  network takes the input into the shared hidden layers and then separately connects them for pleasure difference and arousal prediction.
Revised: The SER neural network takes the input into the shared hidden layers and then separately connects them for pleasure difference and arousal prediction.

Original: The separated layer can learn dimension-dependent information and can optimize for its own goal.
Revised: The separated layer can learn dimension-dependent information and can optimize for its own goal.

Original: The architecture of the  network is shown in Fig.
Revised: The architecture of the SER neural network is shown in Fig.

Revised: The depth of the shared hidden layer in the model is 4 and the dimension of each layer is [2048, 512, 64, 64].

Revised: There is a rectified linear unit (ReLu) activation function in between the shared hidden layers for introducing non-linearity into the output.

Revised: Then the dimension of the separated hidden layer is 6.

Revised: All of these hyper-parameters are decided by the preliminary experiments we conducted.

Revised: [Figure]

Original: The objective function of this task is to minimize the mean square error between the ground truth differences and the predicted differences.
Revised: The objective function of this task is to minimize the mean square error between the ground truth differences and the predicted differences.

Original: We denote the loss function of pleasure as following (arousal is same):
Revised: We denote the loss function of pleasure as following (arousal is same):

Original: [Equation]
Revised: [Equation]

Original: where y pleasure is the ground truth value of pleasure, and ˆ y pleasure is the value of pleasure predicted by the  network.
Revised: where y pleasure is the ground truth value of pleasure and ˆ y pleasure is the value of pleasure predicted by the SER neural network.

Original: [Figure]

Original: We follow the optimization procedure in [35] and make it a bal- ance of weighted pleasure difference and weighted arousal difference.
Revised: We follow the optimization procedure in [36] and make it a bal- ance of weighted pleasure difference and weighted arousal difference.

Original: The following formula describes the overall loss for the net- work:
Revised: The following formula describes the overall loss for the SER neural network:

Original: [Equation]
Revised: [Equation]

Original: where α adjusts weights of pleasure loss and arousal loss.
Revised: where α adjusts weights of pleasure loss and arousal loss.

Original: We choose 0.55 as the value of α for the best performance after experimenting with different values.
Revised: We choose 0.55 as the value of α for the best performance after experimenting with different values.

Original: Please note that the neural network in the mechanism is designed for two purposes: (1) to predict emotion-changing sentences (high- lighted sentences in Fig. 3) on  the interface and(2) to predict the emotion value distribution of the new data.
Revised: The training was conducted on a GPU machine with Pytorch version 1.5 by mini-batch training using Adam optimisation (learning rate 0.0002) [23], the batch size is 16, and the epoch is 30.

Original:  As for predicting high- lighted sentences, we set a threshold of predicted numerical difference as 0.5, above which the latter sentence of two consecutive sentences will be highlighted.
Revised: Highlighting prediction. As for predicting highlighted sentences, we set a threshold of predicted numerical difference as 0.5, above which the latter sentence of two consecutive sentences will be highlighted.

Original: We choose to use mean square error as the objective function, instead of directly using those loss functions designed for binary classification problems.
Revised: Please note that we still choose to use mean square error as the objective function, instead of directly using those loss func- tions designed for binary classification problems.

Original: We train theneural network on training set for 30 epochs.
Revised: This is because the second purpose of the network mentioned at the beginning of this subsection, which is we need the SER network to predict the distribution of new data.

Original: 4.2 Quantification of Relative Labels
Revised: 4.2 Quantification of Relative Labels

Original: In this section and Sect.
Revised: In this section and Sect.

Original: 4.3, we take pleasure as the example, as arousal is processed in the same way.
Revised: 4.3, we take pleasure as the example, as arousal is processed in the same way.

Original: Givena set of N sentences, we assume there is a single ranking R of pleasure value by:
Revised: Given N sentences of one speaker in a conversation, we assume there is a single ranking R of pleasure value by:

Original: [Equation]
Revised: [Equation]

Original: where sentence s 1 has a higher or equal pleasure value than sen- tence s 2 , and so on.
Revised: where sentence s 1 has a higher or equal pleasure value than sen- tence s 2 , and so on.

Original: We use an extended Bradley-Terry Model [4] to describe the probability of this situation:
Revised: We use an extended Bradley-Terry Model [4] to describe the probability of this situation:

Original: [Equation]
Revised: [Equation]

Original: where π i represents the latent pleasure value of s i , and r de- notes the reliability of annotators, estimated by a prior distribution,
Revised: where π i represents the latent pleasure value of s i , and r de- notes the reliability of annotators, estimated by a prior distribution,

Original: Gamma Distribution.
Revised: Gamma Distribution.

Original: It is trying to estimate π by maximum likeli- hood.
Revised: It is trying to estimate π by maximum likeli- hood.

Revised: By using this method, we could get the latent pleasure values of those N sentences from the relative labels of an annotator.

Original: Due to the sequential comparisons acquired by a continuum label- ing interface, we cannot acquire a full set of pairwise comparisons of the set.
Revised: Due to the sequential comparisons acquired by a continuum label- ing interface, we cannot acquire a full set of pairwise comparisons of the set.

Original: For example, if sentence s 2 is “lower” than sentence s 1 and sentence s 3 is “higher” than s 2 , we cannot interpret the relationship between s 1 and s 3 .
Revised: For example, if sentence s 2 is “lower” than sentence s 1 and sentence s 3 is “higher” than s 2 , we cannot interpret the relationship between s 1 and s 3 .

Original: Thus we add regularization to complement the full comparison set.
Revised: Thus we add regularization to complement the full comparison set.

Original: 4.3 Estimation of Absolute Labels
Revised: 4.3 Estimation of Absolute Values

Original: The final goal is to obtain absolute labels of pleasure in the absolute scale as IEMOCAP from  latent pleasure values  obtained in Sect.
Revised: The final goal is to estimate the absolute values of pleasure Y from the latent pleasure values Y L obtained in Sect.

Original: Here me make an assumption that both the latent pleasure values andground-truth absolute pleasure values are normally distributed random variables.
Revised: Given sentences of one speaker in a conversation, we make an assumption that both the latent pleasure values and absolute pleasure values follow normal distributions.

Original: Then we denote latent pleasant values Y L of a group of unseen sentences computed by Equation 4 as:
Revised: Wedenote the latent pleasant values Y L of the sentences computed from Equation 4 as:

Original: [Equation]
Revised: [Equation]

Original: where µ L is the mean of latent pleasure values, and σ L is the standard deviation.
Revised: where µ L is the mean of latent pleasure values, and σ L is the standard deviation.

Original: Then we try to obtain the final estimation Y E ofthe absolute pleasure valuesby affine transforming Y L to make it follow the predicted distribution by the network:
Revised: We denote the distribution of absolute pleasure values Y P pre- dicted by the SER neural network (the second purpose of the SER neural network described in the Sect. 4.1):

Original: [Equation]
Revised: [Equation]

Original: where µ P is the mean of predicted  pleasure values by the  networkin Sect. 4.1, and σ P is the standard deviation.
Revised: where µ P is the mean of predicted absolute pleasure values by the SER neural network and σ P is the standard deviation.

Revised: Then we try to obtain the final estimation Y of the absolute plea- sure value by affine transforming Y L to make it follow the Y E ’s distribution.

Original: We denote the affine transformation as:
Revised: We denote the affine transformation as:

Original: [Equation]
Revised: [Equation]

Original: where A can be calculated by σ P / σ L , and b can be calculated by µ
Revised: where A can be calculated by σ P / σ L , and b can be calculated by µ

Original: P − A µ L .
Revised: P − A µ L .

Original: After the affine transformation, we clip off the final  estimations Y 
Revised: After the affine transformation, we clip off the final absolute pleasure estimations Y to the range of the scale used in

Original: L to the range of the scale used in IEMOCAP, i.e. from 1 to 5.
Revised: IEMOCAP, i.e. from 1 to 5.

Original: We evaluate the  absolute emotion value estimation using the con- cordance correlation coefficient (CCC).
Revised: We evaluate the estimated absolute emotion values using the con- cordance correlation coefficient (CCC).

Original: CCC measures the agreement between the ground truth emotion values and the predicted emotion values.
Revised: CCC measures the agreement between the ground truth emotion values and the estimated emotion values.

Original: It is a commonly used reliable evaluation metric in
Revised: It is a commonly used reliable evaluation metric in

Original: SER, as it is penalized in proportion to the deviation when predic- tions shift in value [38].
Revised: SER, as it is penalized in proportion to the deviation when predic- tions shift in value [39].

Original: The range of CCC is from -1 to 1, with a perfect agreement at 1.
Revised: The range of CCC is from -1 to 1, with a perfect agreement at 1.

Original: 5 U SER S TUDY
Revised: 5 U SER S TUDY

Original: We conducted a user study to examine the effectiveness of RelRoll .
Revised: We conducted a user study to examine the effectiveness of Rel- Roll .

Original: In particular, we compared three conditions (shown in Fig.
Revised: In particular, we compared three conditions (The main differences are shown in Fig.

Revised: 5): (1) RelRoll with highlight suggestions ( Highlight ), the same as described in Sect.

Original: RelRoll with highlight suggestions ( Highlight ), (2) RelRoll without highlight suggestions ( Relative ),  and (3) a traditional absolute labeling interface as a baseline ( Absolute ).
Revised: 3.2, (2) RelRoll without highlight suggestions ( Relative ), the interface described in Sect. 3.2 with highlight suggestions ablated, and (3) a traditional absolute labeling interface as a baseline ( Absolute ), same as described in Sect.

Original: We aimed to answer these Research Questions (RQs) in this study:
Revised: We aimed to answer these Research Questions (RQs) in this study:

Original: RQ1 : Would relative labels improve the annotation agreement (inter-annotator agreement), and would highlight suggestions further improve it?
Revised: RQ1 : Would relative labels improve the annotation agreement (inter-annotator agreement), and would highlight suggestions further improve it?

Original: RQ2 : Would highlight suggestions and relative labeling improve the annotation accuracy?
Revised: RQ2 : Would highlight suggestions and relative labeling improve the annotation accuracy?

Original: Would relative labels from non-expert annotators be valid and reliable?
Revised: Would relative labels from non-expert annotators be valid and reliable?

Original: RQ3 : Compared with the traditional absolute labeling interface, would RelRoll be easier for non-expert annotators to use?
Revised: RQ3 : Compared with the traditional absolute labeling interface, would RelRoll be easier for non-expert annotators to use?

Original: Would it lower the learning curve of annotation?
Revised: Would it lower the learning curve of annotation?

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  5.1 Pilot Study
Revised: [Table] [Table] [Table] 5.1 Pilot Study

Original: We ran a pilot study with three participants to check the interface design and the choice of dialogue.
Revised: We ran a pilot study with three participants to check the interface design and the choice of dialogue.

Original: We tested keyboard operation and mouse operation, and received all preferences on keyboard oper- ations.
Revised: We tested keyboard operation and mouse operation, and received all preferences on keyboard oper- ations.

Original: We tested four dialogues of different base emotions (anger, happiness, sadness, and neutral) with participants to check whether the base emotion can be a factor influencing the annotation results.
Revised: We tested four dialogues of different base emotions (anger, happiness, sadness, and neutral) with participants to check whether the base emotion can be a factor influencing the annotation results.

Original: While we found similar annotation accuracy results among different base emotions, participants reported fatigue when the dialogue was long in duration.
Revised: While we found similar annotation accuracy results among different base emotions, participants reported fatigue when the dialogue was long in duration.

Original: Thus we considered the duration when we chose the dialogue for the user study, limiting it to 3 minutes or less.
Revised: Thus we considered the duration when we chose the dialogue for the user study, limiting it to 3 minutes or less.

Original: In addition, we found original sentence-level split meta information is not perfectly suitable for non-experts to annotate (e.g., there are[Figure] short fillers such as “Um.
Revised: In addition, we found original sentence-level split meta information is not perfectly suitable for non-experts to annotate (e.g., there are short fillers such as “Um.

Original: Okay.” ).
Revised: Okay.” ).

Original: We filtered out sentences not suitable for the user study.
Revised: We filtered out sentences not suitable for the user study.

Original: 5.2 Participants
Revised: 5.2 Participants

Original: We recruited 36 participants through the Amazon Mechanical Turk Masters service [14] to realize a between-subject experiment design, where 12 participants were assigned to each condition.
Revised: We recruited 36 participants through the Amazon Mechanical Turk Masters service [14] to realize a between-subject experiment design, where 12 participants were assigned to each condition.

Original: To guarantee all participants would be “non-experts”, we asked them two questions in the pre-test questionnaire: (1) whether they had previous experience in labeling emotional data and (2) whether they had been trained on emotion comprehension to ensure that all of them were non-expert annotators.
Revised: To guarantee all participants would be “non-experts”, we asked them two questions in the pre-test questionnaire: (1) whether they had previous experience in labeling emotional data and (2) whether they had been trained on emotion comprehension to ensure that all of them were non-expert annotators.

Original: Only those who answered “no” to both questions could proceed with the experiment.
Revised: Only those who answered “no” to both questions could proceed with the experiment.

Original: We also required that they were native or fluent as native English speakers, to reduce the hinder on language.
Revised: We also required that they were native or fluent as native English speakers, to reduce the hinder on language.

Original: As for the operation on the keyboard, we required that they had access to a desktop or laptop with arrow keys on the keyboard to finish the experiment.We paid each participant [Table] [Table]
Revised: As for the operation on the keyboard, we required that they had access to a desktop or laptop with arrow keys on the keyboard to finish the experiment.

Original: [Table] [Table] [Table] 10 USD as compensation for their participation in the experiment.
Revised: We paid each participant 10 USD as compensation for their participation in the experiment.

Original: Participants included 16 females and 20 males from the USA and India, the mean age of whom was 38.14 years old (min=22, max=63, SD=8.81).
Revised: Participants included 16 females and 20 males from the USA and India, the mean age of whom was 38.14 years old (min=22, max=63, SD=8.81).

Original: Table 1 presents the detailed demographic information of each condition.
Revised: Table 1 presents the detailed demographic information of each condition.

Original: 5.3 Procedure and Task
Revised: 5.3 Procedure and Task

Original: Participants completed the experiment through a web browser such as Chrome or Firefox.
Revised: Participants completed the experiment through a web browser such as Chrome or Firefox.

Original: For 3 conditions, participants went through the same procedure, except for the interface differences.
Revised: For 3 conditions, participants went through the same procedure, except for the interface differences.

Original: Each par- ticipant first watched a tutorial video of basic dimensional emotion concepts and an introduction to the interface (5 - 10 minutes).
Revised: Each par- ticipant first watched a tutorial video of basic dimensional emotion concepts and an introduction to the interface (5 - 10 minutes).

Original: In addition, the video also showed examples of speech of different emotion values to help them better understand related concepts.
Revised: In addition, the video also showed examples of speech of different emotion values to help them better understand related concepts.

Original: Then, they were asked to practice labeling a short dialogue (2minutes).
Revised: Then, they were asked to practice labeling a short dialogue (2minutes).

Original: Note that participants were allowed to watched the in- troduction videos and to practice multiple times.
Revised: Note that participants were allowed to watch the intro- duction videos and to practice multiple times.

Original: After practice, they proceeded to  labeling tasks (12 - 15 minutes).
Revised: After practice, they proceeded to the labeling tasks (12 - 15 minutes).

Original: As we required users to focus on one speaker  one dimension in each task, there were four tasks, Female-Pleasure , Female-Arousal , Male-Pleasure , and Male-Arousal .
Revised: As we required users to focus on one speaker and one dimension in each task, there were four tasks, Female-Pleasure , Female-Arousal , Male-Pleasure , and Male-Arousal .

Original: To obtain unbiased feedback from non-expert annotators and to avoid order effects, we counter-balanced the order of four tasks.
Revised: To obtain unbiased feedback from non-expert annotators and to avoid order effects, we counter-balanced the order of four tasks.

Original: Finally, they answered a post-study questionnaire and completed a background survey (15 minutes).
Revised: Finally, they answered a post-study questionnaire and completed a background survey (15 minutes).

Original: The total time of the experiment is around 40 minutes.
Revised: The total time of the experiment is around 40 minutes.

Original: We selected one dialogue used in the pilot study for the user study, which was 2 minutes 40 seconds containing fifteen sentences by Female Speaker and sixteen sentences by Male Speaker.
Revised: We selected one dialogue used in the pilot study for the user study, which was 2 minutes 40 seconds containing fifteen sentences by Female Speaker and sixteen sentences by Male Speaker.

Original: The neural network predicted highlights above sentences in the Highlight condition.
Revised: The neural network predicted highlights above sentences in the Highlight condition.

Original: Here we assumed a mapping rule from absolute emotion value differences to relative labels as: ∆ emotion > 1 . 

Revised: Here we assumed a mapping rule from absolute emotion value differences to relative labels as ∆ emotion > 1 .

Original: = ↑↑ , 0 .
Revised: = ↑↑ , 0 .

Original: 5 ≤ ∆ emotion ≤ 1 . 

Revised: 5 ≤ ∆ emotion ≤ 1 .

Original: = ↑ , − 0 .
Revised: = ↑ , − 0 .

Original: 5 < ∆ emotion < 0 .
Revised: 5 < ∆ emotion < 0 .

Original: 5 = —, − 1 . 0 ≤ ∆ emotion ≤ 0 .
Revised: 5 = —, − 1 . 0 ≤ ∆ emotion ≤ 0 .

Original: 5 = ↓ , and ∆ emotion < − 1 . 

Revised: 5 = ↓ , and ∆ emotion < − 1 .

Original: = ↓↓ .
Revised: = ↓↓ .

Original: For example, if the latter sentence s 1 ’s ground truth pleasure value is 4.0 and the former sentence s 0 is 3.0, then it is mapped to a relative label, ↑ ( ∆ pleasure = 1 . 0).
Revised: For example, if the latter sentence s 1 ’s ground truth pleasure value is 4.0 and the former sentence s 0 is 3.0, then it is mapped to a relative label, ↑ ( ∆ pleasure = 1 . 0).

Original: F1 score was 0.914 for pleasure and 0.933 for arousal.
Revised: F1 score was 0.914 for pleasure and 0.933 for arousal.

Original: Fig.
Revised: Fig.

Original: 6 shows the confusion matrix of the highlight prediction results.
Revised: 6 shows the confusion matrix of the highlight prediction results.

Original: Predicted highlights mean highlights acquired from network predictions, while True highlights mean highlights transformed from  ground truth.
Revised: Predicted highlights mean highlights acquired from network predictions, while True highlights mean highlights transformed from the ground truth.

Original: Both used the mapping rule stated above.
Revised: Both used the mapping rule stated above.

Original: 5.4 Measurement 5.4.1 Annotation agreement
Revised: 5.4 Measurement 5.4.1 Annotation agreement

Original: We evaluate the inter-annotator agreement by three metrics, percent agreement, Krippendorff’s Alpha, and Fleiss’ Kappa, following the previous work [2,31].
Revised: We evaluate the inter-annotator agreement by three metrics, percent agreement, Krippendorff’s Alpha, and Fleiss’ Kappa, following the previous work [2,32].

Original: Percent agreement is commonly used and intuitive, which is also suitable for conducting the Two-Way ANOVA and Tukey HSD Test on sentence-level to compare the agreement of different conditions.
Revised: Percent agreement is commonly used and intuitive, which is also suitable for conducting the Two-Way ANOVA and Tukey HSD Test at the sentence level to compare the agreement of different conditions.

Original: Both Krippendorff’s alpha and Fleiss’ kappa are reliability coefficients, taking into account the agreement expected by chance.
Revised: Both Krippendorff’s alpha and Fleiss’ kappa are reliability coefficients, taking into account the agreement expected by chance.

Original: They are obtained by taking all sentences and all annotations from each condition into  computation.
Revised: They are obtained by taking all sentences and all annotations from each condition into the computation.

Original: Thus, there is no statistical significance test on either coefficient.
Revised: Thus, there is no statistical significance test on either coefficient.

Original: 5.4.2 Annotation accuracy
Revised: 5.4.2 Annotation accuracy

Original: As for Highlight condition and Relative condition, we measures the relative annotation accuracy of each participant.
Revised: As for Highlight condition and Relative condition, we measures the relative annotation accuracy of each participant.

Original: Relative labels were treated as ordinal labels for computation, the accuracy of which was measured by Spearman’s rank correlation coefficient ( ρ ) [42].
Revised: Relative labels were treated as ordinal labels for computation, the accuracy of which was measured by Spearman’s rank correlation coefficient ( ρ ) [44].

Original: Spearman’s rank correlation coefficient is suitable for measuring ordinal data that have uneven intervals, the range of which is from -1 to 1.
Revised: Spearman’s rank correlation coefficient is suitable for measuring ordinal data that have uneven intervals, the range of which is from -1 to 1.

Original: CCC measures the absolute annotation accuracy in Absolute condition and estimated absolute values in Highlight condition and Relative condition.
Revised: CCC measures the absolute annotation accuracy in Absolute condition and estimated absolute values in Highlight condition and Relative condition.

Original: Two-Way ANOVA and Tukey HSD Test are conducted for both relative annotation accuracy and absolute annotation accuracy.
Revised: Two-Way ANOVA and Tukey HSD Test are conducted for both relative annotation accuracy and absolute annotation accuracy.

Original: 5.4.3 Task completion time
Revised: 5.4.3 Task completion time

Original: We record the time usage of each task.
Revised: We record the time usage of each task.

Original: The time usage refers to the time between participants click “Start” and “Next” button.
Revised: The time usage refers to the time between participants click “Start” and “Next” button.

Original: We also count the times of participants press left arrow key for rewinding, which can also be an indicator of labeling speed and an inspector of labeling behaviors.
Revised: We also count the times of participants press left arrow key for rewinding, which can also be an indicator of labeling speed and an inspector of labeling behaviors.

Original: 5.4.4 Questionnaire on user experience
Revised: 5.4.4 Questionnaire on user experience

Original: We use a five-scale Likert to ask participants to rate three factors of the interface: (1) Easiness, (2) Intuitiveness, and (3) Helpfulness.
Revised: We use a five-scale Likert to ask participants to rate three factors of the interface: (1) Easiness, (2) Intuitiveness, and (3) Helpfulness.

Original: Two-Way ANOVA and Tukey HSD Test are conducted on the rating results.
Revised: Friedman’s Test and Tukey HSD Test are conducted on the rating [Table] [Table] [Table]

Revised: [Table]

Revised: results.

Original: We also ask participants to give reasons of their ratings.
Revised: We also ask participants to give reasons of their ratings.

Original: In addition, we allow participants to give a free-form comments on the advantages and disadvantages of the interface, or to give any comments on their experience of any part of the experiment procedure.
Revised: In addition, we allow participants to give a free-form comments on the advantages and disadvantages of the interface, or to give any comments on their experience of any part of the experiment procedure.

Original: 5.5 Results 5.5.1 Annotation agreement
Revised: 5.5 Results 5.5.1 Annotation agreement

Original: Fig.
Revised: Fig.

Original: 7 shows the results of inter-annotator agreement on percent agreement (P), and Table 2shows the results of Krippendorff’s Alpha ( α ) and Fleiss’ Kappa ( κ ).
Revised: 7 shows the results of inter-annotator agreement on percent agreement (P), and Table 2 shows the results of Krippendorff’s Alpha ( α ) and Fleiss’ Kappa ( κ ).

Original: As for Krippendorff’s Alpha, we use the 0.667 threshold as the standard of quality [24].
Revised: As for Krippendorff’s Alpha, we use the 0.667 threshold as the standard of quality [25].

Original: As for Fleiss’ Kappa, we use the 0.400 threshold as the standard of quality [41].
Revised: As for Fleiss’ Kappa, we use the 0.400 threshold as the standard of quality [43].

Original: Percent Agreement.
Revised: Percent Agreement.

Original: Highlight shows an improvement as compared to Relative and Absolute .
Revised: Highlight shows an improvement as compared to Relative and Absolute .

Original: As for pleasure, there is a statis- tical significance in Highlight (P=68.5%) vs. Relative (P=57.1%)
Revised: As for pleasure, there is a statis- tical significance in Highlight (P=68.5%) vs. Relative (P=57.1%)

Original: ( p < 0 . 05),
Revised: ( p < 0 . 05),

Original: Highlight vs. Absolute (P=39.1%)
Revised: Highlight vs. Absolute (P=39.1%)

Original: ( p < 0 . 01), and Relative vs. Absolute ( p < 0 . 01).
Revised: ( p < 0 . 01), and Relative vs. Absolute ( p < 0 . 01).

Original: As for arousal, there is a statistical significance in Highlight (P=69.3%) vs. Relative (P=62.3%)
Revised: As for arousal, there is a statistical significance in Highlight (P=69.3%) vs. Relative (P=62.3%)

Original: ( p < 0 . 05),
Revised: ( p < 0 . 05),

Original: Highlight vs. Absolute (P=41.0%) ( p < 0 . 01), and
Revised: Highlight vs. Absolute (P=41.0%) ( p < 0 . 01), and

Original: Relative vs. Absolute ( p < 0 . 01).
Revised: Relative vs. Absolute ( p < 0 . 01).

Original: Krippendorff’s Alpha and Fleiss’ Kappa.
Revised: Krippendorff’s Alpha and Fleiss’ Kappa.

Original: As for pleasure, We do not find any condition exceeding the threshold on Krippendorff’s Alpha or Fleiss’ Kappa.
Revised: As for pleasure, We do not find any condition exceeding the threshold on Krippendorff’s Alpha or Fleiss’ Kappa.

Original: However, Highlight shows an improvement of 17.8% of Alpha and 12.8% of Kappa as compared to Relative , and both Highlight and Relative show a great improvement than
Revised: However, Highlight shows an improvement of 17.8% of Alpha and 12.8% of Kappa as compared to Relative , and both Highlight and Relative show a great improvement than

Original: Absolute .
Revised: Absolute .

Original: As for arousal, we find both Highlight and Relative exceed the threshold on Krippendorff’s Alpha and Fleiss’ Kappa.In
Revised: As for arousal, we find both Highlight and Relative exceed the threshold on Krippendorff’s Alpha and Fleiss’ Kappa.

Original: [Table] addition, Highlight shows an improvement of 0.8% of Alpha and 4.7% of Kappa as compared to Relative , and both Highlight and
Revised: In addition, Highlight shows an improvement of 0.8% in Alpha and 4.7% in Kappa as compared to Relative , and both Highlight and

Original: Relative show a great improvement over Absolute .
Revised: Relative show a great improvement over Absolute .

Original: 5.5.2 Annotation accuracy
Revised: 5.5.2 Annotation accuracy

Original: Fig.
Revised: Fig.

Original: 8a shows the relative annotation accuracy ( ρ ) results of High- light condition and Relative condition.
Revised: 8a shows the relative annotation accuracy ( ρ ) results of High- light condition and Relative condition.

Original: As for pleasure, Highlight ( ρ =0.930, 95% CI=[0.899, 0.982]) is higher than Relative ( ρ =0.764, 95% CI=[0.708, 0.820]).
Revised: As for pleasure, Highlight ( ρ =0.930, 95% CI=[0.899, 0.982]) is higher than Relative ( ρ =0.764, 95% CI=[0.708, 0.820]).

Original: As for arousal, Highlight ( ρ =0.962, 95% CI=[0.948, 0.977]) is higher than Relative ( ρ =0.865, 95% CI=[0.831, 0.898]).
Revised: As for arousal, Highlight ( ρ =0.962, 95% CI=[0.948, 0.977]) is higher than Relative ( ρ =0.865, 95% CI=[0.831, 0.898]).

Original: Statistical significance has been found in both pleasure ( p < 0 . 01) and arousal ( p < 0 . 01).
Revised: Statistical significance has been found in both pleasure ( p < 0 . 01) and arousal ( p < 0 . 01).

Original: Fig.
Revised: Fig.

Original: 8bshows the results of the absolute absolute annotation ac- curacy measured by CCC. As for pleasure, statistical significance is found on Highlight (CCC=0.713, 95% CI=[0.694, 0.732]) vs. Rela- tive (CCC=0.542, 95% CI=[0.491, 0.592]) ( p < 0 . 01),Highlight vs.
Revised: 8b on Highlight (CCC=0.713, 95% CI=[0.694, 0.732]) vs. Relative (CCC=0.542, 95% CI=[0.491, 0.592]) ( p < 0 . 01),

Original:  Absolute (CCC=0.231, 95% CI=[0.166, 0.297]) ( p < 0 . 01), and Rel- ative vs. Absolute ( p < 0 . 01).
Revised: Highlight vs. Absolute (CCC=0.231, 95% CI=[0.166, 0.297]) ( p < 0 . 01), and - tive vs. Absolute ( p < 0 . 01).

Original: As for arousal, statistical significance is found on Highlight (CCC=0.780, 95% CI=[0.760, 0.800]) vs. Rel- ative (CCC=0.720, 95% CI=[0.695, 0.744]) ( p < 0 . 01), 
Revised: As for arousal, statistical significance is found on Highlight (CCC=0.780, 95% CI=[0.760, 0. ]) vs. Relative (CCC=0.720, 95% CI=[0.695, 0.744]) ( p < 0 . 01), Highlight =

Original: Highlight vs. Absolute (CCC=0.539, 95% CI=[0.442, 0.636]) ( p < 0 . 01), and Relative vs. Absolute ( p < 0 . 01).
Revised: =

Original: Note that in Absolute condition, annotation accuracy iseven lowerthan the network prediction performance. 
Revised: 01), and accuracy is lower prediction performance. of prediction on

Original: The accuracy of network prediction on both pleasure (CCC=0.522, dotted line in Fig. 8b) and arousal (CCC=0.671, dashed line in Fig.
Revised: =0.522,

Original: 8b).
Revised: [Figure]

Original: 5.5.3 Task completion time
Revised: 5.5.3 Task completion time

Original: While participants labeled the same amounts of data across the three conditions, we find  learning effect in Highlight condition, as both time usage (average time (s) in Fig. 9a: 1st=178.3, 2nd=160.3, 3rd=156.1, 4th=152.0) and rewinding counts (average counts: 1st=3.33, 2nd=1.42, 3rd=0.92, 4th=0.58 in Fig.
Revised: While participants labeled the same amounts of data across the three conditions, we find a learning effect in Highlight condition, as both time usage (average time (s) in Fig. 9a: 1st=178.3, 2nd=160.3, 3rd=156.1, 4th=152.0) and rewinding counts (average counts: 1st=3.33, 2nd=1.42, 3rd=0.92, 4th=0.58 in Fig.

Original: 9b) decreased as tasks went on.
Revised: 9b) de- creased as tasks went on.

Original: We haven’t found the same behavior in Relative condition or Absolute condition.
Revised: We haven’t found the same behavior in Relative condition or Absolute condition.

Original: We give a discussion on it in Sect. 5.6.
Revised: We give a discussion on it in Sect. 5.6.

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 5.5.4 Questionnaire on user experience
Revised: 5.5.4 Questionnaire on user experience

Original: Fig.
Revised: Fig.

Original: 10 shows the results of participants’ feedback on Easiness, Intuitiveness, and Helpfulness.
Revised: 10 shows the results of participants’ feedback on Easiness, Intuitiveness, and Helpfulness.

Original: Easiness.
Revised: Easiness.

Original: There’s a statistical significance in Highlight vs. Ab- solute ( p < 0 . 01).
Revised: There’s a statistical significance in Highlight vs. Ab- solute ( p < 0 . 01).

Original: Participants mentioned the “reference” in interface design reducing the memory load in their reasons.
Revised: Participants mentioned the “reference” in interface design reducing the memory load in their reasons.

Original: P3 (Highlight): “It helped me because I could refer to the change in emotion from audio sequence to audio sequence just by looking at the arrow (up or down) on the previous audio clip that I played and labeled.
Revised: P3 (Highlight): “It helped me because I could refer to the change in emotion from audio sequence to audio sequence just by looking at the arrow (up or down) on the previous audio clip that I played and labeled.

Original: Also, the transcript below reminds me how that sentence sounded.”
Revised: Also, the transcript below reminds me how that sentence sounded.”

Original: P9 (Highlight): “It’s a really simple interface that allows me to focus on the speech and not on remembering the complex rules and definitions of emotions.
Revised: P9 (Highlight): “It’s a really simple interface that allows me to focus on the speech and not on remembering the complex rules and definitions of emotions.

Original: Only remembering the last sentence is not hard, at least for me.”
Revised: Only remembering the last sentence is not hard, at least for me.”

Original: Participants also mentioned the continuum and relative labeling interface design reducing oneself’s variations and errors.
Revised: Participants also mentioned the continuum and relative labeling interface design reducing oneself’s variations and errors.

Original: P17 (Relative): “It helps to find out and suppress the variations.”
Revised: P17 (Relative): “It helps to find out and suppress the variations.”

Original: P10 (Highlight): “It was so easy for labeling to reduce identifying errors when in a flow.
Revised: P10 (Highlight): “It was so easy for labeling to reduce identifying errors when in a flow.

Original: Otherwise back and forth can be annoying.”
Revised: Otherwise back and forth can be annoying.”

Original: P2 (Highlight): “It might reduce the variation reported because of the flow between turns.
Revised: P2 (Highlight): “It might reduce the variation reported because of the flow between turns.

Original: It makes you ignore or average the variation within a turn.”
Revised: It makes you ignore or average the variation within a turn.”

Original: Intuitiveness.
Revised: Intuitiveness.

Original: There’s a statistical significance between Highlight vs. Absolute ( p 
Revised: There’s a statistical significance between Highlight vs. Absolute ( p < 0 . 05) and Relative vs. Absolute ( p < 0 . 05). Most reasons were that up key and down key were according to the emotion change. P4 (Highlight):

Original: [Table]
Revised: [Table]

Original: and
Revised: “The

Original: [Table]
Revised: [Table]

Original: vs. [Table] [Table] Most reasons were that up key and down key were according to the emotion change
Revised: are

Original: [Table]
Revised: [Table]

Original: Same 
Revised: Same with the forward/

Original: [Table]
Revised: [Table]

Original: Helpfulness.
Revised: Helpfulness.

Original: There’s a statistical significance in Highlight vs. Relative ( p < 0 . 05) and Highlight vs. Absolute ( p < 0 . 01).
Revised: There’s a statistical significance in Highlight vs. Relative ( p < 0 . 05) and Highlight vs. Absolute ( p < 0 . 01).

Original: Participants mentioned that highlights helped confirm their decisions.P3 (
Revised: Partic- ipants mentioned that highlights helped confirm their decisions.

Original: [Table]
Revised: [Table]

Original: ): “It helps to confirm mylabeling.”
Revised: “It helps to confirm my

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ): “Make us sure that the reaction is going to change.”
Revised: “Make us sure that the reaction is going to change

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ): “Help people that do not feel emotion know what emotion changes should sound like.”
Revised: : “Help people that do not feel emotion know what emotion changes should sound like.”

Original: They also mentioned that highlights helped lowerthe mental effort.
Revised: They also mentioned that highlights helped lower mental effort

Original: P1 (Highlight): “Yes, the highlight is instrumental to completing the task with further support and I take those sentences more seriously.”
Revised: [Table] [Table] highlight is instru- mental to completing the task with further support and I take those sentences more seriously

Original: P7 (Highlight): “I definitely think the yellow helped because it alerted me to the system’s labels and made me focus more on those sections.”
Revised: [Table] [Table] definitely think the yellow helped because it [Table]

Original: P12 (
Revised: me [Table] the

Original: [Table]
Revised: [Table]

Original: ): “Yes because I could concentrate on the part I have to judge.”
Revised: made me focus more [Table] those [Table]

Revised: I could

Revised: [Table]

Original:  Participants also mentioned some advantages of the interface and their labeling experience.
Revised: part I [Table] judge.” Participants also mentioned some advantages of the interface and their labeling experience.

Original: Many mentioned the “quickness” in the non-stop playing and keyboard interaction design.
Revised: Many mentioned the “quickness” in the non-stop playing and keyboard interaction design.

Original: P3 (Highlight): “The direction keys made it very easy to use.
Revised: P3 (Highlight): “The direction keys made it very easy to use.

Original: I could make my selections quickly.”
Revised: I could make my selections quickly.”

Original: P7 (Highlight): “It will allow quick and accurate annotations.”
Revised: P7 (Highlight): “It will allow quick and accurate annotations.”

Original: P17 (Relative): “This helps people that might be slower than others in making their choices.”
Revised: P17 (Relative): “This helps people that might be slower than others in making their choices.”

Original: As for “accurate annotation” mentioned by P7, our understanding is that the participant just felt that keyboard operation was more reliable than mouse-clicking on the graphical interface.
Revised: As for “accurate annotation” mentioned by P7, our understanding is that the participant just felt that keyboard operation was more reliable than mouse-clicking on the graphical interface.

Original: There is a main disadvantage from Absolute group not found in the Highlight and Relative group, wanting more practice and tutorials.
Revised: There is a main disadvantage from Absolute group not found in the Highlight and Relative group, wanting more practice and tutorials.

Original: P27 (Absolute): “The toolwas fun to use
Revised: P27 (Absolute): “The tool

Original: but I feel like more tutorials on how to rate it would help with becoming more proficient with the use of this tool.”

Original: P31 (Absolute): “I think the tutorial is not sufficient for understanding the rating rules.”

Original: P34 (Absolute): “The practice is helpful for the operation but not for how to rate emotions.

Original: I gave the rating based on my feelings.”

Original: This means that our interface design gives rise to a short learning curve when compared to the traditional interface.

Original: Meanwhile, there are also limitations pointed out by participants.

Original: Two mentioned that highlights were different from their decision.

Original: [Table]
Revised: [Table]

Original: : “Most sentences had emotion changes and it would not highlight them.” [Table] [Table] ): “To be honest, I find the highlight sometimes didn’t sit well with what I personally felt.”
Revised: light sometimes didn’t sit well with what I personally felt.”

Original: Another unignorable limitation is that there is cross-talk in the audio, which makes the task difficult and confusing.
Revised: Another unignorable limitation is that there is cross-talk in the audio, which makes the task difficult and confusing.

Original: It is mentioned in all groups.
Revised: It is mentioned in all groups.

Original: [Table] : “I think it can be difficult to compare one speaker’s emotion between sections when the other speaker talks for a while in between [Table] [Table]
Revised: P1 (Highlight): “I think it can be difficult to compare one speaker’s emotion between sections when the other speaker talks for a while in between.”

Original: : “There was a bit

Original: [Table] cross [Table] talk that the system didn’t deal with.”
Revised: P15 (Relative): “There was a bit too much cross-talk that the system didn’t deal with.”

Original: [Table] : “Several times, there were parts in the audio when two speakers were speaking and it was hard to concentrate on the speaker to label.”
Revised: P27 (Absolute): “Several times, there were parts in the audio when two speakers were speaking and it was hard to concentrate on the speaker to label.”

Original: 5.6 Results Discussion
Revised: 5.6 Results Discussion

Original: We answer the Research Questions raised at the beginning of this section, and give an insight into what we found in the user study.
Revised: We answer the Research Questions raised at the beginning of this section, and give an insight into what we found in the user study.

Original: Our approach can improve the annotation agreement and elicit valid and reliable labels from non-expert annotators.
Revised: Our approach can improve the annotation agreement and elicit valid and reliable labels from non-expert annotators.

Original: It also improve the user experience according to the questionnaire.
Revised: It also improve the user experience according to the questionnaire.

Original: Relative labels improved the annotation agreement and high- light suggestions further improved it (RQ1) .
Revised: Relative labels improved the annotation agreement and high- light suggestions further improved it (RQ1) .

Original: As results shown in
Revised: As results shown in

Original: Sect.
Revised: Sect.

Original: 5.5.1, we found the percent agreement of Highlight improved as compared to Relative and then Absolute .
Revised: 5.5.1, we found the percent agreement of Highlight improved as compared to Relative and then Absolute .

Original: We also considered the agreement expected by chance, and we found Highlight exceeded the standard of quality of both Krippendorff’s Alpha and Fleiss’
Revised: We also considered the agreement expected by chance, and we found Highlight exceeded the standard of quality of both Krippendorff’s Alpha and Fleiss’

Original: Kappa in arousal.
Revised: Kappa in arousal.

Original: We did not find the same results in pleasure.
Revised: We did not find the same results in pleasure.

Original: We interpret it as that different cultural backgrounds and experiences lead to the chance agreement on pleasure [1], as people understand the pleasure difference in different ways.
Revised: We interpret it as that different cultural backgrounds and experiences lead to the chance agreement on pleasure [1], as people understand the pleasure difference in different ways.

Original: We also noticed that participants found highlights were not per- fectly consistent to their perceptions.
Revised: We also noticed that participants found highlights were not per- fectly consistent with their perceptions.

Original: We admitted the fact that the  network did not perfectly predicted highlights.
Revised: We admitted the fact that the SER network did not perfectly predict highlights.

Original: However, after a deeper investigation of the percent agreement between correctly predicted sentences and incorrectly predicted sentences, no significance was found between them.
Revised: However, after a deeper investigation of the percent agreement between correctly predicted sentences and incorrectly predicted sentences, no significance was found between them.

Original: Based on these results, we conclude that even though the network performance in this user study might not be perfect, participants could still reach an agreement, even though it interfered  the annotation accuracy of some participants to some degree.
Revised: Based on these results, we conclude that even though the network performance in this user study might not be perfect, participants could still reach an agreement, even though it interfered with the annotation accuracy of some participants to some degree.

Original: RelRoll improved the annotation accuracy, and  relative labels elicited by RelRoll were valid (RQ2) .
Revised: RelRoll improved the annotation accuracy, and the relative labels elicited by RelRoll were valid (RQ2) .

Original: Results in Sect.
Revised: Results in Sect.

Original: 5.5.2 shows that Highlight successfully elicited labels of sufficient accuracy from non-expert annotators.
Revised: shows that Highlight successfully elicited labels of sufficient ac- curacy from non-expert annotators.

Original: In contrast, Absolute failed to get labels of sufficient quality, even lower than the neural network prediction.
Revised: In contrast, Absolute failed to get labels of sufficient quality, even lower than the neural network prediction.

Original: We believe that it is because non-experts could not give reliable absolute annotations when first using the traditional absolute labeling interface.
Revised: We believe that it is because non-experts could not give reliable absolute annotations when first using the traditional absolute labeling interface.

Original: The interface of RelRoll was easier, more intuitive, and more helpful for non-experts, which lowered the learning curve of annotation (RQ3) .
Revised: The interface of RelRoll was easier, more intuitive, and more helpful for non-experts, which lowered the learning curve of annotation (RQ3) .

Original: Results in Sect.
Revised: Results in Sect.

Original: 5.5.4 prove the improved user experience of Highlight .
Revised: 5.5.4 prove the improved user experience of Highlight .

Original: From the questionnaire, we also found the learning curve of annotation was lowered.
Revised: From the questionnaire, we also found the learning curve of annotation was lowered.

Original: Learning annotation is profound because participants need to learn the relationship between numerical values and emotions in an absolute interface.
Revised: Learning annotation is profound because participants need to learn the relationship between numerical values and emotions in an absolute interface.

Original: As for the absolute interface, participants need to learn the full range of relationships between numerical values and emotions, which can be infeasible [1].
Revised: As for the absolute interface, participants need to learn the full range of relationships between numerical values and emotions, which can be infeasible [1].

Original: Meanwhile, highlights would allow users to only focus on the most possible sentences whose pleasure or arousal changes.
Revised: Meanwhile, highlights would allow users to only focus on the most possible sentences whose pleasure or arousal changes.

Original: However, the learning effect is also found in Highlight condition.
Revised: However, the learning effect is also found in Highlight condition.

Original: More time and more rewind counts were found in the beginning tasks.
Revised: More time and more rewind counts were found in the beginning tasks.

Original: We believe this is because (1) users needed to learn the indication of highlights by reviewing their decisions and (2) users were not confident with the highlight predictions when first using the interface.
Revised: We believe this is because (1) users needed to learn the indication of highlights by reviewing their decisions and (2) users were not confident with the highlight predictions when first using the interface.

Original: 6 L IMITATIONS A ND F UTURE W ORK
Revised: 6 L IMITATIONS A ND F UTURE W ORK

Original: We discussed about the limitations in RelRoll and potential future directions in this section.
Revised: We discussed about the limitations in RelRoll and potential future directions in this section.

Original: Dependence on experts’ annotations.
Revised: Dependence on experts’ annotations.

Original: RelRoll is a mechanism of eliciting reliable annotations by combining experts and non-experts.
Revised: RelRoll is a mechanism of eliciting reliable annotations by combining experts and non-experts.

Original: This approach tries to make annotations by non-experts following the distribution of experts’ annotations.
Revised: This approach tries to make annotations by non-experts following the distribution of experts’ annotations.

Original: Thus it is vital to select representative and reliable data annotated by experts as training set before using this mechanism.
Revised: Thus it is vital to select representative and reliable data annotated by experts as training set before using this mechanism.

Original: In the future, we plan to find an optimized method to select the data for the training set .
Revised: In the future, we plan to find an optimized method to select the data for the training set .

Original: Nuances in design decisions.
Revised: Nuances in design decisions.

Original: Even though we validated RelRoll through a user study, there are nuances in design decisions we have not validated.
Revised: Even though we validated RelRoll through a user study, there are nuances in design decisions we have not validated.

Original: There can be different designs on the number of levels of relative labels, which can be interesting to explore.
Revised: There can be different designs on the number of levels of relative labels, which can be interesting to explore.

Original: In addition, we chose one approach to mapping between absolute differences to relative labels.
Revised: In addition, we chose one approach to mapping between absolute differences to relative labels.

Original: There could be other rules to do the mapping depending on practical use cases.
Revised: There could be other rules to do the mapping depending on practical use cases.

Revised: Mechanism scalability.

Revised: It may become infeasible for to use Rel- Roll annotate all emotional dimensions by playing an audio recording only time.

Revised: However, it is related to participants’ expertise in emotion understanding.

Revised: One possible solution is to predict other dimensions when non-expert participants annotate one specific di- mension.

Original:  Dealing With Cross-Talk.
Revised: In the user study, we tested only one 2 minutes 40 seconds conversation. However, if removing overhead, we could acquire three times more than the current results. An extra user study can be conducted to prove the validity of a larger dataset. Dealing With Cross-Talk.

Original: Participants from the user study mentioned that cross-talk influenced their understanding and decisions.
Revised: Participants from the user study mentioned that cross-talk influenced their understanding and decisions.

Original: Since our mechanism requests users to focus on one speaker in each task, it’s important to reduce the effect of cross-talk.
Revised: Since our mechanism requests users to focus on one speaker in each task, it’s important to reduce the effect of cross-talk.

Original: However, cross-talk can be common in spontaneous speech, which refers to unsmooth or inappropriate turn-taking in linguistics.
Revised: However, cross-talk can be common in spontaneous speech, which refers to unsmooth or inappropriate turn-taking in linguistics, even though sometimes they are considered natural or unnoticeable for speakers.

Original: Voice separation can be applied to extract the speech by the speaker to label when there’s a mixed speech of multiple speakers talking at the same time.
Revised: Voice separation can be applied to extract the speech by the speaker to label when there’s a mixed speech of multiple speakers talking at the same time.

Original: It will reduce users’ difficulty in understanding the speech and introduce a better experience.
Revised: It will reduce users’ difficulty in understanding the speech and introduce a better experience.

Original: Other Future Directions.
Revised: Other Future Directions.

Original: It is not supported for users to express uncertainty (e.g., unsure about labels) in the current interface
Revised: It is not supported for users to express uncertainty (e.g., unsure about labels) in the current interface

Original: de- sign.
Revised: de- sign.

Original: It is interesting to consider and quantify the uncertainty in future work.
Revised: It is interesting to consider and quantify the uncertainty in future work.

Original: There can be more applications of RelRoll on other scoring annotation tasks, such as image quality assessment, speech intelligibility rating, etc.
Revised: There can be more applications of RelRoll on other scoring annotation tasks, such as image quality assessment, speech intelligibility rating, etc.

Original: In addition, there can be more applications on different data formats, such as videos, streams of images, etc.
Revised: In addition, there can be more applications for different data formats, such as videos, streams of images, etc.

Original: Last but not the least, while RelRoll can be effective on continuous data like speech, it can be difficult to adapt it to discrete data.
Revised: Last but not the least, while RelRoll can be effective on continuous data like speech, it can be difficult to adapt it to discrete data.

Original: One possible solution is grouping the discrete data into a sequence of an appropriate order before presenting them to users.
Revised: One possible solution is grouping the discrete data into a sequence of an appropriate order before presenting them to users.

Original: Then the same mechanism can be applied to solve the discrete problem similarly to the continuous problem.

Original: 7 C ONCLUSION
Revised: 7 C ONCLUSION

Original: This paper presents a mechanism allowing non-expert annotators to give reliable scoring annotations.
Revised: This paper presents a mechanism allowing non-expert annotators to give reliable scoring annotations.

Original: The key idea is to reduce the labeling difficulty by suggesting changes and estimating absolute values from relative labels.
Revised: The key idea is to reduce the labeling difficulty by suggesting changes and estimating absolute values from relative labels.

Original: User study results showed that the interface of RelRoll can improve the inter-annotator agreement and obtain reliable and valid labels.
Revised: User study results showed that the interface of RelRoll can improve the inter-annotator agreement and obtain reliable and valid labels.

Original: In addition, it can improve nonexperts annotation accuracy as compared to  absolute annotation interface.
Revised: In addition, it can improve nonexperts annotation accuracy as compared to the absolute annotation interface.

Original: We also found that this interface can leverage the user experience from the quantitative results and qualitative feedback.
Revised: We also found that this interface can leverage the user experience from the quantitative results and qualitative feedback.

Original: We hope RelRoll can inspire more research on different scoring annotation tasks other than speech emotion annotation, which can involve more non-experts in originally expert-oriented labeling tasks. 
Revised: We hope RelRoll can inspire more research on different scoring annotation tasks other than speech emotion annotation, which can involve more non-experts in originally expert-oriented labeling tasks. A CKNOWLEDGMENTS

Revised: This work was supported by JST SPRING, Grant Number JP- MJSP2108.