Original: An Empirical Study on the Effect of Quick and Careful Labeling Styles in Image AnnotationAuthor 1* Author 1* Author 1*
Revised: An Empirical Study on the Effect of Quick and Careful Labeling Styles in Image Annotation

Original: Organization or School

Original: Organization or School

Original: Organization or School

Original: A 
Revised: A BSTRACT

Original: BSTRACT

Original: Giving a label to difficult data takes a lengthy amount of time, particularly when non-expert annotators try to select the best possible label.
Revised: Assigning a label to difficult data requires a long time, particularly when non-expert annotators attempt to select the best possible label.

Original: However, there are no detailed studies exploring a label selection style during annotation.
Revised: However, there have been no detailed studies exploring a label selection style during annotation.

Original: It can be very important and may affect theannotation efficiency and quality.
Revised: This is very important and may affect the efficiency and quality of annotation.

Original: In this study, we explore the effects of labeling styles in data annotation and machine learning.
Revised: In this study, we explored the effects of labeling style on data annotation and machine learning.

Original: We ran an empirical study comparing “quick labeling” and “careful labeling” styles in image labeling tasks with three levels of difficulty.
Revised: We conducted an empirical study comparing “quick labeling” and “careful labeling” styles in image-labeling tasks with three levels of difficulty.

Original: We also ran a machine learning experiment usingthe labeled images from the two labeling styles.
Revised: Additionally, we performed a machine learning experiment using labeled images from the two labeling styles.

Original: The results showed that quick and careful labeling styles have both advantages and disadvantages in  annotation efficiency, label quality, and machine learning performance.
Revised: The results indicated that quick and careful labeling styles have both advantages and disadvantages in terms of annotation efficiency, label quality, and machine learning performance.

Original: Specifically,it shows that careful labeling improves label accuracyonly when the task is moderately difficult whereas it is a waste of time when the task is easy or extremely difficult.
Revised: Specifically, careful labeling improves label accuracy when the task is moderately difficult, whereas it is time-consuming when the task is easy or extremely difficult.

Original: 1 I NTRODUCTION
Revised: 1 I NTRODUCTION

Original: A large data size with high quality isalways necessary to obtain better machine learning results.
Revised: A large, high-quality dataset is necessary to obtain better machine learning results.

Original: However, it is too costly if we recruit a large number of expert annotators (who have sufficient domain knowledge) to work on it.
Revised: However, it is expensive to recruit a large number of expert annotators (who have sufficient domain knowledge) to work on it.

Original: Recruiting non-expert annotators (typically, crowd workers) is cheaper and easier, so it is often the only viable option in practice [20, 21].
Revised: Recruiting non-expert annotators (typically crowd workers) is cheaper and easier; therefore, it is often the only viable option in practice [20, 21].

Original: However,the label quality isa criticalissue in non-expert data annotation (i e. crowdsourcing tasks)
Revised: However, label quality is critical in non-expert data annotation (i e., crowdsourcing tasks)

Original: Various annotation methods and tools have been introduced for addressing this issue [13, 14, 29, 32, 40].
Revised: Various annotation methods and tools have been introduced to address this issue [13, 14, 29, 32, 40].

Original: However, there are no detailed studies looking into this issue from a human perspective (i e. cognitive psychology) such as to investigate the effect of a label selection style during annotation (i.e. how a user makes a label decision).
Revised: However, there are no detailed studies examining this issue from a human perspective (i.e., cognitive psychology), such as investigating the effect of a label selection style during annotation (i e., how a user makes a label decision).

Original: This can be important and may affect annotation efficiency and quality.
Revised: This is important and it could affect annotation efficiency and quality.

Original: We expect that different labeling styles used by annotators  affect  annotation efficiency as well as annotation quality.
Revised: The different labeling styles used by annotators could affect the annotation efficiency and quality.

Original: This paper presents an empirical study comparing two labeling styles (quick labeling and careful labeling) for a manual image labeling task, with three datasets under different levels of data difficulty, easy (MNIST), moderately difficult (Fashion-MNIST), and extremely difficult (Kuzushiji-MNIST).
Revised: This study presents an empirical study comparing two labeling styles (quick labeling and careful labeling) for a manual image labeling task with three datasets under different levels of data difficulty: easy (MNIST), moderately difficult (Fashion-MNIST), and extremely difficult (Kuzushiji-MNIST).

Original: We then conducted a machine learning experimentby using the labeled images using quick labeling and careful labeling styles and compared the machine learning results (classification accuracy), see Figure 1.
Revised: Thereafter, we conducted a machine learning experiment using the labeled images with quick labeling and careful labeling styles and compared the machine learning results (classification accuracy), as shown in Figure 1.

Original: These results demonstrated that a labeling style affects annotation efficiency (task completion time) and label accuracy.
Revised: These results indicated that the labeling style affects annotation efficiency (task completion time) and label accuracy.

Original: The careful labeling style showed significantly higher accuracy than the quick labeling style when the data set is moderately difficult.
Revised: The careful labeling style exhibited significantly higher accuracy than the quick labeling style when the dataset was moderately difficult.

Original: Moreover, the results of the machine learning experiment showed that the labeled images (training data) collected via the careful labeling style can also achieve better machine learning performance (higher classification accuracy) than the labeled image collected via the quick labeling style.
Revised: Moreover, the results of the machine learning experiment indicated that the labeled images (training data) collected via the careful labeling style could achieve better machine learning performance (higher classification accuracy) than that collected via the quick labeling style.

Original: However, theresults showed that the careful labeling style did not bring benefits when the images are easy (i.e. label accuracy is already high via the quick labeling style) or extremely difficult (i.e. the careful labeling style cannot significantly improve  label accuracy).
Revised: However, the careful labeling style did not bring benefits when the images were easy (i.e., label accuracy was already high via the quick labeling style) or extremely difficult (i.e., the careful labeling style could not significantly improve the label accuracy).

Original: We discussed the effects of the two labeling styles according to three different levels of data difficulty (easy, moderately difficult, and extremely difficult), and  discussed three factors that need to be carefully considered when selecting an appropriate labeling style for an annotation task.
Revised: We discussed the effects of the two labeling styles according to three different levels of data difficulty (easy, moderately difficult, and extremely difficult), and we discussed three factors that need to be carefully considered when selecting an appropriate labeling style for an annotation task.

Original: This study makes the following contributions:
Revised: This study makes the following contributions:

Original: • Identifying labeling styles as a variable in non-expert image annotation and machine learning.
Revised: • Identifying labeling styles as a variable in non-expert image annotation and machine learning.

Original: • An empirical study comparing quick labeling and careful labeling styles, demonstrating the benefits of the careful labeling style in an image annotation task.
Revised: • An empirical study comparing quick labeling and careful labeling styles, demonstrated the benefits of the careful labeling style in an image annotation task.

Original: • A machine learning experiment using labeled images via different labeling styles, demonstrating the effects of  labeling styles in image classification accuracy.
Revised: • A machine learning experiment using labeled images with different labeling styles, demonstrated the effects of the labeling styles on image classification accuracy.

Original: LEAVE 0.5 INCH SPACE AT BOTTOM OF LEFT

Original: COLUMN ON FIRST PAGE FOR COPYRIGHT BLOCK * email address

Original:  Manual Data Annotation and Challenge
Revised: 2. Manual Data Annotation and Challenges

Original: A manual data annotation is a basic practice for machine learning.
Revised: Manual data annotation is a basic practice in machine learning.

Original: Large size is often necessary to improvethe machine learning results.
Revised: A large size is often necessary to improve machine learning results.

Original: Some popular datasets including ImageNet [28], AudioSet [30], and YouTube-8M [31],all of which were manually labeled by human annotators, were applied.
Revised: Popular datasets including ImageNet [28], AudioSet [30], and YouTube-8M [31], which were manually labeled by human annotators, were applied.

Original: The manual data annotation process is extremely tedious and time-consuming, and many studies have proposed annotation tools for assistingthe tedious process of manual data annotation.
Revised: The manual data annotation process is extremely tedious and time-consuming, and many studies have proposed annotation tools for assisting manual data annotation.

Original: For example, LabelMe [10] is a web-based image annotation tool that allows multiple annotators to label an image and share their labeling results instantly.
Revised: For instance, LabelMe [10] is a web-based image annotation tool that allows multiple annotators to label an image and share their labeling results instantly.

Original: ESP [11] is an image annotation tool combined with a computer game that provides an enjoyable labeling process for annotators, whereas TagATune [27] is an audio annotation tool that shares the same idea.
Revised: ESP [11] is an image annotation tool combined with a computer game that provides an enjoyable labeling process for annotators, and TagATune [27] is an audio annotation tool that shares the same idea.

Original: VIA [12] is an annotation tool that allows annotators to define and describe spatial regions in images, audio segments, and video frames, and iVAT [32] is a video annotation tool that supports manual, semi-automatic, and automatic video annotation.
Revised: VIA [12] is an annotation tool that allows annotators to define and describe spatial regions in images, audio segments, and video frames. iVAT [32] is a video annotation tool that supports manual, semiautomatic, and automatic video annotation.

Original: Most of these tools provide supportive, efficient, and enjoyable systems for improving the tedious processes of manual data annotation.
Revised: Most of these tools provide supportive, efficient, and enjoyable systems for improving the tedious processes of manual data annotation.

Original: These tools generally assume thatthe annotators have sufficient domain knowledge in the labeling tasks.
Revised: These tools generally assume that annotators have sufficient domain knowledge for labeling tasks.

Original: However, data annotation tasks often rely on non-expert annotators who lack sufficient domain knowledge because access to a sufficient number of expert annotators is limited and costly [18, 19].
Revised: However, data annotation tasks often rely on non-expert annotators who lack sufficient domain knowledge because access to a sufficient number of expert annotators is limited and expensive [18, 19].

Original: In such cases, labeling tasks can be significantly difficult for non-expert annotators, and the labeled data may contain numerous errors [23] [33] [36].
Revised: Therefore, labeling tasks can be significantly difficult for nonexpert annotators and the labeled data may contain numerous errors [23] [33] [36].

Original: These annotation tools may not be able to address this issue whenthe annotators are non-experts.
Revised: However, these annotation tools may not be able to address this issue when annotators are non-experts.

Original: 2. Annotation Workflows for Improving  Label Quality
Revised: 2. Annotation Workflows for Improving the Label Quality

Original: Many data annotation workflows have been proposed for improving the label quality, particularly for annotation tasks conducted through crowdsourcing.
Revised: Many data annotation workflows have been proposed to improve the label quality, particularly for annotation tasks conducted through crowdsourcing.

Original: Revolt [13] is a collaborative crowdsourcing labeling workflow that applies concepts from expert annotation workflows (label-check-modify).
Revised: Revolt [13] is a collaborative crowdsourcing labeling workflow that applies concepts from expert annotation workflows (label-check-modification).

Original: This specific workflow can producea higher label quality than a traditional labeling workflow.
Revised: This specific workflow can produce higher label quality than a conventional labeling workflow.

Original: Pairwise HITS [14] is a labeling workflow fora quality estimation that allows annotators to compare a pair of labeled data and select a better one.
Revised: Pairwise HITS [14] is a labeling workflow for quality estimation that allows annotators to compare a pair of labeled data and select the better one.

Original: Fang et al.
Revised: Fang et al.

Original: [32] introduced a two-round workflow to improve the quality ofthe crowdsourced image labeling.
Revised: [32] introduced a two-round workflow to improve the quality of crowdsourced image labeling.

Original: In the first round,  annotators select a label for the target images (several labels might be assigned to each image).
Revised: During the first round, the annotators select a label for the target images (several labels are assigned to each image).

Original: In the second round,  annotators are required to select the best label for each image (referring to the results from other annotators).
Revised: During the second round, the annotators are required to select the best label for each image (referring to the results from other annotators).

Original: In addition, Baba [29] introduced two types of labeling workflows (parallel and interactive) that allow multiple annotators to be involved in an annotation task in different ways to improve the label quality.
Revised: Baba [29] introduced two types of labeling workflows (parallel and interactive) that allow multiple annotators to be involved in an annotation task in different ways to improve the label quality.

Original: In addition, several studies  used the concept of hierarchical classification ina data annotation to increase the labeling efficiency and label quality [15] [16].
Revised: In addition, various studies have used the concept of hierarchical classification in data annotation to increase the labeling efficiency and label quality [15] [16].

Original: The main concept of these annotation workflows is the gathering of knowledge from multiple people.
Revised: The main concept of these annotation workflows is the gathering of knowledge from multiple individuals.

Original: This is a typical workflow for improving the label quality by involving a group of annotators (non-exerts or experts) to collaboratetogether for a labeling task.
Revised: This is a typical workflow for improving the label quality by involving a group of annotators (non-experts or experts) to collaborate for a labeling task.

Original: In this study, we aim to explore the “labeling styles” rather than the “labeling workflows” during a data annotation task.
Revised: In this study, we aim to explore the “labeling styles” rather than the “labeling workflows” during a data annotation task.

Original: In addition to machine learning, data annotation has been usedas an approach in various research areas.
Revised: In addition to machine learning, data annotation has been used in various research areas.

Original: For example, social scientists annotate data to discover interesting phenomena and establish theories [7] [9], and data annotation is often used in the process of analyzing qualitative data such as a thematic analysis approach [8].
Revised: For instance, social scientists annotate data to discover interesting phenomena and establish theories [7] [9], and data annotations, such as a thematic analysis approach, is often used to analyze qualitative data [8].

Original: Data annotation is not only a labeling process but also a cognitive process by which annotators view data, organize concepts, and make labeling decisions.
Revised: Data annotation is not only a labeling process but also a cognitive process by which annotators view data, organize concepts, and make labeling decisions.

Original: Concept organization plays an important role in data annotation, and Kulesza et al.
Revised: Concept organization plays a crucial role in data annotation. Kulesza et al.

Original: [17] indicated that annotators often organize their conceptual similarity by observing more items during data annotation.
Revised: [17] indicated that annotators often organize their conceptual similarity by observing more items during data annotation.

Original: Chang et al.
Revised: Chang et al.

Original: [40] shared the same concept and proposed a spatial layout labeling interface fora concept organization during the annotation process.
Revised: [40] shared the same concept and proposed a spatial layout labeling interface for concept organization during the annotation process.

Original: We believe that cognitive processes (i.e., how to make a labeling decision) in a manual data annotation task is important and might affect the label quality and cost.
Revised: We believe that cognitive processes (i.e., how to make a labeling decision) in a manual data annotation task are important and they could affect the label quality and cost.

Original: 2. Intuitive and Systematic Decision-Making
Revised: 2. Intuitive and Systematic Decision-Making

Original: A cognitive style (or thinking style) is a term used in cognitive psychology to describe how individuals organize and process information, and finally make decisions [1] [2] [3].
Revised: Cognitive style (or thinking style) is a term used in cognitive psychology to describe ways in which individuals organize and process information, and finally make decisions [1] [2] [3].

Original: Intuitive and systematic decision-making are two types of cognitive styles.
Revised: Intuitive and systematic decision-making are two types of cognitive style.

Original: Intuitive decision-making isknown as a type of associative thinking that relies on intuition, and systematic decision-making isknown as a type of rule-based thinking that relies ona logical evaluation [4].
Revised: Intuitive decision-making is a type of associative thinking that relies on intuition, and systematic decision-making is a type of rule-based thinking that relies on logical evaluation [4].

Original: Both cognitive styles have advantages and disadvantages.
Revised: Both cognitive styles have advantages and disadvantages.

Original: For example, intuitive decision-making requires a shorter time than systematic decision-making.
Revised: For instance, intuitive decision-making requires less time than systematic decision-making.

Original: However, systematic decision-making contains a deeper consideration process than intuitive decision-making.
Revised: However, systematic decisionmaking involves a deeper consideration process than intuitive decision-making.

Original: These two cognitive styles have been used and discussed in various fields.
Revised: These two cognitive styles have been used and discussed in several fields.

Original: Sagiv et al.
Revised: Sagiv et al.

Original: [3] conducted a study analyzing different intuitive and systematic cognitive styles used by art, accounting, and mathematics students.
Revised: [3] analyzed different intuitive and systematic cognitive styles used by art, accounting, and mathematics students.

Original: Their study found that different students prefer different cognitive styles in their class, and  cognitive style is consistent with an individual’s personal attributes.
Revised: They established that different students prefer different cognitive styles in their class, and the cognitive style is consistent with an individual’s personal attributes.

Original: Ma-Kellams and Lerner [5] compared intuitive and systematic cognitive styles in the understanding of other people’s feelings and found that a systematic cognitive style can produce better empathic accuracy than an intuitive cognitive style.
Revised: MaKellams and Lerner [5] compared intuitive and systematic cognitive styles to understand the feelings of other people, and they established that a systematic cognitive style can produce better empathic accuracy than an intuitive cognitive style.

Original: In addition, Hwang and Lee [6] explored the impacts of intuitive and systematic cognitive styles of consumers on their visual attention patterns in online shopping environments.
Revised: Hwang and Lee [6] explored the impact of intuitive and systematic cognitive styles of consumers on their visual attention patterns in online shopping environments.

Original: The results showed that consumers pay different visual attention to web pages when they use different cognitive styles to make purchasing decisions.
Revised: The results indicated that consumers pay different visual attention to webpages when they use different cognitive styles to make purchasing decisions.

Original: Such studies showed the effects of the cognitive styles used in various activities.
Revised: These studies have shown the effects of the cognitive styles used in various activities.

Original: In this study, we share a similar concept to explore the effects ofthe cognitive styles (labeling styles) ina manual data annotation, where annotators complete a labeling task (i.e. select an appropriate label for an image) through intuitive (quick labeling) and systematic decision-making (careful labeling).
Revised: In this study, we share a similar concept to explore the effects of cognitive styles (labeling styles) in manual data annotation, where annotators complete a labeling task (i.e., select an appropriate label for an image) through intuitive (quick labeling) and systematic decision-making (careful labeling).

Original: 3 L ABELING S TYLE AND USER INTERFACE
Revised: 3 L ABELING S TYLE AND USER INTERFACE

Original: We define two labeling styles of manual image annotation based on the theory of cognitive psychology: quick labeling and careful labeling.
Revised: We defined two labeling styles of manual image annotation based on the theory of cognitive psychology: quick labeling and careful labeling.

Original: Quick labeling.
Revised: Quick labeling.

Original: The style of quick labeling refers tothe concept of intuitive decision-making.
Revised: Quick labeling refers to intuitive decisionmaking.

Original: In this labeling style, annotatorsare asked to select a label for an image as quickly as possible even when they lack confidence in the target image and label.
Revised: Here, annotators select a label for an image as quickly as possible even when they lack confidence in the target image and label.

Original: They are strongly encouraged to select a label within a short time.
Revised: They are strongly encouraged to select labels within a short time.

Original: Careful labeling.
Revised: Careful labeling.

Original: The style of careful labeling refers to the concept of systematic decision-making.
Revised: Careful labeling refers to the concept of systematic decision-making.

Original: In this labeling style, annotatorsare asked to selecta label for an image  as carefully as possible particularly when they are not confident with the target image and label.
Revised: Here, annotators select an image label as carefully as possible, particularly when they are not confident of the target image and label.

Original: They are strongly encouraged to spend sufficient time before making a label decision.
Revised: They are strongly encouraged to spend sufficient time before making a label decision.

Original: A labeling system was developed to evaluate the quick labeling and careful labeling styles in a manual image annotation task.
Revised: A labeling system was developed to evaluate the quick labeling and careful labeling styles in a manual image annotation task.

Original: Figure 2 shows a screenshot of the image labeling interface.
Revised: Figure 2 shows a screenshot of the image-labeling interface.

Original: The left side of the interface lists the labels (10 labels/categories in 
Revised: The left side of the interface lists the labels (10 labels/categories in the

Original: Fashion-MNIST dataset), and the right side of the interface presents the target image.
Revised: Fashion-MNIST dataset), and the right side of the interface represents the target image.

Original: During the labeling task,  annotators were asked to select an appropriate label from the label list and apply it to the target image.
Revised: During the labeling task, the annotators were asked to select an appropriate label from the label list and apply it to the target image.

Original: After selecting a label for an image (by clicking  a label), the system automatically moves to the next image.
Revised: After selecting a label for an image (by clicking on a label), the system automatically moves to the next image.

Original: The annotators are not allowed to return to the previous images after selectinga label for an image.
Revised: The annotators were not allowed to return to the previous images after selecting an image label.

Original: 4 U SER STUDY
Revised: 4 U SER STUDY

Original: A user study was conducted to compare the quick labeling and careful labeling styles applied to an image labeling task.
Revised: A user study was conducted to compare the quick labeling and careful labeling styles applied to an image-labeling task.

Original: Our goal is to observe the effects of the two labeling styles inthe image labeling tasks, especially in the three different levels of data difficulty (easy, moderately difficult, and extremely difficult).
Revised: We aimed to observe the effects of the two labeling styles in image labeling tasks, specifically at the three different levels of data difficulty (easy, moderately difficult, and extremely difficult).

Original: We comparedthe quick labeling and careful labeling styles in terms of label accuracy and labeling time of the given image labeling tasks.
Revised: We compared quick labeling and careful labeling styles in terms of label accuracy and labeling time for the given image labeling tasks.

Original: 4. Apparatus
Revised: 4. Apparatus

Original: In order to control user study quality, especially the use of quick labeling and careful labeling styles during the image labeling tasks. We outsourced the execution of the user study to a professional company, and the company asked their employees to participate in the user evaluation process asa part of their job.
Revised: To control user study quality, specifically the use of quick labeling and careful labeling styles during image-labeling tasks, we outsourced the execution of the user study to a professional company, which asked their employees to participate in the user evaluation process as part of their job.

Original: The total cost was approximately $1,600, i.e., $640 for the quick labeling task ($53 per participant) and $960 for the careful labeling task ($80 per participant).
Revised: The total cost was approximately $1600, that is, $640 for the quick labeling task ($53 per participant) and $960 for the careful labeling task ($80 per participant).

Original: During the user study, the participants were asked to sit in front of a desktopcomputer and complete the given image labeling tasks (Figure 3).
Revised: During the user study, the participants were asked to sit in front of a desktop and complete the given image-labeling tasks (Figure 3).

Original: 4. Participants
Revised: 4. Participants

Original: A total of 24 participants (12 men and 12 women, 18 to 49 years in age) were invited by the company to participate in the user study.
Revised: Twenty-four participants (12 men and 12 women, 18–49 years) were invited by the company to participate in the user study.

Original: All the participants are Japanese (i.e. understand Japanese Hiragana letters).
Revised: All the participants were Japanese (i.e., understood Japanese Hiragana letters).

Original: Most of the participants (n = 19) had no prior experience with data annotation, four had less than half a year of experience, and one had between half to a full year of experience.[Figure]
Revised: Most of the participants (n = 19) had no prior experience with data annotation, four had less than half a year of experience, and one had between half and a full year of experience.

Original: [Figure]
Revised: 4. Dataset

Original: Three datasets, MNIST [24], Fashion-MNIST [25], and Kuzushiji-MNIST [26], were used forthe labeling tasks in the user study.
Revised: Three datasets, MNIST [24], Fashion-MNIST [25], and Kuzushiji-MNIST [26], were used for labeling tasks in the user study.

Original: Each dataset contains 60,000 training images and 10,000 testing images in ten categories (labels).
Revised: Each dataset contained 60,000 training images and 10, testing images in ten categories (labels).

Original: Figure 4 shows the ten categories for each dataset.
Revised: Figure 4 shows the ten categories for each dataset.

Original: These three datasets have varying levels of difficulty.
Revised: The datasets had varying levels of difficulty.

Original: MNIST is an “easy” dataset.
Revised: MNIST is an “easy” dataset.

Original: These handwritten number digits are not difficult for human users to recognize even  they are in difficult cases, see Figure 5 (a).
Revised: These handwritten number digits are not difficult for human users to recognize, even when they are in difficult cases, as shown in Figure 5(a).

Original: Fashion-MNIST is a “moderately difficult” dataset, which is more difficult than the MNIST dataset It is because it contains some difficult (confusing) items (e.g. “Pullover” and “T-shirt/Top”).
Revised: Fashion-MNIST is a “moderately difficult” dataset. It is because it contains some difficult (confusing) items (e.g., “Pullover” and “T-shirt/Top”).

Original: Figure 5 (b) shows examples of easy and difficult (confusing) cases.
Revised: Figure 5(b) shows examples of easy and difficult (confusing) cases.

Original: Kuzushiji-MNIST is an “extremely difficult” dataset, which is much more difficult than the MNIST and Fashion-MNIST datasets.
Revised: Kuzushiji-MNIST is an “extremely difficult” dataset.

Original: These handwritten Japanese Hiragana letters are very difficult to be recognized (even for Japanese users), especially when the letters are in difficult cases.
Revised: The handwritten Japanese Hiragana letters are very difficult to recognize (even for Japanese users), specifically when the letters are in difficult cases.

Original: Figure 5 (c) shows examples of easy and difficult cases.
Revised: Figure 5(c) shows examples of easy and difficult cases.

Original: We randomly selected 100 images (10 images for each label) from each dataset.
Revised: We randomly selected 100 images (10 images per label) from each dataset.

Original: We then created twelve 100-image datasets (e.g., Datasets 1–12) in which none of the images overlapped.
Revised: Thereafter, we created 12 100-image datasets (e.g., Datasets 1–12) with no overlapping images (1200 images from each dataset, and ten split into 12 non-overlapping subsets).

Original: These twelve 100-image datasets were used forthe 12 participants in the quick labeling task and 12 participants in the careful labeling task (the same datasets were used for both labeling styles).
Revised: The 12 100-image datasets were used for 12 participants in the quick labeling task and 12 participants in the careful labeling task (the same datasets were used for both labeling styles).

Original: 4. Task and Condition
Revised: 4. Task and Condition

Original: The image labeling tasks involved labeling 300 images ( images for each dataset) by each participant.
Revised: The image labeling tasks involved labeling 300 images (100 images for each dataset) for each participant.

Original: During the image labeling task, the participants were requested to select an appropriate label from a 10-category list (10 labels) for each image.
Revised: During the image labeling task, the participants were requested to select an appropriate label from a 10-category list (10 labels) for each image.

Original: A between-subject method was used, by which 12 of the participants were asked to complete the labeling task using the quick labeling style and the other 12 participants were asked to complete the labeling task using the careful labeling style.
Revised: A between-subject method was used, in which 12 of the participants were asked to complete the labeling task using the quick labeling style, and the other 12 participants were asked to complete the labeling task using the careful labeling style.

Original: Quick labeling condition.
Revised: Quick Labeling Conditions.

Original: The participants of the quick labeling task were provided  the following instructions for the labeling task: “Please select a label for an image as quickly as possible.
Revised: The participants of the quick labeling task were provided with the following instructions for the labeling task: “Please select a label for an image as quickly as possible.

Original: In this case, if you are not sure about the target images and labels, 
Revised: Here, if you are unsure about the target images and labels, please

Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: please just select the most appropriate one (i.e., make a guess based on your intuition).
Revised: simply select the most appropriate one (i.e., make a guess based on your intuition).

Original: Please do not spend too much time considering  before making a label selection.
Revised: Please do not spend too much time considering this before selecting the label.

Original: We STRONGLY ENCOURAGE you to select a label for an image within 5 SECONDS. A 5-second timer is provided on the labeling interface.
Revised: We STRONGLY ENCOURAGE you to select a label for an image within 5 s. A 5 s-timer is provided on the labeling interface.

Original: After selecting a label for an image, the system will automatically show the next image.
Revised: After selecting a label for an image, the system will automatically show the next image.

Original: This means that you are not allowed to change your selected label.”
Revised: This means that you are not allowed to change your selected label.”

Original: Our labeling system for the label-quick task contains a 5 s timer for labeling each image.
Revised: Our labeling system for the label-quick task contains a 5 s-timer for labeling each image.

Original: An alert message “Time’s up!
Revised: An alert message “Time’s up!

Original: Please select a label now” is shown when the timer ends.
Revised: Please select a label now” is displayed when the timer ends, as shown in

Original: See Figure 6.
Revised: Figure 6.

Original: The system does not automatically move to the next image until the participants select a label for the target image, even  the timer ends.
Revised: The system did not automatically move to the next image until the participants selected a label for the target image, even when the timer ended.

Original: The timer was designed as a reminder for the participants during annotation.
Revised: The timer was designed as a reminder to the participants during annotation.

Original: In addition,the participants were not allowed to return to the previous image after selecting a label for the image.
Revised: In addition, participants were not allowed to return to the previous image after selecting a label for the image.

Original: Careful labeling condition.
Revised: Careful Labeling Conditions.

Original: The participants in the careful labeling task were provided  the following instruction for the task: “Please select a label for an image as carefully as possible.
Revised: The participants in the careful labeling task were provided with the following instructions: “Please select a label for an image as carefully as possible.

Original: There is no time limitation for labeling each image.
Revised: There is no time limitation for labeling each image.

Original: You have sufficient time to think carefully before making a label decision, particularly when the target images are difficult or when you are not confident with the images and labels.
Revised: You have sufficient time to think carefully before making a label decision, particularly when the target images are difficult or when you are not confident about the images and labels.

Original: We STRONGLY
Revised: We STRONGLY

Original: ENCOURAGE you to spend sufficient time (there is no time limitation) before making a label decision.
Revised: ENCOURAGE you to spend sufficient time (there is no time limitation) before making a label decision.

Original: After selecting a label for an image, the system will automatically show the next image.
Revised: After selecting a label for an image, the system will automatically show the next image.

Original: This means that you are not allowed to change your selected label.”
Revised: This means that you are not allowed to change your selected label.”

Original: The labeling system for the careful labeling task is the same as the labeling system used for the quick labeling task, the only difference being that there is no timer shown on the labeling interface (see Figure 2).
Revised: The labeling system for the careful labeling task is the same as that used for the quick labeling task; but there is no timer shown on the labeling interface (Figure 2).

Original: The participants were allowed to spend as much time as necessary for making a label selection.
Revised: The participants were allowed to spend as much time as necessary to select a label.

Original: In addition, the participants were not allowed to return to the previous image after selecting a label for an image.
Revised: In addition, the participants were not allowed to return to the previous image after selecting a label for an image.

Original: The participants were informed that they can receive 1.5 times higher rewards for the careful labeling task than  a normal labeling task (i.e. quick labeling task).
Revised: The participants were informed that they could receive 1.5 times higher rewards for the careful labeling task than for a normal labeling task (i.e., quick labeling task).

Original: 4. Procedure
Revised: 4. Procedure

Original: An oral overview and detailed written instructionswere provided by an instructor to the participants.
Revised: The instructor provided an oral overview and detailed written instructions to the participants.

Original: The evaluation itself is composed of three parts (in order): instruction and trial (5– min), labeling tasks (20–45 min), and questionnaire (3–5 min).
Revised: The evaluation itself was composed of three parts (in order)

Original: The entire evaluation process was completed between 40 andmin (depending on the labeling styles).
Revised: The entire evaluation process was completed within 40–60 min (depending on the labeling style).

Original: After providing instructions on the labeling interfaces and the given tasks, the participants were allowed to practice on a small labeling task (to label three images for each dataset that differed from the images used in the formal tasks) before starting the given image labeling tasks.
Revised: After providing instructions on the labeling interfaces and the given tasks, the participants were allowed to practice on a small labeling task (to label three images for each dataset that differed from the images used in the formal tasks) before starting the given image labeling tasks.

Original: 4. Measurement
Revised: 4. Measurement

Original: Our labeling system automatically recorded and measured the time and accuracy of the image labeling tasks completed by the participants.
Revised: Our labeling system automatically recorded and measured the time and accuracy of the image-labeling tasks completed by the participants.

Original: The timer started when the participants clicked on “START” and stopped when they clicked on “FINISH.”
Revised: The timer started when the participants clicked on “START” and stopped when they clicked on “FINISH.”

Original: In addition, the system recorded the time spent by the participants for each image labeling during the image labeling process.
Revised: In addition, the system recorded the time spent by the participants for each image-labeling process.

Original: Following the image labeling tasks, the participants were asked to answer a questionnaire regarding the labeling process.
Revised: After the image labeling tasks, the participants were asked to answer a questionnaire regarding the labeling process.

Original: The questionnaire contained 3 Likert scale questions for each labeling style (see section 6.4).
Revised: The questionnaire contained three Likertscale questions for each labeling style (Section 6.4).

Original: 5 M ACHINE L EARNING E XPERIMENT
Revised: 5 M ACHINE L EARNING E XPERIMENT

Original: In the user study, a total of 7,200 labeled images were collected (2,400 for each dataset and 1,200 for each labeling style in a dataset).
Revised: In the user study, 7200 labeled images were collected (2400 for each dataset and 1200 for each labeling style in a dataset).

Original: The training dataset contains errors made by the participants (i.e., the accuracy rate of the training data is not 100%).
Revised: The training dataset contained errors made by the participants (i.e., the accuracy rate of the training data was not 100%).

Original: We used them as  training data to run a machine learning experiment to evaluate the effects ofthe labeling styles (data collected via the quick labeling and careful labeling styles) in machine learning accuracy (image classification).
Revised: We used them as the training data to perform a machine learning experiment to evaluate the effects of labeling styles (data collected via the quick labeling and careful labeling styles) on machine learning accuracy (image classification).

Original: Three common machine learning algorithms (logistic regression, K-nearest neighbors, and support vector machine) were selected for the case study in the machine learning experiment.
Revised: Three common machine learning algorithms (logistic regression, K-nearest neighbors, and support vector machine) were selected for the case study in the machine learning experiment.

Original: We did not use more advanced techniques (e.g. deep learning) because the training dataset is too small and our goal is not to pursue high machine learning accuracy but  compare the difference between the two labeling styles.
Revised: We did not use more advanced techniques (e.g., deep learning) because the training dataset was too small and our goal was not to pursue high machine learning accuracy but to compare the difference between the two labeling styles.

Original: The testing data used in the machine learning experiment were 30,000 images (10,000 for each dataset), which are different from the training dataset (7,200 labeled images).
Revised: The testing data used in the machine learning experiment were 30,000 images (10,000 for each dataset), which were different from the training dataset (7200 labeled images).

Original: 6 R ESULTS 6. Task Completion Time
Revised: 6 R ESULTS 6. Task Completion Time

Original: Figure 7 shows the task completion time for  different labeling styles and datasets.
Revised: Figure 7 shows the task completion times for the different labeling styles and datasets.

Original: The results from the quick labeling task showed that the participants spent an average of 4 min and 23 s, 5 min and 6 s, and 5 min and 37 s to label the 100 images with  MNIST, Fashion-MNIST, and Kuzushiji-MNIST datasets, respectively.
Revised: The results from the quick labeling task indicated that the participants spent an average of 4 min and 23 s, 5 min and 6 s, as well as 5 min and 37 s to label the 100 images with the MNIST, Fashion-MNIST, and Kuzushiji-MNIST datasets, respectively.

Original: The results showed that the participants spent an average of 4 min and 42 s, 6 min and 58 s, and 10 min and 13 s to label the 100 images in the careful labeling task.
Revised: The results indicated that the participants spent an average of 4 min and 42 s, 6 min and 58 s, and 10 min and 13 s labeling the 100 images in the careful labeling task.

Original: The results of an unpaired t-test on the task completion time showed that the difference was insignificant (p > 0.05) between the quick labeling and careful labeling styles in the MNIST dataset, whereas there were significant differences (p < 0.01) in the Fashion-MNIST and Kuzushiji-MNIST datasets.
Revised: The results of an unpaired t-test of the task completion time indicated that the difference was insignificant (p > 0.05) between the quick labeling and careful labeling styles in the MNIST dataset, whereas there were significant differences (p < 0.01) in the Fashion- MNIST and Kuzushiji-MNIST datasets.

Original: This indicates that the careful labeling style requires a longer time to completethe labeling tasks than the quick labeling style when the images are moderatelydifficult and extremely difficult.
Revised: This indicates that the careful labeling style requires a longer time to complete labeling tasks than the quick labeling style when the images are moderately and extremely difficult.

Original: However, the task completion time is comparable between the quick and careful labeling styles when the images are easy.[Figure]
Revised: However, the task completion time was comparable between the quick and careful labeling styles when the images were easy.

Original:  Annotation Accuracy
Revised: 6. Annotation Accuracy

Original: Figure 8 shows the accuracy of the labels given by the participants in the quick labeling and careful labeling styles.
Revised: Figure 8 shows the accuracy of the labels given by the participants in the quick labeling and careful labeling styles.

Original: The results from the quick labeling task showed that the accuracy was 97.58%, 72.08%, and 58.08% with the three datasets,respectively, and the results from the careful labeling tasks showed accuracy of 97.58%, 76.83%, and 60.08%, respectively.
Revised: The results from the quick labeling task indicated accuracies of 97.58%, 72.08%, and 58.08% for the three datasets, and the results from the careful labeling tasks indicated accuracies of 97.58%, 76.83%, and 60.08%.

Original: The analysis ofthe accuracy using an unpaired t-test showed that the difference was insignificant (p > 0.05) in the MNIST and Kuzushiji-MNIST dataset, whereas there was a significant difference (p < 0.05) between the two labeling styles in the Fashion-MNIST dataset.
Revised: The analysis of accuracy using an unpaired t-test showed that the difference was insignificant (p > 0.05) in the MNIST and Kuzushiji-MNIST datasets, whereas there was a significant difference (p < 0.05) between the two labeling styles in the Fashion-MNIST dataset.

Original: This indicates that the careful labeling style can help non-expert annotators to select labels more correctly when the images are moderately difficult (Fashion-MNIST), whereas no clear benefit was observed when the images were easy (MNIST) and extremely difficult (Kuzushiji-MNIST).
Revised: This indicates that the careful labeling style can help non-expert annotators to select labels more correctly when the images are moderately difficult (Fashion- MNIST), whereas no clear benefit was observed when the images were easy (MNIST) and extremely difficult (Kuzushiji-MNIST).

Original: This indicated thatit is a waste of money for conducting a careful labeling task  when the task is  easy ortoo difficult.
Revised: This indicates that conducting a careful labeling task is expensive when the task is either easy or difficult.

Original:  Temporal EffectTask Completion Time
Revised: 6. Temporal Effect

Original:  Figure 9 shows the average time for the labeling process for the first half (1-50 images) and second half (51-100 images) in the three datasets, using the quick labeling style.
Revised: Task Completion Time Figure 9 shows the average time for the labeling process for the first half (1–50 images) and second half (51–100 images) in the three datasets using the quick labeling style.

Original: The results showed that the participants spent an average of 2 min 6 s and 2 min 17 s to complete the first half and second half in the MNIST dataset,min 36 s and 2 min 30 s in the Fashion-MNIST dataset and 2 min 29 s and 3 min 8 s in the  Kuzushiji-MNIST dataset.
Revised: The results indicated that the participants spent an average of 2 min 6 s and 2 min 17 s, 2 min 36 s and 2 min 30 s, and 2 min 29 s and 3 min 8 s to complete the first and second halves of the MNIST, Fashion- MNIST, and Kuzushiji-MNIST datasets, respectively.

Original: The results of  paired t-test showed that the difference was not significant (p > 0.05) between the firsthalf and second half of the labeling process in the MNIST and Fashion-MNIST datasets, but the difference was significant (p < 0.05) in the Kuzushiji-MNIST dataset.
Revised: The results of the paired t-test indicated that the difference was not significant (p > 0.05) between the first and second halves of the labeling process in the MNIST and Fashion-MNIST datasets, but the difference was significant (p < 0.05) in the Kuzushiji-MNIST dataset.

Original: This indicates that there is a temporal effect in all levels of data difficulty when using the quick labeling style.
Revised: This indicates that there is a temporal effect at all levels of data difficulty when using the quick labeling style.

Original: However, the results interestingly showed that the participantseven spent a longer time completing the second half of the image labeling task.
Revised: However, the results interestingly indicated that the participants spent a longer time completing the second half of the image-labeling task.

Original: Figure 10 shows the average time for the labeling process for the first half (1-50 images) andthe second half (51-100 images) using the careful labeling style in different datasets.
Revised: Figure 10 shows the average time for the labeling process for the first half (1–50 images) and second half (51–100 images) using the careful labeling style in different datasets.

Original: The results showed that the participants spent an average of 2 min 24 sandmin 18 s to complete the first half and second half in the MNIST dataset, 4 min 5 s and 2 min 53 s in the Fashion-MNIST dataset, and 4 min 21 s and 5 min 52 s in the  Kuzushiji-MNIST dataset.
Revised: The results indicated that the participants spent an average of 2 min 24 s and 2 min 18 s, 4 min 5 s and 2 min 53 s, and 4 min 21 s and 5 min 52 s to complete the first and second halves of the MNIST, FashionMNIST, and Kuzushiji-MNIST datasets, respectively.

Original: The results of  paired t-test showed that the difference was not significant (p > 0.05) between the firsthalf and second half of the labeling process in the MNIST datasets, but the difference was significant (p < 0.05) in the Fashion-MNIST and KuzushijiMNIST dataset.
Revised: The results of the paired t-test indicated that the difference was not significant (p > 0.05) between the first and second halves of the labeling process in the MNIST dataset, but the difference was significant (p < 0.05) in the Fashion-MNIST and Kuzushiji-MNIST datasets.

Original: This indicates that there is no temporal effect in the use of the careful labeling style when the images are easy.
Revised: This indicates that there is no temporal effect in the use of the careful labeling style when the images are easy.

Original: However,there was a temporal effect  when the images are moderately difficult.
Revised: However, a temporal effect was observed when the images were moderately difficult.

Original: The participants significantly increased their labeling speed in the second half of the image labeling task.
Revised: The participants significantly increased their labeling speed in the second half of the image-labeling task.

Original: In addition, the participant spent more time for the second half when the images are extremely difficult, which is the same as the quick labeling style.
Revised: In addition, the participants spent more time in the second half when the images were extremely difficult, which is the same as the quick labeling style.

Original: Annotation Accuracy
Revised: Annotation Accuracy

Original: Figure 11 shows the accuracy of the labeling process in the first half (1-50 images) and the second half (51-100 images) via the 
Revised: Figure 11 shows the accuracy of the labeling process in the first half (1–50 images) and the second half (51–100 images) via the quick labeling style.

Original: [Figure]
Revised: The results indicated that the accuracy rates were 97.5% and 97.67% in the first and second halves of the

Original: MNIST
Revised: MNIST

Original: [Figure]
Revised: [Figure]

Original: i
Revised: K

Original: [Figure]
Revised: [Figure]

Original:  ( m i nu t e )
Revised: T i m e ( m i nu t e )

Original: Label Quickly Label Carefully 
Revised: Label Quickly Label Carefully **

Original: [Figure]
Revised: [Figure]

Original: * p<0. 
Revised: 5m37s 10 m13s ** p<0. ** n.s. MNIST 80% 60% 40% 20% 72.08% 58.08% F

Original: [Figure]
Revised: [Figure]

Revised: MNIST

Revised: K-MNIST 97.58%

Original: A cc u r a cy
Revised: A cc u r a cy

Original: R a te
Revised: R a t

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: quick labeling style. The results showed that the accuracy rate was 97.5% and 97.67% in the first half and second half in the MNIST dataset, 73.17% and 71% in the Fashion-MNIST dataset, and 56.67% and 59.5% in the Kuzushiji-MNIST dataset.
Revised: MNIST dataset, 73.17% and 71% in the Fashion-MNIST dataset, as well as 56.67% and 59.5% in the Kuzushiji-MNIST dataset.

Original: The results of  paired t-test showed that the difference was not significant (p > 0.05) between the firsthalf and second half of the labeling process in all datasets.
Revised: The results of the paired t-test indicated that the difference was not significant (p > 0.05) between the first and second halves of the labeling process for all datasets.

Original: This indicates that there is no temporal effect in all levels of data difficulty when using the quick labeling style.
Revised: This indicates that there is no temporal effect at any level of data difficulty when using the quick labeling style.

Original: The label accuracy was not significantly affected (improved) by different labeling styles.
Revised: The label accuracy was not significantly affected (improved) by different labeling styles.

Original: Figure 12 shows the accuracy of the labeling process in the first half (1-50 images) and the second half (51-100 images) via the careful labeling style.
Revised: Figure 12 shows the accuracy of the labeling process in the first half (1–50 images) and the second half (51–100 images) via the careful labeling style.

Original: The results showed that the accuracy rate was 97.17% and 98% in the firsthalf and second half in the
Revised: The results indicated that the accuracy rates were 97.17% and 98% in the first and second halves of the

Original: MNIST dataset, 73.67% and 80% in the Fashion-MNIST dataset, and 62.83% and 57.33% in the Kuzushiji-MNIST dataset.
Revised: MNIST dataset, 73.67% and 80% in the Fashion-MNIST dataset, as well as 62.83% and 57.33% in the Kuzushiji-MNIST dataset.

Original: The results of  paired t-test showed that the difference was not significant (p > 0.05) between the firsthalf and second half of the labeling process in the MNIST and Kuzushiji-MNIST datasets.
Revised: The results of the paired t-test indicated that the difference was not significant (p > 0.05) between the first and second halves of the labeling process for the MNIST and Kuzushiji-MNIST datasets.

Original: However, the difference was significant (p < 0.05) in the FashionMNIST dataset.
Revised: However, the difference was significant (p < 0.05) for the Fashion-MNIST dataset.

Original: This indicates that there is no temporal effect when the images are easy and extremely difficult, with the use of the careful labeling style.
Revised: This indicates that there is no temporal effect when the images are easy and extremely difficult to use with the careful labeling style.

Original: However,there was a temporal effect  when the images are moderately difficult.
Revised: However, a temporal effect was observed when the images were moderately difficult.

Original: The participants can significantly improve  label accuracy in the second half of the image labeling task.
Revised: The participants could significantly improve the label accuracy in the second half of the image-labeling task.

Original: To sum up, the careful labeling style brings temporal effect during the labeling process in the task completion time and accuracy rate (i.e. reduce time and increase accuracy) ONLY when the images are moderately difficult.
Revised: In summary, the careful labeling style has a temporal effect during the labeling process in the task completion time and accuracy rate (i.e., reduced time and increased accuracy) only when the images are moderately difficult.

Original: If the images are too easy there is no temporal effect.
Revised: When the images are too easy, there is no temporal effect.

Original: Interestingly, if the images are too difficult task completion time is longer in the second half.
Revised: Interestingly, if the images are too difficult, the task completion time is longer in the second half.

Original: 6. Questionnaire
Revised: 6. Questionnaire

Original: Figure 13 displays how confident the participants felt in the given image labeling tasks.
Revised: Figure 13 shows how confident the participants felt in the given image labeling tasks.

Original: In the MNIST dataset, the results showed that most of the participants felt very confidentand or confident when selecting a label for an image using either the quick labeling (n = 10) or careful labeling (n = 9) styles, whereas none of the participants felt not very confident or not confident.
Revised: In the MNIST dataset, the results indicated that most of the participants felt extremely confident or confident when selecting a label for an image using either the quick labeling (n = 10) or careful labeling (n = 9) styles, whereas none of the participants felt apprehensive.

Original: For the Fashion-MNIST dataset, the results showed that only one participant felt very confident when selecting a label for an image through the quick labeling style and only two participants felt confident when selecting a label for an image through the careful labeling style.
Revised: For the Fashion-MNIST dataset, the results indicated that only one participant felt extremely confident when selecting a label for an image through the quick labeling style and only two participants felt confident when selecting a label for an image through the careful labeling style.

Original: In the Kuzushiji-MNIST dataset, the results showed that no participants felt confident or very confident when selecting a label for an image through either the quick labeling or careful labeling styles.
Revised: In the Kuzushiji-MNIST dataset, the results indicated that no participants felt confident or extremely confident when selecting a label for an image through either the quick labeling or careful labeling styles.

Original: Moreof the participants of the careful labeling style felt not confident (n = 4) or not very confident (n = 7) in comparison to the participants of the quick labeling style (not confident, n = 3; not very confident, n = 6).
Revised: More participants using the careful labeling style felt apprehensive (n = 4) or extremely apprehensive (n = 7) in comparison to the participants using the quick labeling style (apprehensive, n = 3; extremely apprehensive, n = 6).

Original: This indicates that the labeling styles do not affect the subjective impression of the participants’ confidence during annotation.
Revised: This indicates that the labeling styles do not affect the subjective impression of the participants’ confidence during annotation.

Original: However, the ambiguities in the datado affect the confidence of the participants in selecting a label for an image during annotation.
Revised: However, the ambiguities in the data affect the confidence of the participants in selecting a label for an image during annotation.

Original: 6. Results of Machine Learning Experiment
Revised: 6. Results of Machine Learning Experiment

Original: Figure 14 shows the machine learning results (i
Revised: Figure 14 presents the machine learning results (i

Original: [Figure]
Revised: [Figure]

Original: e. image classification accuracy) for the three datasets with the quick and careful labeling styles.
Revised: e., image classification accuracy) for the three datasets with the quick and careful labeling styles.

Original: [Figure]
Revised: [Figure]

Revised: MNIST 80% 60% 40% 20%

Original: [Figure]
Revised: [Figure]

Original:  a 
Revised: A cc u r a cy R a t e

Original: [Figure]
Revised: [Figure]

Original: MNIST80%
Revised: MNIST

Original: [Figure]
Revised: [Figure]

Original: F-MNIST
Revised: F-MNIST

Original: K-MNIST
Revised: K-MNIST

Original: [Figure]
Revised: [Figure]

Original:  a cy
Revised: A cc u r a cy

Original: R a 
Revised: R a t e

Revised: Label Quickly Label Carefully

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: In the data analysis, we were not computing accuracy for each participant because the data set is too small.
Revised: In the data analysis, we did not compute the accuracy for each participant because the dataset was too small.

Original: We combined all the annotations and trained a model and measured the performance of the model.
Revised: We combined all annotations, trained the model, and measured the performance of the model.

Original: MNIST Dataset.
Revised: MNIST Dataset.

Original: The accuracy of the training data was 97.58% for both the quick labeling and careful labeling styles (Figure 12).
Revised: The accuracy of the training data was 97.58% for both the quick labeling and careful labeling styles (Figure 12).

Original: Based on the accuracy of the training data, the machine learning performance (accuracy)also showedthat there were almost no differences between the two labeling styles (logistic regression, LQ = 88.63%, LC = 88.57%; K-nearest neighbors, LQ = 87.81%, LC = 87.71%; support vector machine, LQ = 92.44%, LC = 92.53%).
Revised: Based on the accuracy of the training data, the machine learning performance (accuracy) showed almost no differences between the two labeling styles (logistic regression, LQ = 88.63%, LC = 88.57%; K-nearest neighbors, LQ = 87.81%, LC = 87.71%; support vector machine, LQ = 92.44%, LC = 92.53%).

Original: These results were not surprising as the label accuracy of  training data wastotally the same.
Revised: These results were not surprising because the label accuracy of the training data was the same.

Original: Fashion-MNIST Dataset.
Revised: Fashion-MNIST Dataset.

Original: The accuracy of the training data was 72.08% for the quick labeling style and 76.83% for the careful labeling style (Figure 12), which was a significant difference between the two labeling styles.
Revised: The accuracy of the training data was 72.08% for the quick labeling style and 76.83% for the careful labeling style (Figure 12), which is a significant difference between the two labeling styles.

Original: Based on the accuracy, the machine learning performance (accuracy)also showed that there were differences between the two labeling styles (logistic regression, LQ = 66.51%, LC = 68.71%; K-nearest neighbors, LQ = 65.99%, LC = 70.11%; support vector machine: LQ = 67.91%, LC = 72.66%).
Revised: Based on the accuracy, the machine learning performance (accuracy) showed that there were differences between the two labeling styles (logistic regression, LQ = 66.51%, LC = 68.71%; K-nearest neighbors, LQ = 65.99%, LC = 70.11%; support vector machine: LQ = 67.91%, LC = 72.66%).

Original: The differences were between 2.2% and 4.12%.
Revised: The differences were between 2.2% and 4.12%.

Original: Although machine learning algorithms often work well even if the labels given to difficult data contain errors.
Revised: Machine learning algorithms often work well even if the labels given to difficult data contain errors.

Original: Our results indicate that  theimprovement of label accuracy via the careful labeling stylealso can  improve  machine learning accuracy.
Revised: Our results indicate that improving the label accuracy via the careful labeling style can also improve the accuracy of machine learning.

Original: Kuzushiji-MNIST Dataset.
Revised: Kuzushiji-MNIST Dataset.

Original: The accuracy of the training data was 58.08% for the quick labeling style and 60.08% for the careful labeling style (Figure 12).
Revised: The accuracy of the training data was 58.08% for the quick labeling style and 60.08% for the careful labeling style (Figure 12).

Original: It was a2% small difference  between the two labeling styles, but it was not significant according to a paired t-test analysis.
Revised: There was a small difference of 2% between the two labeling styles, but it was not significant according to the paired t-test analysis.

Original: Based on the accuracy of the training data, the machine learning performance (accuracy) showed that the differences were very small(almost the same) between the two labeling styles (logistic regression, LQ = 37.58%, LC = 37.42%; K-nearest neighbors, LQ = 44.17%, LC = 44.23%; support vector machine, LQ = 43.36%, LC = 43.47%).
Revised: Based on the accuracy of the training data, the machine learning performance (accuracy) showed that the differences were significantly small between the two labeling styles (logistic regression, LQ = 37.58%, LC = 37.42%; K-nearest neighbors, LQ = 44.17%, LC = 44.23%; support vector machine, LQ = 43.36%, LC = 43.47%).

Original: This indicates that the small difference of the label accuracy in the training dataalmost cannot affect  machine learning performance.
Revised: This indicates that a small difference in the label accuracy in the training data cannot affect the machine learning performance.

Original: 7 D ISCUSSION 7. Effects of Labeling Styles in Annotation Efficiency and Label Quality
Revised: 7 D ISCUSSION 7. Effects of Labeling Styles in Annotation Efficiency and Label Quality

Original: In psychology, decision-making is a cognitive process in which the cognitive styles ofthe individuals affect the decision-making process and the decision outcomes and quality [37] [38] [39].
Revised: In psychology, decision-making is a cognitive process in which the cognitive styles of individuals affect the decision-making process as well as the decision outcomes and quality [37] [38] [39].

Original: In manual data annotation, a labeling style can be known as a decision-making process (i.e. select an appropriate label for an image) that may affect label outcomes and quality.
Revised: In manual data annotation, a labeling style can be considered as a decision-making process (i.e., selecting an appropriate label for an image) that may affect the outcomes and quality of the label.

Original: In general, a quick labeling style requires less time to complete an annotation task, whereas the careful labeling style requires more time; and we normally expect that a careful labeling style bring higher quality data than a quick labeling style.
Revised: In general, the quick labeling style requires less time to complete an annotation task, whereas the careful labeling style requires more time, and it can result in higher quality data.

Original: However, it depends on  data difficulty and annotation tasks.
Revised: However, it depends on the data difficulty and annotation tasks.

Original: Our results demonstrate that there was no significant difference between the quick and carful labeling styles in the task completion time and label quality, when the data is easy (i.e. MNIST).
Revised: Our results indicate that there was no significant difference between the quick and careful labeling styles in the task completion time and label quality when the data were easy (i.e., MNIST).

Original: However, it shows differences between the two labeling style when  data is getting difficult.
Revised: However, there are differences between the two labeling styles when the data becomes difficult.

Original: For example, the careful labeling style requires more time to complete a labeling task which contains moderately difficult images than the quick labeling style.
Revised: For instance, the careful labeling style requires more time to complete a labeling task that contains moderately difficult images than the quick labeling style.

Original: At the same time, it significantly improves  label quality.
Revised: Moreover, it significantly improves the label quality.

Original: However, if a labeling task contains extremely difficult images the careful labeling style cannot improve  label quality but requiresmuch longer time to complete the labeling task.
Revised: However, if a labeling task contains extremely difficult images, the careful labeling style cannot improve the label quality and it requires longer time to complete the labeling task.

Original: These results imply that a labeling style affects  annotation efficiency (task completion time) and label quality (accuracy rate) in non-expert data annotation.
Revised: These results indicate that the labeling style affects the annotation efficiency (task completion time) and label quality (accuracy rate) in non-expert data annotation.

Original: However, the effects are dependent on the data difficulty of an annotation task.
Revised: However, these effects are dependent on the data difficulty of the annotation task.

Original: In addition, the questionnaire results showed that the subjective impression of  annotator’s confidence during annotation was not affected by the labeling styles in all the labeling tasks with different data ambiguities.
Revised: In addition, the questionnaire results indicated that the subjective impression of the annotator’s confidence during annotation was not affected by the labeling styles in any of the labeling tasks with different data ambiguities.

Original: However, annotator’s confidence  was affected by  data ambiguities (higher confidence in less difficult data, lower confidence in more difficult data).
Revised: However, the confidence of the annotator was affected by the data ambiguities (higher confidence in less difficult data and lower confidence in more difficult data).

Original: 7. Temporal Effects in the Quick and Careful Labeling Tasks
Revised: 7. Temporal Effects in the Quick and Careful Labeling Tasks

Original: Temporal effect has been used to analyze task performance during an image labeling task [16].
Revised: The temporal effect has been used to analyze task performance during image-labeling tasks [16].

Original: It describes how people change their behavior over time, which is a method to analyze the efficiency of an activity ora study [42, 43, 44].
Revised: It describes ways in which people change their behavior over time, which is a method for analyzing the efficiency of an activity or study [42, 43, 44].

Original: Our results showed that there was a significant temporal effect (p < 0.05) during the labeling process via the careful labeling style when the images are moderately difficultimages (Fashion-MNIST).
Revised: Our results indicated that there was a significant temporal effect (p < 0.05) during the labeling process using the careful labeling style when the images were moderately difficult (Fashion-MNIST).

Original: The participants can reduce the task completion time in the second half of the image labeling task by using the careful labeling style.
Revised: The participants could reduce the task completion time in the second half of the image labeling task by using the careful labeling style.

Original: This indicates that the careful labeling style not only can improve label quality but also bring temporal effect during annotation in a labeling task containing moderately difficult images.
Revised: This indicates that the careful labeling style not only improves the label quality but also causes a temporal effect during annotation in a labeling task containing moderately difficult images.

Original: Furthermore, there were significant temporal effects (p < 0.05) during annotation via the careful labeling style when the images are extremely difficult images (Kuzushiji-MNIST).
Revised: Furthermore, there were significant temporal effects (p < 0.05) during annotation using the careful labeling style when the images were extremely difficult (KuzushijiMNIST).

Original: But, the participants spent longer time to complete the second half of the image labeling task than  the first half.
Revised: However, the participants spent longer time completing the second half of the image-labeling task than in the first half.

Original: In addition, there was no significant temporal effect (p > 0.05) during annotation via both the quick and careful labeling styles when the images are easy (MNIST).
Revised: In addition, there was no significant temporal effect (p > 0.05) during annotation using both the quick and careful labeling styles when the images were easy (MNIST).

Original: However, the reason of the temporal effect was not  clearly demonstrated.
Revised: However, the reason for the temporal effect has not been clearly demonstrated.

Original: 7. Effects of Labeling Styles in Machine Learning Performance
Revised: 7. Effects of Labeling Styles on Machine Learning Performance

Original: Data quality plays a critical role in machine learning.
Revised: The data quality plays a critical role in machine learning.

Original: Our user study showed that the careful labeling style can significantly improve the label quality of the image labeling task which contains moderately difficult images, and  slightly improves the label quality when the task contains extremely difficult images.
Revised: Our user study demonstrated that the careful labeling style can significantly improve the label quality of the image labeling task, which contains moderately difficult images, and it slightly improves the label quality when the task contains extremely difficult images.

Original: Our machine learning experiment also showed similar results.
Revised: The machine learning experiment also showed similar results.

Original: The labeled data collected via the careful labeling style can bring better machine learning performance (higher accuracy) than the labeled data collected via the quick labeling style when the images are moderately difficult (Fashion-MNIST).
Revised: Labeled data collected via the careful labeling style can result in better machine learning performance (higher accuracy) than that collected via the quick labeling style when the images are moderately difficult (Fashion-MNIST).

Original: However, the machine learning performance showed almost no difference between the labeled data collected via the two labeling styles when the images are easy (MNIST) andvery small difference when the data is extremely difficult (Kuzushiji-MNIST).
Revised: However, the machine learning performance showed almost no difference between the labeled data collected via the two labeling styles when the images were easy (MNIST), and small differences when the data were extremely difficult (Kuzushiji-MNIST).

Original: Machine learning algorithms often work well even if the labels given to difficult data contains errors.
Revised: Machine learning algorithms often work well even if the labels given to difficult data contain errors.

Original: This means that different label quality may bring no difference in  machine learning accuracy.
Revised: This indicates that different label qualities may result in no difference in the machine learning accuracy.

Original: In such cases, a labeling style maybe not  a variable in machine leaning (only in manual data annotation).
Revised: In such cases, the labeling style may not be a variable in machine learning (only in manual data annotation).

Original: However, our results showed that the improvement of label quality via the careful labeling style also can increase machine learning accuracy when the data is moderately ambiguities.
Revised: However, our results indicated that the improvement in label quality via the careful labeling style could increase machine learning accuracy when the data are moderately ambiguous.

Original: This finding has shown that the careful labeling style can bring benefits for the both data annotation and machine learning.
Revised: This finding indicates that the careful labeling style can benefit both data annotation and machine learning.

Original: However, it depends on data ambiguities.
Revised: However, this depends on data ambiguities.

Original: Our machine learning experiment used basic algorithms that only can shows labeling style is a variable in  machine leaning performance.
Revised: Our machine learning experiment used basic algorithms that only showed labeling style as a variable in the machine learning performance.

Original: A machine learning experiment with advanced techniques (e.g. deep learning with a large-scale dataset)is still necessary to be implemented in the future.
Revised: Therefore, a machine learning experiment with advanced techniques (e.g., deep learning with a large-scale dataset) still needs to be implemented in the future.

Original: 7. Three Factors for Selecting an Appropriate Labeling Style for An Annotation Task
Revised: 7. Three Factors for Selecting an Appropriate Labeling Style for an Annotation Task

Original: Ouruser study and machine learning experiment have shown that different labeling styles have its advantages and disadvantages in different annotation tasks.
Revised: Our study and machine learning experiment have shown that different labeling styles have their advantages and disadvantages for different annotation tasks.

Original: For example, conducting a careful labeling task is costlier (requires longer time to complete a task) than a quick labeling task, but it cannot guarantee the improvement of  label quality in all levels of data difficulty.
Revised: For instance, conducting a careful labeling task is costly (requires longer time to complete a task) than a quick labeling task; however, it cannot guarantee the improvement of the label quality at all levels of data difficulty.

Original: So, it is important for deciding a reasonable labeling style for an annotation task, otherwise it may bejust a waste of time and money if the improvement is not clear.
Revised: Therefore, it is important to decide a reasonable labeling style for an annotation task; otherwise, it may be a waste of time and money if the improvement is not clear.

Original: Here we discuss three factors that should be carefully considered when selecting a labeling style for an annotation task.
Revised: Here, we discuss three factors that should be carefully considered when selecting a labeling style for an annotation task.

Original: (1) Data Difficulty
Revised: (1) Data Difficulty

Original: Our results showed that data difficulty is a key factor that affects annotation results when using different labeling styles.
Revised: Our results indicated that data difficulty is a crucial factor affecting annotation results when using different labeling styles.

Original: For example, the careful labeling style only shows benefit (i.e. improve label quality) when the data is easy and extremely difficult, while the quick labeling study only shows benefit (i.e. require less time) when the data is easy.
Revised: For instance, the careful labeling style only shows benefit (i.e., improves the label quality) when the data is easy and extremely difficult, whereas the quick labeling style only shows benefit (i.e., requires less time) when the data is easy.

Original: According to the results, we suggest that the quick labeling style is a reasonable choose when conduct an easy annotation task.
Revised: Based on the results, we suggest that the quick labeling style is a reasonable choice when conducting an easy annotation task.

Original: However, when an annotation task contains difficult data the careful labeling style can be a worth choose for consideration.
Revised: However, when an annotation task contains difficult data, a careful labeling style can be worthwhile.

Original: (2) Annotator Type and Task Condition
Revised: (2) Annotator Type and Task Conditions

Original: Although this study only focused on non-expert annotators, we believe that  annotator’s experience (i.e. domain knowledge in the given task) is another important factor that may highly affect  annotation results of using different labeling styles.
Revised: Although this study only focused on non-expert annotators, we believe that the annotator’s experience (i.e., domain knowledge in the given task) is an important factor that may significantly affect the annotation results of using different labeling styles.

Original: For example, the data difficultyalso depends on individual experience and subjective impression.
Revised: For instance, the data difficulty depends on individual experiences and subjective impressions.

Original: We suggest thata qualification is important and necessary for recruiting annotators when conductingan annotation task with different labeling styles.
Revised: We suggest that qualification is important and necessary for recruiting annotators when conducting annotation tasks with different labeling styles.

Original: In addition,  task condition (e.g. crowd tasks and in-person task) is also a factor when deciding a labeling style for an annotation task.
Revised: In addition, the task condition (e.g., crowd tasks and in-person tasks) should be considered when deciding the labeling style for an annotation task.

Original: Crowdsourcing is avery popular approach for conducting annotation tasks such as Amazon Mechanical Turk [41].
Revised: Crowdsourcing is a popular approach for conducting annotation tasks, such as Amazon Mechanical Turk [41].

Original: However, the quality of crowd tasks is a critical issue what have been discussed for many years [22, 35].
Revised: However, the quality of crowd tasks is a critical issue that has been discussed for many years [22, 35].

Original: We believe that this issue also happens whenusing different labeling styles  in a crowd task.
Revised: We believe that this issue also occurs when different labeling styles are used in a crowd task.

Original: This is the main reason we recruitedour participants (annotators) for the user study via a professional company.
Revised: Therefore, we recruited participants (annotators) for the user study via a professional company.

Original: This helps usto explore the effects of labeling styles more precisely (in order to prove the research concept).
Revised: This helps us explore the effects of labeling styles more precisely ( to prove the research concept).

Original: However, crowdsourcing is still an indispensable approach for conductingan annotation task.
Revised: However, crowdsourcing remains an indispensable approach for conducting annotation tasks.

Original: We suggest that an online workflow should be carefully designed for controlling the annotation quality even with different labeling styles.
Revised: We suggest that an online workflow should be carefully designed to control the annotation quality, even with different labeling styles.

Original: (3) Instruction for Implementing a Labeling Style
Revised: (3) Instruction for Implementing the Labeling Style

Original: After deciding a labeling style, it is important to make sure that annotators can fellow and implement the labeling style precisely.
Revised: After deciding on the labeling style, it is important to ensure that annotators can follow and implement the labeling style precisely.

Original: In this study, we provideda textual and oral instruction of each labeling style by an instructor before start a formal task (also contain a trial).
Revised: In this study, we provided textual and oral instructions for each labeling style by an instructor before starting a formal task (including a trial).

Original: This is only for the user study.
Revised: This is only for a user study.

Original: We believe that an instruction for using a labeling style is not enough in a realistic annotation task.
Revised: We believe that instructions for using a labeling style are not sufficient in a realistic annotation task.

Original: It is because some annotators may  inherently careful to fellow an assigned labeling style, but and some may  inherently sloppy.
Revised: This is because some annotators may be inherently careful to follow an assigned labeling style, whereas others may be inherently sloppy.

Original: In order to avoid this kind of bias, we suggest that a specific labeling workflow (or labeling interface)should be designed and provided to afford different labeling styles, or “FORCE” annotators to fellow specific steps.
Revised: To avoid this kind of bias, we suggest that a specific labeling workflow (or labeling interface) be designed and provided to afford different labeling styles, or “FORCE” annotators to follow specific steps.

Original: For example, to design a workflow that requires annotators to double-check or spend  certain  timefor consideration before making a label decision when an annotation task  conducted via a careful labeling style.
Revised: For instance, a workflow that requires annotators to double-check or spend a certain amount of time before making a label decision when an annotation task is conducted via the careful labeling style should be designed.

Original: 8 L IMITATION AND FUTURE WORK
Revised: 8 L IMITATION AND FUTURE WORK

Original: One limitation of this study is that the size of the training data (1,200 labeled images collected via each labeling style, for each dataset) was small in the machine learning experiment.
Revised: One limitation of this study is that the size of the training data (1200 labeled images collected via each labeling style for each dataset) was small in the machine learning experiment.

Original: This is the main reason for the much lower machine learning accuracy of our study in comparison with the benchmarks [24] [25] [26].
Revised: This is the main reason for the significantly lower machine learning accuracy in our study compared to the benchmarks [24] [25] [26].

Original: Another limitation is that we only used basic machine leaning algorithms for training and testingwith our collected data.
Revised: Another limitation is that we only used basic machine learning algorithms for training and testing our collected data.

Original: However, the main purpose of this study is not to pursue high accuracy of machine learning results, and instead to focus on the effects of the labeling styles.
Revised: However, the main purpose of this study is not to pursue high accuracy of machine learning results, but to focus on the effects of the labeling styles.

Original: Our results show that the careful labeling style can improve the label accuracy ina manual data annotation as well as increasethe machine learning accuracy.
Revised: Our results indicate that the careful labeling style can improve the label accuracy in manual data annotation as well as increase machine learning accuracy.

Original: We believe that the labeling styles might have an even greater effect in large-scale labeling tasks and advanced machine learning techniques (e.g. deep learning).
Revised: We believe that the labeling styles might have an even greater effect on large-scale labeling tasks and advanced machine learning techniques (e.g., deep learning).

Original: In the future, we plan to run a large-scale user study via crowdsourcing and test more machine learning algorithms.
Revised: In the future, we plan to conduct a large-scale user study via crowdsourcing and test more machine learning algorithms.

Revised: Another limitation is the careful labeling style used in this study.

Revised: In the current instruction (design) of the careful labeling style, the participants were asked to select a label for an image as carefully as possible without time limitations.

Revised: This condition may be insufficient to conduct a precise label-careful task.

Revised: A more specific condition or workflow (e.g., allowing modification or force to double-check) for a careful labeling task may be needed for further investigation.

Original: In addition, we would like toexplore more details regarding thecareful labeling style.
Revised: In addition to the labeling style, the level of data difficulty should be carefully defined.

Revised: For instance, how to define the “too easy” and “too difficult” data for each annotator because different annotators may feel different about the same data.

Revised: In the future, we will explore more details regarding the careful labeling style.

Original: For example, the reason of the temporal effect and the effect of compensation were not clearly demonstrated in current study that we plan to investigate it.
Revised: For instance, the cause of the temporal effect and the effect of compensation were not clearly demonstrated in this study.

Original: Another interesting possibility is the dynamic control of  labeling styles duringan annotation.
Revised: Another interesting possibility is the dynamic control of the labeling styles during annotation.

Original: If a system can judge the difficulty of each data item before an annotation, it might be possible to ask an annotator to use an appropriate labeling style (use a careful labeling style for data with  moderate difficulty only).
Revised: If a system can judge the difficulty of each data item before an annotation, it might be possible to ask an annotator to use an appropriate labeling style (using a careful labeling style for data with only moderate difficulty).

Original: 9 C ONCLUSION
Revised: 9 C ONCLUSION

Original: In this study, we investigated the effects of labeling styles in non- expert data annotation and machine learning.
Revised: In this study, we investigated the effects of labeling style on nonexpert data annotation and machine learning.

Original: We ran a user study to compare a quick labelingstyle anda careful labeling style for a manual image annotation task, and  used the labeled data (as training data) to run machine learning experiment.
Revised: We conducted a user study to compare the quick labeling and careful labeling styles for a manual image annotation task, and we used the labeled data (as training data) to perform a machine learning experiment.

Original: Our results showed that  labeling style is a variable in a data annotation process and machine learning performance.
Revised: Our results indicated that the labeling style is a variable in the data annotation process and machine learning performance.

Original: The careful labeling style improves  label accuracy only when the task is moderately difficult whereas itis only increases the cost without improving accuracy when the task is easy or extremely difficult.
Revised: The careful labeling style improves the label accuracy only when the task is moderately difficult, whereas it only increases the cost without improving accuracy when the task is easy or extremely difficult.

Original: These findings provide insights forhelping annotators to select an appropriate labeling style for an annotation task.
Revised: These findings provide insights for annotators when selecting an appropriate labeling style for an annotation task.

Original: This can be an alternative solution for improving non-expert annotation.
Revised: This could be an alternative solution for improving non-expert annotations.

Original: R EFERENCES
Revised: A CKNOWLEDGMENTS

Original: This work was supported by ******** (removed for the review).
Revised: This work was supported by JST CREST Grant Number JPMJCR17A1, and JST, ACT-X Grant Number JP-MJAX21AG, Japan. R