Hooge et al. 2018 asked the question: "Is human classification by experienced untrained observers a gold standard in fixation detection?" They conclude the answer is no. I have had a close look at their data and their report, and I find that both the data itself, the method of presentation of the data to the human raters, and the analysis of agreement statistics are problematic. I think that data used to address this important question should be very high quality, and the method of presentation of the data should be optimized. Alternatively, the title should be changed to : "Is human classification by experienced untrained observers a gold standard in fixation detection when data quality is very poor  ?" Numerous examples of various types of low quality data are presented herein. In addition, it appears that, by including missing data in the data submitted for assessment of agreement statistics, Hooge et al (2018) have made a serious mistake. Also, the treatment of some inter-fixation intervals as saccades that might have been identified by human experts is completely unjustified. The results regarding these saccades are without merit or meaning .
Hooge et al. asked the question: "Is human classification by experienced untrained observers a gold standard in fixation detection?" They conclude the answer is no. If they had entitled their paper : "Is human classification by experienced untrained observers a gold standard in fixation detection when data quality is very poor , data are error-filled, data presentation was not optimal, and the analysis was seriously flawed ?" , I would have no case to make. In the present report, I will present evidence to support my view that this latter title is justified. The low quality data assessment is based on using a relatively imprecise eye-tracker, the absence of head restraint for any subjects, and the use of infants as the majority of subjects (60 of 70 subjects). Allowing subjects with more than 50\% missing data (as much as 95\%) is also evidence of low quality data . The error-filled assessment is based on evidence that a number of the "fixations" classified by "experts" have obvious saccades within them, and that, apparently, a number of fixations were classified on the basis of no signal at all. The evidence for non-optimal data presentation stems from the fact that, in a number of cases, perfectly good data was not presented to the coders. The flaws in the analysis are evidenced by the fact that entire stretches of missing data were considered classified, and that the measurement of saccade amplitude was based on many cases in which there was no saccade at all. Without general evidence to the contrary, it is correct to assume that some human classifiers under some conditions may meet the criteria for a gold standard, and classifiers under other conditions may not. This conditionality is not recognized by Hooge et al. A fair assessment would conclude that whether or not humans can be considered a gold standard is still very much an open question .
D
2018
R
I have had a close look at their data and their report, and I find that both the data itself, the method of presentation of the data to the human raters, and the analysis of agreement statistics are problematic. I think that data used to address this important question should be very high quality, and the method of presentation of the data should be optimized. Alternatively, the title should be changed to
If they had entitled their paper
A
, data are error-filled, data presentation was not optimal, and the analysis was seriously flawed
R
Numerous examples of various types of
, I would have no case to make. In the present report, I will present evidence to support my view that this latter title is justified. The low quality data assessment is based on using a relatively imprecise eye-tracker, the absence of head restraint for any subjects, and the use of infants as the majority of subjects (60 of 70 subjects). Allowing subjects with more than 50\% missing data (as much as 95\%) is also evidence of
R
are presented herein. In addition, it appears that, by including missing data in the data submitted for assessment of agreement statistics, Hooge et al (2018) have made a serious mistake. Also, the treatment of some inter-fixation intervals as saccades that might have been identified by human experts is completely unjustified. The results regarding these saccades are without merit or meaning
. The error-filled assessment is based on evidence that a number of the "fixations" classified by "experts" have obvious saccades within them, and that, apparently, a number of fixations were classified on the basis of no signal at all. The evidence for non-optimal data presentation stems from the fact that, in a number of cases, perfectly good data was not presented to the coders. The flaws in the analysis are evidenced by the fact that entire stretches of missing data were considered classified, and that the measurement of saccade amplitude was based on many cases in which there was no saccade at all. Without general evidence to the contrary, it is correct to assume that some human classifiers under some conditions may meet the criteria for a gold standard, and classifiers under other conditions may not. This conditionality is not recognized by Hooge et al. A fair assessment would conclude that whether or not humans can be considered a gold standard is still very much an open question