N EARING OR S URPASSING : O VERALL E VALUATION OF H UMAN -M ACHINE D YNAMIC V ISION A BILITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Dynamic visual ability (DVA), a fundamental function of the human visual system, has been successfully modeled by many computer vision tasks in recent decades.
However, the prosperity developments mainly concentrate on using deep neural networks (DNN) to simulate the human DVA system, but evaluation frameworks still simply compare performance between machines, making it tough to determine how far the gap is between humans and machines in dynamic vision tasks.
In fact, neglecting this issue not only makes it hard to determine the correctness of current research routes, but also cannot truly measure the DVA intelligence of machines.
To answer the question, this work designs a comprehensive evaluation framework based on the 3E paradigm – we carefully pick 87 videos from various dimensions to construct the environment, conﬁrming it can cover both perceptual and cognitive components of DVA; select 20 representative machines and 15 human subjects to form the task executors, ensuring that different model structures can help us observe the effectiveness of research development; and propose multiple evaluation indicators to quantify their DVA.
Based on detailed experimental analyses, we ﬁrst determine that the current algorithm research route has effectively shortened the gap.
Besides, we further summarize the weaknesses of different executors, and design a human-machine cooperation mechanism with superhuman performance.
In summary, the contributions include: (1) Quantifying the DVA of humans and machines, (2) proposing a new view to evaluate DVA intelligence based on the human-machine comparison, and (3) providing a possibility of human-machine cooperation.
The 87 sequences with frame-level human-machine comparison and cooperation results, the toolbox for recording real-time human performance, codes for sustaining various evaluation metrics, and evaluation reports forrepresentative models will be open-sourced to help researchers develop intelligent research on dynamic vision tasks.
I NTRODUCTION
Research on visual abilities can be dated back to the last century (Hubel & Wiesel (1959; 1962)).
Neuroscientists divide the human visual system into two categories, namely the static vision ability (SVA) to perceive the details of static objects (Chan & Courtney (1996)), and the dynamic vision ability (DVA) to track moving objects (JW et al.
These two visual abilities are essential in our daily life (Land & McLeod (2000);
Beals et al.
Burg (1966); Kohl et al.
and have been modeled by a series of computer vision tasks.
Recently, with the growth of dataset scale and the abundance of computing resources, most data-driven algorithms achieve higher and higher scores in experimental environments, and are widely employed in various scenarios (Dankert et al.
Weissbrod et al.
; Wei & Kording (2018)).
However, some bad cases hidden under the prosperity development challenge the state-of-the-art (SOTA) algorithms.
For example, visual models usually decrease their perception when encountering unknown-category targets under special illumination conditions ( e .
g ., a vision-based autonomous vehicle crashes into a large truck at night).
This shortcoming is far from humans’ powerful visual abilities and may cause safety hazards, causing us to rethink – with the support of massive datasets and powerful computing resources, why can’t SOTA models achieve a similar visual ability to humans?
Do existing evaluation methods actually measure the visual intelligence of machines? [Figure]
Compared to thriving algorithm research, evaluation methods have received far less attention.
Traditional evaluations mainly focus on the performance comparison between machines (Deng et al.
Russakovsky et al. (2015)), causing the goal to become downward ( i .
e ., a method that exceeds all others is considered excellent).
In fact, these evaluation mechanisms actually measure the machine’s performance rather than intelligence.
When we refer to machine intelligence, a natural association is Turing test (Turing (2009)), which indicates that machine intelligence evaluation requires human participation, and has gradually attracted scholars to propose a series of essential works in decisionmaking problems ( e .
g .,
AlphaGo (Silver et al.
and DeepStack (Brown & Sandholm (2018))).
Using humans as a reference, the goal will become upward – exemplary machines must gradually move closer to human capabilities.
In other words, exploring the gap between humans and machines is very crucial for machine intelligence evaluation .
Based on this idea, recent works have introduced humans to several vision tasks, using human baseline to measure machine intelligence. Some researchers endeavor to analyze the gap between machines and humans in image classiﬁcation (Geirhos et al.
(2018; 2020; 2021)), others investigate the attention areas in static images to explore visual selectivity (Langlois et al.
Regrettably, their research scope is mainly restricted to static vision tasks, while neuroscience and cognitive psychology studies have shown that the correlation between SVA and DVA is naturally low (Long & Penn (1987)). On the other hand, existing research on DVA evaluation for human and machine is entirely separate. As shown in Figure 1 (a), neuroscientists use toy environments to measure human DVA (Pylyshyn & Storm (1988)), while computer scientists evaluate machines through large-scale datasets (Fan et al.
; Huang et al. – neither the environment nor the evaluation mechanism are compatible, causing the comparison of human-machine DVA to be impossible. Therefore, an intuitive question is, how to compare humans and machines in dynamic vision ability?
To answer this question, we should ﬁrst select a suitable computer vision task to represent human DVA, then design a evaluation framework to accomplish human-machine DVA measurement and comparison.
Compared with other tasks like multi-object tracking (MOT) (Ciaparrone et al.
and video object detection (VOD) (Wang et al.
(2018)), single object tracking (SOT, i .
e ., locates a user-speciﬁed moving target in a video)
(Wu et al.
(2015)) is a category-independent task with no constraint on motion continuity, scene change, or object category, and can be regarded as the closest task to represent human DVA (Appendix A.1).
Due to the human-like task deﬁnition of SOT, excellent task executors should not only keep tracking the moving target ( perception ) but also re-locate the target when its position is mutated ( cognition ) (Appendix A.2).
From above analyses, we select SOT as the representative task, and follow 3E paradigm (Hu et al.
(2022a))
to design an overall evaluation framework (Figure 1). The technical difﬁculties and our contributions are as follows. Experimental environment construction (Section 2.1). The ﬁrst difﬁculty in environment construction is compatibility .
Choosing a high-contrast toy environment used in classical neuroscience work is too simple to evaluate DVA accurately.
On the other hand, human psychophysical experiments are expensive and time-consuming, and cannot be assessed on large-scale datasets like machines. Based on this, the second difﬁculty is representativity . With the limitation of dataset scale, the environment should not only fully represent the characteristics and difﬁculties of DVA, but also provide a graded experimental setup for subsequent analyses. As shown in Table 1, to entirely reﬂect the task characteristics and thoroughly compare the human-machine DVA, we choose 87 videos with 244,455 frames to construct the environment.
All videos are carefully picked from various dimensions, ensuring the environment can cover the perceptual and cognitive components of DVA. Experimental executor selection and record (Section 2.2).
As an interdisciplinary experiment, the task executors involve humans and machines. The difﬁculty for human subjects is accurately recording their performance in dynamic vision tasks and ensuring that the recorded results can be quantiﬁed to support subsequent evaluations. Through detailed comparison and analysis, we select the mouse-based method and design a toolbox for human subjects to record their frame-level performance in real-time. For machines, since we aim to study the DVA gap between humans and machines, there are two concerns in model selection: trends ( i .
e ., selecting various algorithms) and gap ( i . e ., focusing on the upper bound of algorithms). Thus, both classic and SOTA methods with different architectures are selected to explore whether the research route has effectively shortened the gap. Consequently, task executors included 20 representative algorithms and 15 subjects (all subjects are computer vision researchers aged 20-30).
All experiments are managed in a strict process . Evaluation metrics design (Section 2.3).
Traditional indicators usually select the positional relationship ( e . g ., intersection over union (IoU) and center distance) between predicted result p t and ground-truth g t to accomplish calculation.
However, the tracking result of humans is a point, which cannot be calculated by IoU (Zheng et al.
; Rezatoﬁghi et al. Thus, we use two center points ( i . e ., the target center c p predicted by executors, and the center point c g of ground-truth) to design three granularities (frame-level, sequence-level, and group-level) for evaluation.
Especially, we consider the inﬂuence of sequence length and revise the group-level indicators to generate a more appropriate evaluation.
Experimental analyses have veriﬁed the validity of proposed metrics. Based on above steps, we quantify the DVA of executors and conduct detailed comparisons (Section 3). By analyzing and summarizing the results, we ﬁnd that: (1) Synthetically, humans have stronger dynamic vision ability than machines.
Experimental results demonstrate that humans outperform machines when both perceptual and cognitive abilities are required.
Particularly, when we mainly examine the perceptual ability in dynamic vision tasks, we also notice that the gap of humanmachine perceptual ability is closing.
(2) Human-machine cooperation is possible and effective in dynamic vision tasks.
Based on human-machine differences in perception and cognition, we design a set of cooperative experiments.
Results indicate that the SOTA machines with humanmachine cooperation can achieve even better performance than humans in some cases.
In summary, this work starts with designing a more intelligent evaluation framework, measures the DVA gap between humans and machines, determines the effectiveness of current algorithm research routes, ﬁnds the weaknesses of different executors, and notes the possibility of human-machine cooperation.
We will open-source the toolbox, code, and metrics used in the experiments to help researchers further develop intelligent research on dynamic vision tasks.
M ETHODS 2. E NVIRONMENT : D ATASETS
For DVA evaluation, researchers initially require subjects to observe moving objects on a high-contrast background.
For example, an early work displays some moving plus (+) signs on the screen(Pylyshyn & Storm (1988)).
Recently, researchers evaluate the DVA of athletes based on a software named
DynVA (Quevedo et al.
(2012)), which includes various stimuli with different color and motion trajectories.
Although above works have developed from simple designs to computer programs, they all have limited target categories ( e .
g ., speciﬁc symbols), simple backgrounds ( e .
g ., colors or some static photographs), and limited motion patterns ( e .
g ., lateral, vertical, and oblique).
Unlike above toy environments that track simple symbols in a high-contrast background, our environment fully models the application scenes and designs various data combinations for ability analyses ( e .
g ., perception, cognition, and robustness).
Concretely, the environment should consider the diversity of video contents and includes the variation of video length (especially long sequences).
Thus, we refer SOTVerse (Hu et al.
(2022a))
and select sequences from representative benchmarks (OTB (Wu et al.
(2013; 2015)), VOT series (Kristan et al.
GOT-10k (Huang et al.
, LaSOT (Fan et al.
(2020)), and VideoCube (Hu et al.
(2022b)))
to form the experimental environment (Table 1 and Appendix C.1).
[Table]
2. E XECUTOR : M ODELS , H UMAN S UBJECTS
As shown in Table [Table] 20 represent machines covering both classic and SOTA methods are selected: 2 CF-based trackers (Henriques et al. (2015); Danelljan et al.
[Table]
10 SNN-based methods (Bertinetto et al [Table] ; Li et al. (2018b); Zhu et al.
; Li et al. (2018a); Yan et al. ; Zhang & Peng (2019); Guo et al.
Xu et al.
; Zhang & Peng (2020); Voigtlaender et al.
(2020)), 5 algorithms that combine CF and SNN (Danelljan et al.
Bhat et al.
; Danelljan et al.
(2020); Mayer et al.
(2021)), and 3 custom networks (Huang et al.
Bhat et al.
; Cui et al.
Please refer to Appendix B for technical details.
For humans, we start the measurement of human DVA based on the following concerns:
Organization.
87 videos with different durations, object classes, and scene categories are played at 25FPS to 15 subjects (aged between 20-30, all computer vision researchers) under a high-quality experimental organization – this organization is known as small-N design (Smith & Little (2018)), and has been widely used in SVA experiments (Geirhos et al.
Approval.
We have obtained the approvals of all subjects.
They all have signed an experimental statement (the template is provided by our IRB, which includes experiment purpose, procedure, risks and discomforts, costs, and conﬁdentiality (Appendix C.2)) before the experiment.
Process.
Previous works mainly use eye-tracker Hu et al.
(2022b)
; Xia et al.
or mouse Geirhos et al.
(2018; 2020; 2021) to record human visual ability.
To select a better measurement method, all subjects tried both mouse and eye-tracker, and they express that the mouse is better.
Besides, we have compared these two devices from practical and theoretical views in Appendix C.3, and ﬁnally select the mouse-based method.
Experiment includes four steps (Appendix C.4): (1) Subjects check instruments; then adjust the seat height, sitting posture, and the distance to the screen.
(2) 2 TEST videos occur sequentially in the screen center.
Subjects observe and remember the target features in the ﬁrst frame, then move the mouse to track the target in subsequent frames.
(3) 17 FORMAL videos belonging to different groups are played sequentially (a rest time occurs between two videos to relieve visual fatigue); subjects should concentrate on the target and maintain the mouse’s position.
(4) Subjects ﬁll in a questionnaire to self-evaluate their performance (Appendix C.5).
2. E VALUATION : M ETRICS
This work evaluates executors under the one-pass evaluation (OPE) mechanism, which initializes an executor in the ﬁrst frame and continuously records the tracking results.
For the t -th frame F t in a sequence s i =
{ F 1 , F 2 , . . .
, F t , . . .
} , we support the predicted result is p t , the ground-truth is g t , and their center points c p and c g are used to design indicators.
Note that target absent is regarded as an empty set ( i .
e ., g t = φ ) and excluded by the evaluation process.
To accurately evaluate executors’ performance, we divide the evaluation dimensions into three granularities, as shown in Figure 2.
[Table]
L1: Frame-level. Precision score (PRE) (Wu et al. in frame-level P L1 equals the center distance d c . Recently, normalized precision score (N-PRE)
(Hu et al.
(2022b)) is proposed to exclude the inﬂuence of target size and frame resolution.
Trackers with a predicted center outside the ground-truth will add a penalty item d cp ( i .
e ., the shortest distance between center point c p and the ground-truth edge).
For trackers whose center point falls into the ground-truth, the center distance d c (cid:48) equals the original precision d c ( i .
e ., d cp = 0 ). Besides, the maximum value in frame F t is used to normalize the result and generates the ﬁnal N-PRE score NP L1 , as shown in Equation 1. [Equation]
L2: Sequence-level.
Precision score in sequence-level P L2 ( θ d ) is deﬁned as the proportion of frames whose center distance d c ≤ θ d .
To illustrate the performance under different thresholds, previous works (Wu et al.
; Fan et al.
(2020)) usually draw the statistical results based on different θ d into a curve named precision plot . Since θ d = 20 is wildly used to rank trackers, we deﬁne P L2 = P L2 ( 20 ) in following experiments.
Similarly, draw statistical results based on different θ d (cid:48) into a curve generates the normalized precision plot .
However, we note that directly select a θ d (cid:48) to rank executors may introduce human factors, thus we use the proportion of frames
whose predicted center c p successfully fall in the ground-truth rectangle g t (Flag=Y in Figure 2 (L2)) as NP L2 .
The calculation is listed in Equation 2, where |·| is the cardinality.
[Equation]
L3: Group-level.
For a group of sequences G = { s 1 , s 2 , . . .
, s t , . . .
} , existing works usually use the mean value of all sequence-level results as the group-level evaluation, as illustrated in Equation 3. [Figure] [Figure] [Equation]
[Figure] It is worth noting that longer sequences challenge all executors in dynamic vision tasks – humans demand a higher concentration of attention [Figure] and algorithms should overcome the cumulative errors.
However, existing metrics all ignore the inﬂuence of the sequence length, causing us to propose new metrics for group-level evaluation [Figure] as illustrated in Equation [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure] [Equation] [Table]  E XPERIMENTS 3. A C OMPREHENSIVE C OMPARISON OF H UMAN - MACHINE D YNAMIC V ISION A BILITY
We ﬁrst interest in the comprehensive performance of human-machine DVA. Subject_Top and
Subject_Bottom represent the best and worst subjects, and Subject_Mean denotes the average. Figure 3 (I) shows the distribution of NP L2 in all sequences. As a sequence-level evaluation, box width represents stability, box position represents performance.
Figure 3 (II) illustrates the normalized precision plot weighted by sequence length and the NP wL3 (group-level evaluation) of all executors.
Clearly, humans score higher than most machines, indicating that the DVA of machines is still far from humans.
Besides, the boxplot distribution shows that machines’ performance is more unstable.
It is worth noting that the SOTA methods have been comparable to Subject_Bottom in most cases, indicating that the research routes do signiﬁcantly close the gap with humans.
As tasks become more complicated (from short-term tracking to global instance tracking), executors not only need to perceive moving targets, but also should have a good cognitive ability to quickly re-locate the target when its position suddenly changes.
Figure 8 illustrates the performance of human-machine DVA under various task constraints.
Based on experimental results, we ﬁnd that:
(1) A typical short-term tracking task only needs perceptron ability
Thus, both humans and most [Figure] MachineMachine R e l e a s e d Y e a r
R e l e a s e d
Y e a r 2022 Human
Machine R e l e a s e d
Y e a r
I
II
[Figure]
machines perform well in tracking the moving target; some SOTA machines even outperform humans in accuracy and stability.
For the long-term tracking task, since the sequence length increases and the target disappearance is allowed during the tracking process, executors should locate the target when it reappears in the frame.
Results demonstrate that humans can maintain high tracking precision, but the performance of most algorithms drops signiﬁcantly.
Global instance tracking, the most difﬁcult task that sequence length increases with shot-cuts and scene transitions, challenges almost all algorithms.
However, humans can still sustain efﬁcient and stable tracking, indicating that the cognitive ability of humans is much better than machines.
Speciﬁcally, the top-3 machines with various model structurescan help us to understand the performance of different modeling processes (Table 2).
MixFormer (Cui et al.
(2022)), a simple end-to-end model based on transformer structure, performs well in both short-term and long-time tracking tasks, [Figure] indicating that the Mixed Attention Module (MAM) and a straightforward detection head can provide powerful tracking ability.
The SiamRCNN (Voigtlaender et al.
model combines a two-stage scheme with a new trajectory-based dynamic planning algorithm, and uses re-detection to accomplish stable tracking.
KeepTrack (Mayer et al.
(2021)) model improved by SuperDiMP
(Danelljan et al.
(2020)) has an enhanced ability to discriminate interferers and perform well in short-time tracking, but lacking a mechanism for processing target mutation decreases its execution in longer sequences.
However, all these three machines have a performance gap with humans in the global tracking task.
Since MixFormer performs best in most cases, we select it as a representation to accomplish more detailed analyses in the following sections.
Cognition.
We further explore the effect of sequence length and shot-cuts, which requires the executor to have better cognition ability to re-locate the target in complex situations.
Figurein Appendix shows the performance variation of human-machine under different videos.
Clearly, the SOTA machine’s performance ﬂuctuates wildly when the shot-cut occurs, while the subject can still maintain a stable tracking ability.
In addition, machines may ultimately lose the target after frequent shot-cuts, but humans can quickly relocate the target in a new shot.
When the sequence length increases, the performance ﬂuctuation of machines will continue to increase, but humans are better than machines in terms of ﬂuctuation amplitude and tracking effect.
It reveals that when the target motion or apparent information is mutated, the performance of machines ﬂuctuates sharply, while humans can keep robust and precise target tracking based on their high cognition ability.
Perceptron.
Given that multiple challenging factors may coexist in a video, we show the performance of human and SOTA machine under different factor combinations in Figure 5.
Obviously, the purple rectangle is primarily related to the perceptive ability, and blue rectangle reﬂects the dynamic changes between consecutive frames.
We can ﬁnd that most cells in the blue rectangle are challenging for both algorithms and humans, but algorithms are slightly better than humans in some factors like fast motion and small targets.
Conversely, task constraints represented by the green rectangle have less impact on humans, but it powerfully inﬂuences machines.
This phenomenon is consistent with the above analysis of human-machine cognitive abilities.
Further analyses about humans.
Former analyses indicate that to err is human .
Thus, we provide detailed human performance analysis in Appendix D.6, which indicates that fast-moving targets and small targets are challenging to track, but tracking in a long sequence is not difﬁcult for humans.
The above results show that current research, which mainly focuses on improving tracking robustness under short-term tracking tasks, has progressed.
However, machines still have a gap with humans when task constraints are widened to a more general situation.
Some bad cases are illustrated in
Figure 14; more results based on different indicators are listed in Appendix D.3 to D.4. [Figure] M [Figure]
P [Figure] I MPROVEMENTS W HEN C OOPERATING WITH H UMANS
The analysis of Section 3.1 shows that humans and machines can adapt to different challenging factors, so an intuitive idea is what if we allow human-machine cooperation?
The cooperation mechanism is designed as follows: when machines fail to track, subjects can provide the current target position and restart machines from the failure frame.
In this cooperation pattern, machines are responsible for performing relatively simple but repetitive tasks, while humans are responsible for supplementing information at critical moments.
Speciﬁcal examples and detailed information are listed in Appendix C.7.
Figure 6 illustrates the performance of humans, machines, and human-machine collaborators.
Compared with machine itself, human-machine collaborator is greatly improved and even outperforms humans in most cases.
Subﬁgures of Figure 3 and Figure 24 indicate that human-machine collaborators have signiﬁcantly improved their performance based on the assistance of humans.
Therefore, humans and machines have superb possibilities for cooperation in dynamic vision tasks, which not only combines strengths and decreases weaknesses, but also saves human resources and improves efﬁciency.
For detailed results based on different indicators, please refer to Appendix E.1 to E.3.
C ONCLUSIONS AND F UTURE W ORK
In this paper, we aim to answer how far the gap is between humans and machines in dynamic vision tasks, and design an overall evaluation framework from three aspects – we choose 87 videos to construct the experimental environment, select 20 representative algorithms and 15 human subjects to form the executors, and quantify their DVA with a strict evaluation process.
Results indicate that SOTA machines are close to the lower limit of human subjects, meaning the DVA gap between humans and machines has signiﬁcantly closed .
Besides, humans and machines excel at different dynamic vision tasks, as humans can continuously track in general environments and longer sequences based on better cognitive ability, while SOTA machines can maintain efﬁcient and stable tracking ability in short-term tracking with excellent perceptual ability.
Finally, we also ﬁnd that the humanmachine cooperative executor can perform even better than humans, which provides a possibility of human-machine cooperation for application scenarios like self-driving systems.
As a preliminary research of the human-machine DVA, we have obtained interesting conclusions above, but some aspects still deserve further exploration. For example, human DVA is usually decoupled into different tasks – SOT concentrates on locating an arbitrarily moving object, MOT aims to track multiple known objects simultaneously, and VOD focuses on detecting all known objects. Obviously, SOT is the basis of other dynamic vision tasks, thus we use it as a representative task for research.
In the future, researchers can apply the proposed evaluation framework to multiple dynamic vision tasks and further explore the DVA of humans and machines. Besides, future work can select various subjects ( e . g ., different age, gender, and occupation (Burg & Hulbert (1961); Melcher & Lund (1992))) to compose a more comprehensive human baseline.
A T ASK D ESCRIPTION A. V ISION T ASK
[Table]
[Figure]
[Table]
[Table]
[Figure]
Image Classiﬁcation.
As a static vision task, image classiﬁcation (Nath et al.
(2014)) allows for classifying a given image as belonging to one of the labeled and pre-deﬁned categories. An excellent model should understand the target features and have a robust classiﬁcation ability in facing challenging factors like target deformation or image blur. Video Object Detection (VOD). VOD (Esteva et al.
(2021)), an essential task for multiple computer vision applications, desires to accurately determine the category and location of each target in videos.
However, the target category is typically limited to pre-deﬁned categories in the training dataset. Multi-object Tracking (MOT). MOT (Geuther et al. in research usually combines with detection.
MOT algorithms detect the object’s position in the ﬁrst frame, then calculate the similarity to determine instances belonging to the same target in consecutive frames. Thus, MOT is a modelspeciﬁc task and mainly focuses on tracking speciﬁc categories ( e . g ., pedestrians or vehicles). Single Object Tracking (SOT).
Unlike the above vision tasks, SOT (Wu et al.
(2015)) is categoryindependent , which means it intends to track a moving target without any assumption about the target category.
This characteristic allows SOT to be suitable for open-set testing with broad prospects.
Besides, some recent research proposes a new task named Global Instance Tracking (GIT)
(Hu et al. (2022b)) to cancel the continuous motion assumption. Thus, the SOT deﬁnition is additionally extended, making it a further step toward human dynamic vision tasks.
A. S INGLE O BJECT T RACKING T ASK
[Figure]
ABSENT Out of View #
ABSENT Out of View #
ABSENT Out of View #1652 # # # #5500 #3000 #
ABSENT Full Occlusion #
ABSENT Full Occlusion # # # # # # #
ABSENT Out of View #
ABSENT Out of View #
ABSENT Out of View #
ABSENT Out of View #
B D ATASETS AND M ACHINES B. D ATASETS
[Table]
We have correctly cited all benchmarks involved in the experiments and carefully checked the licenses involved in each benchmark (CC BY-NC-SA 4.0.).
All datasets used in the experiments are open-sourced, and our usage is under the license scope.
B. M ACHINES
Our algorithm evaluation experiments are performed on a server with 4 NVIDIA TITAN RTX GPUs and a 64 Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz.
We use the parameters provided by the original authors. B.2. C ORRELATION F ILTER B ASED T RACKERS KCF. The CF-based trackers represented by KCF (Henriques et al.
(2015)) incorporate high speed and tracking accuracy, thus becoming a representative tracking framework in the early stage. We use the Python 2.7 version based on link 1 . ECO. ECO (Danelljan et al.
(2017)) combines CNN with CF, aiming to use deep networks to improve feature representation. The ofﬁcial version is released in PyTracking 2 , which is a general python framework for SOT based on PyTorch. The feature representation of ECO is a combination of the ﬁrst and last convolutional layer in the VGG-m network (Chatﬁeld et al. (2014)), along with HOG (Dalal & Triggs (2005))
and Color Names (CN) (Van De Weijer et al. B.2. S IAMESE N EURAL N ETWORK B ASED T RACKERS SiamFC. As the originator of SNN-based trackers, SiamFC (Bertinetto et al.
3 achieves satisfactory tracking performance by matching features between the template region and the search region through a simple network structure.
The backbone of SiamFC is AlexNet (Krizhevsky et al.
(2017)), which is trained on the ILSVRC15 4 dataset for object detection in video with 50 epochs.
SiamRPN.
SiamRPN (Li et al.
(2018b))
5 introduces the region proposal network (Girshick (2015)) to achieve accurate target regression.
This model is trained based on image pairs from ImageNet-VID (Russakovsky et al.
and Youtube-BB (Real et al.
with 50 epochs.
DaSiamRPN.
DaSiamRPN (Zhu et al.
6 uses data augmentation to enhance the discriminative ability, which is trained based on ImageNet-VID (Russakovsky et al. , Youtube-BB (Real et al. (2017)), ImageNet (Deng et al. and COCO (Lin et al.
with 50 epochs.
SPLT. SPLT (Yan et al.
7 designs a veriﬁer to switch global search and local search. SPLT uses MobileNet (Howard et al. for feature extractor and downsamples the spatial resolution of the https:
The are initialized with the ImageNet al -trained models and then tuned SiamRPN++. SiamRPN++ (Li et al.
(2018a)) 8 introduces deeper backbone and selects ResNet (He et al. for feature extraction. The backbone network is pre-trained on ImageNet (Deng et al.
(2009)) for image labeling. Authors train the SiamRPN++ on the training sets of Youtube-BB (Real et al. ImageNet-VID (Russakovsky et al. and COCO (Lin et al.
and to learn a generic notion of how to measure the similarities between general objects for visual tracking.
SiamDW.
SiamDW (Zhang & Peng (2019)) 9 selects deeper and wider backbones for feature extraction.
The backbone network is pre-trained on ImageNet (Deng et al.
for image labeling.
Authors train the proposed cropping inside residual (CIR) units with SiamRPN on Youtube-BB (Real et al.
and ImageNet-VID (Russakovsky et al. (2015)).
Except for backbone improvements, some SNN-based algorithms refer to detection methods to improve the network architecture.
SiamFC++. SiamFC++ (Xu et al.
10 employs an anchor-free structure (Tian et al.
to eliminate the dependence on anchors.
It adopts ImageNet-VID (Russakovsky et al. , COCO (Lin et al. , Youtube-BB (Real et al. , LaSOT (Fan et al.
and GOT-10k (Huang et al.
as basic training set.
Ocean.
Ocean (Zhang & Peng (2020)) 11 also employs the anchor-free structure.
The backbone network is initialized with the parameters pretrained on ImageNet (Deng et al.
The Ocean tracker is trained on the datasets of Youtube-BB (Real et al.
ImageNet-VID (Russakovsky et al.
, GOT-10k (Huang et al. and COCO (Lin et al. SiamCAR. As another anchor-free tracker, SiamCAR (Guo et al.
12 is trained on COCO (Lin et al.
(2014)), ImageNet (Deng et al.
ImageNet-VID (Russakovsky et al.
and Youtube-BB (Real et al. SiamRCNN. SiamRCNN (Voigtlaender et al.
13 utilizes re-detection mechanism and tracklet dynamic programming algorithm to process object disappearance.
SiamRCNN is built upon the
FasterRCNN (Ren et al. (2015)) implementation with a ResNet-101-FPN backbone.
The backbone has been pre-trained on COCO (Lin et al.
and the overall model is trained on ImageNet-VID (Russakovsky et al. (2015)), YouTube-VOS (Xu et al. (2018)), GOT-10k (Huang et al. and
LaSOT (Fan et al.
B.2. C OMBINE C ORRELATION F ILTER AND S IAMESE N EURAL N ETWORK
ATOM. ATOM (Danelljan et al.
14 tries to combine CF and SNN, and proposes a new framework to use the advantages of ofﬂine training and online updating.
The authors use the training splits of LaSOT (Fan et al.
and TrackingNet (Müller et al.
and augment the training data with synthetic image pairs from COCO (Lin et al.
DiMP.
Based on framework proposed by ATOM, DiMP (Bhat et al.
15 optimizes the loss function for stronger discriminative ability.
The backbone network is initialized with the ImageNet (
Deng et al.
(2009)) weights. DIMP is trained on TrackingNet (Müller et al. , LaSOT (Fan et al. (2020)), GOT-10k (Huang et al.
and COCO (Lin et al.
datasets with 50 epochs by sampling 20,000 videos per epoch PrDiMP and SuperDiMP. PrDiMP and SuperDiMP (Danelljan et al. 16 use probabilistic regression to improve the accuracy.
The training splits of the LaSOT (Fan et al. (2020)), GOT-10k (Huang et al. (2021)), TrackingNet (Müller et al.
and COCO (Lin et al.
(2014)) are used, running 50 epochs with 1000 iterations each.
KeepTrack. KeepTrack (Mayer et al.
17 combines the SuperDiMP with a target candidate association network to accomplish robust tracking.
The authors retrain the target candidate association network on hard sequences mined from LaSOT (Fan et al.
Some other works design custom networks to solve speciﬁc problems like target absence or similar instance interference, naturally demonstrating a development from perceptual to cognitive intelligence. GlobalTrack. GlobalTrack
(Huang et al.
18 does not assume motion consistency and performs a full-image search to eliminate cumulative error.
The authors use COCO (Lin et al. (2014)), GOT-10k (Huang et al. and LaSOT (Fan et al. for model training.
KYS.
KYS (Bhat et al. 19 represents scene information as state vectors and combines them with the appearance model to locate the object. The authors use the training splits of TrackingNet (Müller et al. (2018)), GOT-10k (Huang et al.
and LaSOT (Fan et al.
for model training.
MixFormer. MixFormer (Cui et al.
20 designs an end-to-end transformer-based framework to simultaneously accomplish feature extraction and target information integration. The authors use
COCO (Lin et al.
TrackingNet (Müller et al. (2018)), GOT-10k (Huang et al.
and LaSOT (Fan et al.
for model training.
C E XPERIMENT O RGANIZATION C. E XPERIMENTAL E NVIRONMENT
[Figure]
[Figure]
N # # # # # # # #
O # # # # # # # #
P # # # # # # # #
Q # # # # # # # #
C. E XPERIMENTAL S TATEMENT
We have obtained the approvals of all experiments, the template is provided by our IRB.
Every experiment has signed an experimental statement before the experiment.
Here we list the contents:
Purpose You have been asked to participate in a research study that studies the dynamic visual abilities of humans.
We would like your permission to enroll you as a participant in this research study.
The instruments involved in the experiment are a computer screen and a mouse.
The experimental task consisted of gazing at the presented image sequence and manipulating the mouse to point as consistently as possible at the center of the moving target.
You will be given speciﬁc instructions for the task before it begins.
Procedure In this study, you should read the experimental instructions and ensure that you understand the experimental content.
The whole experiment process lasts about one hour, and the experiment is divided into the following steps:
• Read and sign the experimental statement; • Test the experimental instrument, and adjust the seat height, sitting posture, and the distance between your eyes and the screen.
Please ensure that you are in a comfortable sitting position during the experiment; • Two TEST videos will be played.
You should comprehend the speciﬁc instrument operation rules and be familiar with the experimental process through the test videos; • Start the FORMAL experiment.
Please follow the operation to watch the test sequence and complete the relevant operations.
Note that after watching each video sequence, you should have a rest; • After the experiment, you need to ﬁll in a questionnaire.
Risks and Discomforts
The only potential risk factor for this experiment is trace electron radiation from the computer.
Relevant studies have shown that radiation from computers and related peripherals will not cause harm to the human body.
To rule out the impact of COVID-19 on the experiment, all participants completed nucleic acid tests before the experiment to ensure their health.
Everyone needs to wear a mask throughout the experiment.
In addition, the relevant devices are cleaned and disinfected during the experiment.
Costs Each participant who completes the experiment will be paid 200 RMB.
Conﬁdentiality The results of this study may be published in an academic journal/book or used for teaching purposes.
However, your name or other identiﬁers will not be used in any publication or teaching materials without your speciﬁc permission.
In addition, if photographs, audio tapes or videotapes were taken during the study that would identify you, then you must give special permission for their use.
I conﬁrm that the purpose of the research, the study procedures and the possible risks and discomforts as well as potential beneﬁts that I may experience have been explained to me.
All my questions have been satisfactorily answered.
I have read this consent form.
My signature below indicates my willingness to participate in this study.
C. D EVICE S ELECTION
How to measure the DVA of humans is an important section of our work.
Previous works mainly use eye-tracker (Hu et al.
(2022b)
; Xia et al.
(2021)) or mouse (Geirhos et al.
(2018; 2020; 2021)) to record human visual ability.
To select a better measurement method we divide them into active (mouse) and passive (eye-tracker) methods, then compare them from practical and theoretical views:
• From a practical view, we have organized all subjects to experience both mouse and eyetracker, and they all express that the user experience of the mouse is better.
A possible reason is that the eye-tracker passively records eye movements, causing the effect to be limited by many factors ( e .
g ., the device accuracy, the user’s posture, the relative distance between eyes and the eye-tracker, and the user’s glasses).
Besides, many subjects mention that the eye-tracker can only effectively track eye movements for a short period.
When the experimental time is prolonged, the recording result easily shifts, while subjects can only adjust their posture rather than actively correct the capture result.
Conversely, using a mouse can minimize the above problems.
Although mouse may also have some limitations, its accuracy is greater than eye-tracker in our experiments.
For example, when subjects notice that the mouse position is far from the target, they can actively move the mouse and immediately correct the record result.
• From a theoretical view, some researchers (Holmqvist et al.
(2022)) also indicate that the eye-tracker is affected by the tracking ratio ( i .
e ., the amount of eye-tracking data lost).
Blinks or brief vision drifts can cause a lower tracking ratio, which requires the subject to keep a high degree of concentration when using eye-trackers.
To prove this view, we check recent articles and ﬁnd that experiments based on eye-trackers are either for observing the images (subjects have sufﬁcient time to adjust the position per image)
(Xia et al.
or watching a small amount of video ( e .
g ., 6 videos)
(Hu et al.
(2022b)).
Nevertheless, our experiment
requests each participant to watch 17 videos (including long videos).
Thus, while maintaining a high concentration of attention, subjects frequently change their sitting position unconsciously, causing the eye-tracker performs poorly in our experiment.
What’s more, we ﬁnd that previous works (Geirhos et al.
(2018; 2020; 2021)) have allowed subjects to perform image classiﬁcation tasks with a mouse-based method, and a neuro training company named Reﬂexion also uses touch screens to provide DVA training for athletes, which means recording the observation position based on hand operation has been applied in academia and industry.
Thus, we ﬁnally select the mouse-based experimental method.
C.
E XPERIMENT P ROCESS
I
[Figure]
C. Q UESTIONNAIRE
Subjects should ﬁll in a questionnaire to evaluate their performance and make recommendations.
Here we list the contents:
This questionnaire aims to investigate subjects’ opinions and self-evaluation.
Q1.
Your name: Q2.
Your ID (e.g., subject 1): Q3.
Your age: Q4.
Your vision condition (ﬁll in the left and right eyes separately): Q5.
Please rank the following factors in order of perceived difﬁculty during your experiment (order from difﬁcult to easy): • Video sequence is too long; • Frequent shot changes; • Screen shaking; • Target scale is too small; • Similar objects interference; • Target moves fast; • Target is occluded.
Q6.
In addition to the challenging factors in the previous question, what other factors may inﬂuence your observation?
Q7.
Please provide your self-evaluation score (0 to 100) in short sequences (1-3 minutes): Q8.
Please provide your self-evaluation score (0 to 100) in medium sequences (3-10 minutes): Q9.
Please provide your self-evaluation score (0 to 100) in long sequences (more than 10 minutes): Q10.
What was the most impressive sequence in the experiment?
Why are you impressed by it?
(e.g., the video is too long, the content of the video is exceptional, the video has some challenging factors that lead to poor performance, etc.)
Q11.
Do you have any suggestions or comments for this experiment?
C. G ROUP I NFORMATION
[Table]
M M
M M
B B
B B
B1 G
G G
G G
N N
N
N N
A A
A A
A1 H
H H
H H
O O
O O
O
H
H H
H H
I I
I I
I
P P
P
P P
I I
I I
I4 J
J J
J J
Q Q
Q Q
Q J J
J J
J1 K
K K
K K
A
A A
A A
K K
K K
K1 L
L L
L L
B
B B
B B
L L
L L
L3 M
M M
M M
C C
C
C C
M M
M M
M3 N
N N
N N
D D
D
D D
N N
N N
N2 O
O O
O O
E E
E E
E
O O
O O
O3 P
P P
P P
F F
F
F F
P P
P P
P
Video
Q
Q Q
Q Q
G G
G G
G
Q Q
Q Q
Q ( to long, then become short again.
Subjects in group II watch videos of increasing length.
Subjects in group III watch videos of decreasing length.
Group
I
II
III Exp              15             120             2000         17592 18166 22872 25379 28828         8750 10000 5000             5307             1000             1266 8750 10000           1000 17592 18166 22872 25379         610             528             645             531             551             671             550             571             581    8750
Frame     17592 18166 22872
561 Total 50052 50165 51038 50710 50970 50052 50165 51038 50710 50970 50052 50165 51038
C. C OOPERATION M ECHANISM C.7. D ETAILED I NFORMATION ABOUT C OOPERATION M ECHANISM
degree t represents the blur degree value
F t higher
R r r t , . . .
} the set of object relative scale r t represents the object relative scale value
F
Output: P = { p 1 , p 2 , . . p t , . . .
} the set of tracking results, p t ( x pt , y pt , w pt , h pt ) the predicted bounding : find suitable cooperation frames */ 1 set candidate cooperation frames C = φ for i ← 0 to |S| − 1 do while ( b i > MEDIAN ( B
) ∧ ( r i
MEDIAN do
C ← C ∪ i /*
Step
: human-machine cooperation */ 4 set tracking results P = φ set initialization locations I = φ set failure counter α =for i ← 1 to |S| do if i = = 1 then p 1 =
INITIALIZATION ( g 1 ) P ← P ∪ pcontinue if α
∧ i ∈ ) then
P ← As we have metioned in Section 3.2, the cooperation mechanism is designed as follows: when machines fail to track, subjects can provide the current target position and restart machines from the failure frame.
It is worth noting that human-computer cooperation is very complex, and our experiments only explore the possibility of cooperation through a simple and intuitive mechanism. There is still much space for further optimization.
Based on Algorithm 1, the human-machine cooperation is divided into the following steps.
(1) Find suitable cooperation frames. The quality of the cooperation frame is important for tracking in the following frames. Frames with tiny objects or motion blur may decrease appearance and motion information. Thus, we ﬁrst calculate the blur degree and relative target scale of all frames in S , and select frames that include the target with relatively clear and appropriate scale as candidate cooperation frames (please refer to Hu et al.
(2022a) for more information), and generate the set of candidate cooperation position C .
(2) Human-machine cooperation.
We use the IoU value s i (Line 10 in Algorithm 1) to determine tracking failure and consider that after ten consecutive frames of tracking failure (failure counter α ≥ 10 ), the algorithm will cooperate with the human at the nearest candidate cooperation frame and reinitialize (Line 8 in Algorithm 1, g ci represents the object information provided by human).
Besides, considering that many SOTA machines may re-locate the target in several following frames, we assume that s i ≥ 0 .
5 means successfully re-locate the target, and zeroing the failure counter α if the re-location occurs in 10 frames (Line 13 in Algorithm 1).
It is worth noting that our human-machine cooperation experiments are a preliminary exploration, and we hope to ﬁnd that cooperation is possible for all algorithms.
However, for a combination ofalgorithms and 87 videos (1740 sets), it is time-consuming to arrange human subjects to participate in 1740 cooperation experiments.
Therefore, we used the following strategy to organize the experiments.
(1) We ﬁrst use the ground-truth as target information provided to the algorithm at cooperation frames (Line 8 in Algorithm 1, here we use g i to re-initialize the algorithm).
Since the ground-truth of sequences are provided by professional human annotators, which can be regarded as a representative of the highest level of human in static frames.
Then we record all the cooperation frames as I g ( g means this set is generated by ground-truth information).
(2) We show the frames in I g to human subjects, asking them to ﬁnd the target in the current frame and annotate it with a bounding box, then we generate the human annotations
I h (the length of I h is equal to I g ).
(3) We evaluate I h
and I g based on NP L2 score.
Situation 1: If the score is higher than 0.95, we consider the human subject to have the same performance as the original data annotator.
Thus, we no longer organize the cooperation experiment for this algorithm-sequence combination.
Situation 2: On the contrary, if the score is below 0.95, we consider that the human subject has a gap with the ground-truth.
We will arrange for the human subject and the algorithm to track the sequence together from the beginning, ask the human subject to annotate the target position by bounding-box in cooperation frames, and then record the collaborator’s performance (Line 8 in Algorithm 1, here we use g ci to re-initialize the algorithm).
We ﬁnd that most sequences satisﬁed Situation 1.
As the example in Figure 12, the human subject watches the ﬁrst frame and clearly understands that he should locate the blue-clothed player.
Then he can identify the target at frames #1125 and #6353, even if he does not watch the middle frames.
This phenomenon also demonstrates the solid cognitive and memory abilities of humans. [Figure] #0001 INITIALIZE #0001 INITIALIZE ABSENT #
ABSENT #4461# #5149# #
ABSENT #
ABSENT #6353# #6558#6558 #1125#1125 SHOT
[Figure] [Figure]
#6353 INITIALIZE #6353 INITIALIZE #1125 INITIALIZE #1125 INITIALIZE #6558#6558 #
ABSENT #
ABSENT #5149#5149 #4461#4461 #2753#2753 #2376#2376 #1124#1124 #0001 INITIALIZE #
[Figure]
ABSENTABSENT #
ABSENT
[Figure]
Restart
[Figure]
[Figure]
ABSENT # # #
ABSENT # #6558 #
[Figure]
#6353 INITIALIZE #1125 INITIALIZE #6558 #
ABSENT #5149 #4461 #2753 #2376 #
[Figure]
[Figure]
ABSENT [Figure] To better illustrate the cooperation process, we use video G4 (a long video with 25,397 frames andshot-cuts, describing a table tennis match with two players) as an example. The best human subject scores 0.964 ( NP L2 score) since it is easy to locate a player consistently.
However, when the shot is switched, the position variation of the target player (blue-clothed player) will challenge the SOTA algorithm. Thus, it quickly drifts to the interferer (black-clothed player) and keeps wrong tracking, causing the score to decrease to 0.558. But if we allow algorithms to cooperate with humans, when the human subject ﬁnds that the algorithm fails (such as drifting to the distractor), the subject will provide correct target position to the algorithm to avoid persistent mistracking. For our experiment in
G4, the human subject provides target location information 23 times while the algorithm tracks the remaining 25,374 frames.
Although humans only need to participate 0.1% (23/25,397) in cooperation, the human-machine collaborator MixFormer_Human can score 0.977, indicating that cooperation can signiﬁcantly improve the algorithm’s tracking performance.
This result is also in line with the original intention: algorithms accomplish many simple tasks, while humans focus on the key points.
D P ERFORMANCE OF H UMANS AND M ACHINES D. R ESULTS ON D IFFERENT S EQUENCE L ENGTHS
[Figure]
Subﬁgures (I-IV) represent four videos with different combinations of lengths and shots.
Compared with the SOTA machine, humans can still maintain adequate DVA in the face of lengthy sequences and frequent shot switching.
D.
E XAMPLES ON D IFFERENT S EQUENCES
D G O
P \textbf{
P G
E M B
MachineMachineMachine
HumanHumanHuman
Machine
Human
MachineMachine
HumanHuman
Machine
Human
MachineMachine
HumanHuman
Machine
Human
Condition [Figure] Obviously, humans outperform machines in longer sequences with shot-cuts and can better distinguish the target from interfering objects (condition I).
However, when facing challenges such as tracking small targets with fast movements (condition II), limited by hand-brain coordination and mouse movement speed, humans cannot precisely locate such targets at 25FPS.
Note that both humans and machines may fail when multiple challenges are superimposed in one sequence (condition III).
D. B OXPLOTS
PP
QQ
OOO LL
MM
NN KK
JJ FF
GG
HH
II
EE AA
BB
CC
DD
TT
[Figure] [Figure]
D.
A NALYSES OF H UMAN D YNAMIC V ISUAL
A BILITY
I   4
I       4  4 III   4 III   4  4 V   4 V   4  4 II   4 II   4  4 IV   4 IV
VIVI
We request three subjects to track the same videos in different viewing orders.
Figure {+
+} (I-V) demonstrates that subjects’ performance is independent of viewing order – watching long sequences ﬁrst or last has no signiﬁcant difference.
However, we notice that subjects may also make mistakes, like the performance of Exp09 on A4 is signiﬁcantly lower than others.
A4 is a short video about a basketball game, while Exp09 misadmits the target player and keeps tracking a distractor.
It indicates that humans are not absolutely infallible – they may make low-level mistakes due to carelessness.
Figure 23 (VI) shows the self-perceptions of challenging factors in the questionnaire.
Most subjects consider that fast-moving targets and small targets are challenging to track, but tracking in a long sequence is not difﬁcult – this is consistent with previous experimental conclusions in this paper.
E P ERFORMANCE OF H UMAN -M ACHINE C OOPERATIONS E. B OXPLOTS
[Figure]
II
JJ
KK
NN LL
MM
OO
PP
QQ
HH
AA
BB
TT
CC
EE DD
GG FF
[Figure]