Learning Graph-embedded Key-event Back-tracing for Object Tracking in Event Clouds
Abstract
Event data-based object tracking is attracting attention increasingly.
Unfortunately, the unusual data structure caused by the unique sensing mechanism poses great challenges in designing downstream algorithms.
To tackle such challenges, existing methods usually re-organize raw event data (or event clouds) with the event frame/image representation to adapt to mature RGB data-based tracking paradigms, which compromises the high temporal resolution and sparse characteristics.
By contrast, we advocate developing new designs/techniques tailored to the special data structure to realize object tracking.
To this end, we make the ﬁrst attempt to construct a new end-to-end learning-based paradigm that directly consumes event clouds.
Speciﬁcally, to process a non-uniformly distributed largescale event cloud efﬁciently, we propose a simple yet effective density-insensitive downsampling strategy to sample a subset called key-events.
Then, we employ a graph-based network to embed the irregular spatio-temporal information of keyevents into a high-dimensional feature space, and the resulting embeddings are utilized to predict their target likelihoods via semantic-driven Siamese-matching.
Besides, we also propose motion-aware target likelihood prediction, which learns the motion ﬂow to back-trace the potential initial positions of key-events and measures them with the previous proposal.
Finally, we obtain the bounding box by adaptively fusing the two intermediate ones separately regressed from the weighted embeddings of key-events by the two types of predicted target likelihoods.
Extensive experiments on both synthetic and real event datasets demonstrate the superiority of the proposed framework over state-of-the-art methods in terms of both the tracking accuracy and speed.
The code is publicly available at https://github.com/ZHU-Zhiyu/Event-tracking
Introduction
Event cameras, the bio-inspired silicon retinas, have been widely adopted in computer vision and robotics communities [57, 37, 18, 31, 28, 47].
Different from traditional RGB cameras, which synchronously record the light intensity of each pixel, event cameras [13] only sense the changes in dynamic scenes, i.e., if an event senor captures an intensity increment (resp.
decrement)
larger than a threshold, it would trigger an event, composed of the corresponding pixel location, the timestamp, and the polarity indicating brightness increase or decrease.
With the concise information ﬂow, event cameras enjoy numerous advantages, such as high temporal resolution, low latency, high dynamic range, and low power consumption.
Moreover, recent works show that even without being associated with RGB images, only event data could also realize various vision tasks, such as depth estimation [14], semantic segmentation [1], and object detection [32].
Notably, owing to the above-mentioned advantages, event cameras are desired to be a promising object tracking solution [39, 18, 50], aiming to estimate the position of a speciﬁed target object.
However, the unusual data structure of the raw event data poses great challenges in designing tracking algorithms.
Facing such challenges, the existing non-learning-based methods usually build their models based on various representations, such as event frame [40, 17, 29], time surface [33, 55, 7], and event clouds [2, 32, 45].
In view of the great success of deep learning techniques in various ﬁelds, especially RGB data-based object tracking, recently several deep learning-based methods for event data-based object tracking have been naturally proposed [51, 47].
Generally, they quantize and accumulate raw event data along the temporal dimension into event frames/images to adapt to mature RGB data-based object tracking paradigms.
However, such a manner drops the high temporal resolution and sparse characteristics of raw event data to some extent, thus compromising tracking accuracy.
Instead of following the research line of the existing learning-based methods, we are interested in developing a new end-to-end learning-based paradigm that directly consumes raw event data (or event clouds), in which new algorithms/modules considering the unique features of event clouds are involved.
Speciﬁcally, to cope with the irregular structure, we employ a graph-based neural network to map an event cloud into a high-dimensional feature space to embed rich spatio-temporal semantic information.
Meanwhile, to process a large-scale event cloud efﬁciently and being aware of its non-uniform distribution characteristic, we introduce a simple yet effective density-insensitive downsampling strategy to sample a limited number of key-events, on which the feature embedding process is performed.
We then link the embeddings of template and search event clouds via Siamesematching to predict semantic-aware target likelihoods of search key-events.
Besides, observing the high temporal resolution characteristic of event data naturally preserve motion trajectories, we also propose to back-trace the possible initial positions of the search key-event and measure them with the previous proposal to predict their target likelihoods.
Finally, we obtain the bounding box by adaptively fusing the two intermediate ones separately regressed from the weighted embeddings of key-events by the two types of predicted target likelihoods.
We conduct extensive experiments on both real and synthetic event data to demonstrate the superiority of our framework over state-of-the-art methods in terms of both the tracking accuracy and speed , and comprehensive ablation studies to validate the effectiveness of the proposed modules.
In summary, the main contributions of this paper are two-fold:
• we propose the ﬁrst end-to-end learning-based object tracking paradigm directly handling raw event data (or event clouds), featured with carefully-designed modules, e
g., key-event embedding and motion-aware target likelihood prediction, to take advantage of the unique characteristics of event data; and • The superiority performance of our paradigm certiﬁes the great potential of designing new algorithms tailored to the special characteristics of event data to achieve object tracking.
Besides, we will release our framework built from scratch to contribute to this community.
Related Work
Deep learning on event data .
As a kind of asynchronous bio-inspired vision sensors, event cameras [13] have shown remarkable performance in various scenarios, such as quick motion and low light environments, representing a paradigm shift in visual information collection [9].
Deep learningbased frameworks for modeling event data have been proposed.
For example, Phased LSTM [36] extends the LSTM unit by adding a new time gate for classiﬁcation on event data.
EVDL [31] proposes an event-frame representation and constructs a CNN model for prediction of a vehicle’s steering angle.
Then, EV-SegNet [1] introduces the ﬁrst baseline for semantic segmentation with event data.
The emergence of MVSEC [57] dataset has aroused wide interest in the research of optical ﬂow [56, 58, 16] and depth estimation [58, 20].
Unlike the previous works, Rebecq et al.
[41, 42] used simulated events to train a convolutional recurrent neural network for image reconstruction.
RGB data-based object tracking .
Recent methods could be roughly divided into two categories: spatio-only -based and spatio-temporal -based.
Most of ofﬂine Siamese-based tracking methods [3, 26, 25] fall into the ﬁrst category.
For example, SiamFC [3] enhances the tracking accuracy by ﬁrst introducing a correlation layer as a fusion tensor.
SiamRPN [26] employs a single-stage
RPN [43] detector to redetect the template by comparing its features to that of the current frame.
SiamRPN++ [25] uses a spatial-aware sampling strategy to remove the inﬂuence factors, such as padding, and introduce ResNet [19] into the Siamese network-based visual trackers.
However, the spatio-only methods are difﬁcult to deal with fast-moving or similar objects in the scene.
Compared with spatio-only trackers, spatio-temporal -based methods [35, 23, 46, 49] additionally exploit temporal information to improve trackers’ robustness.
For example, MDNet [35] proposes a
CNN-based multi-domain learning framework, which separates domain-independent information from domain-speciﬁc one, to capture shared representations.
RT-MDNet [23] improves MDNet by employing the RoIAlign technique to extract more accurate representations of targets and candidates from a feature map.
Recently, Wang et al.
[46] converted a general object detector to achieve domain adaption and efﬁcient online update.
However, these methods are either difﬁcult to balance performance and speed or cannot handle real-time scenarios and low light scenes well.
Although various advanced techniques have been proposed for boosting the performance of RGB data-based object tracking, the essentially different data modality requires additional efforts to extend these techniques to event data-based object tracking.
Event data-based object tracking has been receiving increasing attention recently, and the proposed methods could be broadly divided into two categories: optimization-based and learning-based.
For the optimization-based category, as the pioneering work, Mitrokhin et al.
[32] proposed to approximate the 3D geometry of the event stream without feature tracking or explicit optical ﬂow computation.
Bryner et al.
[5] presented a method to track the 6-DOF pose of an event camera in a known environment described by a photometric 3D map.
Ramesh et al .
[39] presented the ﬁrst learning-based long-term object tracking algorithm for event cameras, based on a tracking-learning-detection framework.
Then, Li et al.
[27] utilized the features by VGG-Net-16 to represent the appearance of the event-stream object to achieve tracking.
Recently,
Chae et al.
[6] applied a Siamese-matching paradigm [3] to the event-based object tracking task for learning edge-aware similarity.
However, the methods in [27, 6] mainly focus on applying well-developed RGB data-based tracking techniques to event data without well taking the special characteristics of event data into account.
Proposed Method
Notation .
Denote by e :=
[ x, y, t, p ] ∈ R 1 × 4 an event, where x and y are the spatial coordinates on the image plane, whose values are normalized to [0 , 1] , t is the timestamp, and p is the 1-bit polarity, indicating brightness increase or decrease.
Denote by E ∈ R N × 4 an event cloud, where N events in a short period of time [ t 0 , t 0 + ∆ t ] are randomly stacked in row-wise.
Accordingly, for events in a long period of time, we represent them with an event cloud sequence of U non-overlapping event clouds, i.e., (cid:8) E i ∈ R N i × 4 (cid:9) U i =1 .
Let E t be a template event cloud, and B 0 ∈ R 1 × 4 be the annotated bounding box indicating the 2-D position of the object captured by E t ∈ R N t × 4 , which is parameterized by the 2-D spatial locations and the corresponding size (i.e., width and height).
The objective of event-based object tracking is to predict the bounding box of a target object described by
B 0 and E t from each of a given search event cloud sequence { E is ∈ R N si × 4 } U s i =1 .
Overview .
In contrast to existing learning-based methods usually re-organizing raw event data with the frame-based representation, which compromises the high temporal resolution characteristic to some extent and thus limits tracking performance, we directly deal with raw event data in its intrinsic 3-D spatio-temporal space.
Speciﬁcally, as shown in Fig.
1, the proposed framework processes the event clouds of { E is } U s i =1 sequentially via the following three main steps:
Graph-based Key-event Embedding .
This module aims to project an event cloud into a highdimensional feature space to embed rich spatio-temporal semantic information (see Sec. 3.1).
However , the irregular structure underlying event clouds , caused by the sparse and asynchronous sensing manner, makes it intractable to learn the embeddings efﬁciently and effectively.
To this end, we employ a GNN, which has demonstrated the effectiveness in modeling irregularly distributed signals, to embed input raw event data.
Moreover, to process large-scale event clouds efﬁciently and being aware of their noisy and highly non-uniform characteristics, we propose a simple yet effective downsampling strategy, capable of sampling a few key-events evenly and robustly from an input [Figure] 𝑡 𝑡𝑒𝑚𝑝𝑙𝑒𝑡 𝑡 𝑠𝑒𝑎𝑟𝑐ℎ [Figure]
event cloud. Accordingly, we perform the tracking process on key-events instead of the whole event cloud.
Semantic and Motion-aware Target Likelihood Prediction .
This module, a dual-path architecture, aims to predict the possibility of a key-event of the search event cloud belonging to the target object, i.e., the target likelihood .
Basically, from the visual recognition perspective, we propose a semanticdriven Siamese-matching module, in which the similarity of each search key-event to the template key-events is evaluated, based on their embeddings (see Sec. 3.2).
Besides, observing that the event cloud inherently preserves the motion trajectories of events, we also propose a motion-aware target likelihood prediction module, in which the initial position of each key-event is back-traced and compared with the previous proposal (see Sec. 3.3).
We expect these two paths designed from different perspectives will compensate each other for robust and accurate target likelihood prediction.
Conﬁdence-based Object Proposal .
In this module, we ﬁrst regress two intermediate bounding boxes separately from the embeddings of key-events weighted by the two types of predicted target likelihoods and then fuse them adaptively under the guidance of the learned conﬁdence scores, generating the ﬁnal bounding box (see Sec. 3.4).
3.1 Graph-based Key-event Embedding
We ﬁrst sample a limited number of key-events via a simple yet effective density-insensitive downsampling strategy, then embed them into a high-dimensional feature space via a GNN.
Key-event sampling .
As shown in Fig.
2 (a), to sample n k key-events from a typical event cloud, we ﬁrst deﬁne a 2D regular grid that contains n k grid points and has the same scale as the event cloud in the spatial domain.
Then, for each of the grid points, we ﬁnd the spatially closest event in the Euclidean distance sense, which will be thought of as a key-event.
Such a simple downsampling strategy could spread the receptive ﬁeld of the neural network in a more spatially uniform manner.
We demonstrate its superiority against widely-used random sampling [21] in Sec. 4.3.
GNN-based spatio-temporal embedding .
We utilize the graph convolutional layers to encode the irregular 3-D spatio-temporal information of an event.
Speciﬁcally, as shown in Fig.
2 (a), for each key-event, we ﬁrst ﬁgure out its nearest neighbours, then learn the high-dimensional spatio-temporal embeddings of such a set of events via feeding the explicitly encoding 2 of event clouds to multi-layer perceptrons (MLPs).
Meanwhile, adaptive pooling weights are also learned to aggregate neighbouring information in an unordered manner, ﬁnally producing embeddings F t ∈ R n t × C and F s ∈ R n s × C for E t and E is containing n t and n s key-events, respectively, where C is the number of channels.
We refer readers to Appendix for more details. 𝑥 [Figure] 𝑡
[Figure]
[Figure]
related [Figure] Trajectory Sample
[Figure]
3.2 Target Likelihood Prediction via Semantic-driven Siamese-matching
Let p jsem ∈
[0 , 1] be the target likelihood of the j -th search key-event, and P sem := [ p 1 sem ; · · · ; p jsem ; · · · ; p n s sem ] ∈ R n s × 1 .
Encouraged by the learnable combination pattern of transformer, which has shown its great potential in sparse point cloud processing [52], we ﬁrst apply swin-transformer [30] on the embeddings F t and F s , which respectively encode the semantics of the key-events of E t and E is , to equip them with abstract and distinguishable information, generating
S t ∈ R n t × C o and S s ∈ R n s × C
o ( C o > C ) .
To link E t and E is , we then design a cross-attention layer, which adaptively projects S t onto S s :
[Equation]
where the associated matrix M a ∈ R n t × n s is learned via another GNN G ( · , · ) , and S ts ∈ R n t × C o denotes the projected embeddings blending the information of both E t and E is .
Finally, we feed the concatenation of S ts and S s into another MLP to regress P sem .
We refer readers to Appendix for the detailed architecture.
3.3 Target Likelihood Prediction via Motion Trajectory Back-tracking
In this module, we leverage the inherent motion trajectories of event data [58, 34, 40] to predict the target likelihood of each key-event of E is .
The basic idea is linking the current search key-events to the previous target proposal to distinguish them.
Speciﬁcally, we ﬁrst back-trace the initial position of each key-event from its motion trajectory in 3D spatio-temporal space.
Considering that trajectories are usually continuous, the former event points should have relatively smaller moving offsets.
To explicitly model such a characteristic, we apply MLPs to the embeddings of search key-events to learn their motion velocities V ∈ R n s × 2 , which are further used to calculate the motion ﬂow M ∈ R n s × 2 , i.e.,
[Equation]
where T ∈ R n s × 1 records the timestamps of the search key-events, and max ( · ) is the operation selecting the maximum element of the input.
With the motion ﬂow and the positions of the search key-events, denoted as K ∈ R n s × 2 , we can localize their initial positions, denoted by K ini ∈ R n s × 2 , as K ini = M + K .
Finally, we predict the motion-aware target likelihood of each search key-event, denoted as p jmon ∈ [0 , 1] (accordingly, P mon :=
[ p 1 mon ; · · · ; p jmon ; · · · ; p n s mon ] ∈ R n s × 1 ), from K ini together with the previous bounding box {+and a normalized distance (cid:104) ( x − x b ) w b (cid:105) 
 + (cid:104) ( y − y b ) h b (cid:105)independently+} via MLPs, where x b , and y b are the spatial coordinates of the box center, and w b and h b are the width and height of the box, respectively.
We refer readers to Appendix for the detailed architecture.
3.4 Conﬁdence-based Object Proposal
This module regresses a bounding box from the predicted two kinds of predicted target likelihoods
P sem and P mon , as well as the embeddings F s encoding the rich spatio-temporal semantic informa- tion of E is .
Speciﬁcally, taking the semantic-aware branch as an example, we formulate the bounding box regression as
[Equation]
where B sem ∈ R 1 × 4 denotes the intermediate bounding box, based on P sem , H sem ∈ R 1 × c encodes the embeddings after adaptive pooling, M p sem ∈ R 1 × n k is the pooling weights, (cid:12) denotes the hadamard product, D ( · ) is the duplication operation, and S ( · ) stands for the softmax function.
We also apply the same procedures to the motion-aware branch, leading to another intermediate bounding box B mon .
Finally, we fuse the two intermediate bounding boxes under the guidance of learned conﬁdence scores, i.e.,
[Equation]
where B o denotes the ﬁnal output bounding box, and M f ∈ R 1 × 2 refers to the learned conﬁdence scores.
We refer readers to Appendix for the detailed architecture.
3.5 Training & Online Tracking
Loss function .
To train the proposed framework end-to-end, we design the following loss function:
[Equation]
where ˆ B := { B o , B sem , B mon } , ˆ P :=
{ P sem , P mon } , B g and P g are the ground-truth bounding box and target likelihood, respectively, L cls ( · , · ) is the binary cross-entropy loss to supervise the target likelihood prediction, L reg ( · , · ) refers to the L 1 loss to supervise the regression of the bounding box, L giou ( · , · ) is the generalized intersection over union (GIOU) loss [44], and L CD computes the Chamfer distance between the back-traced key-events and their potential locations to supervise motion ﬂow estimation, deﬁned as [Equation] where K ini ∈ R n c × 2 denotes the back-projected key-events.
We clip the event clouds within very tiny time interval K c ∈ R n c × 2 , t 0 ≤ t c ≤ t 0 + µ ∆ t , as the potential locations of K ini . Due to the inevitable noise events, we truncate the accidental errors by a threshold γ via a minimization function M ( · ) . we empirically set the hyper-parameters α , β , µ , and γ to 1 e 0 , 1 e 1 , 2 e − 1 , and 1 e − 1 , respectively.
Robust training strategy .
In addition to E t and E is , we also have to feed the proposed framework with the previous bounding boxes.
Accordingly, the motion-aware target likelihood prediction may highly rely on the accuracy of previous predictions.
To mitigate the potential accumulated errors and improve the robustness of our framework, we disturb the input bounding boxes with random
Gaussian noises during training.
Template updating strategy for online tracking .
Considering that the appearance of an object may undergo a large variance during a tracking process, we also introduce a strategy to update template key-events.
Speciﬁcally, we ﬁrst partition the embeddings of key-events of E t , i.e., S t , into two sets equally: the constant set S tc and the variant set S tv .
Then, after each search procedure, we stack in the embeddings of the key-events of E is (i.e., S s from) with a high target likelihood, i.e., p > τ , where τ is empirically set to 9 e − 1 , to S tv and randomly pop out embeddings from S tv to keep the total size of S t unchanged.
Experiments 4.1 Experiment Settings
Datasets .
We evaluated the proposed method with both real and synthetic event datasets.
The real event dataset FE108 (MIT) [51] provides event data of spatial dimensions 346 × 260, dynamic range 120dB, and minimum latency time 20 µs , captured by the DAVIS 346 camera.
The ground-truth bounding boxes were captured by the Vicon motion capture system.
FE108 contains 108 sequences [Figure] of total length 1.5 hours, recording the motion of 21 different types of objects under four challenging scenarios.
There are 59K event clouds for validation and 140K event clouds for training.
Considering that most objects in FE108 are rigid, which may be insufﬁcient to validate the performance on nonrigid objects, we also constructed a new synthetic dataset Event LaSOT (Evt-LaSOT) , simulated from the widely-used RGB data-based object tracking dataset LaSOT (Apache-2.0) [12] via Vid2E[15].
Evt-LaSOT consists of eight sequences 3 randomly drawn from LaSOT , and there are 18K event clouds in total.
Compared methods .
We compared the proposed method with SOTA RGB data-based object tracking algorithms, including SiamRPN [26], ATOM [10], DiMP [4], SiamFC++ [48], and PrDiMP [11], and the event data-based branch of FENet [51].
Moreover, we modiﬁed two SOTA RGB data-based object tracking methods, i.e., PrDiMP [8] and TransT [8], to adapt them to the event frame representation [51], leading to two strong baselines, namely E-PrDiMP and E-TransT. For fair comparisons, we trained all methods with the training set of FE108 and evaluated them on the testing set of FE and Evt-LaSOT .
Implementation details .
To train our framework in a batch manner, we sampled 10K event points for each event cloud.
We set the value of ∆ t as the interval of annotations, i.e., 25 ∼ 50 ms for the
FE108 dataset according to its 20/40 FPS annotations.
We implemented the framework with Pytorch on Ubuntu 18.04 and trained it with 4 × NVIDIA-3090 and a batch size of 22 samples in each GPU , about 300K iterations.
We adopted the Adam optimizer [24] with the weight decay of 1 e − 5 and the learning rate of 1 e − 5 (resp.
1 e − 4 ) for the backbone (resp.
the two target likelihood prediction and object proposal modules).
We set the spatial search region of the current event cloud as 1.2 × the size of the previous proposal.
4.2 Results
Results on FE108 .
Table 1 and Fig.
4 (a) and (b) show the quantitative comparisons of different methods on FE108 , where it can be seen that the event-based methods generally surpass the RGB databased methods under the LL and HDR scenarios, demonstrating the advantage of event data.
Besides, the signiﬁcant improvement of E-TansT with the input changed from EF- 1 to EF- 3 demonstrates that the high temporal resolution characteristic of event data is of importance. Our method achieves the [Table]
[Table] [Table] best or second best performance in terms of different metrics under most scenarios, and especially, it signiﬁcantly exceeds the compared methods under the fast motion scenario.
The reason may be that the compared methods are disturbed by the large motion offset in the accumulated event frames, as shown in Fig.
3 (b).
Although the proposed method is slightly worse than or comparable to the compared event frame-based methods under very rare scenarios, we want to notice that those methods beneﬁt a lot from the RGB data-based tracking, a long-standing ﬁeld featured with great efforts devoted and mature techniques, while our new paradigm is still plain to some extent, and it is highly expected that its performance raises gradually with more advanced modules incorporated in the future.
Results on Evt-LaSOT .
Since these synthetic event data from RGB data do not have high temporal resolution and are driven from estimated optical-ﬂow, which may not correctly reﬂect the real movement of an object, for the sake of fairness, we only compared the proposed method with eventbased methods on this dataset.
As shown in Table 2 and Figs.
4 (c) and (d), the proposed method trained with real event data from FE108 is well generalized on Evt-LaSOT and outperforms all the compared methods to a signiﬁcant extent in terms of all the three metrics, which is credited to the new paradigm of directly modeling raw event data.
Besides, since our method directly consumes raw event data without pre-processing required by existing event frame/image-based methods, it is more efﬁcient, reﬂected by the higher FPS.
We also refer readers to the video demo contained in
Supplementary Material for the visual comparison of different methods.
Comparison with 3D point cloud-based trackers. Recently 3D point cloud-based object tracking achieves remarkable growth, which aims to predict the location of a target object from a 3D point cloud (PC) sequence [38, 54, 22, 53]. Although event clouds could be thought of as 3D point clouds, they are still different. Speciﬁcally, 3D point clouds by the LiDAR sensor record the geometric information of 3D entities, which are generally collected by measuring the reﬂection time of the emitted pulses of infrared light hitting nearby objects/scenes.
Only the part of the object/scene visible to the LiDAR sensor will be perceived (one emitted pulse will generate at most one 3D point).
Thus,
[Figure]
[Table] [Figure] the captured 3D points are distributed on the surface of the 3D entity. However, even data are acquired by measuring the pixel intensity variations in a short time period, and for a typical pixel, there may be multiple events (i.e., the events have the same x and y dimensions but different t dimensions).
Thus, the events are distributed within a 3D cube. Such a data distribution difference may make it inappropriate to apply 3D point cloud-based trackers to event data. Here, we also conducted experiments to compare the proposed method with SOTA 3D point cloud-based trackers on FE dataset.
As listed in Table 3, the proposed method exceeds the 3D point cloud-based trackers [38, 54] to a large extent, i.e., 7.4% and 15.5% improvements in terms of the success rate and precision respectively, demonstrating the demand of designing event-specialized tracking algorithms.
4.3 Ablation Studies
Event downsampling strategy .
We constructed a baseline by replacing the proposed downsampling strategy with random sampling.
By comparing rows 4) and 6) of Table 4, it can be seen that without using the proposed downsampling strategy, the values of RSR and RPR decrease by 4 . 8% and 8 .
4% , respectively, validating the effectiveness of this strategy.
Moreover, such a downsampling process can accelerate the network convergence during training.
Motion-aware target likelihood prediction & Training strategy .
To validate the effectiveness of this module, we constructed a baselines without using it.
As shown in rows 1), 2), and 3) of Table 4, it can be seen that this module augmented by the training strategy achieves 2.7 % gains in terms of
RSR.
Besides, without using the training strategy, this module even harms the accuracy, validating the effectiveness of the training strategy on mitigating the accumulated errors.
We trained the 3D point cloud-based trackers with the same experiment settings as ours and also tried our best to adapt those methods to event data for optimal performance.
method
RSR
OP
[Table] OP 0 . RPR P2B
PTTR [54] 47.
Ours 54.
[Table]
Flow estimation module. By comparing rows 3) and 6) of Table [Table] it can be seen that this module improves the value of RSR by 1 % , validating its effectiveness.
Besides, we also visualized the ﬂow of key-events Fig.
3, where it can be seen that although there are some noise events, the predicted ﬂow could correctly present the motion of both the background and target object.
Online template updating strategy .
The effectiveness of this strategy is validated by comparing rows 5) and 6) of Table 4, where it can be see that the value of RSR improves by 0 .
4% when including this strategy.
4.4 Discussions
Limitations: although the proposed method demonstrates the great potential of raw event data-based object tracking, there are still some promising directions that are worth studying to further boost tracking performance.
First, noisy events are inevitable, as shown in Fig.
3, which may disturb the target likelihood prediction; hence, a typical event data denoising algorithm could be applied as pre-processing to remove the interference.
Second, in addition to the prior knowledge of the continuous motion trajectory preserved in event data, the velocity information of an object hidden in event data, which may also be continuous, could be taken into account.
Third, for the long-term object tracking, it is necessary to investigate more delicate template updating strategies.
Finally, according to RGB data-based object tracking, the powerful pre-trained backbones are essential for improving tracking performance, and thus, it is highly desired to introduce pre-trained backbone on event data into event-based object tracking.
Conclusion
We have presented the ﬁrst end-to-end learning-based object tracking paradigm on raw event data.
Owing our critical modeling of the tracking process for exploring the special characteristics of raw event data, including the delicate downsampling method, the motion-aware module, and the noise injecting training strategy, the proposed framework achieves competitive performance and efﬁciency in comparison with state-of-the-art methods.
Last but not least, despite that we demonstrate the great potential of designing new algorithms tailored to the special characteristics of event data to achieve object tracking, as a beginning of this research line, our framework is still relatively plain, compared to the well-developed RGB data-based object tracking, and it could be expected that the performance will improve gradually with more efforts and advanced techniques incorporated in the future. Acknowledgements. Funding in direct support of this work: Hong Kong Research Grants Council under Grant 11202320, Grant 11219422, and Grant 11218121.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section Appendix A. •
Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] We have discussed the limitations in Sec. 4.4.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Existing visual object tracking algorithms may cause public privacy issues. However, our tracking algorithms is specially designed with event-based vision, which contains less visual patterns than traditional RGB images. Thus, we think it would put lower threaten for public privacy.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] No theoretical results.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Our algorithm is robust to the random seed.
We tested our model with different random seeds and showed the results in Table 5.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to Sec. 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We have correctly cited the FE108 dataset in Sec. 4.1.
(b) Did you mention the license of the assets?
[Yes] Please refer to Sec. 4.1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We have included the source code and pre-trained model in the supplementary material.
And we will also release the simulated Evt-LaSOT.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] Since there is no random process for those image-based methods during inference.
Their result is constant. (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] There is no person related data in our dataset.
If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
No research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Details of the Network Architecture
In this section
[Figure]
we provide the detailed network architectures of different modules involved in our framework. We want to notice that the source code and pre-trained model are also included in the supplementary ﬁle.
[Figure] [Figure] [Figure] s p r opo s a l
N o i s e i n j e c t i on & dup li c a t e
C a t w i t h e x p li c i t l y en c
od
i ng
A.1 Graph-based Key-event Embedding
Fig.
A1 shows the detailed network architecture of this module.
“Graph aggregation" means searching the 8 nearest neighbours of each key-event.
[Figure]
A.2 Semantic-driven Siamese-matching
Fig.
A2 shows the detailed network architecture of this module.
Note that the spatially downsampling operation contained in the swin-layers is to resample key-events further using the proposed down-sampling strategy, and feed it with its 3 nearest neighbours to the MLP for dimension reduction.
A.3 Motion-aware Key-event Back-tracing
Fig.
A3 shows the detailed network architecture of this module.
A.4 Conﬁdence-based Object Proposal
Fig.
A4 shows the detailed network architecture of this module.
The semantic-driven likelihood, self-attention, and motion-aware target likelihood are derived from the corresponding modules.
Appendix B Details of Evaluation Metrics
Denote by B ∈ R N × 4 the predicted bounding boxes of event sequences with N event clouds, and
B gt ∈ R N × 4 the ground-truth bounding boxes in the form of [ cx, cy, w, h ] with cx and cy being the spatial position of the central of a bounding-box, and w and h being the width and height.
representation.
The used metrics for quantitative evaluation are deﬁned as follows:
[Equation]
N with I ( True ) = 1 , I ( False ) =
[Equation]
where IOU ( · , · ) computes the intersection over union of two bounding boxes
[Equation]