A simple and efficient hybrid framework for evaluating algorithms for tracking objects in video sequences is presented. The framework unifies state-of-the-art evaluation metrics with diverse requirements in terms of reference information, thus overcoming weaknesses of individual approaches. With foundations on already demonstrated and well known metrics, this framework assumes the role of a flexible and powerful tool for the research community to assess and compare algorithms. Introduction: The large number of proposed algorithms for video object tracking has led to the need for a flexible framework that enables the comparison and characterisation of such algorithms. Many researchers define their own test sequences, processes and metrics for evaluation, making it extremely difficult to replicate results and compare algorithms. Proposals for evaluation frameworks and metrics [1-3] already exist. Although some proposals focus on an evaluation without ground-truth (GT) [4, 5] , the results typically provide insufficient discriminative information, particularly when the noise/errors in the background/foreground segmentations increase. As a result, evaluations based on metrics using GT are commonly preferred. The generation of this type of information is cumbersome, especially when detailed pixel-based references (reference segmentations (RS)) are needed. To minimise the GT generation effort, the use of bounding boxes (BB) to locate the object and provide a coarse dimension of the object has been the preferred strategy. Proposals such as [3, 6] consist of using this type of GT and compute a set of measures encompassing the entire sequence. Although some analysis about split/merge and fragmentation errors is sometimes made, typically there is no information about the temporal evolution of the error, making it more difficult to identify failure points or events in the sequences. Moreover, as the size of the BB is typically greater than the enclosing object, there may also be a loss of spatial resolution. In [1] a novel framework was described for evaluating video segmentation and tracking algorithms. It uses RS as GT and partitiondistance (PD) metrics [7] , and is capable of computing an overall error measure as well as exhibit its temporal evolution over the sequence. The drawback is the requirement for GT in the form of RS, which are harder to generate than the BB GT. This Letter proposes a framework, based on the PD metrics, capable of providing more information than that typically obtained through the use of BB, while minimising the need for RS (Fig. 1) , thus making it more flexible and easier to use by the research community. Given that the application of metrics using a single type of information has been strongly documented, this Letter focuses on the experiments with the combination of GT information.
In REF an initial proposal to combine different types of GT and their use in the computation of PD metrics was made.
Hybrid framework for evaluating video object tracking algorithms
Electronics Letters
Computer Science