Vehicle Speed Estimation Using Computer Vision And Evolutionary Camera Calibration
Abstract
Currently, the standard for vehicle speed estimation is radar or lidar speed signs which can be costly to buy and maintain.
However, most major cities already implement networks of traﬃc surveillance cameras that can be utilized for vehicle speed estimation using computer vision.
This work implements such a system using homography estimation, YOLOv4 object detector, and an object tracker capable of vehicle speed estimation.
The homography component uses world plane-image plane point correspondences, located by humans.
Moreover, a new method is developed speciﬁcally for this use case, using the estimation of density evolutionary algorithm.
It aims at correcting the points misalignment in between planes.
In addition, a basic direct linear transformation (DLT) and a random sample consensus robust version of DLT are implemented for comparison.
Finally, the results show that the proposed homography method reduces the projection error from world to image point by 97%, when compared to the other two methods, and the complete workﬂow can successfully estimate speed distributions expected from vehicles on urban traﬃc and handle dynamic changes in vehicle speed.
Introduction
Each year, approximately 1.35 million people die on roadways worldwide due to speeding drivers; public entities have to deal with the cost of damage control and lidars for speed estimation are expensive to buy and maintain.
In addition, most major cities have implemented a grid of surveillance cameras in areas of interest to tackle this problem.
This work 1 proposes a three-component workﬂow to develop a speed estimation system using only images and point correspondences between the image plane and world plane.
First, homography estimation is employed, assuming a ﬂat road to back-project image coordinates to longitude-latitude coordinates.
A human operator deﬁnes a set of correspondences between the two planes to estimate the homography matrix for each scene.
Then, an object detector locates vehicles on the scene.
Later, an object tracker receives those vehicle detections and uses intersection over union (IoU) to assign them to vehicle tracks, and estimate speed by computing haversine distance between two points on a ﬁxed window of time.
Moreover, three implementations of homography estimation were compared.
The ﬁrst is a basic Direct Linear Transformation (DLT) method, the second is a robust version of
DLT using Random Sample Consensus (RANSAC), and the third is the proposed novel method based on the Estimation of Density evolutionary algorithm (EDA).
This proposed method modiﬁes the longitude-latitude inputs to minimize the projection error loss function.
Therefore, correcting the misalignment between a point in the image plane and the world plane.
Previous Work
There are many works that try to address the problem of vehicle speed estimation using machine vision.
For instance, Tang et al.
[13] introduced a methodology that consists in the calibration of a pinhole camera model based on vanishing points, followed by object detection using YOLO [10], and a customized tracking system.
This tracker uses various visual a spatial features to produce reliable tracks.
This work achieved the ﬁrst place on the NVIDIA AI City Challenge 2018, achieving a
RMSE of 4.0963 mi/h.
Then, the work of [15] proposes a similar workﬂow.
However, the calibration is based on perspective transformation leveraging road markings.
Their work assumes that markings follow standard dimensions.
Furthermore, the authors used them to establish a real world coordinate system, and ﬁnd a homography matrix.
This method only uses a single scene and achieved an average error of 3.2 km/h.
Methodology 3.1 Algorithm Design
This method has three main components: homography matrix estimation, object detection, and object tracking.
Figure 1 depicts the system workﬂow with the aforementioned components.
The ﬁgure also includes all inputs and outputs, which are explained in detail in the following subsections.
3.2 Data Collection
To compute homography matrices, a video dataset with both image and world points representing the same spots is needed.
At the time of writing this work, no such dataset was found.
Instead, a compilation of 10 live feeds from diﬀerent cameras in unconstrained conditions were recorded using the Department of Transportation of Seattle portal ( https://web6.seattle.gov/travelers/ ).
In addition, street addresses from each scene were extracted.
Then, those addresses were visited on Google Maps to ﬁnd the scene and manually extract world points that matched the image plane.
Moreover, as the dataset was built from scratch, no beforehand speed annotations of the vehicles are available.
Table 2 of Appendix A presents the metadata of all the videos.
3.3 Homography Estimation for Camera Calibration
A critical component for a system, capable of measuring the speed of the objects in the scene using only visual information, is camera calibration.
Assuming a ﬂat road, let 𝑠 ∈ R be a scale factor, 𝒑 (cid:48)
= (cid:0) 𝑥 (cid:48) 𝑦 (cid:48) 1 (cid:1)
𝑇 and 𝒑 = (cid:0) 𝑥
𝑦 1 (cid:1) 𝑇 be the homogeneous coordinates of a point in the image and world plane, respectively.
Then, the camera model [12] is expressed as:
[Equation]
The parameters of H , the homography matrix, can be obtained with Direct Linear Transformation (DLT) [4] with a minimum of four point correspondences.
In this work, given a scene 𝑖 , a set of 𝑛 points are manually identiﬁed both for the pixel coordinates from a reference image of the scene and for the latitude/longitude coordinates taken from Google Maps.
Each correspondence between two points should represent the same location on the scene.
Furthermore, three methods are evaluated: a base DLT homography extraction with no robust method for error minimization, a RANSAC [5] DLT robust method, and a proposed technique, designed speciﬁcally for roadside camera calibration, based on evolutionary algorithms.
Given the initial manual annotations, the proposed calibration method consists of an iterative, localized correction of each starting latitude-longitude point on a scene to estimate image coordinates that match the annotations as much as possible.
Intuitively, this procedure tries to alleviate the miss-alignments between real-world and image point correspondence introduced by human operation.
It follows an
Estimation of Density evolutionary algorithm (EDA) [1], guided by the projection error:
[Equation]
where 𝑝 𝑖 is a point selected by a human operator on the image plane, and ˆ 𝑝 𝑖 is the estimated point given a longitude-latitude coordinate.
The EDA algorithm works by establishing an initial population 𝑁 of point correspondences with a pre-deﬁned range where a random variation of the original points can be generated.
Then, generation by generation, 𝐾 individuals are selected with the least projection error.
Each generation, the mean and variance of the survivors are calculated to generate oﬀspring and the process repeats until error convergence.
Note that, although all points from both planes are manually selected, the reasons for subjecting longitude-latitude points to correction instead of the image coordinates is that projection error uses image coordinates as target values and, most importantly, the world plane has no perspective.
This allows a uniform initial search range for every point.
3.4 Object Detection with YOLOv
To be able to detect vehicles on the scene, YOLOv4 detection model, created by Bochkovskiy et al.
[3], was utilized.
It is a deep convolutional neural network that consists of three main components: backbone, neck, and head.
First, a network called CSPDarknet53 is embedded into YOLO as its feature extractor backbone.
Based on its original version Darknet-53, it is implemented as a Cross
Stage Partial Network (CSPNet) [14].
For the neck of the detector, a Spatial Pyramid Pooling (SPP) block was included over the backbone to generate a ﬁxed-length vector regardless of image size [7].
YOLOv4 detector also performs feature extraction at three spatial dimensions to improve detection at varying object sizes.
The mechanism follows a Feature Pyramid Network (FPN)
[9] in which the feature maps gradually decrease in the spatial dimension but are later up-scaled with deconvolution layers.
Finally, the detection head is taken from the previous version, YOLOv3 [11], in which each output feature map is subjected to a 1 × 1 convolutional layer with shape 1 × 1 × ( 𝐵 ( 5 + 𝐶 )) , where 𝐵 is the number of objects that can be detected on each image cell and 𝐶 the number of classes.
3.5 Object Tracking and Speed Estimation
The object tracker follows a tracking by detection scheme from Bochinsky et al.
[2], with modiﬁcations to keep track of the speed in km/h along the path of the objects.
At the ﬁrst image frame, the tracker assigns all detections to new active tracks.
Then, the algorithm updates itself each frame by receiving new detections and assigning them to the tracks that maximize the intersection over union (IoU) using bounding boxes.
In addition, an assignment threshold is deﬁned to avoid identity hĳacking or location jumps [8].
Furthermore, the speed estimation process is made every 𝐾 number of frames for each track independently, using the following formula:
[Equation]
where 𝑑 ( 𝑝, 𝑞 ) is distance in kilometers, 𝑓 𝑠 is the number of frames per second taken by the camera, and 𝑓 − 1 𝑠 represent the time spent each speed estimation.
Each time the speed estimation is performed for a vehicle, 𝑝 takes the value of 𝑞 , and 𝑞 becomes the newest location available for that vehicle.
To calculate distances, recorded locations in image points have to be transformed to the real world plane using the inverse of the homography matrix H − 1 .
Then, the distance in kilometers is computed using the haversine distance [6]:
[Equation]
where 𝑝 𝑥 , 𝑞 𝑥 are latitudes and 𝑝 𝑦 , 𝑞 𝑦 are longitudes.
Results and Discussion 4.1 Homography Estimation
Figure 2 presents quantitative results for the base, RANSAC, and proposed version of the calibration.
On average, both RANSAC and base algorithms got an error of 7.99 pixels, while the proposed returned only 0.24.
This represents a reduction of 97% in projection error.
In particular, video 4 had the highest error on both base and RANSAC algorithms, with estimations signiﬁcantly deviating from targets.
On the other hand, with the proposed methodology, the estimations match the annotations correctly, as can be seen in ﬁgures 3a and 3b, which depict a graphical comparison with the calibration methodologies.
This pattern repeats for all the other videos, except for videos 6 to 10.
These videos had negligible error for all the three methodologies, and they all had only 4 point correspondences, placed at the corners of the region of interest.
4.2 Speed Estimation
Distributions of speed across all the region of interest were computed for every video instance to assess the performance of the speed estimation methodology.
These distributions are depicted in
[Figure]
Figure 4 using boxplots.
Also, Table 1 shows statistics computed on the same videos: number of samples, mean speed, median speed, and standard deviation.
Most of the distributions per video tend to be skewed towards 0 km
/h.
This is due to traﬃc lights that are placed next to the cameras that were ﬁlming the area, and cars often had to stop, or were parked.
Moreover, the trackers were still measuring vehicles at full stop.
Then, the majority of speed values that were obtained, per video, are less than 70 km/h, as can be seen in Table 1.
However, there is still some exceptions.
For instance, Figure 4 shows that video 4, have its fourth quartile on ranges greater than 100 km/h. As this methodology estimates the speed of each vehicle independently, these measures were stored as time series.
Figure 5 shows the speed timeline for ten diﬀerent vehicles for video 7, which were randomly chosen.
It can be seen that the tracker algorithm is able to measure the change in velocity of the vehicles.
There are no sudden spikes in speed or noisy recordings, and the change in speed follows soft curves.
These insights show that the tracker measures are on par with the real behavior of drivers on urban areas.
[Table]
Conclusions
There are two main conclusions that stand out from the results of this work.
The ﬁrst is that the proposed calibration method, employing EDA evolutionary algorithm, resulted in a dramatic reduction of the projection error of all scenes, when compared to both the base DLT and RANSAC DLT methods.
Second, in all scenes, the speed estimation distributions along with the vehicles speed timeseries resembled the behavior of drivers on an urban city
A Appendix A.1 Experimental setup
All experimentation was performed on an Acer Aspire-E5-576G laptop running Ubuntu 20. with Intel ® Core ™ i5-8250U CPU, Nvidia GeForce MX150 GPU with Compute Uniﬁed Device
Architecture (CUDA) enabled, 12 Gb of DDR4 RAM, and 500 GB SSD M.2 NVME.
Although
CUDA is enabled, it could not be used for deep learning acceleration due to the limited graphics memory available.
Thus, CPU had to be utilized to make the computations of the object detection model, signiﬁcantly increasing the processing time.
For future work, more robust hardware is needed to assess the performance of this workﬂow in real-time.
A.2 Metadata of the dataset
All the addresses, date of recording and number of points extracted can be seen in Table 2.
[Table] [Table]
February,
[Table]
E
[Table]
Way S &
[Table]
February,
[Table] [Table]
February
[Table] [Table]
February, 22,
Airport Way S & S Industrial Way
[Table]
March, 10,
[Table]
Ave
[Table]
St
[Table]
March, 10,
Airport Way S & S Lander St
[Table]
March, 10,
E Marginal Way S @
Hudson St
[Table]
March, 10, 1st Ave & Seneca St
[Table]
March, 10,
Fairview Ave & Denny
Way 1920 ×10:
A.3 Algorithm and hyperparameters used for the proposed homography estimation method
The homography estimation process using estimation of density evolutionary algorithm (EDA) follows
Algorithm 4, which also uses auxiliary functions deﬁned in algorithms 1, 2, and 3.
Furhtermore, the hyperparameters are:
• Initial population ( 𝑁 ):• Selected population per generation ( 𝐾 ):• Number of generations ( 𝜏 ):• Initial variability range ( 𝛾 ): 10%
Output : 𝜇, 𝜎 variance vectors.
1 𝜎 𝑥 , 𝜎 𝑦 ← compute variance vector per coordinate of for all point sets; 2 𝜎 ← 𝑐𝑜𝑛𝑐𝑎𝑡𝑒𝑛𝑎𝑡𝑒 ( 𝜎 𝑥 , 𝜎 𝑦 ) ; 3 𝜇𝑥, 𝜇𝑦 ← compute means vector per coordinate of for all point sets; 4 𝜇 ← 𝑐𝑜𝑛𝑐𝑎𝑡𝑒𝑛𝑎𝑡𝑒 ( 𝜇𝑥, 𝜇𝑦 ) ;