Two-Stream Convolutional Networks for Autonomous Steering of Self Driving Cars
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose a methodology to extend the concept of Two-Stream Convolutional
Networks to perform end-to-end learning for self-driving cars with temporal cues.
The system has the ability to learn spatiotemporal features by simultaneouslymapping raw images and pre-calculated optical ﬂows directly to steering commands.
Although optical ﬂows encode temporal-rich information, we found that 2D-CNNsare prone to capturing features only as spatial representations.
We show how the useof Multitask Learning favors the learning of temporal features via inductive transferfrom a shared spatiotemporal representation.
Preliminary results demonstrate acompetitive improvement of 30% in prediction accuracy and stability compared tostate-of-the-art methods trained on the Comma.ai dataset.
Introduction
Decision making in the spatiotemporal domain is a key issue for autonomous driving systems [1].
The current paradigm is the implementation of a Convolutional Neural Network (CNN) to performdirect steering commands regression from raw images [2].
A forward-facing camera records videofootage of the road, which is tagged with the vehicle’s cinematic measurements, such as speed,wheel angle and acceleration.
The data is then used to train the CNN regressor with a supervisedlearning approach.
This system has proven to be efﬁcient on extracting spatial cues [3] and achievesrespectable accuracy.
The main limitation is that frames are analyzed individually without taking intoaccount previous actions.
Therefore, any temporal information is lost due to this abstraction.
Following its introduction by Symonyan and Zisserman (2014) [4], Two-Stream Convolutional
Networks have become widely adopted as the preferred method for action recognition in videos.
Themain advantage is the exploitation of the temporal dynamics captured by the optical ﬂow betweenadjacent frames.
In this case, the temporal stream is prone to decoding features instead of making anestimation of the motion ﬁeld.
To the best of our knowledge, the use of this architecture in regressiontasks, especially autonomous steering, has not been studied in depth.
It is widely believed that the use of temporal features together with spatial characteristics could leadto better model predictions.
Currently, the main research efforts are focused on the use of Recurrent
Neural Networks (RNN) [5] and 3D-Convolutional Neural Networks (3D-CNN) [6].
In contrastto intuition, the use of 2D-CNNs could bring some advantages compared to RNNs, in particular, afaster and more straightforward training.
Also, 2D-CNNs are less likely to overﬁt when performingtemporal tasks.
On the other hand, 3D-CNNs must perform an implicit estimation of the optical ﬂowwhile trying to learn temporal features through the depth of the input image tensor, which makestraining difﬁcult.
The principal motivation of this work is to demonstrate that Two-Stream Convolutional Networksare a robust alternative to perform end-to-end learning with temporal cues.
This article describes thepreliminary results tested on Comma.ai’s dataset [7] compared to other regression methods.
Methods
Based on the Two-Stream Convolutional Network architecture, it is possible to design a customself-driving system that takes advantage of spatial and temporal features using 2D-CNNs.
Thissystem is composed of two visual streams taken from a single front-facing camera and its opticalﬂow calculated from the previous image.
The optical ﬂow can be estimated efﬁciently in almost realtime using the current accelerated computing technology.
In addition, specialized hardware can beintegrated to obtain live optical ﬂow streams.
A recording device uses the controller area network(CAN) bus to collect steering wheel commands directly from the vehicle’s computer.
This data isused to train a regressor, which provides a prediction of the required steering angle based on inputs.
The network output is decoded and sent to a proportional–integral–derivative controller (PID), toﬁnally steer the position of the wheels.
Figure 1 shows an overview of the proposed self-driving carsystem.
Figure 2 shows the proposed network architecture that incorporates key elements of Two-Stream
Convolutional Networks and 2D-CNNs to steering angle regression.
The ensemble is composed bytwo identical CNN branches similar to [8].
Global spatial pooling is applied in the last convolutionallayers, aggregating information from all dimensions into single real values.
This global aggregationconsiderably reduces the number of parameters and decreases the risk of overﬁtting [9].
The resultingembeddings are merged with a fusion layer of type element-wise multiplication, forming a sharedspatiotemporal representation.
A multi-layered perceptron (MLP) with linear output neurons performsthe regression.
Following the Two-Stream Networks logic, the ﬁrst branch is intended to learn spatialfeatures extracted from raw images [3].
The second learns temporal characteristics from the estimatedrelative displacement of objects between frames.
Based on the inputs, it is possible to train the network to simultaneously predict the current andprevious state of the target variable using Multitask Learning (MTL).
As the goal of architectureis to capture short-term temporal dependencies, we assume that on the horizon studied, the inputsprovide the information needed to perform the regression.
Therefore, it makes no practical sense touse RNNs.
The main motivation of performing MTL is to favor the learning of temporal features viainductive transfer [10].
By consequence, the auxiliary target is dropped during the inference step.
The architecture has about 45 million parameters and is trained using backpropagation.
The model was trained with 6.2 driving hours from the Comma.ai dataset [7], corresponding toapproximately 220K image/optical ﬂow pairs of size 160x320x3 sampled at 10 Hz.
The optical ﬂowwas calculated using the Farnebäck’s [11] implementation available in OpenCV.
The estimated motionﬁeld was mapped to a standardized RGB color wheel.
Dropout and data augmentation strategies,including spatial and photometric transformations were applied to avoid overﬁtting.
Due to the smallsize of the training set, this work is more aimed at obtaining a benchmark rather than training anautopilot ready for production.
A two-step training strategy was used, with 60% of the training data for feature learning, and 40%for ﬁne tuning.
During the ﬁrst step, CNN branches are trained independently, the weights obtainedare after used as initialization.
In the second step, we ﬁne-tune the architecture by focusing on the
MLP. We set the following hyper parameters: Loss function of type Mean Squared Error (MSE), Adam optimizer [12] with learning rate 1e-4 for feature learning and 0.5e-4 for ﬁne tuning, and abatch size of 64 for feature learning, and 8 for ﬁne tuning.
The model trained for 30 epochs in thefeature learning step and for 1 epoch for ﬁne tuning.
The architecture was implemented using the
Keras functional API running on a machine with a single NVIDIA M6 GPU and 120Gb of RAM.
The test set comprised 53 minutes of video taken from the second and third ﬁles of the Comma.aidataset.
This was done to prevent data leakage due to similar frames in the training and test partitionsin accordance with [5].
As no examples of these folders were included in the training set, theregression task becomes harder, measuring the generalization capabilities of the model.
The main evaluation metric was the Root Mean Squared Error (RMSE), that measured the accuracyof the steering angle regression.
We also used the Whiteness, similar to the implemented in [5], thatmeasured the smoothness of the predictions over time.
This quantity is based on the square of theﬁrst time derivative calculated according to the expression (1).
The lower the value of the whiteness,the smoother the prediction.
[Equation]
Results
The ﬁrst experiment is intended to evaluate the effect of using optical ﬂows instead of raw imagesto train single 2D-CNNs.
For this, we trained two widely used architectures to perform steeringangle regression directly from images and optical ﬂows respectively.
The main idea is to assess thecapability of CNNs to learn temporal features from the displacement of objects between consecutiveframes.
The results of this experience are shown in table 1.
[Table]
The above results show that whichever CNN architecture is utilized, the optical ﬂow producessigniﬁcant accuracy gains in the test set compared to raw images.
We also see some improvementassociated to the architecture sophistication of Inception V3 [8] compared to simple CNNs.
Theseresults suggest that the object displacement representation is convenient to learn speciﬁc featuresimproving generalization.
The fact that the whiteness did not improve, implies that the networks arelearning features only as spatial representations. [Table]
In the second experiment, we evaluate the proposed architecture alongside other widely-used steeringangle regression methods.
Table 2 shows that Two-Stream Convolutional Network outperformedstate-of-the-art 2D-CNN models in prediction accuracy and stability, with a whiteness closer to thehuman driver.
[Table]
Figure 3 shows the scatter plot of the test set instances in paired coordinates of instantaneous whiteness(x-axis) and steering angle (y-axis), along with predictions of three different CNN regression models.
As the whiteness is proportional to the square of the ﬁrst derivative, the ﬁgure is a spatiotemporalrepresentation of the steering angle.
The proposed architecture (right) helps to resolve the bias in thespatial and temporal axis of individual CNN streams trained with raw images (left) and optical ﬂows(center).
This improvement is achieved by the incorporation of a shared spatiotemporal representationand MTL from a time-related task.
The learning of the previous steering angle as an auxiliary task,produces an inductive bias [10] in the temporal domain, reducing the whiteness.
This ﬁltering effectwould be difﬁcult to achieve using only single-target individual CNNs.
Discussion
The main contribution of this work is to demonstrate that Two-Stream Convolutional Networks area robust alternative for end-to-end learning with temporal cues.
We found that the use of opticalﬂows instead of raw images can signiﬁcantly increase the accuracy of current self-driving systems.
The main reason for this is that optical ﬂows provide a convenient representation that can be easilylearned by CNNs.
However, the use of optical ﬂows alone is not sufﬁcient to incorporate temporaldependencies, as CNNs are biased to learning characteristics as spatial representations.
The proposed Two-Stream architecture learns how to effectively combine spatial and temporalinformation from a shared spatiotemporal representation that encodes the relative movement of objectsbetween two consecutive frames.
The introduction of a related auxiliary task (the previous steeringangle) guides the network to discover short-term temporal dependencies via inductive transfer.
Then,an important bias correction is achieved in the spatial and temporal domains improving generalization.
The results show a competitive average performance increase in prediction accuracy and stability of30%, when compared to widely-used regression methods trained on the Comma.ai data set
The nextsteps of this work will focus on the study of MTL for autonomous driving regression tasks.