  We propose an algorithm for automatic, targetless, extrinsic calibration of a
LiDAR and camera system using semantic information. We achieve this goal by
maximizing mutual information (MI) of semantic information between sensors,
leveraging a neural network to estimate semantic mutual information, and matrix
exponential for calibration computation. Using kernel-based sampling to sample
data from camera measurement based on LiDAR projected points, we formulate the
problem as a novel differentiable objective function which supports the use of
gradient-based optimization methods. We also introduce an initial calibration
method using 2D MI-based image registration. Finally, we demonstrate the
robustness of our method and quantitatively analyze the accuracy on a synthetic
dataset and also evaluate our algorithm qualitatively on KITTI360 and RELLIS-3D
benchmark datasets, showing improvement over recent comparable approaches.
