A sparse-response deep belief network based on rate distortion theory
A novel deep belief network based on rate distortion theory for feature extraction is proposed. Sparse response regularization induced by L1-norm of codes is used to achieve a small rate. KL divergence is considered as a distortion function. Hierarchical representations mimicking computations in the cortical hierarchy are learnt. More discriminative representation than other algorithms in deep belief networks is yielded.
deep belief network rate distortion theory kullbackleibler divergence information entropy unsupervised feature learning