  Tensor network (TN), which originates from quantum physics, shows broad
prospects in classical and quantum machine learning (ML). However, there still
exists a considerable gap of accuracy between TN and the sophisticated neural
network (NN) models for classical ML. It is still elusive how far TN ML can be
improved by, e.g., borrowing the techniques from NN. In this work, we propose
the residual matrix product state (ResMPS) by combining the ideas of matrix
product state (MPS) and residual NN. ResMPS can be treated as a network where
its layers map the "hidden" features to the outputs (e.g., classifications),
and the variational parameters of the layers are the functions of the features
of samples (e.g., pixels of images). This is essentially different from NN,
where the layers map feed-forwardly the features to the output. ResMPS can
naturally incorporate with the non-linear activations and dropout layers, and
outperforms the state-of-the-art TN models on the efficiency, stability, and
expression power. Besides, ResMPS is interpretable from the perspective of
polynomial expansion, where the factorization and exponential machines
naturally emerge. Our work contributes to connecting and hybridizing neural and
tensor networks, which is crucial to understand the working mechanisms further
and improve both models' performances.
