  We consider solutions of the matrix KP hierarchy that are elliptic functions
of the first hierarchical time $t_1=x$. It is known that poles $x_i$ and matrix
residues at the poles $\rho_i^{\alpha \beta}=a_i^{\alpha}b_i^{\beta}$ of such
solutions as functions of the time $t_2$ move as particles of spin
generalization of the elliptic Calogero-Moser model (elliptic Gibbons-Hermsen
model). In this paper we establish the correspondence with the spin elliptic
Calogero-Moser model for the whole matrix KP hierarchy. Namely, we show that
the dynamics of poles and matrix residues of the solutions with respect to the
$k$-th hierarchical time of the matrix KP hierarchy is Hamiltonian with the
Hamiltonian $H_k$ obtained via an expansion of the spectral curve near the
marked points. The Hamiltonians are identified with the Hamiltonians of the
elliptic spin Calogero-Moser system with coordinates $x_i$ and spin degrees of
freedom $a_i^{\alpha}, \, b_i^{\beta}$.
