  The soft SVD is a robust matrix decomposition algorithm and a key component
of matrix completion methods. However, computing the soft SVD for large sparse
matrices is often impractical using conventional numerical methods for the SVD
due to large memory requirements. The Rank-Restricted Soft SVD (RRSS) algorithm
introduced by Hastie et al. addressed this issue by sequentially computing
low-rank SVDs that easily fit in memory. We analyze the convergence of the
standard RRSS algorithm and we give examples where the standard algorithm does
not converge. We show that convergence requires a modification of the standard
algorithm, and is related to non-uniqueness of the SVD. Our modification
specifies a consistent choice of sign for the left singular vectors of the
low-rank SVDs in the iteration. Under these conditions, we prove linear
convergence of the singular vectors using a technique motivated by alternating
subspace iteration. We then derive a fixed point iteration for the evolution of
the singular values and show linear convergence to the soft thresholded
singular values of the original matrix. This last step requires a perturbation
result for fixed point iterations which may be of independent interest.
