  Sparse regression has recently been applied to enable transfer learning from
very limited data. We study an extension of this approach to unsupervised
learning -- in particular, learning word embeddings from unstructured text
corpora using low-rank matrix factorization. Intuitively, when transferring
word embeddings to a new domain, we expect that the embeddings change for only
a small number of words -- e.g., the ones with novel meanings in that domain.
We propose a novel group-sparse penalty that exploits this sparsity to perform
transfer learning when there is very little text data available in the target
domain -- e.g., a single article of text. We prove generalization bounds for
our algorithm. Furthermore, we empirically evaluate its effectiveness, both in
terms of prediction accuracy in downstream tasks as well as the
interpretability of the results.
