  Using pretrained word embeddings has been shown to be a very effective way in
improving the performance of natural language processing tasks. In fact almost
any natural language tasks that can be thought of has been improved by these
pretrained embeddings. These tasks range from sentiment analysis, translation,
sequence prediction amongst many others. One of the most successful word
embeddings is the Word2vec CBOW model proposed by Mikolov trained by the
negative sampling technique. Mai et al. modifies this objective to train CMOW
embeddings that are sensitive to word order. We used a modified version of the
negative sampling objective for our context words, modelling the context
embeddings as a Taylor series of rate matrices. We show that different modes of
the Taylor series produce different types of embeddings. We compare these
embeddings to their similar counterparts like CBOW and CMOW and show that they
achieve comparable performance. We also introduce a novel left-right context
split objective that improves performance for tasks sensitive to word order.
Our Word2rate model is grounded in a statistical foundation using rate matrices
while being competitive in variety of language tasks.
