Paper:
Deep learning natural language processing models often use vector word embeddings, such as word2vec or GloVe, to represent words. A discrete sequence of words can be much more easily integrated with downstream neural layers if it is represented as a sequence of continuous vectors. Also, semantic relationships between words, learned from a text corpus, can be encoded in the relative configurations of the embedding vectors. However, storing and accessing embedding vectors for all words in a dictionary requires large amount of space, and may strain systems with limited GPU memory. Here, we used approaches inspired by quantum computing to propose two related methods, word2ket and word2ketXS, for storing word embedding matrix during training and inference in a highly efficient way. Our approach achieves a hundred-fold or more reduction in the space required to store the embeddings with almost no relative drop in accuracy in practical natural language processing tasks.

Review:
This paper explores two related methods to reduce the number of parameters required (and hence the memory footprint) of neural NLP models that would otherwise use a large word embedding matrix. Their method, inspired by quantum entanglement, involves computing word embeddings on-the-fly (or by directly computing the output of the "word embedding" with the first linear layer of network). They demonstrate their method can save an impressive amount of memory and does not exhibit big performance losses on three nlp tasks that they explore.
This paper is clearly written (with only a couple of typos) but does not yet reach publication standard. Whilst the empirical performance of their approach is promising from the perspective of saving reducing memory requirements, more experiments are required and more careful comparisons to baselines and other methods in the literature for saving memory/parameters. In general the related work and experimental sections are weak and brief, with only superficial analysis. There is  lack of careful analysis and insight into their results, as well as a careful comparisons to other work in this area.
The choice of tasks to evaluate on is broad, which is a strength, but is missing simpler tasks that one would expect to see, such as a text classification dataset, or simple bag-of-vectors style models. In addition, the choice of models are somewhat outdated baselines. It seems that transformers would be an ideal setting for their approach, as transformers have rather high dimensional word embedding matrices, but the authors do not run experiments with the now-ubiquitous Transformer.
The quantum inspiration is largely a distraction, and I think the paper would benefit from this element being scaled back or removed in order to free up space for more experiments.
The authors acknowledge one key weakness of their approach, that both training and inference time are increased (by 28% or 55% longer for DocQA depending on compression)  but much more work could be done to understand the best way to  mitigate for longer training and inference times.
The authors argue that reducing the memory footprint of models is vital to address hardware limitations for training and inference for large models like BERT or ROBERTA, but this argument is not particularly strong. Generally current limitations for training these kinds of models  are the long training times and being able to fit large batches onto our hardware, and the vocabulary matrix is only a constant factor here. And since training time is a bottleneck, the added value of saving memory vs slowing the training speed by 30-50% is debatable. 
Here are some questions for the authors that come to mind when reviewing:
How does your method compare to other published methods on your benchmarks? 
which choices for r and k lead to the best time/memory/performance tradeoff? how does this compare to other compression methods (on your tasks)
Seq2Seq models usually involve multiplying the the output hidden state with a vocab matrix before softmaxing over all the vocabulary produce word probabilities - did you account for this? Does your method work for the output vocab matrix? 
Did you investigate pre-training word2ket like word2vec or Glove?