Paper:
Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples. To address the issues, we propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. In contrast to standard GANs that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier for comparing the text quality between a pair of samples. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. This self-improvement reward mechanism allows the model to receive credits more easily and avoid collapsing towards the limited number of real samples, which not only helps alleviate the reward sparsity issue but also reduces the risk of mode collapse. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity, and yields more stable performance compared to the previous GANs for text generation.

Review:
This paper introduces a Self-Adversarial Learning (SAL) mechanism in GAN based text generation, aiming at tackling the problem of mode collapse and sparse rewards problem. Specifically, motivated by “self-play” mechanism in RL community, instead of using a binary classifier as discriminator in original GAN, SAL employs a comparative discriminator which is a pairwise classifier with three classes: “better”, “worse” and “indistinguishable”. The authors provide lots of experimental results and ablation study showing the efficiency of the proposed mechanism in comparison with previous GAN models.
Decision: weak reject. 
This paper is well motivated: clearly sparse rewards and mode collapse are two problems need to be solved in GAN based text generation, however, the following concerns prevent me from finding this paper acceptable in ICLR:
The “self-play” idea is widely used in RL. In RL, the comparison between policies can be determined directly by the game simulation results. In text generation, such comparison is more difficult to be judged. This paper appears to assume that the generated sentences are “worse” than the real samples, which is similar to the original definition of discriminator in GAN, and the generated sentences in earlier epochs are “worse” than that in later epochs, which needs further justified.
As a simple extension to GAN, I’m not convinced that the problem of mode collapse could be solved by proposed mechanism. If the generator falls into a local minimum, a collapsed mode, the proposed mechanism will never pull the generator out of that. Moreover, what is the theoretical foundation of the proposed evaluation metric on quality-diversity trade-off, NLL_{gen} + NLL_{oracle}?
The setting of the important set of hyper-parameters, reward weights, is unclear in the paper. The reward weights directly influence the reward in training, thus should play an important role of model performance. More discussion about this should be provided.
Moreover, in the paper, only comparison between proposed mechanism with GAN based models are shown. Comparison with more recent models like RelGAN should be provided. And comparison with other state-of-the-art text generation model should be discussed.
Feedbacks: 
References regarding experimental results in table are incorrect. For example, the results in synthetic data should be in “Table 2” and COCO image caption dataset should be in “Table 3”. 
Some imprecise parts, for example, in Equation (5) and (6), it should be G_\theta(Y_{1:t-1} and G_\theta(y_t | Y_{1:t-1} .
I’m curious about what the performance would be like if the weakly supervision by regarding sentences generated in later training stage are “better” than the sentences generated in earlier training stage is removed in training comparative discriminator. This is different from the CAL model in the ablation study.
What is the influence of different values of rewards weight? 
Also about the rewards weight, in the description of Scheduled rewarding, the rewards weight is described to be linearly changed with training iteration, while in Appendix C.3, the rewards weight is described to be fixed. This is very confusing.
A minor issue: for image captioning, a lot other metrics are widely used in measuring the model performance, e.g. CIDEr, SPICE and so on. Those metrics could be helpful for audience to understand how the model performs in comparison with other captioning models.
Another minor issue: for image captioning and WMT (conditional generation), the detailed model structures are not described in paper, which is not very friendly to audiences with relatively little knowledge in related areas.