Paper:
Neural approaches to sequence labeling often use a Conditional Random Field (CRF) to model their output dependencies, while Recurrent Neural Networks (RNN) are used for the same purpose in other tasks. We set out to establish RNNs as an attractive alternative to CRFs for sequence labeling. To do so, we address one of the RNN's most prominent shortcomings, the fact that it is not exposed to its own errors with the maximum-likelihood training. We frame the prediction of the output sequence as a sequential decision-making process, where we train the network with an adjusted actor-critic algorithm (AC-RNN). We comprehensively compare this strategy with maximum-likelihood training for both RNNs and CRFs on three structured-output tasks. The proposed AC-RNN efficiently matches the performance of the CRF on NER and CCG tagging, and outperforms it on Machine Transliteration. We also show that our training strategy is significantly better than other techniques for addressing RNN's exposure bias, such as Scheduled Sampling, and Self-Critical policy training.

Review:
I found the paper difficult to follow. The method proposed is not well motivated, and  the literature review explains well the novelty. Here are some questions/points for discussion:
- the token-level MLE training is not what causes the exposure bias: one can train with MLE and still avoid it by generating appropriate sequences using the RNN, as in scheduled sampling. The problem with MLE (or cross entropy) is that the labels to be predicted might not be the correct ones. See the paper by Ranzato et al. (ICLR2016) for a good discussion of the issue: https://arxiv.org/pdf/1511.06732.pdf
- The criticism against previous works for not comparing agains CRFs seems odd: CRFs are given the number of labels, words, etc. to predict, typically the same as the number of words to be tagged. If one  has this, as well as binary rewards for each decision, then there is little benefit for RL/IL based approaches to be used. The point for them is the use of non-decomposable loss functions such as BLEU, which are not common in tagging, but in tasks like MT, where CRFs can't be used. In fact, for the transliteration experiments in the paper, the CRF approach is padded to perform the task, which highlights that it is not the right comparison. 
- the approach proposed seems very similar to MIXER, which also learns a regressor to predict the reward for each action. A direct comparison both in terms of how the approaches operate and empirically is needed.
- why is it a problem that previous works by Ranzato, Bahdanau and Paulus combine MLE and RL? You are using the same supervision, ie. the labeled corpus.
- the adjusted training seems to essentially not reward correct predictions (top branch in the equation). Why is this a good idea?
- In figure 1 it is not clear at all that the proposed approach works; depending on the epoch the ranking among the three variants differs
- what does it mean for one method to surpass the other in flexibilty? If anything the requirement for immediate rewards after every action restricts flexibility, as one can't use non-decomposable loss functions such as BLEU which are prety common in NLP.
- How is the training efficiency measured in the paper?
- Why not compare against MIXER, as well as more recent work by Leblonde et al. (2018): https://arxiv.org/abs/1706.04499 ? I don't see why the Rennie et al. 2017 method is picked for comparison.
- It is not true that in IL one needs a gold standard policy, one can learn with sub-optimal policies, see Sun et al. (2018): https://arxiv.org/pdf/1703.01030.pdf
- It is odd to say that an approach proposed earlier (Dagger) reduces to a variant of a later proposed one (Scheduled sampling), the reduction should be the other way around
- are the randomly initialized character embeddings for transliteration tuned during training?
- How were the alignments for training the CRF obtained?