Paper:
We present an end-to-end reinforcement learning approach to hierarchical text classification where documents are labeled by placing them at the right positions in a given hierarchy. While existing "global" methods construct hierarchical losses for model training, they either make "local" decisions at each hierarchy node or ignore the hierarchy structure during inference. To close the gap between training/inference and capture label dependencies in an end-to-end manner, we propose to learn a label assignment policy to determine where to place the documents and when to stop. The proposed method, HiLAP, directly optimizes metrics over the hierarchy, makes inter-dependent decisions during inference, and can be combined with different text encoding models for end-to-end training. Experiments on three public datasets show that HiLAP yields an average improvement of 33.4% in Macro-F1 and 5.0% in Samples-F1, outperforming state-of-the-art methods by a large margin.

Review:
This papers uses the label hierarchy to drive the search process over a set of labels using reinforcement learning. The approach offers clever and promising techniques to force the inference process in structured classification to converge, but experiments seem to lack apple-to-apple comparisons.
However, I think the authors should rather present this work as structured classification, as labels dependencies not modeled by the hierarchy are exploited, and as other graph structure could be exploited to drive the RL search.
I tend to see hierarchical classification as an approach to multi-label classification justified by a greedy decomposition that reduced both training and test time. This view has been outmoded for more than an decade, first as flat approaches became feasible, and now as end-to-end  structured classification is implementable with DNNs (see for instance David Belanger work with McCallum)
Compared to other structured classification approaches whose scope is limited by the complexity of the inference process, this approaches is very attractive. The authors open the optimization black box of the inference process by adding a few very clever tricks that facilitate convergence:
- Intermediate rewards based on the gain on F1 score
- Self critical training approach
- "Clamped" pre-training enabled by the use of state embeddings that are multiplied my a transition to any state in the free mode, and just the next states in the hierarchy in the clamped mode
- Addition of a flat loss to improve the quality of the document representation
While those tricks may have been used for other applications, they seem new in the context of hierarchical/multi-label/structured classification.
While the experiments appear thorough, they could be the major weakness of this paper. The results the authors quote as representative of other approaches seem in fact entirely reproduced on datasets that were not used on the original papers, and the authors do not try an apple-to-apple comparison to determine if this 'reproduction' is fair. None of the quoted work used the 2018 version of Yelp, and I could only find RCV1 Micro-F1 experiments in Johnson and Yang, who report a 84% micro-F1, far better than the 76.6% reported on their behalf here, and better than the 82.7% reported  by the authors. I read note 4 about the difference in the way the threshold is computed, but I doubt it can explain such a large difference. I did not check everything, but could not find and apple-to-apple comparison?
Have the network architecture been properly optimized in terms of hyper-parameters?
In particular, having tried Kim CNN on large label sets, I suspect the author settings using a single layer after the convolution is sub-optimal. I concur with the following paper than an additional hidden layer is essential: Liu et al "Deep Learning for Extreme Multi-label Text Classification". I also note the 32 batch size could be way too small for sparse label sets (I tend to use a batch size of 512 on this type of data).