E ven the Simplest Baseline Needs Careful Re-investigation: A Case Study on XML - CNN (2022)

Abstract: The power and the potential of deep learning models attract many researchers to design advanced and sophisticated architectures. Nevertheless, the progress is sometimes unreal due to various possible reasons. In this work, through an astonishing example we argue that more efforts should be paid to ensure the progress in developing a new deep learning method. For a highly influential multi-label text classification method XML-CNN, we show that the superior performance claimed in the original paper was mainly due to some unbelievable coincidences. We re-examine XML-CNN and make a re-implementation which reveals some contradictory findings to the claims in the original paper. Our study suggests suitable baselines for multi-label text classification tasks and confirms that the progress on a new architecture cannot be confidently justified without a cautious investigation.

## introduction deep learning has been a popular research topic in nlp due to its superior performance. the intrinsic structure of deep learning allows researchers to enhance the model performance by introducing more complex network architectures. nevertheless, the increasing complexity brings difficulties to ensure the true architectural progress. for example, @xcite have shown that lstm architectures with appropriate regularization are either competitive or superior to more recent models. as another example, @xcite report that the lack of hyperparameter tuning in an influential work @xcite makes the progress of subsequent network developments questionable. complex architectures are more difficult to train, involve more hyperparameters, and are riskier to unintentional implementation. because new architectures are usually modified from previous ones, a questionable work may make the research progress unclear. therefore, re-examining or reproducing influential architectures are now considered important in the community. in this work, we re-examine xml-cnn @xcite , an influential work in extreme multilabel text classification (xmtc), as a case study to demonstrate the demands of inspecting existing architectures. xml-cnn has been viewed as an essential baseline in subsequent works @xcite @xcite @xcite with more than hundreds of citations. xml-cnn roots from kim-cnn @xcite , a classical architecture for multi-class text classification. the authors of xml-cnn proposed several modifications from kim-cnn to accommodate the xmtc task and empirically claim that all modifications bring significant improvements. despite xml-cnn's popularity, we identified two serious implementation issues that make the original claims uncertain. first, the authors introduced dynamic max-pooling into xml-cnn, but the implementation is actually far from the intended formulation. second, a bug in the experiment code caused the dimensions of convolution operations accidentally swapped. the two issues coincidentally make xml-cnn competitive, leading the authors to illusively claim superiority over kim-cnn and usefulness of dynamic max-pooling in the original paper @xcite . our contribution can be summarized as follows. ## xml-cnn: cnn for multi-label text classification for multi-label text classification, each instance is an n-word document that is associated with a subset of l possible categories. the relationship between the document and the categories can be modeled by a convolutional neural network (cnn), as pioneered for multi-class text classification by @xcite . the architecture was later extended to xml-cnn @xcite for multilabel text classification. here we introduce the two architectures along with a focus on the key modifications. ## investigation into xml-cnn in this section, we point out a significant gap between the formulations in the xml-cnn paper and the authors' implementations. we first reproduce the results in @xcite to ensure what the authors have done. we then confirm the reported superiority of xml-cnn over kim-cnn is actually due to some coincidences. ## the true performance of xml-cnn after showing the gap between the implementation and the formulation in @xcite , the true performance of xml-cnn should be re-examined. in this section, we conduct a comprehensive ablation study for xml-cnn to investigate the claims in the original paper. we then investigate more deeply on dynamic max-pooling to determine its usefulness for xmtc tasks. the results bring us a similarly competitive but simpler baseline for xmtc tasks. ## conclusion this work aims to highlight the importance of validating existing works. from the investigation of xml-cnn, we learned that there are many pitfalls when developing new architectures. we correct the issues in the authors' implementation, carefully re-examine the claims about xml-cnn and recommend suitable baselines for future studies. though not proposing a new method, we hope this work encourages the community to reproduce and re-examine influential works. this may help the community build future works on top of correct materials. this work is partially supported by asus intelligence cloud services and the ministry of science and technology of taiwan via the grants most 110-2628-e-002-013. we also thank the national center for high-performance computing (nchc) of national applied research laborato-ries (narlabs) in taiwan for providing computational resources. a the challenges of reproducing @xcite a.1 dataset the authors evaluated xml-cnn on six datasets from the extreme multi-label repository. 9 unfortunately, some of the datasets have been changed on the repository to different number of data/labels (with a similar name). furthermore, for some of them, the repository does not provide the rawtext documents, making it hard to preprocess the documents to the embedding needed for xml-cnn. fortunately, we find the repository of atten-tionxml. 10 (you et al., 2019) where two datasets (eur-lex and amazon-670k) of raw text match the statistics of the datasets in the xml-cnn work. we then choose the smaller eur-lex as the first attempt to reproduce xml-cnn faithfully.