Parameter Selection: Why We Should Pay More Attention to It (2021)

Abstract: The importance of parameter selection in supervised learning is well known. However, due to the many parameter combinations, an incomplete or an insufficient procedure is often applied. This situation may cause misleading or confusing conclusions. In this opinion paper, through an intriguing example we point out that the seriousness goes beyond what is generally recognized. In the topic of multilabel classification for medical code prediction, one influential paper conducted a proper parameter selection on a set, but when moving to a subset of frequently occurring labels, the authors used the same parameters without a separate tuning. The set of frequent labels became a popular benchmark in subsequent studies, which kept pushing the state of the art. However, we discovered that most of the results in these studies cannot surpass the approach in the original paper if a parameter tuning had been conducted at the time. Thus it is unclear how much progress the subsequent developments have actually brought. The lesson clearly indicates that without enough attention on parameter selection, the research progress in our field can be uncertain or even illusive.

## introduction the importance of parameter selection in supervised learning is well known. while parameter tuning has been a common practice in machine learning and natural language processing applications, the process remains challenging due to the huge number of parameter combinations. the recent trend of applying complicated neural networks makes the situation more acute. in many situations, an incomplete or an insufficient procedure for parameter selection is applied, so misleading or confusing conclusions sometimes occur. in this opinion paper, we present a very intriguing example showing that, without enough attention on parameter selection, the research progress in our field can be uncertain or even illusive. in the topic of multi-label classification for medical code prediction, @xcite is an early work applying deep learning. the evaluation was conducted on mimic-iii and mimic-ii @xcite , which may be the most widely used open medical records. for mimic-iii, besides using all 8,922 labels, they follow @xcite to check the 50 most frequently occurring labels. we refer to these two sets respectively as mimic-iii-full and mimic-iii-50. we will specifically investigate mimic-iii-50. based on @xcite , many subsequent works made improvements to push the state of the art. examples include @xcite @xcite @xcite @xcite @xcite . for the data set mimic-iii-full, @xcite tuned parameters to find the model that achieves the best validation performance. however, when moving to check the set mimic-iii-50, they applied the same parameters without a separate tuning. we will show that this decision had a profound effect. many works directly copied values from @xcite for comparison and presented superior results. however, as demonstrated in this paper, if parameters for mimic-iii-50 had been separately tuned, the approach in mullenbach et al. (2018 ) easily surpasses most subsequent developments. the results fully indicate that parameter selection is more important than what is generally recognized. this paper is organized as follows. in section 2, we analyze past results. the main investigation is in section 3, while section 4 provides some discussion. some implementation details are in the appendix. code and supplementary materials can be found at http://www.csie.ntu.edu.tw/ ˜cjlin/papers/parameter_selection. the task considered in @xcite is to predict the associated icd (international classification of diseases) codes of each medical document. here an icd code is referred to as a label. for the data set mimic-iii-full, caml, which includes an attention layer, was shown to be significantly better than cnn on all criteria; see the paper @xcite has been highly influential. by exactly using their training, validation, and test sets for experiments, many subsequent studies have proposed new and better approaches; see references listed in section 1. most of them copied the cnn and caml results from @xcite as the baseline for comparison. ## investigation we investigate the performance of the cnn and caml approaches in @xcite for the set mimic-iii-50. some implementation details are left in supplementary materials. ## discussion and conclusions the intention of this paper is to provide constructive critiques of past works rather than place blame on their authors. for the many parameter combinations, it is extremely difficult to check them. however, what our investigation showed is that if resources or time are available, more attention should be paid to the parameter selection. for mullenbach et al. (2018), as they have done a comprehensive selection on a super-set mimic-iii-full, the same procedure on the simpler mimic-iii-50 is entirely feasible. the decision of not doing so leads to a weak baseline in the subsequent developments. in conclusion, besides proposing new techniques such as sophisticated networks, more attention should be placed on the parameter selection. in the future this helps to ensure that strong baselines are utilized to check the progress. so that all documents in this batch have the same number of tokens. thus results of the forward operation depend on the batch size. this setting causes issues in validation because a result independent of the batch size is needed. further, for many applications one instance appears at a time in the prediction stage. thus we follow @xcite to use batch @xmath0. after the convolutional layer, @xcite consider the tanh activation function. for both convolutional and linear layers, a bias term is included. before the training process, @xcite sort the data according to their lengths. in the stochastic gradient procedure, data are not reshuffled. therefore, instances considered in each batch are the same across epochs. while this setting is less used in other works, we follow suit to ensure the reproducibility of their results. in the stochastic gradient procedure, we follow @xcite to set 200 as the maximal number of epochs. this setting is different from the default 100 epochs in the software libmultilabel employed for our experiments. in most situations, the program does not reach the maximal number of epochs. instead, it terminates after the validation p@5 does not improve in 10 epochs. this criterion also follows from @xcite . all models were trained on one nvidia tesla p40 gpu compatible with the cuda 10.2 platform and cudnn 7.6. note that results may slightly vary if experiments are run on different architectures.