Post-Hoc Interpretation of Transformer Hyperparameters with Explainable Boosting Machines (2022)

Abstract: Hyperparameter tuning is important for achieving high accuracy in deep learning models, yet little interpretability work has focused on hyperparameters. We propose to use the Explainable Boosting Machine (EBM), a glassbox method, as a post-hoc analysis tool for understanding how hyperparameters influence model accuracy. We present a case study on Transformer models in machine translation to illustrate the kinds of insights that may be gleaned, and perform extensive analysis to test the robustness of EBM under different data conditions.

## introduction deep neural networks have revolutionized the field of ai, bringing about impressive improvements in accuracy at various tasks. there is now a growing interest in interpreting what the model is doing that leads to these high accuracies @xcite . a better understanding is useful in many ways: it can provide researchers a more in-depth view of the problem, assist developers to debug the model, or give users a way to act on the model result. our goal is to improve our understanding of neural network hyperparameters. while there are many research efforts on explaining a model's prediction or interpreting a model's parameters, there has been little work on hyperparameters. hyperparameters like number of layers and learning rate are important factors that impact model performance. in practice, many engineering hours are spent on tuning hyperparameters. we believe methods and tools for interpreting hyperparameters are needed to help practitioners tune more effectively; there are also applications in the growing field of automl @xcite , where our understanding of hyperparameters can help guide researchers design more effective search spaces. in the following, we first develop further the idea of post-hoc interpretation of hyperparameters and contrast it with other types of interpretability research (section 3). then we briefly describe the ebm, which is the glassbox model used in our interpretability framework (section 2). section 4 present a case study on transformers in machine translation tasks, to illustrate how our framework can be used to understand which hyperparameters are important, how its influence changes according to different hyperparameter values, and whether pairwise interactions are present. finally, section 5 analyzes the robustness of ebm: it helps characterize under what conditions are the interpretability results valid. the contribution of this paper is two-fold: first, we advocate a framework for understanding hyperparameters with ebms and present a case study on machine translation transformers to illustrate its usefulness. second, we perform extensive experiments on ebms to characterize the conditions where interpretability results are robust. ## explainable boosting machine let's define input x as a feature vector representing the hyperparameter setting model m λ , and y as the scalar output response variable s(m λ ). we use ebm as introduced in @xcite @xcite and implemented in @xcite . ebm is a generalized additive model with the form: an attractive aspect of ebm is that f j (x j ) is based on a single feature, and can be of arbitrary shape. see examples of f j in figure 3 : on the left, we see that f @xmath0; on the right, we see a different f @xmath1, then drop sharply when bpe increases to 50k. since the f j (•) are summed linearly to predict the response variable (accuracy or bleu score), we can obtain an intuitive understanding of how each hyperparameter impacts the final accuracy. in other words, since ebm is an additive model, it is straightforward to infer the contribution of each feature function; at the same time, the ability to learn arbitrary shapes for the feature function allows for enhanced interpretability. refer to the aforementioned papers for details of how the ebm is trained. ## interpreting hyperparameters proposed framework: our goal is to gain insights about hyperparameters for a class of deep neural networks. we require the existence of a set of models with different hyperparameter settings trained on the same dataset. for example, assume a set of transformer @xcite models {m λ }, λ ∈ λ where λ represents the hyperparameter space, m λ represents a model with a specific hyperparameter setting (e.g. 6-layer encoder, 2-layer decoder, 8 heads, 256 word embedding size); each model has an accuracy metric s(m λ ), and a glassbox model is fit on pairs p ≜ {(m λ , s(m λ ))}. assume there is a person building the models (model builder) and a person analyzing the models after the fact (model analyzer); they may or may not be the same person. our framework consists of three steps: given n models that are trained on dataset d, we find that embedding size influences bleu more than attention heads. but we choose ebm due to its excellent visualization ability. note that while there is a considerable amount of work on interpreting a transformer's parameters such as attention weights @xcite @xcite , these methods are not readily applicable due to the non-differentiability and heterogeneity of hyperparameters. thus, an external model f : λ → r that treats hyperparameters as input features is more amenable. this external model is essentially finding hyperparameter "features" that are predictive of accuracy. as long as this model is glassbox in the sense that it's internals are viewable, then we are able to interpret the results in step 3. ## case study: post-hoc interpretation we now provide a case study on transformer hyperparameters for machine translation to illustrate the kinds of insight we can learn from the proposed post-hoc interpretation framework. ## analysis of ebm robustness to ensure the validity of our post-hoc interpretation framework, we need to analyze the robustness of ebm to different kinds of data sizes and distributions. specifically, one important requirement for our framework is the availability of p ≜ {(m λ , s(m λ ))}; one might not be able to fully control how this data is acquired. it may be a byproduct of an extensive grid search, a manual and focused hyperparameter tuning guided by an engineer's intuition, or an automl experiment. this implies the that hyperparameters may not be sampled uniformly from the space λ, and the number of samples for ebm fitting may not be very large. in order to gain better understanding of ebm's robustness under different conditions, we conduct four experiments. we first study how ebm's fitting ability would be affected if the size or the distribution of training data changes. we then make connections to hyperparameter optimization (hpo), and examine ebm's performance on data generated by sampling from two different hpo methods. finally, we investigate the generalization ability of ebm. to be more specific, we test whether a ebm model trained on one dataset can perform well on another dataset. the experiments following are all conducted on sw-en except for the one in section 5.4. we split the sw-en dataset, the largest dataset among the six datasets provided by @xcite which contains 767 (configuration, bleu score) pairs, into a train set with 614 samples and a test set with 153 samples. an ebm regressor is trained on a subset of the train set and its performance on the test set is reported. we repeat the process 5 times with different random seeds to generate 5 different train-test splits. thus, results reported below are all averaged over 5 runs. ## related work previous work that explores the effect of choices of hyperparameters can be mainly divided into two categories: the prescriptive approach aims to offer advice on the configurations by large-scale experimental runs and those developing tools to improve the understanding of the hyperparameters, as cited in section 3. our work follows the descriptive approach, which seeks to interpret trends from a set of already-trained models. related are some studies that measure hyperparameter importance @xcite and @xcite applied a functional anova framework to assess the importance, while @xcite adopted a variant term, hyperparameter tunability, conditioned on the difference on the performance of default and optimal settings of hyperparameters. exploration of the hyperparameter space also appears in research on hpo interpretability @xcite @xcite . @xcite attempted to interpret the hpo process with a variant of the partial dependence plot and showed what the surrogate model learned about the search space and how the final model is found. ## conclusions in this work, we propose a framework for interpreting the hyperparameters of a set of transformer models. our framework work uses ebm as a posthoc analysis tool, and we show that as a glassbox model, ebm is effectively at interpreting hyperparameters. while the computational needs of generating training data for ebm may seem large at first glance, we emphasize that we advocate for post-hoc analysis. in other words, the analysis is performed on the results of whatever hyperparameter search the model builder needs to perform to deploy a model. our mt case study demonstrates the kinds of insights one can glean regarding the relationship between hyperparameter configurations and transformer performance; for example, we discover that that not all hyperparameters are equally important, and some hyperparameters exhibit non-monotic corelation with bleu scores. further, we conducted a series of analyses to test the robustness of ebm's fitting ability under varying data sizes and distributions. we show that ebm fits well under limited data, yet struggles with transfer across different mt datasets. it should also be noted that the conclusions drawn from mt tasks might not be applicable to other transformer-based tasks. hyperparameter tuning is often viewed as a critical yet un-intuitive part of the model building process. we hope that our proposal provides a first step in unveiling the mysterious masks of hard-tointerpret hyperparameters in deep learning models.