# **5.1 Should researchers use a penalty parameter other than the plug-in default?**

As discussed above, the “plug-in” penalty parameter behaves well asymptotically, but W ̈uthrich and Zhu (2023) show that it can underselect variables in finite samples when the number of control variables in reasonably large. The alternative is cross-validation. Chernozhukov et al. (2024) suggest that cross-validation instead raises the risk of overfitting, and can perform poorly in moderately sized samples as a result. We examine how cross-validation performs in practice using both our replication data and simulations.

Figure 5 shows simulation results, varying the number of observations between 100 and 10 _,_ 000.<sup>12</sup> Panels (a) and (b) show that the cross-validation method tends to select far more variables compared to the plug-in approach, for both steps 1 and 2. For example, when _n_ = 500 the median number of variables selected that are predictive of the dependent variable using the plug-in penalty parameter is one, compared to five for cross-validation. Since the oracle estimator includes five control variables, this suggest that there might be some power advantage of using cross-validation to select the penalty parameter in smaller samples. However, panels (a) and (b) also show that cross-validation often selects too many variables. The method is also unstable, with a high variation in the number of variables selected. This could reduce precision, due to reasons discussed above. Overall, panel (c) shows that the median standard error tends to be smaller for cross-validation in smaller samples, and equal to the plug-in parameter in very large samples. However, the variation in standard errors is larger, especially in smaller samples. That is, using cross-validation risks overfitting.<sup>13</sup> Table 3 examines the performance of cross-validation in real field experiments by re-estimating each treatment effect using cross-validation instead of the plug-in penalty parameter. Comparing the number of variables selected to those presented in Table 1 we confirm that cross-validation tends to select substantially more variables: a mean (median) of 26.96 (13) controls compared to 3.7 (2) controls with the plug-in penalty parameter. The risk of overfitting is seen with a maximum of 264 controls selected. The median standard error is marginally smaller with cross-validation, at 98.4% of the plug-in standard error. However, there are some extreme cases where the standard error is substantially larger. Overall, given that cross-validation does not offer clear improvements, lacks theoretical justification, and has this risk of overfitting, there does not seem to be a compelling case for applied researchers to use cross-validation instead of the plug-in penalty.

Table A2 shows the change in significance when using cross-validation as compared to Ancova and PDS Lasso. Compared to the plug-in penalty parameter, the cross-validation approach leads to just as many

> 12We further set _p_ = 20, _β_ 1 = 0 _._ 3, and _βj_ = 0 _._ 05 for _j ∈_ (2 _,_ 5).

> 13Two other approaches to reduce the risk of over-fitting from cross-validation are the (i) adaptive Lasso (Zou, 2006), or (ii) setting the penalty parameter such that cross-validated error is one standard error larger than the minimum. They can be specified in the _dsregress_ command using the ” _selection(adaptive)_ ” and ” _selection(cv, serule)_ ” options, respectively. Our simulations showed that the adaptive lasso performed similar to simple cross-validation, although tended to still over-select in small samples. The 1 SE rule selected fewer variables than using the plug-in penalty parameter.

17