_ANDR EE_<sup>_ ́_</sup> _SEPTEMBER 28, 2022_ _10_ _D. Estimation_ The model is estimated non-parametrically, using the Cubist algorithm described by Quinlan (1992); Witten et al. (2016) and implemented by Kuhn et al. (2012).<sup>11</sup> This is a piece-wise linear model that combines decision trees, boosting, and neighborhood smoothing, to capture smoothed versions of Random Foresttype of nonlinearities that can be locally extrapolated. In particular, where a Random Forest or Boosted Trees algorithm can only interpolate (by returning for instance a median within the range of values seen) at terminal nodes, the terminal node in Cubist ends in a linear regression that can extrapolate. This generally also results in smoother transitions around the cutoffs of decision trees, which is more suitable for numeric data. Due to the local linear nature, local extrapolations generally remain reasonably stable, which is a benefit over neural network predictions that can rapidly turn explosive outside observed data intervals.

Cubist models have done well on a variety of prediction problems, often reaching accuracy not far below that of deep learning methods on complex prediction tasks while maintaining full model interpretability and providing often better performance on smaller data sets. See for instance (Morellos et al., 2016; Ng et al., 2019; Andr ́ee, 2021a; Sbahi et al., 2021) for applications. It also runs much faster than other ensembling methods such as Gradient Boosting Machines and eXtreme Gradient Boosting methods (Hagenauer et al., 2019). A final benefit of Cubist is that the full rule structure can be printed so it retains a relatively high degree of transparency compared to other “black-box” machine learning models.

# Training on small data

The central modeling challenge is that the desire for predictions itself stems from a lack of data, and so the learning algorithm has to be optimized to learn efficiently from a small amount of data. An insufficient size of training data is well-known to result in unsatisfactory generalization performance of machine learning algorithms and researchers have suggested a variety of synthetic data techniques to combat the issue.<sup>12</sup>

> 11Cubist was developed as an extension of the better-known M5 regression trees model by incorporating pruning, neighborhood smoothing and boosting. Essentially is uses a computationally efficient strategy to recursively partition the data space and fit simple piece-wise linear prediction models within each partition, whose predictions are combined using neighborhood averaging of local model predictions. The advantages over M5 are that it can produce smoother transitions across numeric outputs, and complete in much faster runtime.

12Niyogi et al. (1998) discussed that the natural way for an intelligent learner to counter the small data problem and successfully generalize is by exploiting prior information that may be available about the domain or that can be learned from prototypical examples. They also show that the notion of using prior knowledge by expanding the training data with virtual data is mathematically equivalent to incorporating the prior knowledge as a regularizer, providing a theoretical motivation for the use of synthetic data. The literature on small data prediction has subsequently shown that synthetic data can help improve a model’s ability to predict rare outcomes (Menardi and Torelli, 2014) and has since suggested a wide variety of sampling techniques that have gone beyond simple up-sampling methods to generate entirely new cases that mix attributes adaptively using non-parametric neighborhood methods (Chawla et al., 2002; Han et al., 2005; Douzas and Bacao, 2019; Douzas et al., 2022). In contrast, models have also