Introduction Many marketing problems require the analyst to understand the interactions of a large number of potentially inter-related variables. For example, grocery retailers rely heavily on accurate sales forecasts at SKU level when making business decisions in a wide range of areas including marketing, production, inventory, and finance etc. Sales and promotional effects in any one SKU are potentially affected by marketing and sales activities in a large number of other categories – in other words, there are intra and inter-category variables that may affect the target variable(s). However, identifying important variables from such a large set of possibilities poses a serious modelling challenge- it is the subject of this paper. In a retail forecasting system, product sales history, intra-category promotional schedules, and inter-category promotional schedules are all potential rich sources of information which may influence forecasting accuracy. When building product sales forecasting models, a series of related but fundamental questions must be answered: which sources of information should be inputted into the forecasting model? To what extent do different sources of information contribute to forecasting accuracy improvements? And critically, how to manipulate the high dimensional information to generate better forecasts? The main challenge to be faced is that the dimensionality of promotional explanatory variables grows very rapidly when cross-product promotional information is considered, potentially much larger than the length of SKU sales history. The model may be easily or even cannot be estimated. To build a forecasting model for a SKU, when considering both intra- and inter-category promotional interactions, the number of candidate explanation variables is usually in the order of tens of thousands. With high dimensionality, important predictors can be highly correlated with some unimportant ones, and the maximum spurious correlation also grows with dimensionality (Fan and Lv, 2008). Traditional methods which deal with the problem of high dimensionality include the subset selection method, the penalized L-1 likelihood method, and the information summary approach. The subset selection method and the penalized L-1 likelihood method both try, but using different mechanisms, to find out the most influential variables affecting the dependent variable. However, in the retail context, store managers may promote similar products
Figure 5 MAE improvements of ADL-intra-all and ADL-inter-all over ADL-own Categories such as Cold cereal, Salty snacks and Soup achieve limited forecasting improvements from extra information. In the category Yogurt and Milk, however, both models improve the forecasts over the ADL-own model significantly. In order to show the value of intra- and inter-category information at category level, in Figure 5, we illustrate the MAE improvements of ADL-intra-all and ADL-inter-all over among different categories. In categories, such as Frankfurters, Margarine/Butter, Carbonated beverages, Milk and Peanut butter, the contribution from intercategory information is relatively large, ranging from 12% to 78%, compared with that in other categories. For Mayonnaise Coffee and Cold cereal, including the intercategory information in the model could even worsen the forecasts. An explanation is that the useful predictors from other categories may be too weak to compensate for the loss by including the extra volume of noisy information from these categories. In order to investigate whether the proposed LASSO Granger is an effective way to identify the category level interactive structure, we compare the forecasting results of the proposed LASSO Granger with the results of a fully connected structure based on under rolling scheme. The full connection means that when forecasting the sales of SKUs in one category, all other 14 categories are considered as influential categories. The comparison results are illustrated in Figure 6. All the MAEs of LASSO Granger across categories are smaller or equal to that of using structure of full connections. This means that connections (Figure 3) identified by LASSO Granger enhance the model’s forecasting abilities by reducing the redundant noisy data for some categories. Figure 6 Forecasting comparisons between Full connection and LASSO Granger Table 5 one week ahead forecasting comparison between one-stage and three-stage LASSO Model Estimation Scheme MAE RMSE MASE AvgRelMAE MPE ADL-inter-top5 one-stage fixed 7.024 15.024 0.783 1.013 5.04