We test methods, based on cognitively-simple decision rules, that predict which products consumers select for their consideration sets. Drawing on qualitative research we propose disjunctions-of-conjunctions (DOC) decision rules that generalize well-studied decision models such as disjunctive, conjunctive, lexicographic, and subset conjunctive rules. We propose two machine-learning methods to estimate cognitively-simple DOC rules. We observe consumers' consideration sets for global positioning systems for both calibration and validation data. We compare the proposed methods to both machine-learning and hierarchical-Bayes methods each based on five extant compensatory and non-compensatory rules. On validation data the cognitively-simple DOC-based methods predict better than the ten benchmark methods on an information theoretic measure and on hit rates; significantly so in all but one test. An additive machinelearning model comes close on hit rate. Our results are robust with respect to format by which consideration is measured (four formats tested), sample (German representative vs. US student), and presentation of profiles (pictures vs. text). We close by illustrating how DOC-based rules can affect managerial decisions. Consideration sets, non-compensatory decisions, consumer heuristics, statistical learning, machine learning, revealed preference, conjoint analysis, cognitive complexity, cognitive simplicity, environmental regularity, lexicography, logical analysis of data, decision trees, combinatorial optimization. Two-stage, consider-then-choose decision rules are particularly relevant in the automobile market, but modeling and forecasting such decision rules is of general interest. When consumers face a large number of alternative products, as is increasingly common in today's retail and web-based shopping environments, they typically screen the full set of products down to a smaller, more-manageable consideration set which they evaluate further (e.g., Bronnenberg and Vanhonacker 1996; DeSarbo et al., 1996; Hauser and Wernerfelt 1990; Jedidi, Kohli and DeSarbo, 1996; Mehta, Rajiv, and Srinivasan, 2003; Montgomery and Svenson 1976; Payne 1976; Roberts and Lattin, 1991; Shocker et al., 1991; Wu and Rangaswamy 2003) . Consideration sets for packaged goods are typically 3-4 products rather than the 30-40 products on the market (Hauser and Wernerfelt 1990; Urban and Hauser 2004) . Forecasting consideration sets can explain roughly 80% of the explainable uncertainty in consumer decision making (assuming equally likely choice within the consideration set, Hauser 1978). In complex product categories research suggests that at least some consumers use non-compensatory decision processes when evaluating 3 many products and/or products with many features (e.g., Payne, Johnson 1988, 1993). 2 In this paper we explore machine-learning algorithms based on non-compensatory decision rules that model decisions by consumers in the consideration stage of a consider-thenchoose process. We measure consideration directly for a moderately-complex product, handheld Global Positioning Systems (GPSs) and, assuming a general form of non-compensatory decision rules, we attempt to model the non-compensatory patterns that best predict consumers' consideration decisions. The general form, disjunctions of conjunctions (DOC), is motivated by qualitative data and nests several previously-studied rules. We argue further that modeling and controlling for cognitive simplicity enhances predictive ability. We compare the DOC-based machine-learning algorithms to two sets of benchmarks. The first set includes alternative machine-learning algorithms that assume either compensatory decision rules or previously published non-compensatory decision rules. The second set includes hierarchical Bayes (HB) methods for the same compensatory and non-compensatory rules. In this product category, the proposed DOC-based machine-learning methods predict consideration sets better than the benchmarks using two metrics -hit rates and an information-theoretic measure. In almost all comparisons, predictions are significantly better statistically. We demonstrate that our basic conclusions are robust with respect to format by which consideration is measured (four formats tested), sample (German representative vs. US student), and presentation of profiles (pictures vs. text). We close by illustrating how the modeled noncompensatory patterns affect managerial decisions differently than additive decision rules. We focus on data in which respondents are asked to indicate which of several product 4 profiles (32 in our experiments) they would consider. Respondents are free to select any size consideration set. In some formats respondents classify each profile as considered or not considered; in other formats they do not need to evaluate every profile. We explore situations in which features are described by finitely many levels. Let j index the profiles,  index the levels, f index the features (sometimes called "attributes" in the literature), and h index the respondents. Let J, L, F, and H be the corresponding numbers of profiles, levels, features, and respondents. For ease of exposition only, we do not write J, L, and F as dependent (e.g., L f ). Our models and estimation can (and do) handle such dependency, but the notation is cumbersome. Let  Commonly-studied non-compensatory rules include disjunctive, conjunctive, lexicographic, elimination-by-aspects, and subset conjunctive rules (e.g., Gilbride and Allenby 2004, 2006; Jedidi and Kohli 2005; Montgomery and Svenson 1976; Ordóñez, Benson and Beach 1999; Payne, Bettman, and Johnson 1988; Yee, et. al. 2007 ). Subset conjunctive rules generalize disjunctive and conjunctive rules (Jedidi and Kohli 2005) . For consideration decisions, they also generalize lexicographic rules and deterministic elimination-by-aspects, because any implied ranking of products by lexicographic feature-level orders is indeterminate if we observe only the consideration decision (Hogarth and Karelaia 2005; Johnson, Meyer and Ghose 1989; Montgomery and Svenson 1976; Payne, Bettman, and Johnson 1988; Tversky 1972  Perhaps the most pervasively studied decision rules are additive rules. In an additive rule, consumers consider a profile if its "utility" is above some threshold, T h , which accounts for search and processing costs. If  To study consideration-set decisions we began with a qualitative study that used in-depth interviewing for 38 automobile consumers who were asked to describe their consideration decisions for 100 real automobiles that were balanced to market data. All interviews were videorecorded and the videos were evaluated by independent judges who were blind to any hypotheses about consumers' decision rules (Hughes and Garrett 1990; Perreault and Leigh 1989) . Most respondents made consideration decisions rapidly (89% averaged less than 5 seconds per profile) and most used non-compensatory decision rules (76%). Typically, consumers used conjunctivelike criteria defined on specific levels of features. However, some consumers would consider an automobile if it satisfied at least one of multiple conjunctive criteria (i.e., a disjunction of two or more conjunctions). For example, the following respondent considers automobiles that satisfy either of two criteria. The first criterion is clearly conjunctive (good styling, good interior room, excellent mileage). The second criterion allows cars that are "hotrods." "Hotrods" usually have poor interior room and poor mileage. Depth interviewing is, by necessity, based on a small sample. From the sample we could 7 not determine whether multiple conjunctions were pervasive or were limited to a subset of consumers. However, qualitative interviewing in the handheld GPS category also identified some consumers who used multiple conjunctions. A respondent might be willing to consider a GPS with a B&W screen if the GPS was small and the screen was high resolution, but would require a color screen on a large GPS. Such rules can be written as logical patterns: (B&W screen ∧ small size ∧ high resolution) ∨ (color screen ∧ large size), where ∧ is the logical "and" and ∨ is the logical "or." Patterns might also include negations (¬), for example, a consumer might accept a B&W screen as long as the GPS is less than the highest price of $399: (B&W screen ∧ ¬ $399). To study this phenomenon further, we formalize these qualitative insights with a class of decision rules that generalizes previously-proposed rules. We study rules where a respondent considers a profile if the profile matches one or more target patterns. Because each pattern is a conjunction, these logical rules are disjunctions of conjunctions (DOC). DOC rules generalize disjunctive rules (disjunctions of patterns of size 1), conjunctive rules (patterns of size F), and subset conjunctive rules (patterns of size S).  DOC rules generalize previously proposed non-compensatory decision rules, but they might be too general. For example, any profile can be described by a pattern of size F. Thus, any consideration set of size n can be fit perfectly with a disjunction of n conjunctions of size F. Fortunately, experimental evidence suggests that consumers make consideration decisions with relatively simple rules that enable them to make good decisions while avoiding excess cognitive effort (e.g., Bettman, Luce and Payne 1998; Bröder 2000; Gigerenzer and Goldstein 1996; Gigerenzer and Todd 1999; Hogarth and Karelaia 2005; Payne, Bettman 1988, 1993; Martignon and Hoffrage 2002; Simon 1955; Shugan 1980) . This perspective of simple, efficient, search-and-evaluation rules is consistent with economic theories of consideration-set formation which posit that consumers balance search costs and the option value of utility maximization (Hauser and Wernerfelt 1990; Roberts and Lattin 1991) . To capture this "cognitive simplicity" we define DOC(S) rules as the set of DOC rules with maximum pattern length S. In addition, we either limit the number of patterns, P, or penalize DOC rules that have large P. The basic data we observe, for a set of respondents and profiles, is whether or not a respondent considers a profile (y hj ). We seek to identify the patterns that predict best how respondent h evaluates profiles. Using a calibration sample we seek patterns such that profile j is ob- The number of allowable DOC(S) patterns grows rapidly with S. For example, with the 16 binary features in our empirical test, there would be 32 patterns for S = 1, 512 for S =2, 4,992 for S = 3, and 34,112 for S = 4. There would be almost 20 million patterns of length S = 10. With only 32 binary observations (consider vs. not consider) there is serious concern about overfitting because the vector, h w  , which we seek to estimate, has length equal to this large number of allowable patterns. Machine learning is particularly suited to this pattern-matching task. Qualitative interviews suggest that it was not unreasonable for patterns to be up to length S = 4, which requires we search over 34 thousand patterns to find those that best fit the data. While we might place priors on each pattern and use Bayesian methods, we have not yet been able to develop a Bayesian representation in which the posterior is robust with respect to exogenously-set priors for the large number of parameters. We leave exploration of Bayesian DOC models to future research. Rather than producing posterior probabilities of pattern inclusion, we seek binary indicators of whether or not a pattern is in the best-fit solution. If the data are too noisy or the solution space is too large (even controlling for cognitive simplicity), predictions could over fit the data and predict poorly. To be sensitive to this concern we compare models using predictive tests in which respondents face an entirely new set of profiles and report consideration for those profiles. Although we used cognitive simplicity to motivate small S and P, such constraints or penalties have an alternative interpretation within machine learning -complexity control (e.g., and the complexity-control motivation are consistent with our DOC(S) models -we cannot rule out either with the data in this paper. To further distinguish among potential patterns we use data from the entire sample to help select patterns for respondent h. In an analogy to shrinkage, which enhances accuracy in hierarchical Bayesian models (e.g., Rossi and Allenby 2003), we favor those patterns that fit the largest subset of respondents. While shrinkage alone is sufficient motivation for use in our models, shrinkage is consistent with behavioral theories which suggest that simple rules have evolved because they work well in the general environment in which a sample of consumers often make decisions (e.g., Chase, Hertwig and Gigerenzer 1998). These researchers hypothesize that consumers continue to use similar (simple) rules when faced with new decisions. We now summarize briefly two machine-learning methods. Detailed equations are contained in the Web Appendix. Because we seek the binary vector, We enforce cognitive simplicity (complexity control) by limiting the search to patterns of length S or less and by penalizing pattern length, P. We include shrinkage with terms propor-11 tional to the sum of false positives and false negatives in the sample (sum over all respondents). Formally, our objective function is: DOCMP is equivalent to a set-covering problem and, hence, is an NP-hard problem (Cormen, et. al. 2001 ). Fortunately, efficient greedy approximation algorithms have been In our data, neither leave-one-out-cross-validation nor out-of-sample predictions are 12 particularly sensitive to our choice of "tuning" parameters within ranges that roughly match a priori beliefs. Such robustness is consistent with Evgeniou, Pontil and Toubia (2007). Specifically, we can choose any γ M that is an arbitrarily small number such that sample-level consideration is used only to break ties among patterns. For γ c , cross-validation (and predictive tests) vary little in the range γ c ∈[1, 4.5]. Similarly, we can select a cognitively-simple S to be within ranges that we observe in qualitative interviews (S ~ 2, 3, 4). We report S = 4 for ease of exposition. Logical analysis of data (LAD), which seeks to distinguish "positive" events from "negative" events, is another approach to generate patterns (Boros, et. al. 1997; 2000) . We control cognitive simplicity by limiting the search to at most P patterns of size at most S. We define positive patterns as patterns that match at least one considered profile, but no notconsidered profile. Following the "bottom-up" approach described by Boros, et al, 2000, we begin by generating minimal patterns of length one that match some considered profiles. If such patterns are not contained in any non-considered profile, they are positive patterns. Otherwise, we add aspects to the patterns one by one until we generate positive pattens, or until we reach maximum length (S). We next use a greedy algorithm to identify up to P positive patterns that best fit the data, breaking ties first by giving preference to shorter patterns and then patterns that are positive most frequently in the sample. The union of these positive patterns is a DOC rule. LAD-DOC provides a contrast to DOCMP. It is simpler to formulate and takes less time to run, but shares the characteristics of selecting those patterns that best fit the data subject to cognitive simplicity (S, P) and shrinkage (break ties to fit sample-level consideration). One potential weakness is that our implementation of LAD focuses primarily on avoiding false positives 13 (in the calibration data) rather than a combination of false positives and false negatives. For comparability to DOCMP we set S = 4 and P = 2, but out-of-sample predictions are comparable for P ~ 2, 3, or 4 and S ~ 4 or 5. We choose as benchmarks five decision rules. These rules are estimated with both machine-learning and with hierarchical Bayes methods. The decision rules are: • additive partworth rules • additive q-compensatory rules For the additive and q-compensatory rules we use standard HB choice-based conjoint formulations adapted to our dependent variable (consideration vs. not). We use rejection sampling to enforce the q-compensatory constraint (e.g., Allenby, Arora and Ginter 1995). For subset conjunctive rules we modify an algorithm developed by Gilbride and Allenby (2004) . The modifications reflect differences in data and generalization (S = 1 or F in Gilbride and Allenby 2004). As data, we observe consideration directly while it is a latent construct in the Gilbride-Allenby formulation. To address unordered multi-level features, we do not impose constraints that levels within a feature are ordered. Detailed HB formulations are available in the Web Appendix. 14 For the subset conjunctive rules, we select S = 4 to be consistent with the DOC rules. Predictive tests for other values of S are available from the authors. 2 In addition to detailed formulations, the Web Appendix also contains simulations which compare some of the benchmarks to DOC-based methods on synthetic data. We chose to study GPSs because the number of features and the number of brands available is sufficiently large that we might expect some non-compensatory decision rules. Figure 1 illustrates sixteen features that consumers use to evaluate handheld GPSs. These features were chosen as the most important based on two pretests of 58 and 56 consumers, respectively. Ten Using the sixteen features we generated an orthogonal design of 32 GPS profiles. 4 We then developed four alternative formats by which to measure consideration. These respondent task formats were developed based on qualitative pretests to approximate the shopping environment for GPSs. Each respondent task format was implemented in a web-based survey and pretested extensively with over 55 potential respondents from the target market. At the end of the pretests respondents found the tasks easy to understand and felt that the task formats were reasonable representations of the handheld GPS market. We invited two sets of respondents to complete the web-based tasks: a representative 2 The basic relative comparisons with DOC-based models are similar for S ~ 1, 2, 3, or 4. 3 The simulations are consistent with intuition and are consistent with empirical results in the domain suggested by the empirical data. For example, when the data are generated with a particular decision rule, the estimation models which assume that decision rule tend to predict (out of sample) best. 4 To make the task realistic and to avoid dominated profiles (Johnson, Meyer and Ghose 1989), price was manipulated as a two-level price increment. Profile prices were based on this increment plus additive feature-based costs. We return to the issue of orthogonal designs at the end of this section. sample of German consumers who were familiar with handheld GPSs and a US-based student sample. We first describe results from our primary format using the German sample of representative consumers. We then discuss the other formats, the student sample, and a text-only version. Figure 2 provides screen-shots in English and German for the basic format. A "bullpen" is on the far left. As respondents move their cursor over a generic image in the bullpen, a GPS appears in the middle panel. If respondents click on the generic image, they can evaluate the GPS in the middle panel deciding whether or not to consider it. If they decide to consider the GPS, its image appears in the right panel. Respondents can toggle between current consideration sets and their current not-consider sets. There are many ways in which they can change their mind, for example, putting a GPS back or moving it from the consideration set to the notconsider set, or vice versa. In this format respondents continue until all GPSs are evaluated. Before respondents made consideration decisions, they reviewed screens that described GPSs in general and each of the GPS features. They also viewed instruction screens for the consideration task and instructions that encouraged incentive compatibility. Following the consideration task respondents ranked profiles within the consideration set (data not used in this paper) and then completed tasks designed to cleanse memory. These tasks included short brain-teaser questions that direct respondents' attention away from GPSs. Following the memory-cleansing tasks, respondents completed the consideration task a second time, but for a different orthogonal set of GPSs. These second consideration decisions are validation data and are not used in the estimation of any rules. Respondents were drawn from a web-based panel of consumers maintained by the GfK Group. Initial screening eliminated respondents who had no interest in buying a GPS and no ex-16 perience using a GPS. Those respondents who completed the questionnaire received an incentive of 200 points toward general prizes (Punkte) and were entered in a lottery in which they could win one of the GPSs (plus cash) that they considered. This lottery was designed to be incentive compatible as in Ding (2007) and Ding, Grewal, and Liechty (2005). (Respondents who completed only the screening questionnaire received 15 Punkte.) In total 2,320 panelists were invited to answer the screening questions. The incidence rate (percent eligible) was 64%, the response rate was 47%, and the completion rate was 93%. Respondents were assigned randomly to one of the five task formats (the basic format in Figure   2 , three alternative formats, and a text-only format). After eliminating respondents who had null consideration sets or null not-consider sets in the estimation task, we retained 580 respondents. The average size of the consideration set (estimation data) for the task format in Figure 2 was 7.8 profiles. There was considerable variation among respondents (standard deviation was 4.8 profiles). The average size of the consideration set in the validation task was smaller, 7.2 profiles, but not significantly different. Validation consideration set sizes had an equally large standard deviation (4.8 profiles). PREDICTIVE TESTS Hit rate is an intuitive measure which is used commonly when comparing predictive ability. However, with average consideration sets around 7.2 out of 32 (22.5%), a null model that predicts that no GPSs will be considered will achieve a hit rate of 77.5%. Thus, we follow Srinivasan (1988) , Srinivasan and Park (1997), and Payne, Bettman and Johnson (1993, p. 128) and report the percent improvement relative to a random-prediction null model. Percent improvement is a linear transformation of hit rate, but it is easier to interpret. More critically, the apparent strong performance of "predict nothing considered" suggests that we gain insight with statistics that reward models that actually try to predict consideration. The ability to predict the consideration-set size can reject bad models, but is not sufficient to evaluate a good model. A null model of random prediction (proportional to calibration consideration-set size) predicts the validation consideration-set size accurately but achieves a low hit rate of 65.3% and provides no useful information (0% relative hit-rate improvement). Instead we consider a statistic that is sensitive to false positive predictions, false negative predictions, and predicted consideration-set sizes in the validation data. In particular, we use the Kullback-Leibler divergence (K-L) which measures the expected gain in Shannon's information measure relative to a random model (Chaloner and Verdinelli 1995; Kullback and Leibler 1951; Lindley 1956 ). 5 The K-L percentage is 0% for both the random null model and the "predictnothing-considered" null model. It is 100% for perfect prediction. The K-L percentage rewards models that predict the consideration-set size correctly and favors a mix of false positives and false negatives that reflect true consideration sets over those that do not. It discriminates among models even when the hit rates might otherwise be equal. Together the three statistics, hit rate, relative hit rate improvement, and the K-L percentage, provide a means to assess relative predictive ability (DOC-based models vs. the benchmarks).  Predictive Tests DOC predicts slightly better than DOCMP, but the difference is not significant. Among the benchmarks the additive-rule models predict well, with the machine-learning version significantly better than the HB version on both hit rate and K-L percentage (t = 2.6, p < 0.02; t = 3.7, p < 0.01, respectively). While the DOC-based methods are best or not significantly different than best on all comparisons, the machine-learning additive model is within 1-2 percentage points on hit rate. 6 This is consistent with prior results on the robustness of the linear model for empirical data (e.g., Dawes 1979; Dawes and Corrigan 1974) and consistent with the ability of an additive rule to nest some non-compensatory rules. Estimations based on the DOC generalization predict significantly better than the noncompensatory benchmarks suggesting the generalization improves predictions for at least some of our respondents. 7 The unconstrained additive models, which can represent both qcompensatory and many of non-compensatory models, predict better than the q-compensatory models on both measures, significantly so for the machine-learning algorithms (t = 2.1, p < 0.04 for hit rates; t = 9.4, p < 0.01 for K-L). At the level of the individual respondent, some respondents are fit much better with an unconstrained model and some much better with a qconstrained model. Future research might investigate correlates of these individual differences. For brevity we do not elaborate further on comparisons among the benchmarks themselves. Our data are available for readers who wish to explore machine learning. HB, or other methods for the benchmark rules. 6 LAD-DOC is significantly better than the best (machine-learning) additive model on both hit rate and K-L divergence (t = 2.4, p < 0.02; t = 4.6, p < 0.01), DOCMP is better, but not quite significantly so, on hit rate and significantly better on K-L divergence (t = 1.9, p = 0.06; t = 4.1, p < 0.01). One reason the additive model does less well on the K-L percentage is that it under-predicts the consideration-set size. We examine the predictive ability of the additive model further in the next section. 7 We note the poor performance of the machine-learning subset conjunctive model with S = 16. With S = 16 and a goal of choosing 0 vs. 1 for w hp , the subset-conjunctive integer program tends to over fit the calibration data. Although DOCMP and LAD-DOC are designed to favor cognitive simplicity, unconstrained estimation could conceivably predict better. We re-estimated DOCMP with the γs equal to zero and LAD-DOC without the S and P constraints. For both models, the hit rates are significantly better for the penalized/constrained estimation (p < 0.01 vs. 75.7% DOCMP without γs; p < 0.01 vs. 80.4% LAD-DOC without constraints, respectively). Cognitive simplicity also improves the K-L percentage, but the improvements are not quite significant (p < 0.16 vs. 29.6%; p = 0.07 vs. 32.5%, respectively for unconstrained DOCMP and LAD-DOC). These results are consistent with an hypothesis that predictions improve when cognitive simplicity is enforced, although the marginal significance for K-L percentages suggests that the cognitive-simplicity hypothesis is worth further testing in other contexts. Despite the large number of potential patterns, DOC-based estimation chose relatively simple rules for our data. LAD-DOC predictions do not improve significantly, and often degrade, as we increase either pattern length (S) or the number of patters (P). For DOCMP, 7.1% of the respondents are represented as using two patterns; the remainder with a single pattern. It is interesting that the increased flexibility of the DOC-based estimation methods seems to improve predictive ability relative to alternative non-compensatory rules and their corresponding estimation methods even though only 7.1% of the respondents are modeled with two patterns. There has been significant research in marketing on efficient experimental designs for resolution of the design and, if there were no errors, we should be able to identify DOC patterns accordingly. However, when profiles are removed aspects may no longer be uncorrelated and patterns may not be defined uniquely. As a mild test, we re-estimated three models, DOCMP, machine learning additive, and HB additive, with only 17 of 32 most-popular profiles (#'s 16-17 were tied). DOCMP remained significantly better on the K-L percentages and best or not significantly different than best on hit rates, even though we are now estimating the models with approximately half the observations per respondent. Until the issue of optimal DOC-consideration experimental designs is resolved, the performance of DOC-based estimation methods remains a conservative predictive test. Improved or adaptive experimental designs might improve performance. DOC-based estimation appears to predict hit rates well and provide information (K-L
An integer programming approach is applied in REF to learn disjunctions of conjunctions, providing promising results based on a Bayesian method.
Disjunctions of Conjunctions, Cognitive Simplicity and Consideration Sets
Journal of Marketing Research
Mathematics