  We encounter variables with little variation often in educational data mining
(EDM) and discipline-based education research (DBER) due to the demographics of
higher education and the questions we ask. Yet, little work has examined how to
analyze such data. Therefore, we conducted a simulation study using logistic
regression, penalized regression, and random forest. We systematically varied
the fraction of positive outcomes, feature imbalances, and odds ratios. We find
the algorithms treat features with the same odds ratios differently based on
the features' imbalance and the outcome imbalance. While none of the algorithms
fully solved the problem, penalized approaches such as Firth and Log-F reduced
the scale of the problem. Our results suggest that EDM and DBER studies might
contain false negatives when determining which variables are related to an
outcome. We then apply our findings to a graduate admissions data set and we
propose recommendations for researchers working with the kind of imbalanced
data common to EDM and DBER studies.
