  Question Answering (QA) tasks are used as benchmarks of general machine
intelligence. Therefore, robust QA evaluation is critical, and metrics should
indicate how models will answer any question. However, major QA datasets have
skewed distributions over gender, profession, and nationality. Despite that
skew, models generalize -- we find little evidence that accuracy is lower for
people based on gender or nationality. Instead, there is more variation in
question topic and question ambiguity. Adequately accessing the generalization
of QA systems requires more representative datasets.
