  We present a novel counterfactual framework for both Zero-Shot Learning (ZSL)
and Open-Set Recognition (OSR), whose common challenge is generalizing to the
unseen-classes by only training on the seen-classes. Our idea stems from the
observation that the generated samples for unseen-classes are often out of the
true distribution, which causes severe recognition rate imbalance between the
seen-class (high) and unseen-class (low). We show that the key reason is that
the generation is not Counterfactual Faithful, and thus we propose a faithful
one, whose generation is from the sample-specific counterfactual question: What
would the sample look like, if we set its class attribute to a certain class,
while keeping its sample attribute unchanged? Thanks to the faithfulness, we
can apply the Consistency Rule to perform unseen/seen binary classification, by
asking: Would its counterfactual still look like itself? If ``yes'', the sample
is from a certain class, and ``no'' otherwise. Through extensive experiments on
ZSL and OSR, we demonstrate that our framework effectively mitigates the
seen/unseen imbalance and hence significantly improves the overall performance.
Note that this framework is orthogonal to existing methods, thus, it can serve
as a new baseline to evaluate how ZSL/OSR models generalize. Codes are
available at https://github.com/yue-zhongqi/gcm-cf.
