  We consider the problem of supervised classification, such that the features
that the network extracts match an unseen set of semantic attributes, without
any additional supervision. For example, when learning to classify images of
birds into species, we would like to observe the emergence of features that
zoologists use to classify birds. We propose training a neural network with
discrete top-level activations, which is followed by a multi-layered perceptron
(MLP) and a parallel decision tree. We present a theoretical analysis as well
as a practical method for learning in the intersection of two hypothesis
classes. Since real-world features are often sparse, a randomized sparsity
regularization is also applied. Our results on multiple benchmarks show an
improved ability to extract a set of features that are highly correlated with
the set of unseen attributes.
