  Contextualized representations based on neural language models have furthered
the state of the art in various NLP tasks. Despite its great success, the
nature of such representations remains a mystery. In this paper, we present an
empirical property of these representations -- "average" approximates "first
principal component". Specifically, experiments show that the average of these
representations shares almost the same direction as the first principal
component of the matrix whose columns are these representations. We believe
this explains why the average representation is always a simple yet strong
baseline. Our further examinations show that this property also holds in more
challenging scenarios, for example, when the representations are from a model
right after its random initialization. Therefore, we conjecture that this
property is intrinsic to the distribution of representations and not
necessarily related to the input structure. We realize that these
representations empirically follow a normal distribution for each dimension,
and by assuming this is true, we demonstrate that the empirical property can be
in fact derived mathematically.
