  Probing neural models for the ability to perform downstream tasks using their
activation patterns is often used to localize what parts of the network
specialize in performing what tasks. However, little work addressed potential
mediating factors in such comparisons. As a test-case mediating factor, we
consider the prediction's context length, namely the length of the span whose
processing is minimally required to perform the prediction. We show that not
controlling for context length may lead to contradictory conclusions as to the
localization patterns of the network, depending on the distribution of the
probing dataset. Indeed, when probing BERT with seven tasks, we find that it is
possible to get 196 different rankings between them when manipulating the
distribution of context lengths in the probing dataset. We conclude by
presenting best practices for conducting such comparisons in the future.
