  The dominant approach in probing neural networks for linguistic properties is
to train a new shallow multi-layer perceptron (MLP) on top of the model's
internal representations. This approach can detect properties encoded in the
model, but at the cost of adding new parameters that may learn the task
directly. We instead propose a subtractive pruning-based probe, where we find
an existing subnetwork that performs the linguistic task of interest. Compared
to an MLP, the subnetwork probe achieves both higher accuracy on pre-trained
models and lower accuracy on random models, so it is both better at finding
properties of interest and worse at learning on its own. Next, by varying the
complexity of each probe, we show that subnetwork probing Pareto-dominates MLP
probing in that it achieves higher accuracy given any budget of probe
complexity. Finally, we analyze the resulting subnetworks across various tasks
to locate where each task is encoded, and we find that lower-level tasks are
captured in lower layers, reproducing similar findings in past work.
