  Pre-trained language models (LM) have become go-to text representation
encoders. Prior research used deep LMs to encode text sequences such as
sentences and passages into single dense vector representations. These dense
representations have been used in efficient text comparison and embedding-based
retrieval. However, dense encoders suffer in low resource situations. Many
techniques have been developed to solve this problem. Despite their success,
not much is known about why this happens. This paper shows that one cause lies
in the readiness of the LM to expose its knowledge through dense representation
in fine-tuning, which we term Optimization Readiness. To validate the theory,
we present Condenser, a general pre-training architecture based on Transformer
LMs, to improve dense optimization readiness. We show that fine-tuning from
Condenser significantly improves performance for small and/or noisy training
sets.
