  Cross-lingual word embeddings (CLWE) have been proven useful in many
cross-lingual tasks. However, most existing approaches to learn CLWE including
the ones with contextual embeddings are sense agnostic. In this work, we
propose a novel framework to align contextual embeddings at the sense level by
leveraging cross-lingual signal from bilingual dictionaries only. We
operationalize our framework by first proposing a novel sense-aware cross
entropy loss to model word senses explicitly. The monolingual ELMo and BERT
models pretrained with our sense-aware cross entropy loss demonstrate
significant performance improvement for word sense disambiguation tasks. We
then propose a sense alignment objective on top of the sense-aware cross
entropy loss for cross-lingual model pretraining, and pretrain cross-lingual
models for several language pairs (English to German/Spanish/Japanese/Chinese).
Compared with the best baseline results, our cross-lingual models achieve
0.52%, 2.09% and 1.29% average performance improvements on zero-shot
cross-lingual NER, sentiment classification and XNLI tasks, respectively.
