  Capturing word meaning in context and distinguishing between correspondences
and variations across languages is key to building successful multilingual and
cross-lingual text representation models. However, existing multilingual
evaluation datasets that evaluate lexical semantics "in-context" have various
limitations, in particular, (1) their language coverage is restricted to
high-resource languages and skewed in favor of only a few language families and
areas, (2) a design that makes the task solvable via superficial cues, which
results in artificially inflated (and sometimes super-human) performances of
pretrained encoders, on many target languages, which limits their usefulness
for model probing and diagnostics, and (3) no support for cross-lingual
evaluation. In order to address these gaps, we present AM2iCo, Adversarial and
Multilingual Meaning in Context, a wide-coverage cross-lingual and multilingual
evaluation set; it aims to faithfully assess the ability of state-of-the-art
(SotA) representation models to understand the identity of word meaning in
cross-lingual contexts for 14 language pairs. We conduct a series of
experiments in a wide range of setups and demonstrate the challenging nature of
AM2iCo. The results reveal that current SotA pretrained encoders substantially
lag behind human performance, and the largest gaps are observed for
low-resource languages and languages dissimilar to English.
