  This paper presents our contribution to SemEval-2021 Task 2: Multilingual and
Cross-lingual Word-in-Context Disambiguation (MCL-WiC). Our experiments cover
English (EN-EN) sub-track from the multilingual setting of the task. We
experiment with several pre-trained language models and investigate an impact
of different top-layers on fine-tuning. We find the combination of Cosine
Similarity and ReLU activation leading to the most effective fine-tuning
procedure. Our best model results in accuracy 92.7%, which is the fourth-best
score in EN-EN sub-track.
