  Multilingual language models have shown decent performance in multilingual
and cross-lingual natural language understanding tasks. However, the power of
these multilingual models in code-switching tasks has not been fully explored.
In this paper, we study the effectiveness of multilingual language models to
understand their capability and adaptability to the mixed-language setting by
considering the inference speed, performance, and number of parameters to
measure their practicality. We conduct experiments in three language pairs on
named entity recognition and part-of-speech tagging and compare them with
existing methods, such as using bilingual embeddings and multilingual
meta-embeddings. Our findings suggest that pre-trained multilingual models do
not necessarily guarantee high-quality representations on code-switching, while
using meta-embeddings achieves similar results with significantly fewer
parameters.
