  We describe a new addition to the WebVectors toolkit which is used to serve
word embedding models over the Web. The new ELMoViz module adds support for
contextualized embedding architectures, in particular for ELMo models. The
provided visualizations follow the metaphor of `two-dimensional text' by
showing lexical substitutes: words which are most semantically similar in
context to the words of the input sentence. The system allows the user to
change the ELMo layers from which token embeddings are inferred. It also
conveys corpus information about the query words and their lexical substitutes
(namely their frequency tiers and parts of speech). The module is well
integrated into the rest of the WebVectors toolkit, providing lexical
hyperlinks to word representations in static embedding models. Two web services
have already implemented the new functionality with pre-trained ELMo models for
Russian, Norwegian and English.
