  Neural topic models can augment or replace bag-of-words inputs with the
learned representations of deep pre-trained transformer-based word prediction
models. One added benefit when using representations from multilingual models
is that they facilitate zero-shot polylingual topic modeling. However, while it
has been widely observed that pre-trained embeddings should be fine-tuned to a
given task, it is not immediately clear what supervision should look like for
an unsupervised task such as topic modeling. Thus, we propose several methods
for fine-tuning encoders to improve both monolingual and zero-shot polylingual
neural topic modeling. We consider fine-tuning on auxiliary tasks, constructing
a new topic classification task, integrating the topic classification objective
directly into topic model training, and continued pre-training. We find that
fine-tuning encoder representations on topic classification and integrating the
topic classification task directly into topic modeling improves topic quality,
and that fine-tuning encoder representations on any task is the most important
factor for facilitating cross-lingual transfer.
