  This paper describes the training process of the first Czech monolingual
language representation models based on BERT and ALBERT architectures. We
pre-train our models on more than 340K of sentences, which is 50 times more
than multilingual models that include Czech data. We outperform the
multilingual models on 7 out of 10 datasets. In addition, we establish the new
state-of-the-art results on seven datasets. At the end, we discuss properties
of monolingual and multilingual models based upon our results. We publish all
the pre-trained and fine-tuned models freely for the research community.
