  Large Pre-trained Language Models (PLMs) have become ubiquitous in the
development of language understanding technology and lie at the heart of many
artificial intelligence advances. While advances reported for English using
PLMs are unprecedented, reported advances using PLMs in Hebrew are few and far
between. The problem is twofold. First, Hebrew resources available for training
NLP models are not at the same order of magnitude as their English
counterparts. Second, there are no accepted tasks and benchmarks to evaluate
the progress of Hebrew PLMs on. In this work we aim to remedy both aspects.
First, we present AlephBERT, a large pre-trained language model for Modern
Hebrew, which is trained on larger vocabulary and a larger dataset than any
Hebrew PLM before. Second, using AlephBERT we present new state-of-the-art
results on multiple Hebrew tasks and benchmarks, including: Segmentation,
Part-of-Speech Tagging, full Morphological Tagging, Named-Entity Recognition
and Sentiment Analysis. We make our AlephBERT model publicly available,
providing a single point of entry for the development of Hebrew NLP
applications.
