Similar to the other publicly available LSG models, this model was created using the LSG conversion script found here then MLM pretrained on a sample of the OSCAR and BookCorpus datasets:
https://github.com/ccdv-ai/convert_checkpoint_to_lsg