Spaces:
Runtime error
Runtime error
| # GottBERT: a pure German language model | |
| ## Introduction | |
| [GottBERT](http://arxiv.org/abs/2012.02110) is a pretrained language model trained on 145GB of German text based on RoBERTa. | |
| ## Example usage | |
| ### fairseq | |
| ##### Load GottBERT from torch.hub (PyTorch >= 1.1): | |
| ```python | |
| import torch | |
| gottbert = torch.hub.load('pytorch/fairseq', 'gottbert-base') | |
| gottbert.eval() # disable dropout (or leave in train mode to finetune) | |
| ``` | |
| ##### Load GottBERT (for PyTorch 1.0 or custom models): | |
| ```python | |
| # Download gottbert model | |
| wget https://dl.gottbert.de/fairseq/models/gottbert-base.tar.gz | |
| tar -xzvf gottbert.tar.gz | |
| # Load the model in fairseq | |
| from fairseq.models.roberta import GottbertModel | |
| gottbert = GottbertModel.from_pretrained('/path/to/gottbert') | |
| gottbert.eval() # disable dropout (or leave in train mode to finetune) | |
| ``` | |
| ##### Filling masks: | |
| ```python | |
| masked_line = 'Gott ist <mask> ! :)' | |
| gottbert.fill_mask(masked_line, topk=3) | |
| # [('Gott ist gut ! :)', 0.3642110526561737, ' gut'), | |
| # ('Gott ist überall ! :)', 0.06009674072265625, ' überall'), | |
| # ('Gott ist großartig ! :)', 0.0370681993663311, ' großartig')] | |
| ``` | |
| ##### Extract features from GottBERT | |
| ```python | |
| # Extract the last layer's features | |
| line = "Der erste Schluck aus dem Becher der Naturwissenschaft macht atheistisch , aber auf dem Grunde des Bechers wartet Gott !" | |
| tokens = gottbert.encode(line) | |
| last_layer_features = gottbert.extract_features(tokens) | |
| assert last_layer_features.size() == torch.Size([1, 27, 768]) | |
| # Extract all layer's features (layer 0 is the embedding layer) | |
| all_layers = gottbert.extract_features(tokens, return_all_hiddens=True) | |
| assert len(all_layers) == 13 | |
| assert torch.all(all_layers[-1] == last_layer_features) | |
| ``` | |
| ## Citation | |
| If you use our work, please cite: | |
| ```bibtex | |
| @misc{scheible2020gottbert, | |
| title={GottBERT: a pure German Language Model}, | |
| author={Raphael Scheible and Fabian Thomczyk and Patric Tippmann and Victor Jaravine and Martin Boeker}, | |
| year={2020}, | |
| eprint={2012.02110}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CL} | |
| } | |
| ``` | |