GEOBERTje

A Domain-Adapted Dutch Language Model Trained on Geological Borehole Descriptions

Description

GEOBERTje is a language model built upon the BERTje architecture, comprising 109 million parameters. It has been further trained using masked language modeling (MLM) on a dataset of approximately 300,000 borehole descriptions in the Dutch language from the Flanders region in Belgium. It can serve as the base language model for a variety of geological applications. For instance, by leveraging the model's understanding of geological terminology and borehole data, professionals can streamline the process of interpreting subsurface information and generating detailed 3D representations of geological structures. This capability opens up new possibilities for improved exploration, interpretation, and analysis in the field of geology.

Hugging Face API

How to use GEOBERTje as a pipeline:

from transformers import pipeline

pipe = pipeline("fill-mask", model="hghcomphys/geobertje-base-dutch-uncased")
pipe("grijs fijn zand met enkele [MASK]")

How to load it directly:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("hghcomphys/geobertje-base-dutch-uncased")
model = AutoModel.from_pretrained("hghcomphys/geobertje-base-dutch-uncased")

Application

To showcase the potential application of GEOBERTje, we fine-tuned it on a limited dataset of 3,000 labeled samples. This fine-tuning allowed the model to classify various lithological categories. For example Grijs kleiig zand, zeer fijn, met enkele grindjes will be classified as main lithology: fijn zand, second lithology: klei, third lithology: grind. Our classifier obtained higher accuracy than conventional rule-based approaches or zero-shot classification using GPT-4.

The figure below shows comparative accuracy of rule-based, GPT-4, and GEOBERTje models in classifying main, secondary, and tertiary lithology from geological drill core descriptions.

Citation

If you use GEOBERTje or fine-tune the model, please include this citation.

@misc{ghorbanfekr2024classificationgeologicalboreholedescriptions,
      title={Classification of Geological Borehole Descriptions Using a Domain Adapted Large Language Model}, 
      author={Hossein Ghorbanfekr and Pieter Jan Kerstens and Katrijn Dirix},
      year={2024},
      eprint={2407.10991},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.10991},

References

Please see this repository for further details on the training of GEOBERTje and its fine-tuning for the lithology classification task.
The reference datasets are available from here.
Check out our paper on arxiv.