|
--- |
|
language: |
|
- nl |
|
thumbnail: >- |
|
geobertje.png |
|
tags: |
|
- GEOBERTje |
|
--- |
|
|
|
# GEOBERTje |
|
**A Domain-Adapted Dutch Language Model Trained on Geological Borehole Descriptions** |
|
|
|
<img src="geobertje.png" width="400"> |
|
|
|
|
|
## Description |
|
GEOBERTje is a language model built upon the [BERTje](https://github.com/wietsedv/bertje) architecture, comprising 109 million parameters. |
|
It has been further trained using masked language modeling (MLM) on a dataset of approximately 300,000 borehole descriptions |
|
in the Dutch language from the Flanders region in Belgium. |
|
It can serve as the base language model for a variety of geological applications. |
|
For instance, by leveraging the model's understanding of geological terminology and borehole data, |
|
professionals can streamline the process of interpreting subsurface information and generating detailed 3D representations of geological structures. |
|
This capability opens up new possibilities for improved exploration, interpretation, and analysis in the field of geology. |
|
|
|
|
|
## Hugging Face API |
|
How to use GEOBERTje as a pipeline: |
|
```python |
|
from transformers import pipeline |
|
|
|
pipe = pipeline("fill-mask", model="hghcomphys/geobertje-base-dutch-uncased") |
|
pipe("grijs fijn zand met enkele [MASK]") |
|
``` |
|
|
|
How to load it directly: |
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("hghcomphys/geobertje-base-dutch-uncased") |
|
model = AutoModel.from_pretrained("hghcomphys/geobertje-base-dutch-uncased") |
|
``` |
|
|
|
## Application |
|
To showcase the potential application of GEOBERTje, we **fine-tuned** it on a limited dataset of 3,000 labeled samples. |
|
This fine-tuning allowed the model to classify various lithological categories. |
|
For example `Grijs kleiig zand, zeer fijn, met enkele grindjes` will be classified as main lithology: `fijn zand`, second lithology: `klei`, third lithology: `grind`. |
|
Our classifier obtained higher accuracy than conventional rule-based approaches or zero-shot classification using GPT-4. |
|
|
|
The figure below shows comparative accuracy of rule-based, GPT-4, and GEOBERTje models in classifying |
|
main, secondary, and tertiary lithology from geological drill core descriptions. |
|
<img src="accuracy.png" hgiht="400"> |
|
|
|
## Citation |
|
If you use GEOBERTje or fine-tune the model, please include this citation. |
|
```bibtex |
|
@misc{ghorbanfekr2024classificationgeologicalboreholedescriptions, |
|
title={Classification of Geological Borehole Descriptions Using a Domain Adapted Large Language Model}, |
|
author={Hossein Ghorbanfekr and Pieter Jan Kerstens and Katrijn Dirix}, |
|
year={2024}, |
|
eprint={2407.10991}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2407.10991}, |
|
``` |
|
|
|
## References |
|
- Please see [this](https://github.com/VITObelgium/geobertje) repository for |
|
further details on the training of GEOBERTje and its fine-tuning for the lithology classification task. |
|
- The reference datasets are available from [here](https://huggingface.co/datasets/hghcomphys/geological-borehole-descriptions-dutch). |
|
- Check out our paper on [arxiv](https://arxiv.org/abs/2407.10991). |
|
|
