laurabernardy
/

LuxGPT2-basedGER

Text Generation

lëtzebuergesch

text generation

transfer learning

Inference Endpoints

text-generation-inference

Model card Files Files and versions Community

LuxGPT2-basedGER / README.md

laurabernardy's picture

Update README.md

96c0794 over 1 year ago

|

raw history blame contribute delete

No virus

2.51 kB


	---
	language:
	- "lb"
	license: "mit"
	tags:
	- "luxembourgish"
	- "lëtzebuergesch"
	- "text generation"
	- "transfer learning"
	model-index:
	- name: "LuxGPT2-basedGER"
	results:
	- task:
	type: "text-generation" # Required. Example: automatic-speech-recognition
	name: "Text Generation" # Optional. Example: Speech Recognition
	dataset:
	type: "LuxembourgishTestDataset" # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
	name: "Luxembourgish Test Dataset" # Required. A pretty name for the dataset. Example: Common Voice (French)
	metrics:
	- type: "accuracy" # Required. Example: wer. Use metric id from https://hf.co/metrics
	value: "0.34" # Required. Example: 20.90
	- name: "LuxGPT2-basedGER"
	results:
	- task:
	type: "text-generation" # Required. Example: automatic-speech-recognition
	name: "Text Generation" # Optional. Example: Speech Recognition
	dataset:
	type: "LuxembourgishTestDataset" # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
	name: "Luxembourgish Test Dataset" # Required. A pretty name for the dataset. Example: Common Voice (French)
	metrics:
	- type: "perplexity" # Required. Example: wer. Use metric id from https://hf.co/metrics
	value: "45.89" # Required. Example: 20.90
	---
	## LuxGPT-2 based GER

	GPT-2 model for Text Generation in luxembourgish language, trained on 711 MB of text data, consisting of RTL.lu news articles, comments, parlament speeches, the luxembourgish Wikipedia, Newscrawl, Webcrawl and subtitles. Created via transfer learning with an German base model, feature space mapping from LB on Base feature space and gradual layer freezing.
	The training took place on a 32 GB Nvidia Tesla V100
	- with One Cycle policy for the learning rate
	- with the help of fastai's LR finder
	- for 53.4 hours
	- for 20 epochs and 7 cycles
	- using the fastai library


	## Usage
	```python
	from transformers import AutoTokenizer, AutoModelForCausalLM
	tokenizer = AutoTokenizer.from_pretrained("laurabernardy/LuxGPT2-basedGER")

	model = AutoModelForCausalLM.from_pretrained("laurabernardy/LuxGPT2-basedGER")
	```
	## Limitations and Biases
	See the [GPT2 model card](https://huggingface.co/gpt2) for considerations on limitations and bias. See the [GPT2 documentation](https://huggingface.co/transformers/model_doc/gpt2.html) for details on GPT2.