|
--- |
|
tags: |
|
- generated_from_trainer |
|
model-index: |
|
- name: gpt2-arxiv |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# gpt2-arxiv |
|
|
|
A [gpt2](https://huggingface.co/gpt2) powered predictive keyboard trained on ~1.6M manuscript abstracts from the ArXiv. |
|
This model uses https://www.kaggle.com/datasets/Cornell-University/arxiv |
|
|
|
```python |
|
from transformers import pipeline |
|
from transformers import GPT2TokenizerFast |
|
|
|
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2") |
|
llm = pipeline('text-generation',model='pearsonkyle/gpt2-arxiv', tokenizer=tokenizer) |
|
|
|
texts = llm("Directly imaged exoplanets probe", |
|
max_length=50, do_sample=True, num_return_sequences=5, |
|
penalty_alpha=0.65, top_k=40, repetition_penalty=1.25, |
|
temperature=0.95) |
|
|
|
for i in range(5): |
|
print(texts[i]['generated_text']+'\n') |
|
``` |
|
|
|
- *The reflectance of Earth's vegetation suggests* `that large, deciduous forest fires are composed of mostly dry, unprocessed material that is distributed in a nearly patchy fashion. The distributions of these fires are correlated with temperature, and also with vegetation...` |
|
- *Directly imaged exoplanets probe* `the atmospheres of giant planets. The detection of such planets requires high-quality imaging with high contrast and angular resolution, as well as` |
|
- *We can remotely sense an atmosphere by observing its reflected, transmitted, or emitted light in varying geometries. This light will contain information on* `the planetary conditions including atmospheric temperature and cloud properties, which is essential for understanding how the planet interacts with the atmosphere and how it affects the climate. The primary science objective of this paper is to develop a methodology that can be applied to any kind of observation and measurement data, and to provide a framework that enables the detection and characterization of the atmospheres of exoplanets` |
|
|
|
|
|
## Model description |
|
|
|
[GPT-2](https://huggingface.co/transformers/v2.2.0/pretrained_models.html): 12-layer, 768-hidden, 12-heads, 117M parameters |
|
|
|
## Intended uses & limitations |
|
|
|
Coming soon... |
|
- Predictive Keyboard using text generation |
|
- Realtime reference recommendations using nearest neighbors of embeddings |
|
|
|
Be careful when generating a lot of text or when changing the sampling mode of the language model. It can sometimes produce things that are not truthful, e.g., |
|
|
|
- The surface of Mars is composed of a thin layer of water ice, that was discovered by the Cassini spacecraft after its impact on the Earth's surface. |
|
|
|
## Training procedure |
|
|
|
~49 hours on a 3090 training for 1.25M iterations |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 5e-05 |
|
- train_batch_size: 16 |
|
- eval_batch_size: 4 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 10 |
|
|
|
### Framework versions |
|
|
|
- Transformers 4.25.1 |
|
- Pytorch 1.13.1 |
|
- Tokenizers 0.13.2 |
|
|
