|
--- |
|
license: mit |
|
base_model: microsoft/deberta-v3-small |
|
tags: |
|
- regression |
|
model-index: |
|
- name: deberta-v3-small-sp500-edgar-10k-markdown-1024-vN |
|
results: [] |
|
datasets: |
|
- BEE-spoke-data/sp500-edgar-10k-markdown |
|
language: |
|
- en |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# pszemraj/deberta-v3-small-sp500-edgar-10k |
|
|
|
|
|
this predicts the `ret` column of the training dataset, given the `text` column. |
|
|
|
|
|
|
|
<details> |
|
|
|
<summary>Click to expand code example</summary> |
|
|
|
```py |
|
import json |
|
from transformers import pipeline |
|
from huggingface_hub import hf_hub_download |
|
|
|
model_repo_name = "pszemraj/deberta-v3-small-sp500-edgar-10k" |
|
pipe = pipeline("text-classification", model=model_repo_name) |
|
pipe.tokenizer.model_max_length = 1024 |
|
|
|
# Download the regression_config.json file |
|
regression_config_path = hf_hub_download( |
|
repo_id=model_repo_name, filename="regression_config.json" |
|
) |
|
with open(regression_config_path, "r") as f: |
|
regression_config = json.load(f) |
|
|
|
|
|
def inverse_scale(prediction, config): |
|
"""apply inverse scaling to a prediction""" |
|
min_value, max_value = config["min_value"], config["max_value"] |
|
return prediction * (max_value - min_value) + min_value |
|
|
|
|
|
def predict_with_pipeline(text, pipe, config, ndigits=4): |
|
result = pipe(text, truncation=True)[0] # Get the first (and likely only) result |
|
score = result["score"] if result["label"] == "LABEL_1" else 1 - result["score"] |
|
# Apply inverse scaling |
|
scaled_score = inverse_scale(score, config) |
|
return round(scaled_score, ndigits) |
|
|
|
|
|
text = "This is an example text for regression prediction." |
|
|
|
# Get predictions |
|
predictions = predict_with_pipeline(text, pipe, regression_config) |
|
print("Predicted returns (frec):", predictions) |
|
``` |
|
|
|
</details> |
|
|
|
## Model description |
|
|
|
This model is a fine-tuned version of [microsoft/deberta-v3-small](https://huggingface.co/microsoft/deberta-v3-small) on BEE-spoke-data/sp500-edgar-10k-markdown |
|
|
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.0005 |
|
- Mse: 0.0005 |
|
|
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 4 |
|
- eval_batch_size: 4 |
|
- seed: 30826 |
|
- gradient_accumulation_steps: 16 |
|
- total_train_batch_size: 64 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_ratio: 0.05 |
|
- num_epochs: 3.0 |
|
- mixed_precision_training: Native AMP |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Mse | |
|
|:-------------:|:-----:|:----:|:---------------:|:------:| |
|
| 0.0064 | 0.54 | 50 | 0.0006 | 0.0006 | |
|
| 0.0043 | 1.08 | 100 | 0.0005 | 0.0005 | |
|
| 0.0028 | 1.61 | 150 | 0.0006 | 0.0006 | |
|
| 0.0025 | 2.15 | 200 | 0.0005 | 0.0005 | |
|
| 0.0025 | 2.69 | 250 | 0.0005 | 0.0005 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.38.0.dev0 |
|
- Pytorch 2.2.0+cu121 |
|
- Datasets 2.16.1 |
|
- Tokenizers 0.15.2 |