Add model documentation

README_model.md

@@ -0,0 +1,55 @@
+---
+language: en
+tags:
+- log-analysis
+- pythia
+- hdfs
+license: mit
+datasets:
+- honicky/log-analysis-hdfs-preprocessed
+metrics:
+- cross-entropy
+- perplexity
+base_model: EleutherAI/pythia-70m
+---
+
+# pythia-70m-hdfs-logs
+
+Fine-tuned Pythia-14m model for HDFS log analysis, specifically for anomaly detection.
+
+## Model Description
+
+This model is fine-tuned from `EleutherAI/pythia-70m` for analyzing HDFS log sequences. It's designed to understand and predict patterns in
+HDFS log data so that we can detect anomalies using the perplexity of the log sequence. THhe HDFS sequence is handy because it has labels
+so we can use it to validate that the model can predict anomalies.
+
+We will use this model to understand the ability of a small model to predict anomalies in a specific dataset.  We will study model scale
+and experiment with tokenization, intialization, data set size, etc. to find a configuration that is minimal in size and fast, but can
+effectively predict anomalies.  We will then attempt build a model that is more robust to different log formats.
+
+- Huggingface Model: [honicky/pythia-14m-hdfs-logs](https://huggingface.co/honicky/pythia-14m-hdfs-logs)
+
+## Training Details
+- Base model: EleutherAI/pythia-70m
+- Dataset: https://zenodo.org/records/8196385/files/HDFS_v1.zip?download=1 + preprocessed data at honicky/log-analysis-hdfs-preprocessed
+- Batch size: 16
+- Max sequence length: 405
+- Learning rate: 0.0001
+- Training steps: 2000
+- Weights and Biases run: https://wandb.ai/honicky/log-analysis-pythia/runs/jomdv9lz
+
+
+## Special Tokens
+- Added `<|sep|>` token for event ID separation
+
+## Intended Use
+This model is intended for:
+- Analyzing HDFS log sequences
+- Detecting anomalies in log patterns
+- Understanding system behavior through log analysis
+
+## Limitations
+- Model is specifically trained on HDFS logs and may not generalize to other log formats
+- Limited to the context window size of 405 tokens
+
+