semvec/pypi-packages huggingface dataset:
complexity-concept 0.3.0
Llama-based architecture with INL Dynamics Lite stabilization
Requires Python: >=3.8
Keywords: llm transformer dynamics llama pytorch deep-learning

# Complexity Model

A modern transformer architecture with **2024 optimizations** and **Token-Routed MLP** innovation.

## Innovations

### 1. Token-Routed MLP (Original)
Routes tokens to specialized experts based on token ID:

```
Token IDs 0-25K     → Expert 0 (frequent tokens)
Token IDs 25K-50K   → Expert 1
Token IDs 50K-75K   → Expert 2
Token IDs 75K-100K  → Expert 3 (rare tokens)
```

### 2. Flash Attention (SDPA)
Uses PyTorch 2.0+ `scaled_dot_product_attention` for:
- 2-4x faster attention
- O(n) memory vs O(n²)
- Automatic backend selection

### 3. QK Normalization (2024)
Normalizes Q and K before attention:
- Stabilizes training
- Prevents attention collapse
- Used in Gemma, Cohere, etc.

### 4. Sliding Window Attention (Optional)
Mistral-style local attention:
- Efficient for long sequences
- Configurable window size

## Benefits

| Metric | Standard | Complexity |
|--------|----------|------------|
| Attention speed | 1x | 2-4x (Flash) |
| MLP compute/token | 100% | ~25% (1 expert) |
| Training stability | baseline | better (QK Norm) |
| PPL | baseline | better (specialization) |

## Architecture

```
complexity/
├── core/
│   ├── normalization.py    # RMSNorm
│   ├── rotary.py           # RoPE
│   ├── attention.py        # GQA + Flash + QK Norm
│   ├── mlp.py              # Standard SwiGLU
│   ├── token_routed_mlp.py # Token-Routed MLP
│   └── layer.py            # Decoder layer
└── models/
    ├── config.py           # ComplexityConfig
    ├── modeling.py         # ComplexityForCausalLM
    └── utils.py            # create_complexity_model()
```

## Usage

```python
from complexity import create_complexity_model

# Create model with all innovations (default)
model = create_complexity_model("base")

# Forward pass
outputs = model(input_ids, labels=labels)
loss = outputs.loss
```

## Model Sizes

| Size | Params | Hidden | Layers | Experts |
|------|--------|--------|--------|---------|
| tiny | ~15M | 256 | 6 | 4 |
| 20m | ~20M | 320 | 8 | 4 |
| small | ~50M | 512 | 8 | 4 |
| base | ~125M | 768 | 12 | 4 |
| medium | ~350M | 1024 | 24 | 4 |
| large | ~760M | 1536 | 24 | 4 |
| 1b | ~1B | 2048 | 24 | 4 |

## Training

```bash
# Train tokenizer first
python train_tokenizer.py --dataset Pacific-Prime/mixed-inl --vocab-size 100000

# Train model
python train_complexity.py --size small --dataset Pacific-Prime/mixed-inl
```

## Comparison

| Component | Llama | INL-LLM v3 | Complexity |
|-----------|-------|------------|------------|
| Attention | GQA + RoPE | GQA + RoPE | **GQA + RoPE + Flash + QK Norm** |
| MLP | SwiGLU | SwiGLU + MoE | **Token-Routed SwiGLU** |
| Norm | RMSNorm | RMSNorm | RMSNorm |
| Flash Attention | No | No | **Yes (SDPA)** |
| QK Norm | No | No | **Yes** |
| Sliding Window | No | No | **Optional** |

## License

MIT
sebastos 0.1.1
local query&sync of standardebooks
Requires Python: >=3.12

# sebastos: a sync client for Standard Ebooks

The folks over at [Standard Ebooks](https://standardebooks.org/) provide a fabulous collection of public domain books,
beautifully produced.

I recently purchased a Kobo e-reader, and want to load it with some of those ebooks, getting updates as new books become available, and without having to download and copy a bunch of files. This client makes that easy.

## Getting started

Make a directory for `sebastos` to run from. If you have your ereader plugged in and mounted, this can be on the device itself.

For example, on my Mac with my Kobo plugged in and mounted:

```bash
cd /Volumes/KOBOeReader
mkdir sebastos
cd sebastos
uvx sebastos init
```

The `init` subcommand will scan the standardebooks github organisation for ebooks, and download their metadata. This process
will take a few minutes to complete. At any later time, you can use the `update` subcommand to redo this scan. This will only download metadata for new ebooks, so it will be substantially quicker than the initial scan.

Edit the `.sebastos/config.toml` file and set up as many `[[query]]` blocks as you like, to download by author, subject, title, and so on.
Then run `uvx sebastos sync` and matching books will be downloaded. If you've pared back your queries, books may be deleted, although the tool will ask you to confirm before it does so.

If you'd like to see the metadata that queries run against, just run `uvx sebastos list`

## Donations

I'm conscious that Standard Ebooks rely upon donations to operate. You can [donate here](https://standardebooks.org/donate) - please do so if you find some value in this tool.