semvec/pypi-packages huggingface dataset:
syntaxi 0.1.7
Make your tokenizer more syntax-friendly.
License: MIT
Requires Python: >=3.11,<4.0

![pypi](https://img.shields.io/pypi/v/syntaxi.svg)

# 🚕 syntaxi

Make your tokenizer more syntax-friendly.

Syntaxi will encode capital words using a special shift-token, allowing words to be effectively capital-invariant. "Dog" and "dog" are the same word. Without Syntaxi, your language model needs to learn these words as if they were not the same.

Let your language model learn to think in terms of shift tokens, rather than learning words twice.

## Getting started

### Requirements

Python 3.11+, it's 2024.

Syntaxi only depends on `regex` for Unicode property escapes, and uses HuggingFace's `tokenizers` for convenience.

### Installation

```
pip install syntaxi
```

### Example

Load an existing, pre-trained HuggingFace tokenizer to be patched by Syntaxi.

**Create directly using `Tokenizer.from_pretrained`**
```py
import syntaxi

tokenizer = syntaxi.huggingface_tokenizer("nilq/baby-tokenizer")
encoded = tokenizer.encode("My dog is a Dog, and my Dog is a dog.")

encoded.tokens
# ['[SHIFT]', '▁my', '▁dog', '▁is', '▁a', '▁', '[SHIFT]', '▁dog,', '▁and', '▁my', '▁', '[SHIFT]', '▁dog', '▁is', '▁a', '▁dog.']

tokenizer.decode(encoded.ids)
# "My dog is a Dog, and my Dog is a dog."
```

**Manually patch tokenizer**
```py
import syntaxi
from tokenizers import Tokenizer

tokenizer: Tokenizer = ...

# Original `tokenizer` stays the same.
syntaxi_tokenizer = syntaxi.patched_tokenizer(tokenizer)
```
microprocessors 0.1.8
A simulation of the RiSC-16 (Ridiculously Simple Computer - 16 bit) processor.
License: MIT

# Microprocessors

## Example of Library Use

`````python
from microprocessors.architechture import RiSC16

risc = RiSC16()

# Properly queue instructions
risc.encode_instruction('ADDI', 1, 0, 5)  # R1 = 5
risc.encode_instruction('LUI', 2, None, 1)  # R2 = 1 << (16 - immediate_bits)
risc.encode_instruction('ADD', 3, 1, 2)  # R3 = R1 + R2
risc.encode_instruction('NAND', 4, 1, 2)  # R4 = ~(R1 & R2)
risc.encode_instruction('SW', 3, 0, 10)  # Memory[10] = R3
risc.encode_instruction('LW', 5, 0, 10)  # R5 = Memory[10]
risc.encode_instruction('BEQ', 1, 2, 2)  # if R1 == R2, skip next 2 instructions
risc.encode_instruction('ADDI', 6, 0, 1)  # R6 = 1 (skipped if R1 == R2)
risc.encode_instruction('JALR', 7, 0)  # R7 = PC + 1; PC = R0

# Run all and visalize register at final state
risc.run_program()
risc.visualize_state()
`````

## Documentation
Refer to the in-line comments and method docstrings for detailed usage of each feature.

## Contribution
Contributions are welcome! Feel free to submit pull requests, suggest features, or report bugs.

## License
This library is distributed under the MIT license. See `LICENSE` for more information.

## Contact
- **Author**: Alexandre Le Mercier
- **Date**: February 20, 2024
- **Email**: [alexandre.le.mercier@ulb.be](mailto:alexandre.le.mercier@ulb.be)
- **LinkedIn**: [Alexandre Le Mercier](https://www.linkedin.com/in/alexandre-le-mercier-7b5594283/details/experience/)

Happy coding!