semvec/pypi-packages huggingface dataset:
qwen-tokenizer 0.2.0
Lightweight tokenizer for qwen.
License: MIT
Requires Python: >=3.8

# Qwen Tokenizer

## Introduction

Qwen Tokenizer is an efficient and lightweight tokenization libraries which doesn't require heavy dependencies like the `transformers` library, Qwen Tokenizer solely relies on the `tokenizers` and `regex` library, making it a more streamlined and efficient choice for tokenization tasks.

## Installation

To install Qwen Tokenizer, use the following command:

```bash
pip install qwen_tokenizer
```

## Basic Usage

Below is a simple example demonstrating how to use Qwen Tokenizer to encode text:

```python
from qwen_tokenizer import qwen_tokenizer

# Sample text
text = "Hello! 毕老师！1 + 1 = 2 ĠÑĤÐ²ÑĬÑĢ"

# Encode text
result = qwen_tokenizer.encode(text)

# Print result
print(result)
```

### Output

```
[9707, 0, 6567, 107, 243, 101049, 6313, 16, 488, 220, 16, 284, 220, 17, 9843, 254, 71354, 147667, 95199, 29456, 71354, 149472, 71354, 144806]
```

## License

This project is licensed under the MIT License.
rich2md 0.1.0
Convert Rich tables to Markdown tables
License: MIT
Requires Python: <4.0,>=3.10

# Rich2MD

This is a simple tool to convert [Rich](https://github.com/Textualize/rich) tables to Markdown tables.

## Installation

```bash
pip install rich2md
```

## Usage

There are two functions:

- `rich_table_to_df` – converts the Rich `Table` object to a Pandas DataFrame.
- `rich_table_to_md_table` – converts the Rich `Table` object to a Markdown table and returns it as a string.

Example:

```python
from rich.table import Table
from rich2md import rich_table_to_md_table

table = Table("A", "B")
table.add_row("my", "mom")
table.add_row("your", "dad")
table_md = rich_table_to_md_table(table)
print(table_md)
```