semvec/pypi-packages huggingface dataset:
wordpiece-rs 0.1.0
A fast WordPiece tokenizer implementation in Rust with Python bindings
License: MIT
Requires Python: >=3.7
Keywords: nlp, tokenizer, wordpiece, rust, machine-learning

# WordPiece Tokenizer in Rust

A fast implementation of the WordPiece tokenizer in Rust with Python bindings using PyO3. This implementation uses a trie-based approach for O(n) time complexity, making it significantly faster than traditional O(n²) implementations.

## Performance

The tokenizer uses a trie data structure to efficiently find the longest matching subwords, resulting in:
- O(n) time complexity for tokenization (vs O(n²) in naive implementations)
- O(m) space complexity where m is the total size of the vocabulary
- Fast prefix matching using a character-based trie
- Efficient token ID lookup

## Key Features

- Fast tokenization using Rust
- Python bindings via PyO3
- Unicode normalization (NFKC)
- Configurable unknown token and maximum input length
- Support for custom vocabularies

## Installation

### From Source

1. Make sure you have Rust and Python installed
2. Install maturin: `pip install maturin`
3. Build and install:
```bash
cd wordpiece_rs
maturin develop
```

## Usage

```python
import wordpiece_rs

# Create a vocabulary (token -> id mapping)
vocab = {
    "[UNK]": 0,
    "[CLS]": 1,
    "[SEP]": 2,
    "want": 3,
    "##ed": 4,
    "to": 5,
    "go": 6,
    "home": 7,
}

# Initialize the tokenizer
tokenizer = wordpiece_rs.WordPieceTokenizer(vocab)

# Tokenize text
tokens = tokenizer.tokenize("wanted to go home")
print(tokens)  # ['want', '##ed', 'to', 'go', 'home']

# Encode text to token IDs
ids = tokenizer.encode("wanted to go home")
print(ids)  # [3, 4, 5, 6, 7]

# Decode token IDs back to text
text = tokenizer.decode([3, 4, 5, 6, 7])
print(text)  # "wantedtogohome"
```

## Customization

You can customize the tokenizer by providing optional parameters:

```python
tokenizer = wordpiece_rs.WordPieceTokenizer(
    vocab,
    unk_token="<UNK>",  # Default: "[UNK]"
    max_input_chars_per_word=100  # Default: 200
)
```

## License

MIT License
django-social-links-field 0.1.6
A Django form field and widget for managing social media links via models.JSONField
License: MIT
Requires Python: <4.0,>=3.10

# Django Social Links Field

## Overview

`social-links-field` is a Django package that provides a custom model field, form field, and widget for managing social media links with ease. It allows developers to store and manage multiple social media links for a model with built-in validation and a user-friendly interface.


![Admin Panel Screenshot](./social_links_field/screenshot.png?raw=true)

## Features

- Custom JSON-based field for storing social media links
- Predefined social media platforms
- Dynamic form widget for adding/removing social links
- Built-in validation
- Easy integration with Django forms and admin interface



## Installation

Install the package using pip:

```bash
pip install django-social-links-field
```


## Quick Start

### 1. Add to INSTALLED_APPS

In your Django project's `settings.py`:

```python
INSTALLED_APPS = [
    ...
    'social_links_field',
    ...
]
```

### 2. Use in Models

```python
from django.db import models
from social_links_field.models import SocialLinksField

class UserProfile(models.Model):
    name = models.CharField(max_length=100)
    social_links = SocialLinksField()
```

### 3. In Django Admin

The field automatically works with Django admin:

```python
from django.contrib import admin
from .models import UserProfile

@admin.register(UserProfile)
class UserProfileAdmin(admin.ModelAdmin):
    # No special configuration needed
    pass
```

## Supported Social Media Platforms

- Facebook
- Instagram
- Twitter
- LinkedIn
- GitHub
- YouTube
- Custom

Configure the dropdown values from `settings.py`
```
SOCIAL_LINKS_FIELD_MEDIA_TYPES = [
    # value, label
    ("facebook", "Facebook"),
    ("instagram", "Instagram"),
    ("twitter", "Twitter"),
    ("linkedin", "LinkedIn"),
    ("github", "GitHub"),
    ("youtube", "YouTube"),
]

```

## Data Structure

Social links are stored as a list of dictionaries:

```python
[
    {
        'type': 'facebook', 
        'link': 'example_user', 
        'label': 'My Facebook Profile'
    }
]
```

## Form Usage

```python
from django import forms
from .models import UserProfile

class UserProfileForm(forms.ModelForm):
    class Meta:
        model = UserProfile
        fields = ['name', 'social_links']
```

## Customization

### Adding Custom Social Media Platforms

You can extend `SOCIAL_MEDIA_TYPES` in your project:

```python
from social_links_field.models import SOCIAL_MEDIA_TYPES

SOCIAL_MEDIA_TYPES += [
    ('tiktok', 'TikTok'),
    ('telegram', 'Telegram')
]

```
You can customize html/css/js by directly overriding default widget html template
```
social_links_field/social_links_widget.html
```
## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

MIT License