semvec/pypi-packages huggingface dataset:
cleansetext 1.1.0
A Python library for cleaning text data
License: MIT
Requires Python: >=3.7

# CleanseText

![](https://github.com/aflah02/cleansetext/actions/workflows/python-publish.yml/badge.svg)
![](https://github.com/aflah02/cleansetext/actions/workflows/python-package.yml/badge.svg)

This is a simple library to help you clean your textual data.

## Why do I need this?

Honestly there are several packages out there which do similar things, but they've never really worked well for my use cases or don't have all the features I need. So I decided to make my own.

The API design is made to be readable, and I don't hesitate to create functions even for trivial tasks as they make reaching the goal easier.

## How to Install?

`pip install cleansetext`

## Sample usage

```
from cleansetext.pipeline import Pipeline
from cleansetext.steps import *
from nltk.tokenize import TweetTokenizer
tk = TweetTokenizer()

# Create a pipeline with a list of preprocessing steps
pipeline = Pipeline([
    RemoveEmojis(),
    RemoveAllPunctuations(),
    RemoveTokensWithOnlyPunctuations(),
    ReplaceURLsandHTMLTags(),
    ReplaceUsernames(),
    RemoveWhiteSpaceOrChunksOfWhiteSpace()
], track_diffs=True)

# Process text
text = "@Mary I hate you    and everything about you ...... 🎉🎉 google.com"
text = tk.tokenize(text)

print(text)
# Output: ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '...', '🎉', '🎉', 'google.com']

print(pipeline.process(text))

# Output:
# ['<USER>', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '<URL>']

pipeline.explain(show_diffs=True)

# Output:
# Step 1: Remove emojis from text | Language: en
# Diff: ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '...', '🎉', '🎉', 'google.com'] -> ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '...', 'google.com']
# Step 2: Remove all punctuations from a list of words | Punctuations: !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
# Diff: ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '...', 'google.com'] -> ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '...', 'google.com']
# Step 3: Remove tokens with only punctuations from a list of words | Punctuations: !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
# Diff: ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '...', 'google.com'] -> ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', 'google.com']
# Step 4: Remove URLs and HTML tags from a sentence | Replace with: <URL>
# Diff: ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', 'google.com'] -> ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '<URL>']
# Step 5: Remove usernames from a sentence | Replace with: <USER>
# Diff: ['@Mary', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '<URL>'] -> ['<USER>', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '<URL>']
# Step 6: Remove whitespace from a sentence or chunks of whitespace
# Diff: ['<USER>', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '<URL>'] -> ['<USER>', 'I', 'hate', 'you', 'and', 'everything', 'about', 'you', '<URL>']

```
django-db-cascade-2 0.3.5
Fork of django-db-cascade for Django 2.0.  Optionally use postgres db ON CASCADE DELETE on django foreign keys
License: MIT

# django-db-cascade-2

### Installation for Django 3
`pip install django-db-cascade-2`

### Installation for Django 2
`pip install django-db-cascade-2==0.2.3`

settings.py:
```
DATABASES = {
    'default': {
        'ENGINE': 'django_db_cascade.backends.postgresql_psycopg2',
        # ... etc ...
    }
}
```

### Usage
```
from django.db import models
from django_db_cascade.fields import ForeignKey, OneToOneField
from django_db_cascade.deletions import DB_CASCADE

class Thing(Common):
    account = ForeignKey('self', DB_CASCADE)
```

### Caveats
- DB_CASCADE only supports Postgres
- DB_CASCADE does not support django on_delete signals
- DB_CASCADE will not cascade delete multiple inherited tables as expected
- DB_CASCADE will not trigger CASCADE on another model. E.g. Model A points to model B, via DB_CASCADE. Model B points to model C, via CASCADE. A will cascade delete B, B will django delete C, but __deleting A will not delete C__!
- DB_CASCADE on a ManyToMany of A <---> B, only A_B set records will be cascade deleted (deleting A will not delete B)

### How it works
1. Minimal subclassing of the django postgresql backend and the django ForeignKey field
3. Added a new possible value for ForeignKey's on_delete kwarg, called DB_CASCADE
4. When you use DB_CASCADE, the migration framework will recognize a change, and write new sql
6. example SQL generated:
    ```
    ALTER TABLE mytable ADD CONSTRAINT myconstraint FOREIGN KEY (mycolumn)
    REFERENCES myothertable myothercolumn ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED
    ```

### Future proof
If, and when, DB_CASCADE ever gets into django, editing these generated migrations should be very easy.

Generated migrations:
```
migrations.AlterField(
    model_name='modelname',
    name='fieldname',
    field=django_db_cascade.fields.ForeignKey(on_delete=django_db_cascade.deletions.DB_CASCADE)
)
```

Changing them over, if django ever handles DB_CASCADE natively, might look like:
```
migrations.AlterField(
    model_name='modelname',
    name='fieldname',
    field=django.db.models.ForeignKey(on_delete=models.DB_CASCADE)
)
```

### Ticket
The ticker where django has discussed bringing DB_CASCADE to django:
https://code.djangoproject.com/ticket/21961