semvec/pypi-packages huggingface dataset:
textsplitter 1.0.3
A Python library to split large text into smaller chunks based on the maximum token size and other criteria
Requires Python: >=3.6

# Text Splitter

A Python library to split large text into smaller chunks based on the maximum token size and other criteria.

## Features

- Split text into smaller chunks based on maximum token size
- End chunks at sentence boundaries
- Preserve formatting
- Remove URLs
- Replace entities
- Remove stopwords

## Installation

To install the library, run the following command:

pip install textsplitter


## Usage

Here's a simple example of how to use the TextSplitter:

```python
from textsplitter import TextSplitter

sample_text = "Your sample text goes here..."

text_splitter = TextSplitter(max_token_size=20, end_sentence=True, preserve_formatting=True,
                             remove_urls=True, replace_entities=True, remove_stopwords=True, language='english')

chunks = text_splitter.split_text(sample_text)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i + 1}:\n{chunk}")
get_new_id 0.1.1
Generate new ID as UUID

# Generate new ID as UUID


Generate new ID as UUID

Usage

---

python get_new_id.py : direct call as main program to generate any source ID

import get_new_id.py : as part of add_semantic_identification to semantic record into SPHERES repo

Parameters

---

-s, --source        : Source EX = External Semantic, IN = Internal Semantic, CO = Constructed Semantic,

    BA = Baseline data, SA = Sample data, SP = SPHERES platform ID, RE = Real data

-c, --count         : Return list of sourceID with requested amount of IDs

-p, --platform      : Platform name or Platfrom environment as prefix (i.e. SPHERES, TST)

-v, --version       : Platform version as prefix (i.e. 0001), 4 digits is suggested minimum