semvec/pypi-packages huggingface dataset:
sentok 0.1.2
A fast and dynamic python package for converting paragraphs into sentences with customizable thresholds for adaptive sentence segmentation.
License: MIT
Requires Python: >=3.9
Keywords: sentence tokenization, natural language processing, text segmentation


# sentok

**Sentok** is a fast and dynamic Python package for converting paragraphs into sentences. It offers customizable thresholds for adaptive sentence segmentation and is built on top of pandas for high performance and easy adjustment. The package allows you to easily convert paragraphs into a list of sentences or a DataFrame with probability columns.

## Features

- **High Performance**: Efficient handling of large texts.
- **Dynamic Configuration**: Customizable parameters and regular expressions.
- **Simple Logic**: Easy to understand and extend.

## Installation

### Via pip

To install the latest version directly from the GitHub repository, use:

```bash
pip install sentok
```

Or

```bash
pip install git+https://github.com/kothiyarajesh/sentok.git
```

### Building from Source

1. Clone the repository:

    ```bash
    git clone https://github.com/kothiyarajesh/sentok.git
    ```

2. Navigate to the project directory:

    ```bash
    cd sentok
    ```

3. Install the package:

    ```bash
    python setup.py install
    ```

## Usage

### Python Script

Here’s a simple example of how to use the `sentok` library in a Python script:

```python
import sentok

# Display current weights used by the tokenizer
# Uncomment the following line to view the current weights in use:
# print(sentok.get_weights())

# Adjust weights only if necessary for specific use cases
# For example, updating the set of start characters:
# sentok.set_weights({'start_chars': list('ABCDEFGHIJKLMNOPQRSTUVWXYZ')})

# Sample text for sentence tokenization
text = """Natural language processing (NLP) is a captivating domain that merges computer science, artificial intelligence, and linguistics. It empowers computers to comprehend, interpret, and produce human language in a manner that is both useful and insightful. NLP finds application in various fields, such as text analysis, speech recognition, and machine translation. For example, advanced language models like GPT-3 have showcased exceptional skills in generating text that resembles human writing and in answering queries. As technology progresses, NLP continues to advance, enhancing its precision and expanding its scope of applications."""

# Tokenize the sample text into sentences using the default threshold of 0.65
# Adjust the threshold as needed based on your text's quality.
sentences = sentok.sent_tokenize(text, 0.64)

# Print each extracted sentence
for sentence in sentences:
    print('->', sentence)

# Print the total number of sentences extracted
print('Total Sentences:', len(sentences))

# Obtain a DataFrame with tokenization features for further analysis or model training:
df = sentok.get_sent_tokenize_df(text)
print(df)
```

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
MultiFeatures 1.1.0
A versatile Python package with multiple features
License: MIT
Requires Python: >=3.7
Keywords: Tools, Python, 12

## Install

```sh
pip install -U multifeatures
```

## Documentation
  - Documenttation available here: [https://multifeatures.devh.in/](https://multifeatures.devh.in/)

## Example Usage

```py
from MultiFeatures.IndianRailway import confirmtkt
confirmtkt_object = confirmtkt.Confirmtkt()
try:
    live_status_data = confirmtkt_object.live_train_status(train_no="12345", doj="01-01-2023", locale="en")
    print(live_status_data)
except NotAValidTrainNumber as error:
    print(f"Error: {error}")
except InternetUnreachable:
    print("Error: Connectivity issue. Please check your internet connection.")
except HTTPErr:
    print("Error: ran into an issue. Please try again later, or Check the data.")
```