semvec/pypi-packages huggingface dataset:
textraer 0.1.0
Text processor with classification using DataBERT.
License: MIT
Requires Python: >=3.6

# Textraer: Dataset Mention Detection in Documents

We introduce "textraer" which leverages a fine-tuned BERT model, specifically DataBERT, to identify sentences in documents that mention datasets. It's designed to process documents provided via URLs, extract text content sentence by sentence, and classify each sentence to determine whether it mentions a dataset.

## Installation

Before using textraer, ensure you have all necessary dependencies installed. This project requires Python 3.x and the following Python libraries: `transformers`, `datasets`, `nltk`, `PyPDF2`, `tqdm`, and `requests`.

You can install these dependencies using pip:

```bash
pip install transformers datasets nltk PyPDF2 tqdm requests
```

## Initialization

First, import DocumentProcessor from textraer and initialize it with the necessary parameters:

```python
from textraer import DocumentProcessor

organization = 'your_huggingface_org'
json_cache_dir = './cache'
tokenizer_model = 'bert-base-uncased'
model_path = 'jamesliounis/DataBERT'

processor = DocumentProcessor(organization, json_cache_dir, tokenizer_model, model_path)
```

Replace `'your_huggingface_org'` with your Hugging Face organization name and adjust the `json_cache_dir`, `tokenizer_model`, and `model_path` as needed.

## Extracting and Classifying Text

### From a PDF URL

To process a PDF document from a URL, extract its text content sentence by sentence, and classify each sentence, use the following methods:

```python
pdf_url = 'https://arxiv.org/pdf/1706.03762.pdf'  # Example PDF URL
mode = 'sent'  # Extract text sentence by sentence

# Extract sentences from the PDF
sentences = processor.get_doc_from_url(pdf_url, mode)

# Classify each sentence
classifications = processor.classify(sentences)

# Print classification results
for sentence, classification in zip(sentences, classifications):
    print(f"Sentence: {sentence}")
    print(f"Classification: {classification}")
```

### Creating a Dataset
If you want to create a dataset from the processed documents and push it to the Hugging Face Hub:

```python
# Assuming 'sentences' contains your extracted sentences
processor.create_dataset(sentences, mode='sent')
```

This method will push the sentences as a dataset to your specified Hugging Face organization.

### Understanding the Output

The classify method returns a list of dictionaries. Each dictionary contains two keys: 'label' and 'score'. The 'label' key can be either 'LABEL_0' or 'LABEL_1', where 'LABEL_1' indicates a mention of a dataset. The 'score' key provides the confidence level of the classification.
wmicprocsdict 0.10
Converts the output of a powershell (Get-...)/wmic command to a dict
License: MIT
Keywords: process, data


# Converts the output of a powershell (Get-...)/wmic command to a dict

## pip install wmicprocsdict

### Tested against Windows 10 / Python 3.11 / Anaconda

```py
Args:
    cmd (str): The command to execute.
    convert_dtypes_with_ast (bool, optional): Whether to convert column data types to their appropriate types using the ast module. Defaults to True.
    format_powershell (bool, optional): Format to: f'powershell "{cmd} | Format-Table *"'. Defaults to False.
    cols (int): Number of columns for the console. Defaults to 9999999.
    lines (int): Number of lines for the console. Defaults to 1.

Returns:
    dict: A dictionary representing the command output.

Example:

from wmicprocsdict import get_dict_from_command

d1 = get_dict_from_command(
    cmd="Get-Process",
    convert_dtypes_with_ast=True,
    format_powershell=True,
    cols=9999999,
    lines=1,
)
d2 = get_dict_from_command(
    cmd=f'powershell "Get-Process | Format-Table *"',
    convert_dtypes_with_ast=True,
    format_powershell=False,
    cols=9999999,
    lines=1,
)
e1 = get_dict_from_command(
    cmd="wmic process",
    convert_dtypes_with_ast=True,
    format_powershell=False,
    cols=9999999,
    lines=1,
)

d3 = get_dict_from_command(
    cmd="Get-Service",
    convert_dtypes_with_ast=True,
    format_powershell=True,
    cols=9999999,
    lines=1,
)

d4 = get_dict_from_command(
    cmd=f'powershell "Get-Service | Format-Table *"',
    convert_dtypes_with_ast=True,
    format_powershell=False,
    cols=9999999,
    lines=1,
)
print(d1)
print(d2)
print(e1)
print(d3)
print(d4)

# {0: {'Id': 56, 'PriorityClass': 'Normal', 'FileVersion': (6, 6, 0, 0), 'HandleCount': 430, 'WorkingSet': 32923648, 'PagedMemorySize': 38731776, 'PrivateMemorySize': 38731776, 'Virt...
# {0: {'Id': 56, 'PriorityClass': 'Normal', 'FileVersion': (6, 6, 0, 0), 'HandleCount': 430, 'WorkingSet': 32923648, 'PagedMemorySize': 38731776, 'PrivateMemorySize': 38731776, 'Virt...
# {0: {'CommandLine': '', 'CreationClassName': 'Win32_Process', 'CreationDate': '20240325180030.969217-180', 'CSCreationClassName': 'Win32_ComputerSystem', 'CSName': '...
# {0: {'RequiredServices': {}, 'CanPauseAndContinue': False, 'CanShutdown': False, 'CanStop': False, 'DisplayName': 'Agent Activation Runtime_c5578', 'DependentServices': {}, 'Machin...
# {0: {'RequiredServices': {}, 'CanPauseAndContinue': False, 'CanShutdown': False, 'CanStop': False, 'DisplayName': 'Agent Activation Runtime_c5578', 'DependentServices': {}, 'Machin...

```