semvec/pypi-packages huggingface dataset:
bearlib-py 0.3.3
Utilities and standardizations for UNCO

# BearLib

## What is BearLib?

BearLib is a library of utilities and standardizations used for writing Python code for UNCO applications and scripts.

## Modules

### `logging`

A module for logging both locally (file and console), as well as remote webhooks (Teams, Discord, Slack, etc)

#### `logging.core`

The core logging functionality. Contains the default file + console code, as well as the ability to load webhooks.

#### `logging.webhooks`

All officially supported webhooks, as well as the `Webhooks` superclass

##### Officially supported webhooks

* Discord
* Teams
* Slack

### `notifiers`

Handlers for sending (non-logging) notifications out to various services

#### `notifiers.email`

SMTP Email support for notifications

### `oracle`

A module for utilities to be used with Oracle SQL databases

#### `oracle.factories`

Some extra cursor rowfactories for returning different types of data besides lists

## Usage

Basic usage with Teams webhook

```py
from bearlib.logging import core, webhooks

_logger = core.Logger(level="WARNING", echo=True, write=False)

teams = webhooks.Teams(
    hook_url="<Teams webhook URL>",
    summary="Example Teams Webhook",
    notify_only=False, # Set to true for no data to be sent
    subtitle="Example usage",
)

_logger.add_webhook(teams)

_logger.log("WARNING", "Something non-critical happened")

_logger.change_level("DEBUG")
_logger.log("DEBUG", "Debugging enabled")

_logger.close_log()
```
color-it 2.1.3
A cross-platform lightweight package for printing colors in the terminal.
Requires Python: >=3.6

# ColorIt
A simple library to add color to your output.

Have you ever wanted to print Colors to the console? I certainly have. There comes a time when you realize it is a necessity. That's why I created ColorIt. ColorIt is a super simple way to print color to the console. 

## How it works
Interally, the library creates custom ANSI sequences with RGB values. This means there are 16 million colors that can be used with `colorit`!

## Installation
To install `colorit`. Run `pip install color-it` on Windows, or `pip3 install color-it` on macOS and Linux, from there it can be imported with `import colorit`

## How to use it

To use ColorIt:

```python
from colorit import *

# Use this to ensure that ColorIt will be usable by certain command line interfaces
init_colorit()

# Foreground
print(color("This text is red", Colors.red))
print(color("This text is orange", Colors.orange))
print(color("This text is yellow", Colors.yellow))
print(color("This text is green", Colors.green))
print(color("This text is blue", Colors.blue))
print(color("This text is purple", Colors.purple))
print(color("This text is white", Colors.white))

# Background
print(background("This text has a background that is red", Colors.red))
print(background("This text has a background that is orange", Colors.orange))
print(background("This text has a background that is yellow", Colors.yellow))
print(background("This text has a background that is green", Colors.green))
print(background("This text has a background that is blue", Colors.blue))
print(background("This text has a background that is purple", Colors.purple))
print(background("This text has a background that is white", Colors.white))

# Custom
print(color("This color has a custom grey text color", (150, 150, 150)))
print(background("This color has a custom grey background", (150, 150, 150)))

# Combination
print(
    background(
        color("This text is blue with a white background", Colors.blue), Colors.white
    )
)

# If you are using Windows Command Line, this is so that it doesn't close immediately
input()
```

As output you get:

![demo](https://user-images.githubusercontent.com/55718659/88487218-1fbcf800-cf51-11ea-8a27-678407774a37.png)

If you would like to try this for yoyr self, you can download `demo.py` and try it out. 

And that's it. That's really all there is to it.

**Additional Note**: `init_colorit()` clears the console, so put this somewhere in your code before printing output.

## Release Log
* Released the first version of ColorIt (v1.0.0)
* Added a license

## Support me on patreon!
Here is the link to my patreon page: https://www.patreon.com/supermazingcoder :D
cd4py 0.1.0
CD4Py: Code De-Duplication for Python
Keywords: python source code de-duplication detection cd4py

# Intro
**CD4Py** is a code de-duplication tool for Python programming language. 
It detects **near** and **exact** duplicate source code files. To train a machine learning model on source code files, 
it is essential to identify and remove duplicate source code files from the dataset. Otherwise, code duplication 
significantly affects the practicality of machine learning-based tools, especially on unseen data.

- [Quick Installation](#quick-installation)
- [Usage](#usage)
  - [Examples](#examples)
- [Approach](#approach)

# Quick Installation
```
$ git clone https://github.com/saltudelft/CD4Py.git & cd CD4Py
$ pip install .
```

# Usage
```
$ cd4py --help
usage: cd4py [-h] --p P --od OD --ot OT [--d D] [--th TH] [--k K] [--tr TR]

Code De-Duplication for Python

optional arguments:
  -h, --help  show this help message and exit
  --p P       Path to Python projects
  --od OD     Output folder to store detected duplicate files.
  --ot OT     Output folder to store tokenized files.
  --d D       Dimension of TF-IDF vectors [default: 2048].
  --th TH     Threshold to identify duplicate files [default: 0.95].
  --k K       Number of nearest neighbor [default: 10].
  --tr TR     Number trees to build the index. More trees gives higher
              precision but slower [default: 20].
```

## Examples
- Run `CD4Py` to identify duplicate files for a Python dataset
```
$ cd4py --p $PYHON_DATASET --ot $TOKENS --od py_dataset_duplicates.jsonl.gz --d 1024
```
Replace `$PYHON_DATASET` with the path to the Python project folders and `$TOKENS` with the path to store 
tokenized project files. Also, note that detected duplicate files will be stored in 
the file `py_dataset_duplicates.jsonl.gz`.

- The following code example shows the removal of duplicate files using the example file `py_dataset_duplicates.jsonl.gz`:
```python
from dpu_utils.utils.dataloading import load_jsonl_gz
import random
# Selects randomly a file from each cluster of duplicate files
clusters_rand_files = [l.pop(random.randrange(len(l))) for l in load_jsonl_gz('py_dataset_duplicates.jsonl.gz')]
duplicate_files = [f for l in load_jsonl_gz('py_dataset_duplicates.jsonl.gz') for f in l]
duplicate_files = set(duplicate_files).difference(set(clusters_rand_files))
```

# Approach
The `CD4Py` code de-duplication tool uses the following procedure to identify duplicate files in a Python code corpus:

1. Tokenize all the source code files in the code corpus using `tokenize` module of Python standard library.
2. Preprocess tokenized source files by only selecting identifier tokens and removing language keywords.
3. Convert pre-processed tokenized files to a vector representation using the TF-IDF method.
4. Perform `k`-nearest neighbor search to find `k` candidate duplicate files for each source code file. 
Next, filter out candidate duplicate files by considering the threshold `t`.
5. Find clusters of duplicate source code files while assuming that similarity is transitive.