semvec/pypi-packages huggingface dataset:
clean-ffmpeg-python-utils 0.0.2
Easy-to-use ffmpeg python utils with simpler arguments
Keywords: python,ffmpeg,video,audio

# clean_ffmpeg_python_utils

Easy-to-use ffmpeg python utils with simpler arguments

# Installing

Install:

```
pip install clean_ffmpeg_python_utils
```

# Usage

```
clean_ffmpeg_python_utils --help

usage: clean_ffmpeg_python_utils --input_filepath INPUT_FILEPATH [--output_extension OUTPUT_EXTENSION] [--output_filepath OUTPUT_FILEPATH] [--fps FPS]
                           [--start_time START_TIME] [--duration DURATION] [--width WIDTH] [--height HEIGHT] [-h]

optional arguments:
  --input_filepath INPUT_FILEPATH
                        (Path, required)
  --output_extension OUTPUT_EXTENSION
                        (Union[str, NoneType], default=None)
  --output_filepath OUTPUT_FILEPATH
                        (Union[pathlib.Path, NoneType], default=None)
  --fps FPS             (Union[int, NoneType], default=None)
  --start_time START_TIME
                        (Union[int, NoneType], default=None)
  --duration DURATION   (Union[int, NoneType], default=None)
  --width WIDTH         (Union[int, NoneType], default=None)
  --height HEIGHT       (Union[int, NoneType], default=None)
  -h, --help            show this help message and exit
```

# Example 1: Convert mp4 to gif

```
clean_ffmpeg_python_utils --input_filepath openai_rubiks_cube.mp4 --output_extension gif
```

# Example 2: Trim video length

```
clean_ffmpeg_python_utils --input_filepath openai_rubiks_cube.mp4 --start_time 25 --duration 10
```

# Example 3: Modify FPS

```
clean_ffmpeg_python_utils --input_filepath openai_rubiks_cube.mp4 --fps 10
```
pcdviz 0.0.3
point cloud viz
Requires Python: >=3.6

# pcdviz
pcdviz is a tool help to visuliaza pointcloud, labels and user-defined shapes. In addition, it also provides tool functions to generate charts and animations, etc. It's inspired by open3d.

## Install
```
pip3 install pcdviz
```

# Quick start
You can use it in many scenarios, here are some examples.

## Display point cloud
If you only need to display the point cloud, specify the point cloud path and run the following command.
```
pcdviz --pcd=data/kitti/training/velodyne/000003.bin --example
```

> --example means example mode, you can remove it in normal mode

![pointcloud](docs/imgs/pointcloud.png)

## Display multi point cloud
If you want to display multiple point clouds at the same time, such as the results of ground detection, or the results of point cloud registration.

From here on we start to use the configuration file method because it is more flexible.
```
pcdviz --cfg=config/multi_pointcloud.yaml --example
```

> --example means example mode, you can remove it in normal mode

![multi_pointcloud](docs/imgs/multi_pointcloud.png)

## Display point cloud and labels
In the field of deep learning, we want to visualize detection results and compare them with ground truth.
```
pcdviz --cfg=config/frame_visualize.yaml --example
```

> --example means example mode, you can remove it in normal mode

![frame_visualize](docs/imgs/frame_visualize.png)

## Display dataset
If you want to view the whole dataset like KITTI, Nuscenes, Waymo. The first frame is initially displayed, and you can switch to the next frame by pressing the button `N`.
```
pcdviz --cfg=config/dataset_visualize.yaml --example
```

> --example means example mode, you can remove it in normal mode

![dataset_visualize](docs/imgs/dataset_visualize.png)

#### KITTI
KITTI directory structure is as follows
```
kitti
└── training
    ├── calib
    │   ├── 000003.txt
    │   ├── 000004.txt
    │   └── 000005.txt
    ├── label_2
    │   ├── 000003.txt
    │   ├── 000004.txt
    │   └── 000005.txt
    └── velodyne
        ├── 000003.bin
        ├── 000004.bin
        └── 000005.bin
```


## Plan
dataset
- Customize the frame order, which is useful when checking data quality
- Automatically filter based on conditions, for example, only display frames where the number of pedestrians is greater than 3

statistics
- Generate statistical charts for datasets
aiforthechurch 0.5
Package for training and deploying doctrinally correct LLMs.

# AI for the Church
Modern LLMs are rooted in secular value systems that are often misaligned with religious organisations. This PyPI package allows anyone to train and deploying doctrinally correct LLMs based on Llama2. Effectively, we are aligning models to a set of values.

## Model fine-tuning
```python
from aiforthechurch import align_llama2
doctrinal_dataset = "/path/to/csv"
align_llama2(doctrinal_dataset)
```

`aiforthechurch` is integrated with HuggingFace shuch that the aligned model will be automatically pushed to your HuggingFace repo of choice at the end of the training.

At aiforthechurch.org we provide tools for generating doctrinal datasets, a few examples are available at huggingface.co/AiForTheChurch.

## Model inference and deployment
We implemented an inference API in the same format as OpenAI's.

```python
import aiforthechurch
aiforthechurch.Completion.create(denomination="catholic", message="Does Jesus love me?")
```

There is also an asynchronous streaming API, just set `stream=True`.

And you can use our code to create your own inference server with the models
you train by editing the `MODELS` dictionary in
`gloohack/deployment/prod_models.py` with a denomination and a path to your
model in huggingface. Start the server by running the following command on your
machine:

```
python gloohack/deployment/inference.py
```

## Model training requirements
If you wish to train your models using this repo you will need access to a machine with over 16GB of GPU memory and 30GB RAM. The full model weights for Llama2-7B amount to almost 30GB, but we use parameter-efficient fine-tuning (PEFT) LoRA to save memory and avoid any catastrophic forgetting during the fine-tuning procedure.


## References
We leaned heavily on open-source libraries like `transformers`, `peft`, and
`bitsandbytes` for this project.
- Dettmers, Tim, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. "[LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/abs/2208.07339)." *arXiv preprint arXiv:2208.07339*.
- Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. "[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)." *arXiv preprint arXiv:2106.09685*.
PyCustomFocus 3.4.0
Full vectorial calculation of focused electromagnetic fields moduled by a custom phase mask
Requires Python: >=3.8

# PyFocus

Author: Fernando Caprile (fcaprile@gmail.com)

Supervisors: Luciano A. Masullo (lmasullo@df.uba.ar), Fernando D. Stefani (fernando.stefani@df.uba.ar)

Affiliation: Physics Department, Faculty of Natural and Exact Sciences, University of Buenos Aires and Center for Bionanoscience Research (CIBION), National Scientific and Technical Research Council (CONICET)

This is PyFocus, a Python package that provides high-level functions and an user interface to perform full vectorial calculations of the focus of an electromagnetic field that has been modulated by a custom phase mask, while providing the option of simulating the free propagation of these fields under the paraxial approximation and the focus through an interface or a multilayer system.

You can find detailed information about the theorical calculations used in this package and a full description of many common use cases in the following paper: https://doi.org/10.1016/j.cpc.2022.108315. If use this package for your research, please cite this work.

For a quick windows executable program, download and run "PyFocus.exe"

See the full documentation and installation tutorial at: https://pyfocus.readthedocs.io/en/latest/
split-words 0.1.3
Split German words
Requires Python: >=3.8.3,<4.0.0

[![pytest](https://github.com/ffreemt/split-words/actions/workflows/routine-tests.yml/badge.svg)](https://github.com/ffreemt/split-words/actions)[![python](https://img.shields.io/static/v1?label=python+&message=3.8%2B&color=blue)](https://www.python.org/downloads/)[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)[![PyPI version](https://badge.fury.io/py/split-words.svg)](https://badge.fury.io/py/split-words)

# split-words

[`CharSplit`](https://github.com/dtuggener/CharSplit) repacked with `poetry`, published as `pypi` `split-words`--
all credit goes to the original author.
```
pip install split-words
```
```
# replace charsplit with split_words in the following, e.g.
from split_words import Splitter
...
```

## CharSplit - An *ngram*-based compound splitter for German
[url](https://github.com/dtuggener/CharSplit)

Splits a German compound into its body and head, e.g.
> Autobahnraststätte -> Autobahn - Raststätte

Implementation of the method decribed in the appendix of the thesis:

Tuggener, Don (2016). *Incremental Coreference Resolution for German.* University of Zurich, Faculty of Arts.

**TL;DR**: The method calculates probabilities of ngrams occurring at the beginning, end and in the middle of words and identifies the most likely position for a split.

The method achieves ~95% accuracy for head detection on the [Germanet compound test set](http://www.sfs.uni-tuebingen.de/lsd/compounds.shtml).

A model is provided, trained on 1 Mio. German nouns from Wikipedia.

### Usage ###
**Train** a new model:
```bash
training.py --input_file --output_file
```
from command line, where `input_file` contains one word (noun) per line and `output_file` is a json file with computed n-gram probabilities.

**Compound splitting**

In python

```python
>> from charsplit import Splitter
>> splitter = Splitter()
>> splitter.split_compound("Autobahnraststätte")
```
returns a list of all possible splits, ranked by their score, e.g.
```python
[(0.7945872450631273, 'Autobahn', 'Raststätte'),
(-0.7143290887876655, 'Auto', 'Bahnraststätte'),
(-1.1132332878581173, 'Autobahnrast', 'Stätte'), ...]
```
By default, `Splitter` uses the data from the file `charsplit/ngram_probs.json`. If you retrained the model, you may specify a custom file with
```python
>> splitter = Splitter(ngram_path=<json_data_file_with_ngram_probs>)
```