semvec/pypi-packages huggingface dataset:
fst 0.0.2
FST in Python3 - NLP

Home-page: https://github.com/RishabhBhatnagar/fst
Author: Rishabh Bhatnagar
Author-email: bhatnagarrishabh4@gmail.com
License: UNKNOWN
Description: FST in Python3 - NLP
        
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.5
Description-Content-Type: text/markdown
make-datasetfolder 0.0.1
A utility to create a PyTorch DatasetFolder from any .csv or .tsv file with file path and class data.
Requires Python: >=3.5

# make-datasetfolder
A utility to create a PyTorch DatasetFolder from any .csv or .tsv file with file path and class data.

## Use Case
In PyTorch, the `DataFolder` and `ImageFolder` classes provide a convenient interface for computer vision datasets structured as such:

```
root/class_x/xxx.ext
root/class_x/xxy.ext
root/class_x/xxz.ext

root/class_y/123.ext
root/class_y/nsdf3.ext
root/class_y/asd932_.ext
```

This utility transforms any dataset with a table containing file paths and class labels into this format.

## Example
Suppse you have `dataset.csv` of the form:
```
sample,class,some_feature,another_feature
img-0001.jpg,0,foo,bar
some/relative/directory/img-0002.jpg,1,foo,bar
...
```

Running `make-dataset-folder -p sample -l class dataset.csv output` will create a folder `output` with the following structure:

```
output/0/img-0001.jpg
output/1/img-0002.jpg
...
```

Using the `-m` flag will move images rather than copy them. This could be useful for large datasets that shouldn't be duplicated on disk.

## Usage
```
usage: make-datasetfolder [-h] [-p PATH_COLUMN] [-l LABEL_COLUMN] [-m] [-f]
                          [-t THREADS]
                          input output

positional arguments:
  input                 Path to input .csv or .tsv
  output                Path to output directory.

optional arguments:
  -h, --help            show this help message and exit
  -p PATH_COLUMN, --path-column PATH_COLUMN
                        Column name or index with file paths (default: 0).
  -l LABEL_COLUMN, --label-column LABEL_COLUMN
                        Column name or index with labels (default: 1).
  -m, --move            Move files instead of copying.
  -f, --force           Overwrite output directory if it already exists.
  -t THREADS, --threads THREADS
                        Number of threads to use (default: number of CPU
                        cores)
```
flexi-hash-embedding 0.0.3
PyTorch Extension Library of Optimized Scatter Operations
Keywords: pytorch,scatter,groupby,embedding,hashing,variable,fixed

# Flexi Hash Embeddings

This PyTorch Module hashes and sums variably-sized dictionaries of 
features into a single fixed-size embedding. 
Feature keys are hashed, which
is ideal for streaming contexts and online-learning
such that we don't have to memorize a
mapping between feature keys and indices.
Multiple variable-length features are grouped by example
and then summed. Feature embeddings are scaled by their
values, enabling linear features rather than just one-hot
features.

Uses the wonderful [torch scatter](https://github.com/rusty1s/pytorch_scatter)
library and sklearn's feature hashing under the hood.

So for example:

```python
>>> from flexi_hash_embedding import FlexiHashEmbedding
>>> X = [{'dog': 1, 'cat':2, 'elephant':4},
         {'dog': 2, 'run': 5}]
>>> embed = FlexiHashEmbedding(dim=5)
>>> embed(X)
tensor([[  1.0753,  -5.3999,   2.6890,   2.4224,  -2.8352],
        [  2.9265,   5.1440,  -4.1737, -12.3070,  -8.2725]],
       grad_fn=<ScatterAddBackward>)
```

## Example
Frequently, we'll have data for a single event or user
that's, for example, in a JSON blob format which lends itself
to features which may be missing, incomplete or never before seen.
Furthermore, there may be a variable number of features defined.
This use case is ideal for feature hashing and for groupby summing
of feature embeddings.

In this example we have a total of six features across our whole dataset,
but we compute three vectors, one for every input row:

![img](https://i.imgur.com/OBkiM7T.png)


In the example above we have a total of six features but they're
spread out across three clients. The first client has three active
features, the second client two features 
(and only one feature that overlaps with the first client)
and the third client has a single feature active.
`Flexi Hash Embeddings` returns three vectors, one for each client,
and not six vectors even though there are six features present.
The first client's vector is a sum of three feature vectors 
(plus_flag, age_under, luxe_flag) 
while the second client's vector is a sum of just two feature vectors
(lppish, luxe_flag)
and the third client's vector is just a single feature.

## Speed

Online feature hasing and groupby are relatively fast. 
For a large batchsize of 4096 with on average 5 features per row
equals 20,000 total features. This module will hash that many features
in about 20ms on a modern MacBook Pro.

## Installation

Install from PyPi do `pip install flexi-hash-embedding`

Install locally by doing `git@github.com:cemoody/flexi_hash_embedding.git`.

## Testing

```bash
>>> pip install -e .
>>> py.test
```

To publish a new package version:

```bash
python3 setup.py sdist bdist_wheel
twine upload dist/*
pip install --index-url https://pypi.org/simple/ --no-deps flexi_hash_embedding
```
flashfreeze 0.1
A modern rewrite of FlashBack

# Flashfreeze

A modern rewrite of flashbake

This is *very* rough, but it works.

You can now install via pip:

`pip install flashfreeze`


Copy `config.yaml.sample` to `~/.flashfreeze/config.yaml` and edit to your liking.

To run:
`flashfreeze`

From the python interpreter:

```python
>>> from flashfreeze import FlashFreeze
>>> f = FlashFreeze()
>>> f.run()
```

This isn't even alpha, folks. Use at your own risk. But if you have issues/bugs/suggestions, please let me know.