semvec/pypi-packages huggingface dataset:
ncdu_compare 0.0.8
Compare ncdu export files and show difference to find what took most space since previous measurement
Requires Python: >=3.8

# ncdu-compare
Compare ncdu export files and shows differences to find out what consumes most space.

## Installation

~~~
# from pypi
pip3 install ncdu-compare

# ... or from git
pip3 install git+https://github.com/yaroslaff/ncdu-compare.git
~~~

## Example usage

Create ncdu snapshots with command:

~~~bash
# now
$ ncdu /some/path/ -0o /tmp/snapshot-1

# after some time, e.g. daily
$ ncdu /some/path/ -0o /tmp/snapshot-2
~~~

Compare it
~~~bash
$ ncdu-compare /tmp/snapshot-1 /tmp/snapshot-2 | sort -n | tail -n 5
~~~

Example real output (first number is growth in bytes for `sort -n`, may be negative):
~~~
2125273220 DIR /home/username/public_html/client-portal/storage/framework/cache (18.96G => 21.09G (2.13G))
2125273220 DIR /home/username/public_html/client-portal/storage/framework/cache/data (18.96G => 21.09G (2.13G))
2125741830 DIR /home/username/public_html/client-portal/storage/framework (18.97G => 21.10G (2.13G))
2153400708 DIR /home/username/public_html/client-portal/storage (19.29G => 21.44G (2.15G))
3134727368 DIR /home/username/public_html/client-portal/public/alavie/customer_folder (36.76G => 39.90G (3.13G))
3146881276 DIR /home/username/public_html/client-portal/public/alavie (52.62G => 55.77G (3.15G))
4649987199 DIR /home/username/public_html/client-portal/public (158.13G => 162.78G (4.65G))
6803387907 DIR /home/username/public_html/client-portal (177.97G => 184.78G (6.80G))
6805286889 DIR /home/username/public_html (179.92G => 186.73G (6.81G))
~~~

## Invalid unicode filenames
Sometimes ncdu can produce invalid JSON files (see https://dev.yorhel.nl/ncdu/jsonfmt). When loading ncdu files ncdu-compare uses (default) 'replace' mode to handle unicode errors. You can  override it with `-e`/`--error` options, e.g. `--error ignore`. More at python doc to [open()](https://docs.python.org/3/library/functions.html#open).

## See also

My other project [Plus Size](https://github.com/yaroslaff/pluss) to detect changes in short-time (e.g. fast growing log files)

[ncdu-diff](https://github.com/lilydjwg/ncdu-diff) C + Python ncdu fork that can compare and diff results
mauve-text 0.4.0
Implementation of the MAUVE to evaluate text generation
Requires Python: >=3.6

# MAUVE

This is a library built on PyTorch and HuggingFace Transformers to measure the gap between neural text and human text
with the MAUVE measure, 
introduced in [this NeurIPS 2021 paper](https://arxiv.org/pdf/2102.01454.pdf) (Outstanding Paper Award) and [this JMLR 2023 paper](https://arxiv.org/pdf/2212.14578.pdf).


MAUVE is a measure of the gap between neural text and human text. It is computed using the Kullback–Leibler (KL) divergences between the two text distributions in a quantized embedding space of a large language model. MAUVE can identify differences in quality arising from model sizes and decoding algorithms.

### [Documentation Link](https://krishnap25.github.io/mauve/)  [GitHub Link](https://github.com/krishnap25/mauve/)

### _MAUVE is also available via [HuggingFace Evaluate](https://huggingface.co/spaces/evaluate-metric/mauve)!_


**Features**:
- MAUVE with quantization using *k*-means. 
- Adaptive selection of *k*-means hyperparameters. 
- Compute MAUVE using pre-computed GPT-2 features (i.e., terminal hidden state), 
    or featurize raw text using HuggingFace transformers + PyTorch.
- Use with other modalities (e.g. images or audio) by directly passing in pre-computed feature embeddings to our API.

More details can be found in the [documentation](https://krishnap25.github.io/mauve).

## Installation

For a direct install, run this command from your terminal:
```
pip install mauve-text
``` 

## Citation
If you find this package useful, or you use it in your research, please cite the following papers:
```
@article{pillutla-etal:mauve:jmlr2023,
  title={{MAUVE Scores for Generative Models: Theory and Practice}},
  author={Pillutla, Krishna and Liu, Lang and Thickstun, John and Welleck, Sean and Swayamdipta, Swabha and Zellers, Rowan and Oh, Sewoong and Choi, Yejin and Harchaoui, Zaid},
  journal={JMLR},
  year={2023}
}

@inproceedings{pillutla-etal:mauve:neurips2021,
  title={MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers},
  author={Pillutla, Krishna and Swayamdipta, Swabha and Zellers, Rowan and Thickstun, John and Welleck, Sean and Choi, Yejin and Harchaoui, Zaid},
  booktitle = {NeurIPS},
  year      = {2021}
}

@inproceedings{liu-etal:mauve-theory:neurips2021,
  title={{Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals}},
  author={Liu, Lang and Pillutla, Krishna and Welleck, Sean and Oh, Sewoong and Choi, Yejin and Harchaoui, Zaid},
  booktitle={NeurIPS},
  year={2021}
}

```
    
## Acknowledgements
This work was supported by NSF DMS-2134012, NSF CCF-2019844, NSF DMS-2023166, the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), the CIFAR "Learning in Machines & Brains" program, a Qualcomm Innovation Fellowship, and faculty research awards.