semvec/pypi-packages huggingface dataset:
removedup 1.0.6
Remove duplicates from parallel corpora
Requires Python: >=3.7

# RemoveDup

A fast, memory efficient Python module to remove duplicates from parallel text corpora.

It's useful for cleaning up datasets that contain duplicate entries for training language models.

## Installation

```bash
pip install removedup
```

## Usage

```python
from removedup import rdup

src, tgt, removed = rdup("source.txt", "target.txt")
print(src, tgt, removed)
# source.txt.dedup
# target.txt.dedup
# <num lines removed>
```

## Notes

Source and target must have the same number of lines. No validation checks are made.

Duplication checks are only made on the source content. If you want to check for duplicates on the target, simply switch the order of the parameters.

## Build

```bash
git clone https://github.com/LibreTranslate/RemoveDup
cd RemoveDup
python setup.py build
```

## Standalone Binary

You can also use removedup as a standalone Windows, macOS or Linux application (but you currently need to build from source, we don't provide binaries).

```
mkdir build
cd build && cmake .. && make -j4
./rdup source.txt target.txt
```

## Contributing

We welcome pull requests!

## License

AGPLv3
freewrite 0.0.2
A textual-based tool for structurng **freewriting**.
Requires Python: >=3.8

# Freewrite

<!--
[![PyPI - Version](https://img.shields.io/pypi/v/hatch-demo.svg)](https://pypi.org/project/freewrite)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/hatch-demo.svg)](https://pypi.org/project/freewrite)
-----
-->

The concept of **freewriting** comes from the book
*Writing Without Teachers* (1973) by Peter Elbow, who has made a summary
publicly available
[here](http://peterelbow.com/pdfs/How_to_Write_Better_through_Freewriting_by_Peter_Elbow.pdf).
The point of **freewritnig** is to write for short uninterrupted bursts,
the emphasis is on continuing writing rather than on what you write.
As one is to keep on writing, there is no time to read what is written nor to
go back and make any corrections.
Mistakes in spelling, grammar, syntax etc. are to be be ignored
during the writing part of the session. *Keep going.*

It is a simple enough exercise and could just as well be done with a pen as with a keyboard.
However, it takes some extra effort to not going back and reading or editing ones text,
effort that can better be focused on writing.
This app prevents this backtracking by not showing you what you write,
until the time is up and you press the stop button.

**Table of Contents**

- [Installation](#installation)
- [License](#license)
- [Usage](#usage)

## Installation


**freewrite** is available thorugh pip and can easily be installed through
 [pipx](https://github.com/pypa/pipx):

```console
pipx install freewrite
```
## Usage

Once installed the app can be started through

```console
freewrite
```

## License

`freewrite` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.