pre-processing 0.4.0.dev1
GPAM text pre-processing
License: Apache-2.0
Requires Python: >=3.7.0

# GPAM Pre-processing

## Installing

**Alert:** Currently only working with python 3.6 and 3.7

To install the dependencies for usage run:

``` bash
$ pip3 install -e .
```

Then download the model `pt_core_news_sm` for spacy lemmatization:

``` bash
$ python3 -m spacy download pt_core_news_sm
```

## Using

Just import the module and use it like:

```python
from pre_processing import Pipeline

text = """
Art. 1º A República Federativa do Brasil,
 formada pela união indissolúvel dos Estados e Municípios e
 do Distrito Federal, constitui-se em Estado Democrático de
 Direito e tem como fundamentos:
"""

result = Pipeline().apply(text, verbose=True)

print(result)
```

Or in batchs (supports parallelizing):

```python
from pre_processing import Pipeline

# enumerate to generate indexes
# Indexes useful for working with pandas
batch = list(enumerate([
    "Art. 1º A República Federativa do Brasil,\
     formada pela união indissolúvel dos Estados e Municípios e\
     do Distrito Federal,constitui-se em Estado Democrático de\
     Direito e tem como fundamentos:",
    "I - a soberania;",
    "II - a cidadania;",
    "III - a dignidade da pessoa humana;",
    "IV - os valores sociais do trabalho e da livre iniciativa;\
     (Vide Lei nº 13.874, de 2019)",
    "V - o pluralismo político.",
    "Art. 2º São Poderes da União, independentes e harmônicos entre si\
     o Legislativo, o Executivo e o Judiciário.",
]))


result = Pipeline().apply_batch(batch)

print(result)
```

## Spellchecker

The spellchecker currently used has been configured for use in **GPAM** from a wrapper of the [fuzzy library](https://github.com/sajari/fuzzy) from [sajari](https://www.sajari.com/) company.

First, we have to train the model that the spellchecker will use
### Training the model for spellchecker

1. Download the ELF binary file (Linux executable):
``` bash
$ wget https://github.com/icaropires/spellchecker-fuzzy-wrapper/releases/download/0.1.0/spellchecker
```

2. Give permission for binary file to be executable:
``` bash
$ chmod +x spellchecker
```

3. Train the model (will generate the `spell_model.json` file):
``` bash
$ ./spellchecker train 2 vocabulary.txt
```
- Where `2` are the training depth, [see more here](https://github.com/sajari/fuzzy).
- And the `vocabulary.txt` is the dictionary of known words.

----
Now you have two options to run the spellchecker:
1. Run spellchecker with `Pipeline.apply()`
2. Run spellchecekr with `Pipeline.apply_batch()`

### 1. To run spellchecker with the apply_batch()
1. First run the `compose-rabbit-master`:
``` bash
$ sudo docker-compose -f docker/compose-rabbit-master.yml up
```

2. Then run the `compose-rabbit-consumer`:
``` bash
$ sudo docker-compose -f docker/compose-rabbit-consumer.yml up
```

3. Just remove the `'spellcheck'` from the `disabled` argument from `apply_batch()`.

4. Batch Distributed **(Optional)**: To run the processes of `apply_batch()` in multiple machines, run the `compose-rabbit-consumer` in other PCs and change the `HOST` environment variable on the compose file, pointing to the **IP** running the `compose-rabbit-master`.

### 2. To run spellchecker with the apply()
1. Run the `compose-http`:
``` bash
$ sudo docker-compose -f docker/compose-http.yml up
```

2. Just remove the `'spellcheck'` from the `disabled_steps` argument from `apply()`.

## Testing

Install all dependencies:

``` bash
$ pip3 install -r requirements.txt
```

Run tests:

``` bash
$ pytest
```

## vocabulary.txt

The vocabulary.txt file contains the words that spellchecker will use 
to check text. This file is divided in three parts separated by line 
breaks, the first part contains the usual vocabulary, the second 
contains the legal vocabulary and the third, the tax terms.

## synonyms.txt

The synonyms.txt file contains words and phrases that have the same meaning, both are synonymous.
The organization of this file is in groups of synonyms, where each group is formed by terms of equal meaning.
Each group is separated from another by line breaks.