Aruana 1.1.1
Aruana is a collection of methods that can be used for simple NLP tasks and for machine learning text preprocessing.
License: Apache


# Aruana NLP for Machine Learning

*  [Aruana](https://aruana.nheeng.com/)

*  [Nhe'eng](https://nheeng.com/)

Aruana is a collection of methods that can be used for simple NLP tasks. It can be used for tasks involving text preprocessing for machine learning. Aruana works mainly with text strings and lists of strings.  

The library is developed in Python 3.

## Installing Aruana

### pip

    $ pip3 install aruana

If you want, you can also install Aruana in a virtual environment:

	$ python -m venv .env

	$ source .env/bin/activate

	$ pip3 install aruana


### Prerequisites

Aruana uses following external Python libraries:

- nltk (3.3)

- tqdm (4.19.5)

- pdoc (0.5.1)

They are all documented in the [requirements.txt](requirements.txt) file.


## Usage examples

To use Aruana, initialize it by choosing one of the three available languages ('en', 'fr', 'pt-br')

	aruana_en = Aruana('en')

### Quick preprocessing

Aruana has the preprocess method, which applies commonly used preprocessed steps on you text.

	text = "At the end of the day, you're solely responsible for your success and your failure. And the sooner you realize that, you accept that, and integrate that into your work ethic, you will start being successful. As long as you blame others for the reason you aren't where you want to be, you will always be a failure."
	preprocessed_text = aruana_en.preprocess(text)
	print(preprocessed_text)

	>>> ['at', 'the', 'end', 'of', 'the', 'day', 'you', 'are', 'sole', 'respons', 'for', 'your', 'success', 'and', 'your', 'failur', 'and', 'the', 'sooner', 'you', 'realiz', 'that', 'you', 'accept', 'that', 'and', 'integr', 'that', 'into', 'your', 'work', 'ethic', 'you', 'will', 'start', 'be', 'success', 'as', 'long', 'as', 'you', 'blame', 'other', 'for', 'the', 'reason', 'you', 'are', 'not', 'where', 'you', 'want', 'to', 'be', 'you', 'will', 'alway', 'be', 'a', 'failur']

If you prefer, you can choose to:

- tokenize the sentence

- stem it

- remove stop words

- pos tag the portuguese sentences

		text = "At the end of the day, you're solely responsible for your success and your failure. And the sooner you realize that, you accept that, and integrate that into your work ethic, you will start being successful. As long as you blame others for the reason you aren't where you want to be, you will always be a failure."

		preprocessed_text = aruana_en.preprocess(text, stem=False, remove_stopwords=True)

		print(preprocessed_text)

		['end', 'day', 'solely', 'responsible', 'success', 'failure', 'sooner', 'realize', 'that', 'accept', 'that', 'integrate', 'work', 'ethic', 'start', 'successful', 'long', 'blame', 'others', 'reason', 'want', 'be', 'always', 'failure']

### List preprocessing

If you have a list of sentences or you are using Pandas, you can pass the entire list for preprocessing by using the preprocess_list method.

	list_of_strings = ['I love you',
						'Please, never leave me alone',
						'If you go, I will die',
						'I am watching a lot of romantic comedy lately',
						'I have to eat icecream' ]

	list_processed = aruana_en.preprocess_list(list_of_strings, stem=False, remove_stopwords=True)

	print(list_processed)

	>>> [['love'], ['please', 'never', 'leave', 'alone'], ['go', 'die'], ['watching', 'lot', 'romantic', 'comedy', 'lately'], ['eat', 'icecream']]

### Defining your own pipeline

Use the single available methods to create a custom pipeline instead of using the quick preprocessing function.

	text = "At the end of the day, @john you're solely responsible for your #success and your #failure. And the sooner you realize that, you accept that, and integrate that into your work ethic, you will start being #successful."
	text = aruana_en.lower_remove_white(text)
	text = aruana_en.expand_contractions(text)
	text = aruana_en.replace_handles(text, 'HANDLE')
	text = aruana_en.replace_hashtags(text, 'HASHTAG')
	text = aruana_en.remove_stopwords(text)
	text = aruana_en.replace_punctuation(text, placeholder='PUNCTUATION')
	text = aruana_en.tokenize(text)
	print(text)

	>>> ['end', 'day', 'PUNCTUATION', 'HANDLE', 'solely', 'responsible', 'HASHTAG', 'HASHTAG', 'PUNCTUATION', 'sooner', 'realize', 'that', 'PUNCTUATION', 'accept', 'that', 'PUNCTUATION', 'integrate', 'work', 'ethic', 'PUNCTUATION', 'start', 'HASHTAG', 'PUNCTUATION']

## Development

### Testing

1. Create a clean test environment

2. Navigate to aruana project on your computer and generate a package using bdist_wheel

		$ python3 setup.py sdist bdist_wheel

3. Install the package

		$ python3 setup.py install

### Docs

Navigate to aruana/aruana and type:

	$ pdoc --html aruana

### Release

Follow the steps below before releasing a new version:

1. Update all necessary documents

2. Generate the package using bdist

3. Install the new version on a clean environment for testing

4. If everything is ok, generate the doc using pdoc

## Contributing

Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.

## Versioning

Use [SemVer](http://semver.org/) for versioning.

## Authors

*  **Wilame Vallantin** - *Initial work* - [Nhe'eng](https://nheeng.com/)

## License

This project is licensed under the Apache License - see the [LICENSE.md](LICENSE.md) file for details

## V. 1.1.1

###  New features

- Adds the random_classification method, useful for random text classification for testing model accuracy
- Adds the replace_with_blob method, useful for creating blobs from a corpus for testing method accuracy
- adds the strings module, with a list of punctuation and diacritic strings
- adds an internal tokenizer
- adds a pos-tagger for portuguese (experimental, version 0.0.1)

### Improvements
- expand_contractions recognizes now more words for portuguese
- Preprocess text now converts emojis to text instead of completely removing them
- Removes NLTK tokenizer and replaces it for an internal tokenizer