semvec/pypi-packages huggingface dataset:
awesome-crawler 1.0.1
Python CLI to crawl any sites in any structures
License: MIT
Requires Python: >=3.7.5

# pycrawler
This is to crawl content of the sites by the *.lxml file(s) as the input.

Supported Python version = 3.7.6

## Installation
```
pip install awesome-crawler [--upgrade]
```

## CLI Usage
* Check version
```
python -m pycrawler --version
```
* Get helps
```
python -m pycrawler --help
```


* To use crawler:prometheus / sinker:mssql
```
python -m pycrawler --target_dir "C:\Users\DAT\Documents\Sources\pycrawler\examples\targets-social" --program "chel"
```
* To use crawler:prometheus / sinker:mongodb
```
python -m pycrawler --target_dir "C:\Users\DAT\Documents\Sources\pycrawler\examples\targets-social" --program "manu"
```
* To use crawler:prometheus / sinker:mongodb / notificator:sendgrid
```
python -m pycrawler --target_dir "C:\Users\DAT\Documents\Sources\pycrawler\examples\targets-social" --program "manc"
```
* To use crawler:prometheus / sinker:api
```
python -m pycrawler --target_dir "C:\Users\DAT\Documents\Sources\pycrawler\examples\targets-social" --program "barc"
```


## CLI build
[See PUBLISH](PUBLISH)


## Development enviroment
#### Virtual enviroment:
```
python -m venv env
```

#### Activate virtual env:
```
WINDOWS:  .\env\Scripts\activate
LINUX:    source env/bin/activate (linux)
```

#### Install dependencies:
```
pip install -r requirements.txt
```

#### Run the tess.py
```
python tess.py
```

#### Config file
[XSD here](pycrawler/config/schema.xsd)


## TODO
* actions in target-field such as: expand
* action order
* mkdocs
* crawler.poseidon: selenium & bs4
* crawler.atlas: scrapy