semvec/pypi-packages huggingface dataset:
simple-crawler 1.2
my simple crawler
Keywords: crawler

## Install

`pip3 install simple-crawler`

Set environment `AUTO_CHARSET=1` to pass `bytes` to beautifulsoup4 and let it detect the charset.

## Classes

* `URL`: define a URL
* `URLExt`: class to handle `URL`
* `Page`: define a request result of a `URL`
    * `url`: type `URL`
    * `content`, `text`, `json`: response content properties from library `requests`
    * `type`: the response body type, is a enum which allows `BYTES`, `TEXT`, `HTML`, `JSON`
    * `is_html`: check whether is html accorrding to the response headers's `Content-Type`
    * `soup`: `BeautifulSoup` contains html if `is_html`
* `Crawler`: schedule the crawler by calling `handler_page()` recusively

## Example

```
from simple_crawler import *


class MyCrawler(Crawler):
    name = 'output.txt'
    aysnc def custom_handle_page(self, page):
        print(page.url)
        tags = page.soup.select("#container")
        tag = tags and tags[0]
        with open(self.name, 'a') as f:
            f.write(tag.text)
        # do some async call

    def filter_url(self, url: URL) -> bool:
        return url.url.startswith("https://xxx.com/xxx")


loop = get_event_loop(True)
c = MyCrawler("https://xxx.com/xxx", loop, concurrency=10)
schedule_future_in_loop(c.start(), loop=loop)
```

## TODO

* [x] Speed up using async or threading
skidward 0.0.1.dev434
scheduler for python scripts

# Skidward

Scheduler For Python Scripts

## Prerequisites
The project was built with Python 3.6.7 and requires a working Postgres and Redis setup.  
Please refer to your operating system's specific installation instructions per package.

* Python 3.6.7
* Postgres (tested on 11 and up)
* Redis (tested on 5 and up)

## Installation
- Install the requirements

        $ pip install -r requirements.txt

- Create user (called a 'role') in postgres and give it ownership over the new db
**Note:** It is recommended to create a user with a password, even for development purposes.

        $ createuser skidwarduser
        $ createdb skidwarddb -O skidwarduser

The application uses environment variables to offer flexible configuration options.  
Before using the application, review the configuration in the `.env` file to match your desired setup.  
A typical connectionstring looks like this:
    `SQLALCHEMY_DATABASE_URI='postgresql://USER:PASSWORD@localhost/DATABASENAME'` 

- Copy the `.env.default` file to the skidward module and rename to `.env`.  

        $ cp .env.default skidward/.env

- Migrate the database

        $ python -m skidward migrate

- Publish available workers on the namespace in the database

        $ python -m skidward publish-workers

- Create an admin user for use in the application

        $ python -m skidward create-admin USER_EMAIL

#### Development
- Install the development requirements

        $ pip install -r requirements-dev.txt

- Set up the git pre-commit hook required for development

        $ pre-commit install

*Optional:* If you want to install demo-workers from a proxy,
navigate to the root of your virtual environment and create a `pip.conf` file.

        $ touch pip.conf
        $ echo "[global]" >> pip.conf
        $ echo "extra-index-url = URL_TO_PROXY" >> pip.conf

#### Contributing to the documentation
All documentation is built with Sphix and can be found in the `docs` folder,
with pages in `docs/src` and images located in `docs/src/images`.

-  Build the docs

        $ cd docs
        $ make html

- Open the result in a web browser (here Firefox is used)

        $ firefox _build/html/index.html

## Running the application
There's an admin interface provided to communicate with the application,  
and then there's the backend side doing the operations.

### 1. Web interface
- Point Flask to the location of the web application

        $ export FLASK_APP=skidward.web

- Run the default setup (in the same session as the previous command)

        $ flask run

### 2. Scheduler side
- Start a new scheduler process (in the fore- or background)

        $ python -m skidward start-scheduler        # Runs in your session
        $ python -m skidward start-scheduler true   # Creates a new process running in the background