semvec/pypi-packages huggingface dataset:
pagecrawler 1.1.2
A simple webscraper
Requires Python: <4.0,>=3.11

# PageCrawler

## How to use
### _request

- call the _request() function, it will first try a request with the request libary and then with selenium
- fill out these keywords: url: str, keyword: str, headers: dict = None, soup:bool=False, max_retry:int=2, wait:int=0
- Explanation:
	- url : request url
	- keyword: the keyword that should be in the website to know whether or not it got the right website, use '' to ignore
	- headers: request header in dicit form, use {} for no headers, leave empty for basic request header
	- soup : Whether or not returned as a soup object
	- max_retry: how often it reties the request (boath the normal and selenium) to get a response containing the keyword

### multi_request
- calls the _request in multiprocessing
- the first argument just uses a list of lists of these 3 arguments: [url, keyword, headers] (lenght of list determines how many request are done)
- new argument: process: int = 1, just determines how many processes are called at the same time
- the rest are just the same as _request, but apply to every request