semvec/pypi-packages huggingface dataset:
webpagetester 0.1.5
Basic Tool for automated testing if libraries exists
License: BSD-3-Clause
Requires Python: >=3.7

[![pipeline status](http://gitlab/Willi/webpagetester/badges/master/pipeline.svg)](http://gitlab/Willi/webpagetester/-/commits/master)
[![coverage report](http://gitlab/Willi/webpagetester/badges/master/coverage.svg)](http://gitlab/Willi/webpagetester/-/commits/master)


```
<gitlab-url>/-/settings/ci_cd
```

### Install

````shell
pip install webpagetester
````

### Test

````shell
webpagetester -f tests/files/working_urls.txt -f tests/files/special_urls.txt -f tests/files/wrong_urls.txt -o report.txt
````

### Usage

enter:
``` shell
webpagetester --help
```
to see all current options.

#### Test single URLs

``` shell
webpagetester -p http://google.de -p https://google.de -p http://this.does.not.exit
```
Returns:
``` shell
http://google.de Web site exists
https://google.de Web site exists
http://this.does.not.exit Web site does not exist
Press Enter to close...
```

#### From File

Create a file with an unknown url in each line.

pages.txt:
``` text
http://google.de
https://google.de
http://this.does.not.exit
```

Then use the following command: 

``` shell
webpagetester -f pages.txt -o report.txt
```

Returns:
``` shell
http://google.de Web site exists
https://google.de Web site exists
http://this.does.not.exit Web site does not exist
Press Enter to close...
```

and in report.txt:
``` text
FOUND:
======
http://google.de
https://google.de


NOT FOUND:
==========
http://this.does.not.exit
```

#### Saving a "Screenshot"

Create a file with an unknown url in each line.

Then use the following command: 

``` shell
webpagetester -p https://google.de -s True
```

Returns:
``` shell
http://google.de Web site exists
Press Enter to close...
```

and a ``https#$$google_de.html``-file
Be aware that links still work so to be safe only look at the files without internet!

## Comparing Links
````shell
links -t tests/files/target.txt -c tests/files/compare3.txt -c tests/files/compare2.txt -o result.txt
````


### Testing

```
python setup.py test
```

### Publishing
````shell
python setup.py sdist bdist_wheel
twine upload dist/*
````

### TODO
* page scanner
* onion urls
* vpn
    - stem.torproject.org
    - or https://medium.com/@jasonrigden/using-tor-with-the-python-request-library-79015b2606cb