semvec/pypi-packages huggingface dataset:
SimpleCrawler 1.0.1
Enter something here
Requires Python: >=3.6
Keywords: python

# SimpleCrawler
* This web crawler can be used to crawl a website from the command line or code

# Install
* Python required (3.6+) https://www.python.org/downloads/
* `pip install SimpleCrawler`

OR

* `git clone https://github.com/jackwardell/SimpleCrawler.git`
* `cd SimpleCrawler`
* `python3 -m venv venv`
* `source venv/bin/activate`
* `pip install --upgrade pip`
* `pip install -r requirements.txt`
* `pip install -e .`
* `pytest`
* `crawl https://www.example.com`

# Rules:
This crawler will:
* Only crawl text/html mime-types
* Only crawl pages that return 200 OK HTTP statuses
* Look at /robots.txt and obey by default (but can be overridden)
* Add User-Agent, default value = PyWebCrawler (but can be changed)
* Ignore ?query=strings and #fragments by default (but can be changed)
* Get links from ONLY href value in <a href='/some-link'>click here</a> tags

Todo:
* Nicer logging
* Crawl client errors and server error pages? (Most websites have 404 & 500 handlers which may have links)
* Parse more than just <a> tags and href attrs e.g. src='/some-link'
* Add a scheduler / wait frequency (Politeness)
* Request timeout


# Use
* just type `crawl <url>` into your command line e.g. `crawl https://www.google.com`

```
$ crawl --help
Usage: crawl [OPTIONS] URL

Options:
  -u, --user-agent TEXT
  -w, --max-workers INTEGER
  -t, --timeout INTEGER
  -h, --check-head
  -d, --disobey-robots
  -wq, --with-query
  -wf, --with-fragment
  --debug / --no-debug
  --help                     Show this message and exit.
```

* optional params:
    - "--user-agent" or "-u"
        - what the User-Agent header param is
        - default = 'PyWebCrawler'
    - "--max-workers" or "-w"
        - max number of worker threads
        - default = 1
    - "--timeout" or "-t"
        - how long to wait for new items from work queue before shutting down
        - default = 10
    - "--check-head" or "-t"
        - whether to send HEAD request before sending GET request
        - why? some GET responses will be large (e.g. pdf) and sending HEAD request first will allow crawler to see MIME type before it makes a GET request, therefore averting the GET request altogether
        - default = False
    - "--disobey-robots" or "-d"
        - whether to disobey robots.txt file
        - default = False
    - "--with-query" or "-wq"
        - whether to allow query args e.g. https://www.example.com/?hello=world -> https://www.example.com/ if not --with-query
        - default = False
    - "--with-fragment" or "-wf"
        - whether to allow fragments e.g. https://www.example.com/#helloworld -> https://www.example.com/ if not --with-fragment
        - default = False
    - "--debug/--no-debug", default=False
        - whether to run the crawl, if debug on, then it wont crawl but will pump out crawler config
        - default = False


OR from code

```
from simple_crawler import Crawler

crawler = Crawler()
found_links = crawler.crawl('https://www.example.com/')
```
trafficlight 1.4.7
Start and stop EC2s using tags. now in color!
Requires Python: >=3.6

# trafficlight  
  
start and stop ec2 instances.  
  
### installation  
  
##### install with pip  
  
`pip install trafficlight`  
  
### dependencies   
  
-aws cli (installed and configured)  
-python3     

### flags  
```
  -h, --help     show this help message and exit  
  --key KEY      optional. use a tag key besides Name  
  -R REGION, --region REGION
                 specify a different region
  -g, --green    start.  
  -r, --red      stop.  
  -L, --leave    do not change instance state.
  -y, --yes      automatically approve instance state change.  
  -H, --host     use hostnames instead of ip addresses.  
  -c, --connect  begin an ssh session.  
  -K, --keyfile  write pem keyfile to trafficlight config.  
  -p, --pipe     output instance ip or hostname via standard out to pipe into other commands.  
  -v, --version  show program's version number and exit  
```
  
### usage  
  
show trafficlight help page, including usage and flags.  
`$ trafficlight -h`  
   
switch instance with tag Name:example. if instance is stopped it will start, if it's running it will stop.   
`$ trafficlight example`   
  
connect to ec2 instance with tag Name:example. if instance is stopped it will start the instance first.  
`$ trafficlight example -c`  
  
specify a different region.   
`$ trafficlight example -R us-west-1`   
   
update trafficlight's config file with keyfile for using -c --connect flag.  
`$ trafficlight -K`  
  
start all instances with tag Name:example. if instances are started already they will stay up.   
`$ trafficlight example -g`   
   
stop all instances tag Name:example. if instances are stopped already they will stay stopped.   
`$ trafficlight example -r`   
   
get instance info for tag, but don't change instance state.   
`$ trafficlight example -L`   
   
stop all instances with tag Products:example.    
`$ trafficlight example --key Products -r`   
   
list all ec2s and do nothing.  
`$ trafficlight`  
  
list all ec2s and stop them.  
`$ trafficlight -r`