semvec/pypi-packages huggingface dataset:
reppy2 0.3.6
Replacement robots.txt Parser in pure Python
License: MIT License
Keywords: utilities

Replaces the built-in robotsparser with a
RFC-conformant implementation that supports modern robots.txt constructs like
Sitemaps, Allow, and Crawl-delay. Main features:

- Memoization of fetched robots.txt
- Expiration taken from the `Expires` header
- Batch queries
- Configurable user agent for fetching robots.txt
- Automatic refetching basing on expiration

This is a patched fork of the last pure Python version that
works on Python 2 and 3.