semvec/pypi-packages huggingface dataset:
robotsparser 0.0.12
Parse robots.txt files and find indexed urls
Keywords: robots.txt,sitemap,crawler,data mining


# robotsparser
Python library that parses robots.txt files

## Functionalities

- Automatically discover all sitemap files
- Unzip gziped files
- Fetch all URLs from sitemaps

## Install
```
pip install robotsparser
```

## Usage

```python
from robotsparser.parser import Robotparser

robots_url = "https://www.example.com/robots.txt"
rb = Robotparser(url=robots_url, verbose=True)
# To initiate the crawl of sitemaps and indexed urls. sitemap_crawl_limit argument is optional
rb.read(fetch_sitemap_urls=True, sitemap_url_crawl_limit=5)

# Show information
rb.get_sitemap_indexes() # returns sitemap indexes
rb.get_sitemaps() # returns sitemaps
rb.get_urls() # returns a list of all urls
```

## Crawl in the background using thread

Crawl in the background and output new entries to file

This is useful for sites where sitemaps are heavily nested and take a long
time to crawl

```python
from robotsparser.parser import Robotparser
import threading

if __name__ == '__main__':
    robots_url = "https://www.example.com/robots.txt"
    rb = Robotparser(url=robots_url, verbose=False)

    sitemap_crawl_proc = threading.Thread(target = rb.read, kwargs = {'fetch_sitemap_urls': False}, daemon=True)
    sitemap_crawl_proc.start()

    while sitemap_crawl_proc.is_alive():
        time.sleep(1)
        print(f"entries_count: {len(rb.get_sitemap_entries())}, indexes: {len(rb.get_sitemap_indexes())}")
        # any logic here to get object data
```
ddfasdf4325-myproject 0.0.2
A small example package
Requires Python: >=3.6


# Make sure you have upgraded version of pip
Windows
```
py -m pip install --upgrade pip
```

Linux/MAC OS
```
python3 -m pip install --upgrade pip
```

## Create a project with the following structure
```
packaging_tutorial/
├── LICENSE
├── pyproject.toml
├── README.md
├── setup.cfg
├── src/
│   └── example_package/
│       ├── __init__.py
│       └── example.py
└── tests/
touch LICENSE
touch pyproject.toml
touch setup.cfg
mkdir src/mypackage
touch src/mypackage/__init__.py
touch src/mypackage/main.py
mkdir tests
```

## pyproject.toml 

This file tells tools like pip and build how to create your project

```
[build-system]
requires = [
    "setuptools>=42",
    "wheel"
]
build-backend = "setuptools.build_meta"
```
build-system.requires gives a list of packages that are needed to build your package. Listing something here will only make it available during the build, not after it is installed.

build-system.build-backend is the name of Python object that will be used to perform the build. If you were to use a different build system, such as flit or poetry, those would go here, and the configuration details would be completely different than the setuptools configuration described below.


# Setup.cfg setup
Using setup.cfg is a best practice, but you could have a dynamic setup file using setup.py

```
[metadata]
name = example-pkg-YOUR-USERNAME-HERE
version = 0.0.1
author = Example Author
author_email = author@example.com
description = A small example package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/pypa/sampleproject
project_urls =
    Bug Tracker = https://github.com/pypa/sampleproject/issues
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir =
    = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

```
# Running the build
### Make sure your build tool is up to date
Windows
```
py -m pip install --upgrade build
```
Linux/MAC OS
```
python3 -m pip install --upgrade build
```


### Create the build
```
py -m build
```













### References
https://packaging.python.org/tutorials/packaging-projects/
patek 0.5.2
A collection of utilities and tools for accelerating pyspark development and productivity.

# **Patek**

 A collection of reusable pyspark utility functions that help make development easier!

## Installation

Patek is available on PyPI and can be installed with pip:

```bash
pip install patek
```

## Usage

------------

### IO Helpers

Patek provides a set of IO helpers to quickly read and write data from/to various sources in PySpark.

#### *Dynamic Delta Table Writer*

The superDeltaWriter function allows you to write data to a Delta table using the merge capability without having to write out every single update and merge condition. This is useful when you have a large number of columns and/or a large number of update conditions.

```python

from patek.io import superDeltaWriter

superDeltaWriter(sparkDataframe, ['key_column1'], 'delta/path', sparkSession, sparkContext, ['update_col1', 'update_col2'])

```

If update columns are not specified, the default is to update all non-key columns that exist in both the source and target tables. Also, if the target table does not exist, it will be created.

#### *Funnel.io Schema to Spark Schema*

The funnelSparkler function allows you to convert a Funnel.io schema to a Spark schema. This is useful to remove ambiguity when reading data from Funnel.io exports into spark dataframes, without having to manually define the schema.

```python
from patek.io import funnelSparkler

dataframe = funnelSparkler('path/to/funnel_schema.json', 'path/to/funnel_export_data', sparkSession, sparkContext, data_file_type='csv')
```

### Utility Functions

Patek provides a set of utility functions to help make development easier.

#### *Determine Key Candidates*

The determine_key_candidates function allows you to determine the key candidates for a given dataframe. This is useful when you have a large number of columns in a dataframe and you want to quickly determine which columns are good candidates for a key.

```python
from patek.utils import determine_key_candidates

key_candidates = determine_key_candidates(sparkDataframe)
print(key_candidates)

# Output:
# a list containing single column key candidates: ['column1', 'column2', 'column3']
# a list containing composite key candidates: [['column1', 'column2'], ['column1', 'column3']]
```

#### *Clean Column Names*

The column_cleaner function allows you to clean column names in a dataframe. It removes special characters and replaces spaces with underscores.

```python
from patek.utils import column_cleaner

# input dataframe columns: ['column?? 1', 'column: 2', 'column-3']

cleaned_dataframe = column_cleaner(sparkDataframe)

# output dataframe columns: ['column_1', 'column_2', 'column_3']
```
pastelaria 0.0.1
CLI for generating API boilerplates
Requires Python: >=3.10

# 🌮 PASTELARIA

![pastelaria](https://raw.githubusercontent.com/thaisribeiro/pastelaria/main/pastelaria.png)

This repository is a cli that automatically creates api boilerplates for some python frameworks

_Made with love by [Thais](https://github.com/thaisribeiro). - CX software engineer at Luizalabs (Magalu)_

# Prerequisites
This repository assumes that Python 3.10 or newer is used.

# Installation

Use `pip` to download project dependencies:

``` shell
pip install pastelaria
```
# Dependencies

```shell
pip install cutie
```
# Command Line Interface (CLI)

A CLI will be installed through the package via pip. To get help on any option, just add the `--help` flag when invoking the CLI.

to get the list of commands do:

```bash
pastelaria --help
``` 
To start building the boilerplate, do:

```bash
pastelaria init
```
Select framework and database options.

⚠️ The project is under construction, due to this some boilerplates are still being built and will be released soon

## Contributing

This is an open source work, feel free to contribute, ideas and criticism are welcome.

First of all, please check out the [CONTRIBUTING](/CONTRIBUTING.md) guide.

Feel free to start a thread about github issues or contact me at thaisribeirodn@gmail.com
## Finally
Thank you for making it this far, we are in the beginning, we expect contributions and patience for the finalization of the CLI