semvec/pypi-packages huggingface dataset:
spark-connectby 1.0.4
connectby hierarchy query in spark
Requires Python: >=3.7,<4.0

# spark-connectby
Spark currently does not support hierarchy query `connect by` as of version 3.5.0. 
And there is a [PR](https://github.com/apache/spark/pull/40744) opened to support recursive CTE query. But that is not available yet.  

This is an attempt to add `connectby` method to [DataFrame](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.html)

# Concept 
Hierarchy query is one of the important feature that many traditional relational databases, such as Oracle, DB2, My SQL, 
Snowflake, [Redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_CONNECT_BY_clause.html), etc.,
would support directly or alternatively by using recursive CTE. 

Example in Redshift:
```sql
select id, name, manager_id, level
from employee
start with emp_id = 1
connect by prior emp_id = manager_id;
```

With this library, we can use `connectby()` on `Dateframe`:
```python
from spark_connectby.query import connectby
from pyspark.sql import SparkSession

schema = 'emp_id string, manager_id string, name string'
data = [[1, None, 'Carlos'],
        [11, 1, 'John'],
        [111, 11, 'Jorge'],
        [112, 11, 'Kwaku'],
        [113, 11, 'Liu']
        ]
df = spark.createDataFrame(data, schema)
df2 = df.connectby(prior='emp_id', to='manager_id', start_with='1')
df2.show()
```
With result:
```
+------+-----+----------+------+
|emp_id|level|manager_id|  name|
+------+-----+----------+------+
|     1|    1|      null|Carlos|
|    11|    2|         1|  John|
|   111|    3|        11| Jorge|
|   112|    3|        11| Kwaku|
|   113|    3|        11|   Liu|
+------+-----+----------+------+
```

# Installation
Python Version >= 3.7 
```
$ pip install --upgrade spark-connectby
```

# Usage
```python
from spark_connectby.query import connectby

df = ...

df.connectby(prior='emp_id', to='manager_id', start_with='1')   # start_with `emp_id` as 1

df.transform(connectby, prior='emp_id', to='manager_id', start_with='1')   # or by using df.transform() method

df.connectby(prior='emp_id', to='manager_id')   # without start_with, it will go through each node

df.connectby(prior='emp_id', to='manager_id', level_col='the_level')  # level column name other than `level`

df.connectby(prior='emp_id', to='manager_id', start_with=['1', '2'])  # start_with a list of top nodes ids. 

```
rst-pypi-ref 0.4.0
reStructuredText custom role to add ref to PyPI.
Requires Python: >= 3.9

============
rst-pypi-ref
============

reStructuredText custom role to refer PyPI packages.

Description
===========

This is python package to provide features to refer to PyPI in docutils (and more).

Included features:

* reStructuredText role
* Sphinx extension

Installation
============

.. code-block:: console

   pip install rst-pypi-ref

Usage
=====

When you write ``:pypi:`rst-pypi-ref``` into reStructuredText source,
this appends ref to PyPI URL for ``rst-pypi-ref``.

Supporting syntax
-----------------

* Package name only: ``:pypi:`rst-pypi-ref```
* Explicit version: ``:pypi:`rst-pypi-ref==0.2.0```
* Change display text: ``:pypi:`PyPI Link <rst-pypi-ref>```

  * There package name in between ``<`` and ``>``.

Simple usage
------------

.. code-block:: console

   $ echo ':pypi:`rst-pypi-ref`' | python -m rst_pypi_ref.cli
   <document source="<stdin>">
       <paragraph>
           <reference refuri="https://pypi.org/project/rst-pypi-ref">
               rst-pypi-ref

With Sphinx
-----------

.. code-block:: python

   extensions = [
       "rst_pypi_ref.sphinx",
   ]

For other docutils app
----------------------

In entrypoint (before parse reST source), Call ``rst_pypi_ref.core.bootstrap``.

.. code-block:: python

   from rst_pypi_ref.core import bootstrap

   bootstrap()

Contibuting
===========

See `CONTIBUTING note <https://github.com/attakei-lab/rst-pypi-ref/blob/main/CONTRIBUTING.rst>`_.
geoserver-api 0.1.0
Python API for configuring and interacting with GeoServer. Wraps around the official HTTP Rest API
Requires Python: >=3.10,<3.13

# GeoServer Config

Configuration for geoserver and additional resources needed to host stuff there.

## Docker in EC2

### EC2 Setup

Launch an instance using [this user data](config/ec2/user_data.sh). It is recommended the instance to have >=8Gb RAM and >=30Gb ephemeral storage. A good alternative is the `t3.large` which costs ~60 USD/month as of 02/10/2023.

Connect to your instance and add docker to user group by running:

```bash
sudo usermod -aG docker $USER
newgrp docker # restarts docker
```

### Launch Geoserver

Pull the geoserver image:

```bash
docker pull docker.osgeo.org/geoserver:2.23.x 
```

Now run the container, installing COG extension and using a local data directory at `/MY/DATADIRECTORY`. As detailed in the documentation: *this setup allows direct management of the file data shared with the container. This setup is also easy to update to use the latest container.*

```bash
docker run --mount type=bind,src=/MY/DATADIRECTORY,target=/opt/geoserver_data -it -p8080:8080 --env INSTALL_EXTENSIONS=true --env COMMUNITY_EXTENSIONS="cog" --env JAVA_TOOL_OPTIONS="-Xmx4096m" docker.osgeo.org/geoserver:2.23.x
```

**The `JAVA_TOOL_OPTIONS` is important to increase the maximum hava heap size available for the app**. The example command provided earlier sets this limit on ~4Gb For coverage stores, a large heap size is necessary since there is a lot of data to bring. You can adjust it according to your use case. Run `java -XX:+PrintFlagsFinal -version | grep MaxHeapSize` for getting the max heat space in the docker container.


### Optimization of Geoserver ImageMosaic

- [See geoserver documentation](https://training.geonode.geosolutionsgroup.com/master/GN2/ADV_LAYERS_PUB/OPTIMIZE_RASTER.html)

## Example Notebooks

- **Coverage Store using Cog Images**: setup a WMS/WMTS-T service using COG rasters located in cloud storage

## TODO

- [ ] Fix creating new datastore with existing database index: [geoserver repo issue](https://osgeo-org.atlassian.net/jira/software/c/projects/GEOS/issues/GEOS-11144)