semvec/pypi-packages huggingface dataset:
conv_html_to_markdown 0.1.31
Curate scraped HTML for easy interpretation by large language models. Build more robust generative AI applications. Convert HTML to Markdown using Regex, BeautifulSoup4, and filter out useless content with Jina Embeddings.
License: MIT
Requires Python: >=3.10

# Convert and Format HTML to Markdown

## Purpose

For converting HTML to Markdown and formatting a dataset of HTML content 
into structured Markdown, with added capabilities of processing text embeddings to identify and remove redundant content.

* No API keys required
-> Because this project uses the open-source Jina Embeddings model locally, it's totally free to use.

## Installation & Setup

First clone the package: `git clone https://github.com/daethyra/conv_html_to_markdown.git`

To get started, run:

`pip install conv_html_to_markdown`

* Run `jina_embeddings.py` to preemptively download the embeddings model.

**Example integration**:

* Please see an example usage in [gpt-crawler](https://github.com/Daethyra/gpt-crawler). This fork of `gpt-crawler` has the `conv_html_to_markdown` package integrated into its processing pipeline. 

**Configuration**:
* You can clone the package repository to configure similarity threshold for removing content, chunk size, maximum number of threads, the file pattern to match when loading files for conversion, and the output file's name.

To do so, edit the parameter values passed into the `main` function in [main.py](./src/conv_html_to_markdown/main.py#async_def_main).
gitlab-download-artifacts 1.4
Download the gitlab artifacts matching a git repo or submodule, or provided repo http url (optionally with @<commit>)
License: MIT

gitlab-download-artifacts
=========================

This Python utility is used to download artifacts from gitlab ci.

It can be used to grab artifacts from a CI job matching a git repo (or submodule) at the current commit, or
from the https url of a repo, optionally at provided commit ref

gitlab-ci usage
---------------

When used in Gitlab CI a project / private token with api access will be required or user:pass specified in the url.
Currently (Jan 2022) the CI Job Token doesn't have enough api access to find the correct artifacts to download.

developer/desktop usage
-----------------------

If run from desktop the default browser will be used, however in this case an ``@<commit hash>`` cannot be used as
the gitlab artifact urls don't support this: https://docs.gitlab.com/ee/api/job_artifacts.html#download-the-artifacts-archive


gitlab-download-artifacts options
---------------------------------

To download build artifacts for the current git project, simply run the tool from within the project folder and refer to the job whose artifacts are needed, eg:

::

  cd myproject
  gitlab-download-artifacts . build

The artifacts from the ``build`` job are downloaded and extracted into the repo folder.


To download the artifacts for a submodule it can be used like:

::

  gitlab-download-artifacts ./submodule build

The artifacts will be downloaded from the commit the submodule is checked out at and expanded into its folder.

Alternatively to just grab from a project page:

::

  gitlab-download-artifacts https://gitlab.com/alelec/gitlab-download-artifacts@65ad5e1 release

Command Line
------------

::

    usage: gitlab-download-artifacts [-h] [-p PROJECT_TOKEN] [-t TIMEOUT] target job

    Download gitlab artifacts

    positional arguments:
      target                                              Path to git repo to grab or repo url (optionally with @<commit>)
      job                                                 Gitlab CI job name to grab from

    optional arguments:
      -h, --help                                          show this help message and exit
      -p PROJECT_TOKEN, --project-token PROJECT_TOKEN     Project / private token to auth with, env: ${PROJECT_TOKEN} by default
      -t TIMEOUT, --timeout TIMEOUT                       Timeout (in minutes) to wait for dependent jobs to be ready


version: 1.4+g4a2cb1e
byrne 1.0.0
A intelligent DynamoDB frontend for Python 3
License: MIT
Requires Python: >=3.10,<4.0

# Byrne
> A intelligent DynamoDB frontend for Python3


## Basic Usage

### Importing the package

```python
    import boto3  # require to build a client
    import byrne
```

### Creating a client

```python
    session = boto3.Session()
    base_client = session.client("dynamodb")
    client = byrne.DynamoDB(base_client)
    client.list_tables()  # ["table", "names"]
```

### Creating a Table

```python
    table_def = byrne.datastructures.TableDefinition(
        "TableName",
        {
            "id": "S"
        },
        byrne.datastructures.KeyDefinition("id")
    )

    client.create_table(table_def)
```

## FAQ

 1. How do I install `byrne`?
    - Simply run `pip3 install byrne` - the PyPI package is automatically kept up to date with this repository.

 2. Why does this require Python 3.7?
    - We use dataclasses heavily to model table metadata and to simplify query expressions.

 3. Why is this better than using the default client in `boto3`?
    - Support for asyncronous result pagination.
    - Allows for integrated object mapping.
    - Configurable value marshalling and unmarshalling.
    - A more Pythonic interface.
    - Builtin support for working with DAX.

 4. How is the package tested?
    - The testing regimen is automated through `tox`.
    - Tests are run against a DynamoDB Local instance for all supported Python versions.
    - The tests are run against the live DynamoDB service in AWS us-east-1 for the latest Python version.
    - Code coverage is tracked through [Coveralls](https://coveralls.io/github/jacobneiltaylor/byrne).

 5. Why is the DAX code path untested?
    - We are working on creating a testing strategy for DAX workflows.
    - DAX is expensive, we can't justify having a dedicated cluster running only for testing.

 6. I can't run the test cases on my local machine! Help!
    - There are 5 test environments defined in `tox.ini`:
        1. `test`: Runs test suite against "real" DynamoDB.
        2. `test_local`: Runs test suite against DynamoDB Local.
        3. `test_coverage`: Same as `test`, with coverage reporting.
        4. `test_coverage_local`: Same as `test_local`, with coverage reporting.
        5. `flake8`: Runs PEP8 conformance testing.
    - The live test cases require AWS credentials on your machine.
    - These credentials need access rights to assume a role used for testing.
    - Access to this role is only granted to trusted contributors.
    - You should be able to run the local environments after executing the `run_dynamodb_local.sh` script.
    - `flake8` should run without any fuss.