semvec/pypi-packages huggingface dataset:
download-webpage-data 1.0.2
A tool to download website contents through Tor with German exit nodes and extract images
License: MIT
Requires Python: >=3.8

# download-webpage-data

A Python tool to download website contents through Tor with German exit nodes and extract images from downloaded websites.

## Prerequisites

- Python 3.8 or higher
- Tor service installed on your system
- `torrc` configuration file (will be created automatically)

## Installation

1. Clone this repository
2. Install dependencies:
```bash
pip install .
```

## Features

### Website Downloader
- Routes all traffic through Tor
- Uses German exit nodes exclusively
- Downloads complete website contents
- Preserves website structure
- Handles errors gracefully
- Supports sites with invalid SSL certificates
- Retries failed downloads with new Tor identity

### Image Extractor
- Extracts all images from downloaded websites
- Supports multiple image formats (jpg, jpeg, png, gif, webp, svg, ico)
- Finds both direct image files and HTML-referenced images
- Preserves original filenames
- Creates organized output structure
- Handles duplicate files

## Usage

### Downloading Websites

1. Ensure Tor service is running on your system:
```bash
# On Manjaro/Arch:
sudo systemctl start tor
```

2. Download a website:
```bash
# Interactive mode
python -m download_webpage_data

# Direct URL mode
python -m download_webpage_data -u https://example.com

# With SSL verification
python -m download_webpage_data --verify-ssl -u https://example.com
```

### Extracting Images

1. After downloading one or more websites, run:
```bash
python -m download_webpage_data.extract_images
```

2. Select the website from the list
3. Images will be extracted to `images/<website>/` directory

## Command-line Options

### Website Downloader
- `-u, --url`: URL to download (if not provided, will prompt)
- `--verify-ssl`: Enable SSL certificate verification (disabled by default)

### Image Extractor
- Interactive menu to select from downloaded websites
- Press 'q' to quit at any time

## Directory Structure

```
.
├── downloads/           # Downloaded websites
│   └── example.com/    # Website content
└── images/             # Extracted images
    └── example.com/    # Images from website
```

## Security Note

This tool is for legitimate use only. Ensure you have permission to download website contents before using this tool.

## License

MIT
PyFTS5 0.1.0
A lightweight wrapper for in‑memory SQLite FTS5 full‑text search in Python.
License: MIT
Requires Python: <4.0,>=3.8
Keywords: sqlite, fts5, full-text search, python, sqlite3

# PyFTS5

**PyFTS5** is a lightweight Python library that wraps an in‑memory SQLite database with FTS5 full‑text search capabilities. It provides a simple, Pythonic API to add documents (with optional IDs) and perform various full‑text searches using common operators. The library also leverages SQLite’s built‑in highlighting function to mark matched terms.

## Features

- **In‑Memory Database:** Uses an in‑memory SQLite database for fast, lightweight searches.
- **FTS5 Integration:** Leverages SQLite’s C‑implemented FTS5 extension for efficient full‑text indexing and search.
- **Document IDs:** Allows you to insert documents with specific IDs, so that search results carry a persistent identifier.
- **Helper Search Methods:** Supports common queries:
  - **Generic search:** Direct MATCH queries.
  - **Phrase search:** Exact phrase matching.
  - **Prefix search:** Match tokens beginning with a given prefix.
  - **Boolean operators:** AND, OR, NOT combinations.
  - **NEAR search:** Find terms within a specified distance.
- **Highlighting:** Optionally return search results with matched tokens highlighted (using SQLite’s `highlight()` function).
- **Context Manager:** Implements `__enter__` and `__exit__` so that the connection is automatically closed when finished.

## Installation

### Via Poetry

Clone the repository and run:

```bash
poetry install
```

If published on PyPI, install via pip:

```bash 
pip install PyFTS5
```

## Usage
Below is a quick example:

```python
from fts_search_db import FullTextSearchDB

# Prepare some documents as (doc_id, content) pairs.
docs = [
    (101, "The quick brown fox jumps over the lazy dog"),
    (102, "Never jump over the lazy dog quickly"),
    (103, "A quick movement of the enemy will jeopardize six gunboats"),
    (104, "Quick thinking leads to quick decisions"),
]

# Use the class as a context manager for automatic cleanup.
with FullTextSearchDB(docs) as fts_db:
    # Perform a generic search (without highlighting).
    results = fts_db.search("quick AND dog")
    for doc_id, content in results:
        print(f"Doc {doc_id}: {content}")

    # Perform a search with highlighting enabled.
    highlighted = fts_db.search("quick AND dog", highlight=True, hl_prefix="<<", hl_suffix=">>")
    for doc_id, hl_text, content in highlighted:
        print(f"Doc {doc_id} (highlighted): {hl_text}")
```

## Testing
Unit tests are written with pytest. To run the tests, simply run:

```bash
poetry run pytest
```

## License
This project is licensed under the MIT License.
yatla 0.5.0
A templating engine with a simple templating language specifically designed for type inference, allowing consumers to provide users with slots of determined types.
License: MIT
Requires Python: <4.0,>=3.10

# YATLA 📑: Yet Another Templating Language (Again!)

YATLA is a templating engine with a simple templating language specifically designed for type inference, allowing consumers to provide users with slots of determined types.

## Roadmap 🗺️

- [x] Add numerical blocks
- [x] Add type inference. Go from a template -> list of typed parameters
- [x] Write own lexer to respect whitespace
- [x] Write own own parsing system
- [x] Add template validation (preventing non-unique identifiers...)
- [x] Publish as pip-installable package
- [x] Add foreach loops
- [x] Add numerical function calls
- [ ] Improve Python interface
- [ ] Think about object accessors

## Sample Template

This sample template gives a whirlwind tour of YATLA's functionality.

```
This a sample template.

This line contains a {{ slot }}.

Slots can also contain mathematical expressions: {{ 3 + factor * 7 }}.
With full support for operator precedence: {{ (3 + factor) * 7 }}.

Foreach loops are also valid. For example:

This is the {{ factor }} times table:
{{ foreach num in num_list }}
    {{ factor }} * {{ num }} = {{ factor * num }}
{{ endforeach }}

There is also a standard library of mathematical functions.
Comparing {{ factor }} and {{ num }}, the largest is {{ Maximum(factor, num) }} and the smallest is {{ Minimum(factor, num) }}.
{{ RoundUp(15, 2) }} = 16, and {{ RoundDown(15, 2) }} = 14. 
```

From the template above, YATLA will identify the parameters in the template, and infer their types:

- `slot`: string or number
- `factor`: number
- `num`: number
- `num_list`: number array

## Contributing

Contributions are welcome! If you encounter any issues or have suggestions for improvement, please [submit an issue](https://github.com/eugene-prout/yatla/issues).

## License

This project is licensed under the MIT License - see the [LICENSE](/LICENSE) file for details.

## Acknowledgements

YATLA is developed and maintained by [Eugene Prout](https://www.prout.tech).