semvec/pypi-packages huggingface dataset:
folder-indexer 0.1.1
A Python script to index a large folder into a parquet file, along with metadata
Requires Python: >=3.10

# folder-indexer-py
A Python script to index a large folder into a parquet file, along with metadata

## Description

This script is useful for searching for files stored on a reasonably slow disk
from backups, especially in where you aren't sure about the files are are
searching for.

Use tools like DBeaver and DuckDB to query and explore the generated index.

## Usage

```bash
pip install folder_indexer

python3 -m folder_indexer -i /path/to/input/folder -o /path/to/output/folder
# --or--
folder_indexer -i /path/to/input/folder -o /path/to/output/folder
```

## Metadata Indexed and Output

The output parquet files have the following columns:

    * file_path
    * folder_path
    * file_name
    * file_size_bytes
    * md5_hash_hex
    * sha256_base64
    * date_created
    * date_modified
    * date_accessed
    * magic_file_type_1
    * first_100_bytes
    * last_100_bytes
    * timestamp_crawled
    * indexing_start_timestamp

The generated parquet files are stored to the output folder with the following naming convention: `partial_file_index_{datetime}.parquet`

At the end of the execution, the individual parquet files are unioned into a single parquet file, with the following name: `00_complete_file_index.parquet`