gptize 0.5.1
Gptize is a tool designed to concatenate the contents of project files for ChatGPT
Requires Python: >=3.9.0


# GPTize

**GPTize** is a tool for merging the contents of project files into a single text document. It is specifically designed to create datasets that can be loaded into ChatGPT for analysis or training. I, [Aleksei Svetlov](https://www.linkedin.com/in/svetlovtech/), created this tool because I was tired of copying file contents and paths to make GPT understand the context of my project. With GPTize, this process is now automated and streamlined.

## Features
- **Clipboard functionality**: The combined content of project files is automatically copied to the clipboard after generation using `pyperclip`.
- **Custom output file naming**: Output file names now include the project name along with the date and time for better traceability.
- **Exception handling for files based on `.gitignore`** (including custom `.gptignore`).
- **Support for specifying a target directory with repository root `.gitignore`**.
- **Support for various encodings when reading files**.
- **Customizable output file naming** based on the input file or directory name.
- **Report generation including all processed files**.
- **Enhanced limit checks for file size and token count**, with warnings logged instead of errors raised when limits are exceeded.
- **Detailed Git integration**: Includes the current branch, last commit, and file status in the generated report.
- **Tokenization support**: Integrated with `tiktoken` for accurate token counting and analysis.
- **File statistics**: Provides detailed information on line count, character count, token count, file size, last modified date, and permissions for every processed file.
- **Warnings for large files**: Alerts if files exceed a predefined line count threshold.

## Installation
To install GPTize, simply use pip:

```bash
pip install gptize
```

This command will install GPTize and all its dependencies. After installation, you can use GPTize from the command line anywhere.

## Usage
To run GPTize, you have several options:

### Basic Usage
Simply invoke GPTize in the command line to process the current directory:

```bash
gptize
```

This will process all files in the current directory and generate a report with a default name like `gptize-output-PROJECT_NAME-YYYYMMDD-HHMMSS.txt`. The content will also be copied to your clipboard.

### Specifying a Directory
To process a specific directory, use:

```bash
gptize /path/to/directory
```

This will process all files in the specified directory and create a report named `gptize-output-PROJECT_NAME-YYYYMMDD-HHMMSS.txt`, and the content will be copied to your clipboard.

### Specifying a Single File
For processing a single file:

```bash
gptize /path/to/file.txt
```

This will process only the specified file and generate a report named `gptize-output-file_name-YYYYMMDD-HHMMSS.txt`, where `file_name` is the name of the input file. The result will also be copied to your clipboard.

### Specifying Repository Root for `.gitignore`
If your `.gitignore` is located in the root of the repository but you want to process files in a different subdirectory, you can use the `--repo-root` option:

```bash
gptize /path/to/directory --repo-root /path/to/repo_root
```

This will apply the `.gitignore` from the repository root to files in the specified directory, and the result will be copied to your clipboard.

### Custom Output File
If you want to specify a custom output file name, use the `-o` or `--output` option:

```bash
gptize -o custom_output.txt
```

This command will override the default naming convention and use `custom_output.txt` as the output file name. The content will still be copied to your clipboard.

### Uploading to ChatGPT
After generating the merged file using GPTize, you can upload it to ChatGPT for improved context understanding. When making requests to ChatGPT, explicitly reference the uploaded file, for instance, using a phrase like `... based on the imported txt file.` This approach significantly enhances the quality of ChatGPT's responses by providing it with specific context.

### Analyzing Statistics
GPTize provides detailed statistics for all processed files:
- Line count, character count, and token count.
- Total project statistics including total lines, tokens, and percentage of GPT-4o context usage.
- Warnings for files exceeding predefined line count thresholds.

### Git Integration
The generated report includes Git details such as:
- Current branch.
- Last commit message, author, and timestamp.
- Untracked or modified files.

## Components
- `gptizer.py`: The main class for file processing.
- `main.py`: The entry point of the application.
- `models.py`: Data models for files and projects.
- `output_builder.py`: Output constructor for report generation.
- `settings.py`: Project settings.

## Author and Maintainer
[Aleksei Svetlov](https://www.linkedin.com/in/svetlovtech/) - Creator and main maintainer.

### Contact Information
- [LinkedIn](https://www.linkedin.com/in/svetlovtech/)
- [Telegram](https://t.me/SvetlovTech)
- Email: svetlovtech@aabee.tech

## License
The project is distributed under the [MIT License](LICENSE).


# CHANGELOG

## [0.5.1] - 2024-12-01
- [Fix] Fixed console summary statistics display

## [0.5.0] - 2024-12-01
- [Feature] Added support for tokenization using `tiktoken`
  - Integrated `tiktoken` for accurate token counting.
  - Introduced new tokenization settings in `Settings` class, including `TOKEN_MODEL_NAME`, `GPT4O_CONTEXT_WINDOW`, and `TOP_TOKEN_FILES_COUNT`.

- [Feature] Git status integration
  - Added functionality to include Git branch, last commit, and file status in the combined output.

- [Feature] Custom `.gptignore` support
  - Implemented a `.gptignore` file to specify additional patterns for file exclusion alongside `.gitignore`.

- [Enhancement] Expanded Python version support
  - Updated GitHub Actions CI to include testing on Python 3.13.

- [Enhancement] Metadata and statistics for processed files
  - File metadata now includes size, last modified timestamp, and permissions.
  - Statistics for line count, character count, and token count added for each file.
  - Enhanced logging to display detailed file statistics and warnings for files exceeding predefined limits.

- [Enhancement] Detailed statistics summary
  - Added summary for total lines, tokens, and characters, along with the percentage of GPT-4o context usage.

- [Enhancement] Clipboard support improvements
  - Enhanced compatibility for clipboard operations, including fallback to `xclip` if default tools are unavailable.

- [Refactor] Cleaned up code and improved maintainability
  - Simplified method parameters and removed redundant comments.
  - Introduced `FileStats` and `FileMetadata` classes for structured storage of file-related data.

- [Fix] Updated requirements
  - Added `tiktoken` and its dependencies to `requirements.in` and `requirements.txt`.

- [Fix] Enhanced error handling
  - Improved error messages and handling for file reading, Git commands, and clipboard operations.

## [0.4.0] - 2024-09-27
- [Feature] Added clipboard copy functionality
  - The combined output file content is now automatically copied to the clipboard using `pyperclip` after the content is generated.
- [Enhancement] Improved output file naming
  - Output file names now include the project name in addition to the date and time. This provides better traceability for output files.
- [Enhancement] Updated `Settings.custom_output_file`
  - The method now accepts the project name as an argument and uses it in the output file name.
- [Enhancement] Reorganized the main logic in `main.py`
  - Fixed a bug where the `gptizer` object was used before initialization.
  - Ensured that output file name generation happens after `gptizer` is properly initialized.

## [0.3.0] - 2024-09-26
- [Feature] Support for specifying a target directory with repository root .gitignore
  - Now `gptize` can be executed from any directory, while still applying `.gitignore` rules from the root of the repository.
  - Added the `--repo-root` argument to specify the root directory of the repository where the `.gitignore` is located.
  - Example usage: `gptize src/py_module/ --repo-root .` allows processing files in `src/py_module/` while applying `.gitignore` rules from the repository root.
- [Feature] Added support for a custom .gitignore for GPTize
  - Now, you can use an additional custom `.gptignore` file along with the repository root `.gitignore`.
  - The custom `.gptignore` can be specified and will be applied in addition to the main `.gitignore`.
  - Example usage: `gptize src/py_module/ --repo-root .` will apply both `.gitignore` and `.gptignore`.

## [0.2.5] - 2023-11-25
- [Modification] Updated File Size and Token Count Checks
  - Modified the `combine_files` method in `gptizer.py` to log a warning instead of raising an error when the total size of the combined content exceeds the `MAX_FILE_SIZE_BYTES_LIMIT` or `MAX_TOKEN_COUNT_LIMIT` defined in `settings.py`.

## [0.2.4] - 2023-11-16
- [Feature] Custom Output File Naming
  - Output files now include the name of the processed file or directory, enhancing traceability and identification.
- [Enhancement] Settings Method for Custom File Names
  - Updated the `Settings` class with a new method to generate output file names incorporating the name of the input file or directory.
- [Modification] Main File Processing Logic
  - Modified `main.py` to adopt the new output file naming scheme.
- [Fix] Minor Bug Fixes and Performance Improvements
  - Addressed various minor bugs and optimized performance.

## [0.2.3] - 2023-11-12
- [Enhancement] Detect binary files and handle errors gracefully
  - Added binary file detection logic in load_file_content method.
  - Improved error handling for file reading.
  - Updated the OutputBuilder to handle binary files properly.