semvec/pypi-packages huggingface dataset:
doc-it-installer 0.1.3
A modular CLI tool to automatically generate documentation for code changes using a local Ollama LLM.
Requires Python: >=3.10

# Doc-It 

**Doc-It** is a powerful, modular command-line tool that leverages local Large Language Models (LLMs) via [Ollama](https://ollama.com/) to **automatically generate professional documentation** for your code changes.  
Track your work, explain complex changes in natural language, and export your documentation to multiple formats — all from your terminal.

---

## Features

- **AI-Powered Explanations**  
  Automatically generates clear, human-readable explanations for code modifications.

- **Fully Local**  
  Uses your local Ollama instance — ensuring your code stays private and secure.

- **Snapshot-Based Tracking**  
  Intelligently detects changes by comparing snapshots of your codebase.  
  _No Git dependency required._

- **Multiple Export Formats**  
  Generate documentation in:
  - Markdown (`.md`)
  - Microsoft Word (`.docx`)
  - LaTeX (`.tex`)

- **Modular & Configurable**  
  Easily configure the tool using a simple `.json` file to:
  - Specify the Ollama model
  - Select files to watch
  - Customize behavior

- **Simple CLI Interface**  
  Clean and intuitive commands for a seamless documentation experience.

---

## Prerequisites

Ensure you have the following installed on your system:

- Python **3.10 or higher**
- [Ollama](https://ollama.com/)
- At least one Ollama model pulled (e.g. Llama 3):

```bash
ollama pull llama3
```

---

## Installation

Install Doc-It from [PyPI](https://pypi.org/project/doc-it-installer/):

```bash
pip install doc-it-installer
```

---

## Usage

Navigate to your project’s root directory and follow this three-step workflow:

### 1️⃣ Initialize Your Project

Sets up the `.docit` directory, initializes a snapshot database, scans your codebase, and creates a high-level summary using AI.

```bash
doc-it init
```

_Run this only once per project._

---

### 2️⃣ Analyze and "Commit" Changes

After modifying your code, use the `commit` command.  
Doc-It compares snapshots and sends the differences to AI for explanation.

```bash
doc-it commit -m "Refactored the user authentication module"
```

💡 The `-m` flag is required to describe your update.

---

### 3️⃣ Generate the Documentation

Create a final documentation file using the `generate` command.

#### Default (Markdown):

```bash
doc-it generate
```

#### Generate a Word document:

```bash
doc-it generate --format docx
```

#### Generate a LaTeX document:

```bash
doc-it generate --format tex
```

---

## Authors

- **Kartik Sharma**
- **Harshvardhan Singh**

---

## License

This project is licensed under the [MIT License](LICENSE).

---

> _Empower your code with intelligent documentation. Let Doc-It do the writing._
stealth-crawler 0.9.0
A stealth web crawler using headless Chrome
Requires Python: >=3.10

# Stealth Crawler

A headless-Chrome web crawler that discovers same-host links and optionally saves HTML, Markdown, PDF, or screenshots. Use as a library or via the `stealth-crawler` CLI.

---

## Features

- Asynchronous, headless Chrome browsing via `pydoll`
- Discovers internal links starting from a root URL
- Optional content saving:
  - HTML
  - Markdown (via `html2text`)
  - PDF snapshots
  - PNG screenshots
- Rich progress bars with `rich`
- Configurable URL filtering (base, exclude)
- Pure-Python API and CLI

---

## Installation

Install the latest stable release for everyday use:

```bash
pip install stealth-crawler
````

Or in an isolated environment with **pipx**:

```bash
pipx install stealth-crawler
```

Or via **Poetry**:

```bash
poetry add stealth-crawler
```

---

## Quickstart

### Command-Line

```bash
# Discover URLs only
stealth-crawler crawl https://example.com --urls-only

# Crawl and save HTML + Markdown
stealth-crawler crawl https://example.com \
  --save-html --save-md \
  --output-dir ./output

# Exclude specific paths
stealth-crawler crawl https://example.com \
  --exclude /private,/logout
```

Run `stealth-crawler --help` for full options.

### Python API

```python
import asyncio
from stealthcrawler import StealthCrawler

crawler = StealthCrawler(
    base="https://example.com",
    exclude=["/admin"],
    save_html=True,
    save_md=True,
    output_dir="export"
)
urls = asyncio.run(crawler.crawl("https://example.com"))
print(urls)
```

---

## Configuration

| Option        | CLI flag       | API param    | Default    |
| ------------- | -------------- | ------------ | ---------- |
| Base URL(s)   | `--base`       | `base`       | start URL  |
| Exclude paths | `--exclude`    | `exclude`    | none       |
| Save HTML     | `--save-html`  | `save_html`  | `False`    |
| Save Markdown | `--save-md`    | `save_md`    | `False`    |
| URLs only     | `--urls-only`  | `urls_only`  | `False`    |
| Output folder | `--output-dir` | `output_dir` | `./output` |

---

## Testing & Quality

* Run tests:

  ```bash
  pytest
  ```
* Check formatting & linting:

  ```bash
  black src tests
  ruff check src tests
  ```

---

## Contributing

1. Fork the repository and create a feature branch.
2. Set up your development environment:

   ```bash
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -e ".[dev]"
   ```

   Or with **uv**:

   ```bash
   uv venv .venv
   source .venv/bin/activate
   uv pip install -e ".[dev]"
   ```
3. Implement your changes, add tests, run:

   ```bash
   black src tests
   ruff check src tests
   pytest
   ```
4. Open a pull request against `main`.

---

## License

This project is licensed under the **GNU General Public License v3.0 or later** (GPL-3.0-or-later).
You are free to use, modify, and redistribute under the terms of the GPL.
See [LICENSE](./LICENSE) for full details.