semvec/pypi-packages huggingface dataset:
masterai-scrapy-extensions 2024.8.1
A Scrapy extension that report your log from your scraped data.
License: MIT
Requires Python: ==3.11.*
Keywords: scrapy, log, report, colorful

# Scrapy Log Report Extension

A Scrapy extension that report your log from your scraped data.

## Usage

This Scrapy extension provides a way to report your log from your scraped data. It will generate a report every `LOGSTATS_INTERVAL` seconds, and send it to your log server.

```bash
# log report demo
{
  "items_add": 0,
  "pages_add": 0,
  "items_rate": 0,
  "pages_rate": 0,
  "items_count": 0,
  "pages_count": 0,
  "spider_name": "douban",
  "log_count/INFO": 8,
  "log_count/DEBUG": 1,
  "log_count/WARNING": 2,
  "item_scraped_add_count": 0,
  "response_received_add_count": 0
}
```

## Installation

First, pip install this package:

```bash
$ pip install masterai-scrapy-extensions
```

## Usage

Enable the extension in your project's `settings.py` file, by adding the following lines:

```python
EXTENSIONS = {
    "masterai_scrapy_extensions.logreport.ReportStats": 100,
}
#
LOGSTATS_INTERVAL = 60
# set the URL to your log server
# method POST is used to send the report data
LOGREPORT_URL = "http://127.0.0.1:5000/api/v1/task/worker/status"

# log color
from masterai_scrapy_extensions import logcolor

logcolor.log_color_init()
```

That's all! Now run your job and have a look at the field stats.

## Settings

The settings below can be defined as any other Scrapy settings, as described on [Scrapy docs](https://doc.scrapy.org/en/latest/topics/settings.html#populating-the-settings).

- `LOGREPORT_URL`: set the interval in seconds to generate the report.
- `LOGSTATS_INTERVAL`: set the URL to your log server,method POST is used to send the report data.
- `COLORLOG_FORMAT`: log color format.
- `COLORLOG_COLORS`: log colors.
- `COLORLOG_DATEFORMAT`: log color date format.
llm-xs 1.0.0
A module for creation of endpoint for the open-sourced LLMs
License: MIT

# <p align = 'center'> 📝 XS Module Documentation </p>
<p align = 'center'>
The workflow of the XS module, with a tagline : <b>xs makes you access</b>
</p>

[![xs-module.png](https://i.postimg.cc/pT38G84v/xs-module.png)](https://postimg.cc/KkrR4Kmp)

This module leverages ***Flask, Pyngrok, and Waitress*** to create a simple API endpoint for text generation of all ***Open Source LLMs***.
 
## 🚀 Features

- 🌐 **API Endpoint Creation**: Easily create an API endpoint to access the text generation function.
- 🔀 **Dynamic Port Allocation**: Automatically find an available port to avoid conflicts.
- 🚧 **Ngrok Tunneling**: Expose your local server to the internet using Ngrok.
- 🔄 **Flushing Tunnels**: Clear previous Ngrok tunnels to avoid the max tunnel limit.

## 📦 Installation

To use this module, you need to install by the following command :

```bash
pip install 'git+https://github.com/xprabhudayal/xs.git'
```

## 📋 Tutorials and Usage
### 1 Watch these videos for better Understanding

<table >
  <tr >
    <td >
      <a href="https://www.linkedin.com/posts/xprabhudayal_interactive-tutorial-on-how-to-use-the-xs-activity-7223727702548090881-ik7s?utm_source=share&utm_medium=member_desktop">
        <img src="https://i.ibb.co/fn1tzsf/xs-intro.png" alt="LinkedIn Video" width="400"/>
      </a>
    </td>
    <td>
    <td>
    </td>
    </td>
    <td>
      <a href="https://www.linkedin.com/feed/update/urn:li:activity:7223539588932227072/">
        <img src="https://i.ibb.co/wWnTJZW/xs.png" alt="LinkedIn Video" width="400"/>
      </a>
    </td>
  </tr>
</table>


### 2. Finding an Available Port
### `xs.find_port()`

 It dynamically finds an available port.

### 3. Forward Function 
### `xs.forward(NGROK_API)`

The `xs.forward()` function sets up the Ngrok tunnel and starts the server using Waitress. `NGROK_API` is required here.

### 4. Connect Text Function 
### `xs.connect_text(url, prompt)`

The `connect_text` function allows remote devices to access the API and use the text generation option. It sends a POST request to the API endpoint with the provided data and prints the generated text response.

### 5. Flushing Tunnels 
### `xs.flush()`

The `flush` function clears all previous Ngrok tunnels. **In the free tier, you can have 3 tunnels only!** This is useful if you encounter the maximum tunnel limit error from Ngrok.


## 🔧 Future Features

- **Port Killing**: Automatically kill processes using specific ports before starting the server.
- **Enhanced Error Handling**: Improve the robustness of error handling mechanisms.

## 🛠 Troubleshooting

- **Ngrok Tunnel Error**: If you encounter a max tunnel limit, use the `flush` function to clear previous tunnels.
- **JSON Decode Error**: Ensure the API response is correctly formatted JSON.

## License 📄
This project is licensed under the MIT License.

## Contact 📧
For any inquiries, please reach me at : [MAIL](mailto:pradachan@tuta.io )


Made with 💖 by Prabhudayal
---

Enjoy using the `xs.py` module! 🚀