semvec/pypi-packages huggingface dataset:
EETQ 1.0.0
Easy and efficient quantization for transformer
Requires Python: >=3.7

# EETQ
<div align='right' ><font size="1"><b><a href="./README_zh.md">中文README</a></b> </font></div>
Easy & Efficient Quantization for Transformers

## Table of Contents
- [EETQ](#eetq)
  - [Table of Contents](#table-of-contents)
  - [Features](#features)
  - [Getting started](#getting-started)
    - [Environment](#environment)
    - [Installation](#installation)
    - [Usage](#usage)
  - [Examples](#examples)
  - [Performance](#performance)

## Features
- **New**🔥: [Implement gemv](https://github.com/huggingface/text-generation-inference/pull/1502) in w8a16, performance improvement 10~30%. 
- INT8 weight only PTQ
  * High-performance GEMM kernels from FasterTransformer, [original code](https://github.com/NVIDIA/FasterTransformer/tree/main/src/fastertransformer/kernels/cutlass_kernels/fpA_intB_gemm)
  * No need for quantization training
- Optimized attention layer using [Flash-Attention V2](https://github.com/Dao-AILab/flash-attention)
- Easy to use, adapt to your pytorch model with one line of code


## Getting started

### Environment

* cuda:>=11.4
* python:>=3.8 
* gcc:>= 7.4.0 
* torch:>=1.14.0 
* transformers:>=4.27.0

The above environment is the minimum configuration, and it is best to use a newer version.

### Installation
Recommend using Dockerfile.


```bash
$ git clone https://github.com/NetEase-FuXi/EETQ.git
$ cd EETQ/
$ git submodule update --init --recursive
$ pip install .
```
If your machine has less than 96GB of RAM and lots of CPU cores, `ninja` might
run too many parallel compilation jobs that could exhaust the amount of RAM. To
limit the number of parallel compilation jobs, you can set the environment
variable `MAX_JOBS`:
```bash
$ MAX_JOBS=4 pip install .
```

### Usage
1. Quantize torch model
```python
from eetq.utils import eet_quantize
eet_quantize(torch_model)
```

2. Quantize torch model and optimize with flash attention
```python
...
model = AutoModelForCausalLM.from_pretrained(model_name, config=config, torch_dtype=torch.float16)
from eetq.utils import eet_accelerator
eet_accelerator(model, quantize=True, fused_attn=True, dev="cuda:0")
model.to("cuda:0")

# inference
res = model.generate(...)
```

3. Use EETQ in [TGI](https://github.com/huggingface/text-generation-inference). see [this PR](https://github.com/huggingface/text-generation-inference/pull/1068).
```bash
text-generation-launcher --model-id mistralai/Mistral-7B-v0.1 --quantize eetq ...
```

4. Use EETQ in [LoRAX](https://github.com/predibase/lorax). See [docs](https://predibase.github.io/lorax/guides/quantization/#eetq) here.
```bash
lorax-launcher --model-id mistralai/Mistral-7B-v0.1 --quantize eetq ...
```

## Examples

Model:
- [examples/models/llama_transformers_example.py](examples/models/llama_transformers_example.py)

## Performance

- llama-13b (test on 3090)
prompt=1024, max_new_tokens=50
<img src="./docs/images/benchmark.jpg" style="zoom:50%;" />
pyevaltool 0.1.8
A package to evaluate the models.
Requires Python: >=3.8
Keywords: testing, evaluate, model

# pycvtools

[![forthebadge made-with-python](http://ForTheBadge.com/images/badges/made-with-python.svg)](https://www.python.org/)                 
[![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/)   

## Functionality of the pyevaltool

- Automated testing of the bot response deployed gpt


```
pip install pyevaltool
```
#### To run
```
from pyevaltool import *
```
#### For GPT test -
```
url = '' # Add the frontend url 
token = '' # add your token or key - Apim-Subscription-Key or Bearer token
# change the chat completion format if required. 
# This code will work only if the question asked is at chat_completion_format['messages'][0]["content"]
chat_completion_format = {'chatId': '', 'siteInfo': {'domain': ''}, 'messages': [{'role': 'user', 'content': ''}], 'model': '', 'stream': False} 
answer_loc = "['choices'][0]['messages'][0]['content']" #in string format
input_file = " " # Here you add your input file link
gpt_test(url,token,chat_completion_format,answer_loc,input_file,output_file = "output.csv" )
```
checkpoint-api 0.1.8
Check Point REST API abstraction

# Check Point API

![coverage](https://gitlab.com/peter.gastinger/checkpoint-api/badges/master/coverage.svg?job=coverage)


## Installation

- `pip install checkpoint-api`


## Basic usage

- get object
```python
from checkpoint_api.checkpoint import CheckpointSession

with CheckpointSession("10.111.30.25", "api", "CGx3IjUSd47TrXKh6zJ7") as cps:
    data = {'name': 'IP_10.14.1.2_xlsrv02', 'details-level': 'standard'}
    response_json = cps.post("show-host", json=data)
    print(response_json)
```

Output:
```
{'color': 'black',
 'comments': '',
 'domain': {'domain-type': 'domain',
            'name': 'SMC User',
            'uid': '41e821a0-3720-11e3-aa6e-0800200c9fde'},
 'groups': [],
 'icon': 'Objects/host',
 'interfaces': [],
 'ipv4-address': '10.14.1.2',
 'meta-info': {'creation-time': {'iso-8601': '2020-05-27T10:26+0200',
                                 'posix': 1590567969972},
               'creator': 'user01',
               'last-modifier': 'user01',
               'last-modify-time': {'iso-8601': '2020-05-27T10:26+0200',
                                    'posix': 1590567969972},
               'lock': 'unlocked',
               'validation-state': 'ok'},
 'name': 'IP_10.14.1.2_xlsrv02',
 'nat-settings': {'auto-rule': False},
 'read-only': False,
 'tags': [],
 'type': 'host',
 'uid': '8874eceb-e217-476e-a16c-b1ec2926b921'}
```

- get gateways and servers for MDS

```python
from checkpoint_api.checkpoint import CheckpointSession
from pprint import pprint

checkpoint_username = "user01"
checkpoint_password = "toosecret"

with CheckpointSession(
    "10.1.235.2",
    checkpoint_user=checkpoint_username,
    checkpoint_pass=checkpoint_password,
) as cps:
    pprint(cps.post("show-gateways-and-servers", json={"details-level": "full"}))

```

- Authenticate via api_key and add new access rule

```python
from checkpoint_api.checkpoint import CheckpointSession
import logging
from datetime import datetime

cp_firewall_policy = "PoC Network"

source_zone = "TRUST"
destination_zone = "UNTRUST"
now = datetime.now()
user = "userxyz"

with CheckpointSession(
    "192.168.20.100", api_key="hc9emRAB+fECy7VJkLEIeg==", read_only=False
) as cps:
    logging.info(
        cps.post(
            "add-access-rule",
            json={
                "layer": cp_firewall_policy,
                "position": {"above": "Cleanup rule"},
                "name": f"Drop_zone_from_{source_zone}_to_{destination_zone}",
                "source": f"GRP_ZONE_{source_zone}",
                "destination": f"GRP_ZONE_{destination_zone}",
                "track": {"type": "Log"},
                "action": "Reject",
                "comments": f"{now}-{user}",
            },
        )
    )
```