semvec/pypi-packages huggingface dataset:
ALI-Agent 0.2.4
A brief description of your project
Requires Python: >=3.9

<div align=center>
<!-- <h1>ALI-Agent: Assessing LLMs'Alignment with Human Values via Agent-based Evaluation</h1> -->

<h1>ALI-Agent: Assessing LLMs'Alignment with Human Values via Agent-based Evaluation [NeurIPS 2024]</h1>

<img src="https://img.shields.io/badge/License-MIT-blue" alt="license">

![world](flow.png)

ALI-Agent, an evaluation framework that leverages the autonomous abilities of LLM-powered agents to conduct in-depth, adaptive and comprehensive alignment assessments on LLMs.  ALI-Agent operates through two principal stages: Emulation and Refinement. During the Emulation stage, ALI-Agent automates the generation of realistic test scenarios.
In the Refinement stage, it iteratively refines the scenarios to probe long-tail risks. Specifically, ALI-Agent incorporates a memory module to guide test scenario generation, a tool-using module to reduce human labor in tasks such as evaluating feedback from target LLMs, and an action module to refine tests.

</div>


<p id="Catalogue"></p>  

## 📋 Catalogue 

- [Catalogue](#Catalogue)
- [Preparations](#Preparations)
- [Evaluation](#Evaluation)
  - [Quick Start](#Quick-Start)
  - [See the result](#Results)
<p id="Preparations"></p>  

## ⚙️ Preparations

### Step 1. Install requirements.txt
Set up a virtualenv and install the [pytorch](https://pytorch.org/get-started/previous-versions/) manually. 

Our experiments have been tested on **Python 3.9.17 with PyTorch 2.0.1+cu117**. 

```bash
conda create --name myenv python=3.9.17
conda activate myenv
```

After that, install all the dependencies listed in the `requirements.txt` file by running the following command:

```bash
pip install -r requirements.txt
```

### Step 2. Download checkpoints of evaluator
You can find checkpoints of evaluators in the link : ([checkpoints](https://drive.google.com/drive/folders/1kXheWR1smR-Q8_4cr0OwvFDqRDhUQuEV?usp=sharing))

Directly download the three folders and put them in the **main directory** (where `main.py` can be found).

<p id="Evaluation"></p>  

## ⌛️ Evaluation
Make sure you are in the **main directory** (where `main.py` can be found).

Replace "OPENAI_API_KEY" in simulation/utils.py with **your own OpenAI API key**.

<p id="Quick-Start"></p> 

### Quick Start

To run the agent on a specified dataset, run code as 

```bash
python main.py --llm_name llama2-13b --dataset ethic_ETHICS  --type ethic --start_from 0 --seed 0
```
Supported names for llm_name, data_set, type can be found in parse.py

To run the agent with web browsing, replace "BING_API_KEY" in simulation/utils.py with **your own BING API key**.

```bash
python main.py --llm_name llama2-13b --web_browsing
```

<p id="Results"></p>  

### See the Results

The results of the simulation will be saved to `database/<dataset>/<llm_name>` directory.
sbcli-fri 11.0.5
CLI for managing SimplyBlock cluster


# Simply Block
[![Docker Image Build](https://github.com/simplyblock-io/sbcli/actions/workflows/docker-image.yml/badge.svg)](https://github.com/simplyblock-io/sbcli/actions/workflows/docker-image.yml)

[![Python Unit Testing](https://github.com/simplyblock-io/sbcli/actions/workflows/python-testing.yml/badge.svg)](https://github.com/simplyblock-io/sbcli/actions/workflows/python-testing.yml)


## Install
Add the package repo from AWS CodeArtifact using [awscli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)

```bash
aws codeartifact login --tool pip --repository sbcli --domain simplyblock --domain-owner 565979732541 --region eu-west-1
```
Install package
```bash
pip install --extra-index-url https://pypi.org/simple sbcli-dev
```

# Components

## Simply Block Core
Contains core logic and controllers for the simplyblock cluster

## Simply Block CLI
Please see this document
[README.md](../main/simplyblock_cli/README.md)


## Simply Block Web API
Please see this document
[README.md](../main/simplyblock_web/README.md)



### local development

FoundationDB requires a client library (libfdb_c.dylib) for the Python bindings to interact with the database.
Depending on the OS architecture, please install the appropriate version from the official github repo

```
wget https://github.com/apple/foundationdb/releases/download/7.3.3/FoundationDB-7.3.3_arm64.pkg
```

setup the code on a management node and the webApp code can be developed by building the `docker-compose-dev.yml` file.


```
sudo docker compose -f docker-compose-dev.yml up --build -d
```
repo-reasoner 0.0.33
Repository reasoner that uses LLM, instead of a real project or framework or library

# Repositories Reasoner

[![CI](https://github.com/dzhovi/repositories-reasoner/actions/workflows/ci.yml/badge.svg)](https://github.com/dzhovi/repositories-reasoner/actions/workflows/ci.yml)
[![PyPI version](https://badge.fury.io/py/repo-reasoner.svg)](https://pypi.org/project/repo-reasoner/)
[![Coverage Status](https://coveralls.io/repos/github/dzhovi/repositories-reasoner/badge.svg)](https://coveralls.io/github/dzhovi/repositories-reasoner)[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/dzhovi/repositories-reasoner/blob/main/LICENSE.txt)
[![Known Vulnerabilities](https://snyk.io/test/github/dzhovi/repositories-reasoner/badge.svg)](https://snyk.io/test/github/dzhovi/repositories-reasoner)

Repositories Reasoner is a command-line application that facilitates llm
capabilities to reason about different aspects
of repository.
Primary task that can be solved using repositories reasoner is to determine
whether repository maintained or not.

**Motivation**. During the work on [CaM](https://github.com/yegor256/cam)
project,
where we're building datasets with open source Java programs,
we [discovered](https://github.com/yegor256/cam/issues/297)
the need for filtering out repositories that not maintained. This repository
is portable command-line tool that filters those repositories.

## How to use

First, install it from [PyPI](https://pypi.org/project/repo-reasoner/0.0.7/)
like that:

```bash
pip install repo-reasoner
```

then, execute:

```bash
repo-reasoner is-maintained --repository=author/repository_name --key=your_gigachat_api_key
```

For `--repository` you should provide a name of **existing** Github reposioty,
in a format `author/repository_name`. The result would be printed into stdout.
If `repo-reasoner` thinks that a given repository is maintained, it will answer `yes`,
and `no` otherwise.
If you feel missed, try `--help` and tool will explain to you what you should do.

## Limitations

In current implementation, you have no other options but gigachat as a llm.