alter-ego-llm 1.6.0
Library to run experiments with LLMs
Requires Python: >=3.8

# alter\_ego

<p align="center">
    <a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
</p>

`alter_ego` is a library that allows you to run experiments with LLMs. `ego` is a command-line helper included with this package.

`alter_ego` allows you to run microexperiments using a simple shorthand. You can also create more advanced experiments. For turn-based interactive experiments, a [builder](https://ego.mg.sb/builder/) is available.

Read our [paper](https://q.mg.sb/ego).

You can browse our source code in this repository. [Here](https://ego.mg.sb/docs/) are autogenerated docs that can make it easier to find what you're looking for.

# Getting started

## Prerequisites

1. Install [Python](https://python.org), at least version 3.8. *If you are on Windows, make sure to install Python into the PATH.*
2. Create a *virtual environment* and activate it. On Linux and macOS, this is very simple. Just open a terminal and execute the following commands:

```console
user@host:~$ python -m venv env
user@host:~$ source env/bin/activate
```

*Note*: In this document, you may have to replace `python` by `python3` and `pip` by `pip3`. This depends on your system's settings.

On Windows, consider using [this tutorial](https://realpython.com/python-virtual-environments-a-primer/#create-it) to create and activate a virtual environment.

Some editors also do this for you.

3. Install `alter_ego` using

```console
(env) user@host:~$ pip install -U alter_ego_llm
```

Note how `(env)` signals that we are in the virtual environment created earlier.

**For the remainder of this document, we assume that your editor's current directory is also your terminal's present working directory.** From within your terminal, you can find out the present working directory using `pwd` — this should show the very same directory as opened in your editor.

## Using `alter_ego` with GPT

*Note*: If you do not want to use GPT for now, simply change `GPTThread` to `CLIThread` in the examples below and skip this section.

1. **Obtain an API key from OpenAI.** [Here is more information.](https://help.openai.com/en/articles/4936850-where-do-i-find-my-api-key) Your API key looks as follows: `sk-***` Copy this to your clipboard.
1. Create a new file in your editor.
1. Put the content of your clipboard into the file `openai_key` in **your current directory**. The file must not have a file extension—it is literally just called `openai_key`.

## Developing a simple microexperiment

*New*: [**📺 WATCH VIDEO TUTORIAL**](https://youtu.be/GPc0a-Fg1bY)

Let's create a minimal experiment using `alter_ego`'s shorthand feature.

1. Create a new file in your editor, `first_experiment.py`. Here's its code:

```python
import alter_ego.agents
from alter_ego.utils import extract_number
from alter_ego.experiment import factorial

def agent():
  return alter_ego.agents.GPTThread(model="gpt-3.5-turbo", temperature=1.0)

prompt = "Estimate the public approval rating of {{politician}} during the {{time}} of their presidency. Only return a single percentage from 0 to 100."

data = factorial(
  prompt,
  politician=["George W. Bush", "Barack Obama"],
  time=["1st year", "8th year"]
).run(agent, extract_number, times=1)

for row in data:
  print(row)
```

Note how we use variables within the prompt. The crucial feature of `alter_ego` is how these variables are automatically replaced based on treatment.

2. In the terminal, run

```console
(env) user@host:~$ python first_experiment.py
```

3. This will take a few seconds and give you output similar to this:

```
{'politician': 'George W. Bush', 'time': '1st year', 'result': None}
{'politician': 'George W. Bush', 'time': '8th year', 'result': None}
{'politician': 'Barack Obama', 'time': '1st year', 'result': 63}
{'politician': 'Barack Obama', 'time': '8th year', 'result': 8}
```

As you see, GPT did not give a valid response for George W. Bush. Let's debug by changing the line with `run` to:

```python
).run(agent, extract_number, times=1, keep_retval=True)
```

Rerunning our script gives:

```python
{'politician': 'George W. Bush', 'time': '1st year', 'result': 62, 'retval': 'Approximately 62%.'}
{'politician': 'George W. Bush', 'time': '8th year', 'result': None, 'retval': "It is difficult to provide an accurate estimate without conducting a specific poll or analysis. However, based on historical data and trends, it is common for a president's approval rating to decline over the course of their second term. Taking into account various factors such as the economic recession and the ongoing Iraq War during George W. Bush's final year in office (2008), it is reasonable to estimate his public approval rating to be around 25-35%. Please note that this estimation is subjective and might not perfectly reflect the actual public sentiment at that time."}
```

(Here, I have shown only two rows of the output.)

As you see, in the cases where GPT returned only a single number, `alter_ego` was able to correctly extract it. Unfortunately, GPT 3.5 tends to refuse requests to just return a single number. GPT 4 works better. If you have access to GPT 4 over the API, you can change `model="gpt-3.5-turbo"` to `model="gpt-4"`. This is the resulting output (where I have omitted the `retval` once again):

```python
{'politician': 'George W. Bush', 'time': '1st year', 'result': 57}
{'politician': 'George W. Bush', 'time': '8th year', 'result': 34}
{'politician': 'Barack Obama', 'time': '1st year', 'result': 57}
{'politician': 'Barack Obama', 'time': '8th year', 'result': 55}
```

[Here](https://ego.mg.sb/docs/source/alter_ego.experiment.html#alter_ego.experiment.Experiment.run) you can view the documentation for `run`. `run` allows you to quickly execute an experiment defined by what highfalutin scientists call a “factorial design.” This is because the possibilities of `politician` (George W. Bush, Barack Obama) were “multiplied” by the possibilities for `time` (1st year, 8th year). 

The nice thing about these microexperiments is that you can easily carry the output forward to Pandas, Polars, etc.—this is because `data` is only a “list of dicts,” and as such it is trivial to convert to a DataFrame. This allows you to analyze data received straight from an LLM.

Of course, you will often want to set the temperature to `0.0` or another low value. This depends on the nature of your use-case.

## Using the builder to construct a turn-based experiment

*New*: [**📺 WATCH VIDEO TUTORIAL**](https://youtu.be/tV5xACU-abw)

We offer a web app to build *simple* experiments between multiple LLMs. The builder can be found [here](https://ego.mg.sb/builder/), with its source code being available [here](https://github.com/mrpg/ego_builder).

1. For now, just read through the app (it showcases an example of a framed ultimatum game) and scroll down.

2. Copy the code shown below “Export or import scenario” on the web app into a new file. Call that file `built.json` in your current project directory. **The file must be called `built.json`.**

3. Open a terminal and execute

```console
(env) user@host:~$ ego run built
```

4. This will show “System instructions” for two separate players. Note how they vary: One player (the first one) is the proposer and the second player is the responder.

5. The proposer is now asked to put in a proposal in JSON. Let's do it:

```json
{"keep": 4.2}
```

6. As you see, the responder is notified and can now `ACCEPT` or `REJECT`. Let's accept:

```
ACCEPT
```

7. This completes the experiment. You will see something like:

```
Experiment c6627c4e-f17f-4cdc-ba47-462eced3e489 OK
```

8. Let's look at the data that was generated. We can get it in CSV format by executing:

```console
(env) user@host:~$ ego data built c6627c4e-f17f-4cdc-ba47-462eced3e489 > data.csv
```

(You need to replace `c6627c4e-f17f-4cdc-ba47-462eced3e489` with your actual experiment ID)

This should tell you that 2 lines were written. If you open `data.csv` in your preferred spreadsheet calculator, you will see the following output:

| choice        | convo                                | experiment                           | i | round | tainted | thread                               | thread\_type | treatment |
|---------------|--------------------------------------|--------------------------------------|---|-------|---------|--------------------------------------|-------------|-----------|
| {"keep": 4.2} | 03ba9edf-99c0-46d6-8c26-42c26683197c | c6627c4e-f17f-4cdc-ba47-462eced3e489 | 1 | 1     | False   | 1ce5faaf-cc1f-438f-8231-8b7e0d96fb07 | CLIThread   | take      |
| "ACCEPT"      | 03ba9edf-99c0-46d6-8c26-42c26683197c | c6627c4e-f17f-4cdc-ba47-462eced3e489 | 2 | 1     | False   | 9043cd57-3e83-42d2-8d62-c783725e05e7 | CLIThread   | take      |

This is obviously easy to post-process in whatever statistics software you use.

If you re-run the experiment, enter garbage instead of the expected inputs and re-export the data, you will see that the `tainted` column becomes `True`. You can check for `tainted` to verify that inputs were received and processed as expected. Note that once *any* Thread responds invalidly, the Conversation will be stopped and all Threads will have `tainted` set to `True`. Thus, `ego data`'s output may be partial.

You can run your scenario five times by doing

```console
(env) user@host:~$ ego run -n 5 built
```

Needless to say, but you can replace `5` by any integer whatsoever.

Feel free to experiment with our builder.

*Note*: Experts can set the environment variable `BUILT_FILE` to have `ego` use a different file name from `built.json`.

## Using `alter_ego` with oTree

*New*: [**📺 WATCH VIDEO TUTORIAL**](https://youtu.be/ouxRFdKOGEw)

[oTree](https://www.otree.org) is a relatively popular framework for web-based experiments.

### Checking out our apps

[Here](https://otree.readthedocs.io/en/latest/install-nostudio.html) is oTree's own guide to installing it on your computer, and [here](https://gitlab.com/gr0ssmann/otree_course#installing-otree) is another one by us.

In general, after installing Python (and having the installer put it in your PATH), run

```console
user@host:~$ python -m venv env
user@host:~$ source env/bin/activate
(env) user@host:~$ pip install -U alter_ego_llm # run this first
(env) user@host:~$ pip install -U otree # then run this
(env) user@host:~$ otree startproject my_project # this creates an oTree “project”, say no to sample games
(env) user@host:~$ cd my_project # this enters the oTree “project”
```

Put your OpenAI API key into the file `openai_key` – note how this file has no extension.

[Go to `otree/` in this repository.](https://github.com/mrpg/ego/tree/master/otree) Copy any of the apps `ego_human` or `ego_chat` into your project directory. The app directory must be on the same level of `settings.py`. In other words, here is how the folder structure looks if you decide to check out `ego_chat` and you faithfully followed all previous instructions:

```
.
├── ego_chat
│   ├── Chat.html
│   ├── __init__.py
│   ├── prompts
│   │   └── system.txt
│   └── Welcome.html
├── openai_key
├── Procfile
├── requirements.txt
├── settings.py
├── _static
│   └── global
│       └── empty.css
└── _templates
    └── global
        └── Page.html
```

Amend `settings.py` as follows:

```python
SESSION_CONFIGS = [
    dict(
        name="ego_chat",
        app_sequence=["ego_chat"],
        num_demo_participants=1,
    ),
]
```

Run `otree devserver` and open your browser to [localhost:8000](http://localhost:8000). From there, you can click `ego_chat` to invoke the app. Enjoy!

By the way, if you wish to use another model, you can change `gpt-4` in line 31 in `ego_chat/__init__.py` to `gpt-3.5-turbo` or [any other supported](https://platform.openai.com/docs/models) value.

*Note*: `alter_ego` saves message histories automatically in `.ego_output` in your oTree project folder. We did this so that nothing ever gets lost.

### In general

You can attach Threads (i.e., LLMs) and Conversations (i.e., bundles of LLMs) to oTree objects (participants, players, subsessions or sessions). This basically works as follows (in your app's `__init__.py`:

```python
from alter_ego.agents import *
from alter_ego.utils import from_file
from alter_ego.exports.otree import link as ai

...

def creating_session(subsession):
    for player in subsession.get_players():
        ai(player).set(GPTThread(model="gpt-3.5-turbo", temperature=1.0))
```

Here, each `player` would get their own personal GPT agent. If you want to assign such an agent to a `group`, just do this:

```python
def creating_session(subsession):
    for group in subsession.get_groups():
        ai(group).set(GPTThread(model="gpt-3.5-turbo", temperature=1.0))
```

Then, within your code, you can access the agent using a *context manager*. Here's an example of a simple `live_method`-based chat if we attached the Thread to the `player` object:

```python
class Chat(Page):
    def live_method(player, data):
        if isinstance(data, str):
            with ai(player) as llm:
                # this submits the player's message and gets the response
                response = llm.submit(data, max_tokens=500)

                # note: if you put the AI on the "group" object or somewhere other than
                # the player, you may want to change this
                return {player.id_in_group: response}
```

If you want to set the system prompt, you can do:

```
def before_next_page(player, timeout_happened):
    with ai(player) as llm:
        # this sets the "system" prompt
        llm.system("You are participating in an experiment.")
```

You can do whatever you want, but always remember to open the LLM's context (using `with ai(...) as llm`) before performing any action. (This extra step is necessary because of oTree's ORM, which otherwise couldn't notice changes deep down in the Thread.)

You can also attach a whole Conversation to the aforementioned oTree objects.

We provide a simple Chat in this repository, see the directory `otree/ego_chat`.

Remember to put your API key into your oTree project folder. `alter_ego` saves message histories automatically in `.ego_output` in your oTree project folder.

## Developing full-fledged experiments

*New*: [**📺 WATCH VIDEO TUTORIAL**](https://youtu.be/WHW0gkT-oHE)

You can use the primitives exposed by this library to develop full-fledged experiments that go beyond the capabilities of our builder. The directory `scenarios/` contains a bunch of examples, including the code for our paper's machine--machine interaction example (`ego_prereg.py`). Watch the video tutorial to get a feeling for what's possible.

# Citation

When using any part of `alter_ego` in a scientific context, cite the following work:

```tex
@article{ego,
  title={Integrating Machine Behavior into Human Subject Experiments: A User-friendly Toolkit and Illustrations},
  author={Engel, Christoph and Grossmann, Max R. P. and Ockenfels, Axel},
  year={2023},
}
```

# License

`alter_ego` is © Max R. P. Grossmann *et al.*, 2023. It is licensed under LGPLv3+. Please see `LICENSE` for details.

This program is distributed in the hope that it will be useful, but **without any warranty**; without even the implied warranty of **merchantability** or **fitness for a particular purpose**. See the GNU Lesser General Public License for more details.