优化

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 *.ttf filter=lfs diff=lfs merge=lfs -text
+*.db filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -9,12 +9,6 @@ __pycache__
 qwen_agent/llm/gpt.py
 qwen_agent/llm/tools.py
 #workspace/*
-
-benchmark/log/*
-benchmark/output_data/*
-benchmark/upload_file/*
-benchmark/upload_file_clean/*
-benchmark/eval_data/
 Qwen-Agent
 
 docqa/*

README.md

@@ -156,97 +156,3 @@ You can watch the following showcase videos to learn about the basic operations
 - Long-form writing based on visited webpages and PDFs [video](https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/qwen_agent/showcase_write_article_based_on_webpages_and_pdfs.mp4)
 - Drawing a plot using code interpreter based on the given information [video](https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/qwen_agent/showcase_chat_with_docs_and_code_interpreter.mp4)
 - Uploading files, multi-turn conversation, and data analysis using code interpreter [video](https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/qwen_agent/showcase_code_interpreter_multi_turn_chat.mp4)
-
-# Evaluation Benchmark
-
-We have also open-sourced a benchmark for evaluating the performance of a model in writing Python code and using Code Interpreter for mathematical problem solving, data analysis, and other general tasks. The benchmark can be found in the [benchmark](benchmark/README.md) directory. The current evaluation results are as follows:
-
-<table>
-    <tr>
-        <th colspan="5" align="center">In-house Code Interpreter Benchmark (Version 20231206)</th>
-    </tr>
-    <tr>
-        <th rowspan="2" align="center">Model</th>
-        <th colspan="3" align="center">Accuracy of Code Execution Results (%)</th>
-        <th colspan="1" align="center">Executable Rate of Code (%)</th>
-    </tr>
-    <tr>
-        <th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th><th align="center">General↑</th>
-    </tr>
-    <tr>
-        <td>GPT-4</td>
-        <td align="center">82.8</td>
-        <td align="center">66.7</td>
-        <td align="center">60.8</td>
-        <td align="center">82.8</td>
-    </tr>
-    <tr>
-        <td>GPT-3.5</td>
-        <td align="center">47.3</td>
-        <td align="center">33.3</td>
-        <td align="center">55.7</td>
-        <td align="center">74.1</td>
-    </tr>
-    <tr>
-        <td>LLaMA2-13B-Chat</td>
-        <td align="center">8.3</td>
-        <td align="center">1.2</td>
-        <td align="center">15.2</td>
-        <td align="center">48.3</td>
-    </tr>
-    <tr>
-        <td>CodeLLaMA-13B-Instruct</td>
-        <td align="center">28.2</td>
-        <td align="center">15.5</td>
-        <td align="center">21.5</td>
-        <td align="center">74.1</td>
-    </tr>
-    <tr>
-        <td>InternLM-20B-Chat</td>
-        <td align="center">34.6</td>
-        <td align="center">10.7</td>
-        <td align="center">24.1</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>ChatGLM3-6B</td>
-        <td align="center">54.2</td>
-        <td align="center">4.8</td>
-        <td align="center">15.2</td>
-        <td align="center">62.1</td>
-    </tr>
-    <tr>
-        <td>Qwen-1.8B-Chat</td>
-        <td align="center">25.6</td>
-        <td align="center">21.4</td>
-        <td align="center">22.8</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>Qwen-7B-Chat</td>
-        <td align="center">41.9</td>
-        <td align="center">23.8</td>
-        <td align="center">38.0</td>
-        <td align="center">67.2</td>
-    </tr>
-    <tr>
-        <td>Qwen-14B-Chat</td>
-        <td align="center">58.4</td>
-        <td align="center">31.0</td>
-        <td align="center">45.6</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>Qwen-72B-Chat</td>
-        <td align="center">72.7</td>
-        <td align="center">41.7</td>
-        <td align="center">43.0</td>
-        <td align="center">82.8</td>
-    </tr>
-</table>
-
-# Disclaimer
-
-This project is not intended to be an official product, rather it serves as a proof-of-concept project that highlights the capabilities of the Qwen series models.
-
-> Important: The code interpreter is not sandboxed, and it executes code in your own environment. Please do not ask Qwen to perform dangerous tasks, and do not directly use the code interpreter for production purposes.

README_CN.md

@@ -157,96 +157,4 @@ python run_server.py --model_server http://{MODEL_SERVER_IP}:7905/v1 --workstati
 - 提取浏览内容使用代码解释器画图 [video](https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/qwen_agent/showcase_chat_with_docs_and_code_interpreter.mp4)
 - 上传文件、多轮对话利用代码解释器分析数据 [video](https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/qwen_agent/showcase_code_interpreter_multi_turn_chat.mp4)
 
-# 评测基准
-
-我们也开源了一个评测基准，用于评估一个模型写Python代码并使用Code Interpreter进行数学解题、数据分析、及其他通用任务时的表现。评测基准见 [benchmark](benchmark/README.md) 目录，当前的评测结果如下：
-
-<table>
-    <tr>
-        <th colspan="5" align="center">In-house Code Interpreter Benchmark (Version 20231206)</th>
-    </tr>
-    <tr>
-        <th rowspan="2" align="center">Model</th>
-        <th colspan="3" align="center">代码执行结果正确性 (%)</th>
-        <th colspan="1" align="center">生成代码的可执行率 (%)</th>
-    </tr>
-    <tr>
-        <th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th><th align="center">General↑</th>
-    </tr>
-    <tr>
-        <td>GPT-4</td>
-        <td align="center">82.8</td>
-        <td align="center">66.7</td>
-        <td align="center">60.8</td>
-        <td align="center">82.8</td>
-    </tr>
-    <tr>
-        <td>GPT-3.5</td>
-        <td align="center">47.3</td>
-        <td align="center">33.3</td>
-        <td align="center">55.7</td>
-        <td align="center">74.1</td>
-    </tr>
-    <tr>
-        <td>LLaMA2-13B-Chat</td>
-        <td align="center">8.3</td>
-        <td align="center">1.2</td>
-        <td align="center">15.2</td>
-        <td align="center">48.3</td>
-    </tr>
-    <tr>
-        <td>CodeLLaMA-13B-Instruct</td>
-        <td align="center">28.2</td>
-        <td align="center">15.5</td>
-        <td align="center">21.5</td>
-        <td align="center">74.1</td>
-    </tr>
-    <tr>
-        <td>InternLM-20B-Chat</td>
-        <td align="center">34.6</td>
-        <td align="center">10.7</td>
-        <td align="center">24.1</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>ChatGLM3-6B</td>
-        <td align="center">54.2</td>
-        <td align="center">4.8</td>
-        <td align="center">15.2</td>
-        <td align="center">62.1</td>
-    </tr>
-    <tr>
-        <td>Qwen-1.8B-Chat</td>
-        <td align="center">25.6</td>
-        <td align="center">21.4</td>
-        <td align="center">22.8</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>Qwen-7B-Chat</td>
-        <td align="center">41.9</td>
-        <td align="center">23.8</td>
-        <td align="center">38.0</td>
-        <td align="center">67.2</td>
-    </tr>
-    <tr>
-        <td>Qwen-14B-Chat</td>
-        <td align="center">58.4</td>
-        <td align="center">31.0</td>
-        <td align="center">45.6</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>Qwen-72B-Chat</td>
-        <td align="center">72.7</td>
-        <td align="center">41.7</td>
-        <td align="center">43.0</td>
-        <td align="center">82.8</td>
-    </tr>
-</table>
-
-# 免责声明
-
-本项目并非正式产品，而是一个概念验证项目，用于演示Qwen系列模型的能力。
-
-> 重要提示：代码解释器未进行沙盒隔离，会在部署环境中执行代码。请避免向Qwen发出危险指令，切勿将该代码解释器直接用于生产目的。
+

benchmark/README.md

@@ -1,248 +0,0 @@
-# Code Interpreter Benchmark
-
-## Introduction
-To assess LLM's ability to use the Python Code Interpreter for tasks such as mathematical problem solving, data visualization, and other general-purpose tasks such as file handling and web scraping, we have created and open-sourced a benchmark specifically designed for evaluating these capabilities.
-
-### Metrics
-The metrics are divided into two parts: code executability and code correctness.
-- Code executability: evaluating the ability of the LLM-generated code to be executed.
-- Code correctness: evaluating whether the LLM-generated code runs correctly.
-
-### Domain
-When evaluating the accuracy of the code execution results for code correctness, we further divide it into two specific domains: `Math`, `Visualization`.
-In terms of code executability, we calculate executable rate of the generated code for `General problem-solving`.
-
-## Results
-- Qwen-7B-Chat refers to the version updated after September 25, 2023.
-- The code correctness judger model for `Visualization` has changed from `Qwen-vl-chat` to `gpt-4-vision-preview` in the version 20231206.
-
-<table>
-    <tr>
-        <th colspan="5" align="center">In-house Code Interpreter Benchmark (Version 20231206)</th>
-    </tr>
-    <tr>
-        <th rowspan="2" align="center">Model</th>
-        <th colspan="3" align="center">Accuracy of Code Execution Results (%)</th>
-        <th colspan="1" align="center">Executable Rate of Code (%)</th>
-    </tr>
-    <tr>
-        <th align="center">Math↑</th><th align="center">Visualization-Hard↑</th><th align="center">Visualization-Easy↑</th><th align="center">General↑</th>
-    </tr>
-    <tr>
-        <td>GPT-4</td>
-        <td align="center">82.8</td>
-        <td align="center">66.7</td>
-        <td align="center">60.8</td>
-        <td align="center">82.8</td>
-    </tr>
-    <tr>
-        <td>GPT-3.5</td>
-        <td align="center">47.3</td>
-        <td align="center">33.3</td>
-        <td align="center">55.7</td>
-        <td align="center">74.1</td>
-    </tr>
-    <tr>
-        <td>LLaMA2-13B-Chat</td>
-        <td align="center">8.3</td>
-        <td align="center">1.2</td>
-        <td align="center">15.2</td>
-        <td align="center">48.3</td>
-    </tr>
-    <tr>
-        <td>CodeLLaMA-13B-Instruct</td>
-        <td align="center">28.2</td>
-        <td align="center">15.5</td>
-        <td align="center">21.5</td>
-        <td align="center">74.1</td>
-    </tr>
-    <tr>
-        <td>InternLM-20B-Chat</td>
-        <td align="center">34.6</td>
-        <td align="center">10.7</td>
-        <td align="center">24.1</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>ChatGLM3-6B</td>
-        <td align="center">54.2</td>
-        <td align="center">4.8</td>
-        <td align="center">15.2</td>
-        <td align="center">62.1</td>
-    </tr>
-    <tr>
-        <td>Qwen-1.8B-Chat</td>
-        <td align="center">25.6</td>
-        <td align="center">21.4</td>
-        <td align="center">22.8</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>Qwen-7B-Chat</td>
-        <td align="center">41.9</td>
-        <td align="center">23.8</td>
-        <td align="center">38.0</td>
-        <td align="center">67.2</td>
-    </tr>
-    <tr>
-        <td>Qwen-14B-Chat</td>
-        <td align="center">58.4</td>
-        <td align="center">31.0</td>
-        <td align="center">45.6</td>
-        <td align="center">65.5</td>
-    </tr>
-    <tr>
-        <td>Qwen-72B-Chat</td>
-        <td align="center">72.7</td>
-        <td align="center">41.7</td>
-        <td align="center">43.0</td>
-        <td align="center">82.8</td>
-    </tr>
-</table>
-
-Furthermore, we also provide the results of `Qwen-vl-plus` as the code correctness judger model for `Visualization` task to serve as a reference.
-
-<table>
-    <tr>
-        <th colspan="3" align="center">Code Correctness Judger Model = Qwen-vl-plus</th>
-    </tr>
-    <tr>
-        <th rowspan="2" align="center">Model</th>
-        <th colspan="2" align="center">Accuracy of Code Execution Results (%)</th>
-    </tr>
-    <tr>
-        <th align="center">Visualization-Hard↑</th>
-        <th align="center">Visualization-Easy↑</th>
-    </tr>
-    <tr>
-        <td>LLaMA2-13B-Chat</td>
-        <td align="center">2.4</td>
-        <td align="center">17.7</td>
-    </tr>
-    <tr>
-        <td>CodeLLaMA-13B-Instruct</td>
-        <td align="center">17.9</td>
-        <td align="center">34.2</td>
-    </tr>
-    <tr>
-        <td>InternLM-20B-Chat</td>
-        <td align="center">9.5</td>
-        <td align="center">31.7</td>
-    </tr>
-    <tr>
-        <td>ChatGLM3-6B</td>
-        <td align="center">10.7</td>
-        <td align="center">29.1</td>
-    </tr>
-    <tr>
-        <td>Qwen-1.8B-Chat</td>
-        <td align="center">32.1</td>
-        <td align="center">32.9</td>
-    </tr>
-    <tr>
-        <td>Qwen-7B-Chat</td>
-        <td align="center">26.2</td>
-        <td align="center">39.2</td>
-    </tr>
-    <tr>
-        <td>Qwen-14B-Chat</td>
-        <td align="center">36.9</td>
-        <td align="center">41.8</td>
-    </tr>
-    <tr>
-        <td>Qwen-72B-Chat</td>
-        <td align="center">38.1</td>
-        <td align="center">38.0</td>
-    </tr>
-</table>
-
-
-
-## Usage
-
-### Installation
-
-```shell
-git clone https://github.com/QwenLM/Qwen-Agent.git
-cd benchmark
-pip install -r requirements.txt
-```
-
-### Dataset Download
-```shell
-cd benchmark
-wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/qwen_agent/benchmark_code_interpreter_data.zip
-unzip benchmark_code_interpreter_data.zip
-mkdir eval_data
-mv eval_code_interpreter_v1.jsonl eval_data/
-```
-
-### Evaluation
-To reproduce the comprehensive results of benchmark, you can run the following script:
-
-```Shell
-python inference_and_execute.py --model {model_name}
-```
-
-{model_name}:
-- qwen-1.8b-chat
-- qwen-7b-chat
-- qwen-14b-chat
-- qwen-72b-chat
-- llama-2-7b-chat
-- llama-2-13b-chat
-- codellama-7b-instruct
-- codellama-13b-instruct
-- internlm-7b-chat-1.1
-- internlm-20b-chat
-
-The benchmark will run the test cases and generate the performance results. The results will be saved in the `output_data` directory.
-
-**Notes**:
-Please install `simhei.ttf` font for proper display in matplotlib when evaluating visualization task. You can do this by preparing `simhei.ttf` (which can be found on any Windows PC) and then running the following code snippet:
-```python
-import os
-import matplotlib
-target_font_path = os.path.join(
-    os.path.abspath(
-        os.path.join(matplotlib.matplotlib_fname(), os.path.pardir)),
-        'fonts', 'ttf', 'simhei.ttf')
-os.system(f'cp simhei.ttf {target_font_path}')
-font_list_cache = os.path.join(matplotlib.get_cachedir(), 'fontlist-*.json')
-os.system(f'rm -f {font_list_cache}')
-```
-
-#### Code Executable Rate
-```Shell
-python inference_and_execute.py --task {task_name} --model {model_name}
-```
-
-{task_name}:
-- `general`: General problem-solving task
-
-
-#### Code Correctness Rate
-```Shell
-python inference_and_execute.py --task {task_name} --model {model_name}
-```
-
-{task_name}:
-- `visualization`: Visualization task
-- `gsm8k`: Math task
-
-
-## Configuration
-The inference_and_exec.py file contains the following configurable options:
-
-- `--model`: The model to test which can be one of `qwen-72b-chat`, `qwen-14b-chat`, `qwen-7b-chat`, `qwen-1.8b-chat`, `qwen-7b-chat`, `llama-2-7b-chat`, `llama-2-13b-chat`, `codellama-7b-instruct`, `codellama-13b-instruct`, `internlm-7b-chat-1.1`, `internlm-20b-chat`.
-- `--task`: The test task which can be one of `all`, `visualization`, `general`, `gsm8k`.
-- `--output-path`: The path for saving evaluation result.
-- `--input-path`: The path for placing evaluation data.
-- `--output-fname`: The file name for evaluation result.
-- `--input-fname`: The file name for evaluation data.
-- `--force`: Force generation and will overwrite the cached results.
-- `--eval-only`: Only calculate evaluation metrics without re-inference.
-- `--eval-code-exec-only`: Only evaluate code executable rate
-- `--gen-exec-only`: Only generate and execuate code without calculating evaluation metrics.
-- `--gen-only`: Only generate without execuating code and calculating evaluation metrics.
-- `--vis-judger`: The model to judge the result correctness for `Visualization` task which can be one of `gpt-4-vision-preview`, `qwen-vl-chat`, `qwen-vl-plus`. It is set to `gpt-4-vision-preview` by default in the version 20231206, and `Qwen-vl-chat` has been deprecated.

benchmark/code_interpreter.py

@@ -1,250 +0,0 @@
-import base64
-import io
-import json
-import logging
-import os
-import queue
-import re
-import subprocess
-import sys
-import time
-import traceback
-import uuid
-
-import matplotlib
-import PIL.Image
-from jupyter_client import BlockingKernelClient
-from utils.code_utils import extract_code
-
-WORK_DIR = os.getenv('CODE_INTERPRETER_WORK_DIR', '/tmp/workspace')
-
-LAUNCH_KERNEL_PY = """
-from ipykernel import kernelapp as app
-app.launch_new_instance()
-"""
-
-_KERNEL_CLIENTS = {}
-
-
-# Run this fix before jupyter starts if matplotlib cannot render CJK fonts.
-# And we need to additionally run the following lines in the jupyter notebook.
-#   ```python
-#   import matplotlib.pyplot as plt
-#   plt.rcParams['font.sans-serif'] = ['SimHei']
-#   plt.rcParams['axes.unicode_minus'] = False
-#   ````
-def fix_matplotlib_cjk_font_issue():
-    local_ttf = os.path.join(
-        os.path.abspath(
-            os.path.join(matplotlib.matplotlib_fname(), os.path.pardir)),
-        'fonts', 'ttf', 'simhei.ttf')
-    if not os.path.exists(local_ttf):
-        logging.warning(
-            f'Missing font file `{local_ttf}` for matplotlib. It may cause some error when using matplotlib.'
-        )
-
-
-def start_kernel(pid):
-    fix_matplotlib_cjk_font_issue()
-
-    connection_file = os.path.join(WORK_DIR,
-                                   f'kernel_connection_file_{pid}.json')
-    launch_kernel_script = os.path.join(WORK_DIR, f'launch_kernel_{pid}.py')
-    for f in [connection_file, launch_kernel_script]:
-        if os.path.exists(f):
-            logging.warning(f'{f} already exists')
-            os.remove(f)
-
-    os.makedirs(WORK_DIR, exist_ok=True)
-
-    with open(launch_kernel_script, 'w') as fout:
-        fout.write(LAUNCH_KERNEL_PY)
-
-    kernel_process = subprocess.Popen([
-        sys.executable,
-        launch_kernel_script,
-        '--IPKernelApp.connection_file',
-        connection_file,
-        '--matplotlib=inline',
-        '--quiet',
-    ],
-                                      cwd=WORK_DIR)
-    logging.info(f"INFO: kernel process's PID = {kernel_process.pid}")
-
-    # Wait for kernel connection file to be written
-    while True:
-        if not os.path.isfile(connection_file):
-            time.sleep(0.1)
-        else:
-            # Keep looping if JSON parsing fails, file may be partially written
-            try:
-                with open(connection_file, 'r') as fp:
-                    json.load(fp)
-                break
-            except json.JSONDecodeError:
-                pass
-
-    # Client
-    kc = BlockingKernelClient(connection_file=connection_file)
-    kc.load_connection_file()
-    kc.start_channels()
-    kc.wait_for_ready()
-    return kc
-
-
-def escape_ansi(line):
-    ansi_escape = re.compile(r'(?:\x1B[@-_]|[\x80-\x9F])[0-?]*[ -/]*[@-~]')
-    return ansi_escape.sub('', line)
-
-
-def publish_image_to_local(image_base64: str):
-    image_file = str(uuid.uuid4()) + '.png'
-    local_image_file = os.path.join(WORK_DIR, image_file)
-
-    png_bytes = base64.b64decode(image_base64)
-    assert isinstance(png_bytes, bytes)
-    bytes_io = io.BytesIO(png_bytes)
-    PIL.Image.open(bytes_io).save(local_image_file, 'png')
-
-    return local_image_file
-
-
-START_CODE = """
-import signal
-def _m6_code_interpreter_timeout_handler(signum, frame):
-    raise TimeoutError("M6_CODE_INTERPRETER_TIMEOUT")
-signal.signal(signal.SIGALRM, _m6_code_interpreter_timeout_handler)
-
-def input(*args, **kwargs):
-    raise NotImplementedError('Python input() function is disabled.')
-
-import os
-if 'upload_file' not in os.getcwd():
-    os.chdir("./upload_file/")
-
-import math
-import re
-import json
-
-import seaborn as sns
-sns.set_theme()
-
-import matplotlib
-import matplotlib.pyplot as plt
-plt.rcParams['font.sans-serif'] = ['SimHei']
-plt.rcParams['axes.unicode_minus'] = False
-
-import numpy as np
-import pandas as pd
-
-from sympy import Eq, symbols, solve
-"""
-
-
-def code_interpreter(action_input_list: list, timeout=30, clear=False):
-    code = ''
-    for action_input in action_input_list:
-        code += (extract_code(action_input) + '\n')
-    fixed_code = []
-    for line in code.split('\n'):
-        fixed_code.append(line)
-        if line.startswith('sns.set_theme('):
-            fixed_code.append('plt.rcParams["font.sans-serif"] = ["SimHei"]')
-            fixed_code.append('plt.rcParams["axes.unicode_minus"] = False')
-    fixed_code = '\n'.join(fixed_code)
-    if 'def solution()' in fixed_code:
-        fixed_code += '\nsolution()'
-
-    return _code_interpreter(fixed_code, timeout, clear)
-
-
-def _code_interpreter(code: str, timeout, clear=False):
-    if not code.strip():
-        return ''
-    if timeout:
-        code = f'signal.alarm({timeout})\n{code}'
-    if clear:
-        code = "get_ipython().run_line_magic('reset', '-f')\n" + START_CODE + code
-
-    pid = os.getpid()
-    if pid not in _KERNEL_CLIENTS:
-        _KERNEL_CLIENTS[pid] = start_kernel(pid)
-        _code_interpreter(START_CODE, timeout=None)
-    kc = _KERNEL_CLIENTS[pid]
-    kc.wait_for_ready()
-    kc.execute(code)
-    result = ''
-    image_idx = 0
-    while True:
-        text = ''
-        image = ''
-        finished = False
-        msg_type = 'error'
-        try:
-            msg = kc.get_iopub_msg()
-            msg_type = msg['msg_type']
-            if msg_type == 'status':
-                if msg['content'].get('execution_state') == 'idle':
-                    finished = True
-            elif msg_type == 'execute_result':
-                text = msg['content']['data'].get('text/plain', '')
-                if 'image/png' in msg['content']['data']:
-                    image_b64 = msg['content']['data']['image/png']
-                    image_url = publish_image_to_local(image_b64)
-                    image_idx += 1
-                    image = '![fig-%03d](%s)' % (image_idx, image_url)
-            elif msg_type == 'display_data':
-                if 'image/png' in msg['content']['data']:
-                    image_b64 = msg['content']['data']['image/png']
-                    image_url = publish_image_to_local(image_b64)
-                    image_idx += 1
-                    image = '![fig-%03d](%s)' % (image_idx, image_url)
-                else:
-                    text = msg['content']['data'].get('text/plain', '')
-            elif msg_type == 'stream':
-                msg_type = msg['content']['name']  # stdout, stderr
-                text = msg['content']['text']
-            elif msg_type == 'error':
-                text = escape_ansi('\n'.join(msg['content']['traceback']))
-                if 'M6_CODE_INTERPRETER_TIMEOUT' in text:
-                    text = f'Timeout. No response after {timeout} seconds.'
-        except queue.Empty:
-            text = f'Timeout. No response after {timeout} seconds.'
-            finished = True
-        except Exception:
-            text = 'The code interpreter encountered an unexpected error.'
-            logging.warning(''.join(
-                traceback.format_exception(*sys.exc_info())))
-            finished = True
-        if text:
-            result += f'\n\n{msg_type}:\n\n```\n{text}\n```'
-        if image:
-            result += f'\n\n{image}'
-        if finished:
-            break
-    result = result.lstrip('\n')
-    if timeout:
-        _code_interpreter('signal.alarm(0)', timeout=None)
-    return result
-
-
-def get_multiline_input(hint):
-    print(hint)
-    print('// Press ENTER to make a new line. Press CTRL-D to end input.')
-    lines = []
-    while True:
-        try:
-            line = input()
-        except EOFError:  # CTRL-D
-            break
-        lines.append(line)
-    print('// Input received.')
-    if lines:
-        return '\n'.join(lines)
-    else:
-        return ''
-
-
-if __name__ == '__main__':
-    while True:
-        print(code_interpreter([get_multiline_input('Enter python code:')]))

benchmark/config.py

@@ -1,66 +0,0 @@
-from parser import InternLMReActParser, ReActParser
-
-from models import LLM, QwenVL, Qwen, QwenDashscopeVLModel
-from prompt import InternLMReAct, LlamaReAct, QwenReAct
-
-react_prompt_map = {
-    'qwen': QwenReAct,
-    'llama': LlamaReAct,
-    'internlm': InternLMReAct,
-}
-
-react_parser_map = {
-    'qwen': ReActParser,
-    'llama': ReActParser,
-    'internlm': InternLMReActParser,
-}
-
-model_map = {'qwen': Qwen, 'llama': LLM, 'internlm': LLM, 'qwen-vl-chat': QwenVL}
-
-model_type_map = {
-    'qwen-72b-chat': 'qwen',
-    'qwen-14b-chat': 'qwen',
-    'qwen-1.8b-chat': 'qwen',
-    'qwen-7b-chat': 'qwen',
-    'llama-2-7b-chat': 'llama',
-    'llama-2-13b-chat': 'llama',
-    'codellama-7b-instruct': 'llama',
-    'codellama-13b-instruct': 'llama',
-    'internlm-7b-chat-1.1': 'internlm',
-    'internlm-20b-chat': 'internlm',
-    'qwen-vl-chat': 'qwen-vl-chat',
-}
-
-model_path_map = {
-    'qwen-72b-chat': 'Qwen/Qwen-72B-Chat',
-    'qwen-14b-chat': 'Qwen/Qwen-14B-Chat',
-    'qwen-7b-chat': 'Qwen/Qwen-7B-Chat',
-    'qwen-1.8b-chat': 'Qwen/Qwen-1_8B-Chat',
-    'llama-2-7b-chat': 'meta-llama/Llama-2-7b-chat-hf',
-    'llama-2-13b-chat': 'meta-llama/Llama-2-13b-chat-hf',
-    'codellama-7b-instruct': 'codellama/CodeLlama-7b-Instruct-hf',
-    'codellama-13b-instruct': 'codellama/CodeLlama-13b-Instruct-hf',
-    'internlm-7b-chat-1.1': 'internlm/internlm-chat-7b-v1_1',
-    'internlm-20b-chat': 'internlm/internlm-chat-20b',
-    'qwen-vl-chat': 'Qwen/Qwen-VL-Chat',
-}
-
-
-def get_react_prompt(model_name, query, lang, upload_fname_list):
-    react_prompt_cls = react_prompt_map.get(model_type_map[model_name],
-                                            QwenReAct)
-    return react_prompt_cls(query, lang, upload_fname_list)
-
-
-def get_react_parser(model_name):
-    react_parser_cls = react_parser_map.get(model_type_map[model_name],
-                                            ReActParser)
-    return react_parser_cls()
-
-
-def get_model(model_name):
-    if model_name in ["qwen-vl-plus"]:
-        return QwenDashscopeVLModel(model=model_name)
-    model_path = model_path_map.get(model_name, None)
-    model_cls = model_map.get(model_type_map[model_name], LLM)
-    return model_cls(model_path)

benchmark/inference_and_execute.py

@@ -1,280 +0,0 @@
-import argparse
-import json
-import logging
-import os
-from parser import ReActParser
-
-import prettytable
-import tqdm
-from code_interpreter import code_interpreter
-from config import (get_model, get_react_parser, get_react_prompt,
-                    model_path_map)
-from datasets import load_dataset
-from metrics.code_execution import eval_code_execution_rate
-from metrics.gsm8k import eval_gsm8k_acc, is_correct
-from metrics.visualization import eval_visualization_acc
-from utils.code_utils import replace_upload_fname
-from utils.data_utils import load_jsonl
-
-logging.basicConfig(
-    format='%(asctime)s - %(levelname)s - %(message)s',
-    datefmt='%Y-%m-%d %H:%M:%S',
-    level=logging.INFO,
-)
-
-WORK_DIR = os.getenv('CODE_INTERPRETER_WORK_DIR', '/tmp/workspace')
-os.makedirs(WORK_DIR, exist_ok=True)
-os.system(f'cp -r upload_file_clean {WORK_DIR}/upload_file')
-os.system('cp -r upload_file_clean ./upload_file')
-
-global_eval_result = {
-    'code_executability': {
-        'math': None,
-        'visualization': None,
-        'general': None,
-    },
-    'code_correctness': {
-        'math': None,
-        'visualization-hard': None,
-        'visualization-easy': None,
-    }
-}
-
-
-def llm_with_plugin(args, query, item=None, exec_limit=3):
-    exec_count = 0
-
-    # Build ReAct prompt
-    upload_fname_list = item[
-        'input_file_path'] if item and 'input_file_path' in item else []
-    lang = item['lang'] if item and 'lang' in item else 'en'
-    react_prompt_obj = get_react_prompt(args.model, query, lang,
-                                        upload_fname_list)
-    planning_prompt = react_prompt_obj.build_prompt()
-
-    # Execute the code when providing the first action in the query
-    if '<|im_start|>' in query:
-        _, prepend_code, __ = ReActParser().parse_latest_plugin_call(query)
-        prepend_code = replace_upload_fname(prepend_code, upload_fname_list)
-        call_plugin(_, [prepend_code], clear=(exec_count == 0))
-        exec_count += 1
-        exec_limit += 1
-
-    # Inference and execute
-    text = ''
-    while exec_count < exec_limit:
-        stop_words_list = react_prompt_obj.get_stop_words_list()
-        output = text_completion(args.llm,
-                                 planning_prompt + text,
-                                 stop_words=stop_words_list)
-
-        if args.gen_only:
-            text += output
-            break
-
-        react_parser = get_react_parser(args.model)
-        action, action_input, output = react_parser.parse_latest_plugin_call(
-            output)
-        if action:
-            action_input = replace_upload_fname(action_input,
-                                                upload_fname_list)
-            observation = call_plugin(action, [action_input],
-                                      clear=(exec_count == 0))
-            output += react_prompt_obj.build_observation(observation)
-            text += output
-            exec_count += 1
-            if 'error:' in observation or 'Traceback' in observation:
-                break
-        else:
-            text += output
-            break
-    return text
-
-
-def text_completion(llm, input_text, stop_words=[]):
-    logging.info('Generating'.center(60, '='))
-    logging.info('Input'.center(60, '-'))
-    logging.info(input_text)
-
-    output = llm.generate(input_text, stop_words)
-
-    logging.info('Output'.center(60, '-'))
-    logging.info(output)
-    return output
-
-
-def call_plugin(plugin_name, plugin_args_list, clear=False):
-    # Relax constraints on plugin name.
-    logging.info('Call code interpreter'.center(60, '='))
-    obs = code_interpreter(plugin_args_list, clear=clear)
-    logging.info(obs)
-    return obs
-
-
-def process_code_interpreter(item, writer):
-    query = item['query']
-    exec_limit = 3 if 'visualization' in item['tags'] else 1
-    response = llm_with_plugin(args=args,
-                               query=query,
-                               item=item,
-                               exec_limit=exec_limit)
-    item['gen'] = response
-
-    writer.write(json.dumps(item, ensure_ascii=False) + '\n')
-    writer.flush()
-
-
-def process_gsm8k(doc, writer):
-    context = doc['question']
-    completion = llm_with_plugin(args=args, query=context)
-    acc = is_correct(completion, doc['answer'])
-    doc['completion'] = completion
-    doc['acc'] = acc
-
-    writer.write(json.dumps(doc, ensure_ascii=False) + '\n')
-    writer.flush()
-
-
-def sequential_processing(args, data_list, process_func, writer):
-    for item in tqdm.tqdm(data_list):
-        process_func(item, writer)
-
-
-process_func_map = {
-    'gsm8k': process_gsm8k,
-    'visualization': process_code_interpreter
-}
-
-
-def gather_eval_result(model_name):
-    for metric in global_eval_result:
-        logging.info(metric)
-        table = prettytable.PrettyTable()
-        table.field_names = ['model'] + list(global_eval_result[metric].keys())
-        row_data = [model_name]
-        for item in global_eval_result[metric].values():
-            item = str(item) if not item else str(round(item, 2))
-            row_data.append(item)
-        table.add_row(row_data)
-        logging.info('\n' + str(table))
-
-
-def eval_metrics(args, test_set, full_output_fname):
-    # metrics
-    assert os.path.exists(
-        full_output_fname), f'Not Found File {full_output_fname}.'
-    inference_res = load_jsonl(full_output_fname)
-    assert len(inference_res) == len(
-        test_set
-    ), f'There are still {len(test_set)-len(inference_res)} cases left.'
-
-    abs_output_fname = os.path.join(os.path.dirname(os.path.abspath(__file__)),
-                                    full_output_fname)
-    if args.task == 'gsm8k':
-        math_code_correctness = eval_gsm8k_acc(abs_output_fname)
-        global_eval_result['code_correctness'].update(math_code_correctness)
-    else:
-        code_executability = eval_code_execution_rate(abs_output_fname,
-                                                      args.task, args.model)
-        global_eval_result['code_executability'].update(code_executability)
-        if args.task in ['all_ci', 'visualization'
-                         ] and not args.eval_code_exec_only:
-            visualization_code_correctness = eval_visualization_acc(
-                abs_output_fname, args.model, args.vis_judger)
-            global_eval_result['code_correctness'].update(
-                visualization_code_correctness)
-
-
-def main(args):
-    current_dir = os.getcwd()
-    os.makedirs(args.output_path, exist_ok=True)
-    full_output_fname = os.path.join(
-        args.output_path,
-        (args.output_fname or f'{args.task}_{args.model}_res.jsonl'))
-
-    if not os.path.exists(full_output_fname):
-        with open(full_output_fname, 'w'):
-            logging.info(f'Create file {full_output_fname} done.')
-
-    # build data
-    if args.task == 'gsm8k':
-        dataset = load_dataset('gsm8k', 'main')
-        test_set = dataset['test']
-    else:
-        eval_data_path = os.path.join(args.input_path, args.input_fname)
-        test_set = [
-            item for item in load_jsonl(eval_data_path)
-            if args.task in item['tags']
-        ]
-    logging.info(f'Test set: {len(test_set)}')
-
-    if args.eval_only:
-        eval_metrics(args, test_set, full_output_fname)
-    else:
-        key = 'question' if args.task == 'gsm8k' else 'query'
-        cache_question = [item[key] for item in load_jsonl(full_output_fname)
-                          ] if not args.force else []
-        data_list = [
-            item for item in test_set if item[key] not in cache_question
-        ]
-        logging.info(f'Left cases: {len(data_list)}')
-
-        # inference
-        writer_mode = 'w' if args.force else 'a'
-        f_output = open(full_output_fname, writer_mode, encoding='utf-8')
-        process_func = process_func_map.get(args.task,
-                                            process_code_interpreter)
-        sequential_processing(args, data_list, process_func, f_output)
-        f_output.close()
-
-        # evaluate
-        if not args.gen_exec_only:
-            eval_metrics(args, test_set, full_output_fname)
-
-    os.chdir(current_dir)
-
-
-def parse_args():
-    parser = argparse.ArgumentParser()
-    parser.add_argument('--model',
-                        type=str,
-                        default='qwen-14b-chat',
-                        choices=list(model_path_map.keys()))
-    parser.add_argument(
-        '--task',
-        type=str,
-        default='all',
-        choices=['all', 'gsm8k', 'visualization', 'general'])
-    parser.add_argument('--output-path', type=str, default='output_data')
-    parser.add_argument('--input-path', type=str, default='eval_data')
-    parser.add_argument('-o', '--output-fname', type=str, default='')
-    parser.add_argument('-i',
-                        '--input-fname',
-                        type=str,
-                        default='eval_code_interpreter_v1.jsonl')
-    parser.add_argument('-f', '--force', action='store_true', default=False)
-    parser.add_argument('--eval-only', action='store_true', default=False)
-    parser.add_argument('--eval-code-exec-only',
-                        action='store_true',
-                        default=False)
-    parser.add_argument('--gen-exec-only', action='store_true', default=False)
-    parser.add_argument('--gen-only', action='store_true', default=False)
-    parser.add_argument('--vis-judger', type=str, default="'gpt-4-vision-preview'",
-                        choices=['gpt-4-vision-preview', 'qwen-vl-chat', 'qwen-vl-plus'])
-    args = parser.parse_args()
-    return args
-
-
-if __name__ == '__main__':
-    args = parse_args()
-    if not args.eval_only:
-        args.llm = get_model(args.model)
-        logging.info(f'Init {args.model} done.')
-
-    if args.task == 'all':
-        for key in ['gsm8k', 'visualization', 'general']:
-            args.task = key
-            main(args)
-    else:
-        main(args)
-    gather_eval_result(args.model)

benchmark/metrics/code_execution.py

@@ -1,257 +0,0 @@
-import logging
-import os
-
-import func_timeout
-from config import get_react_parser
-from func_timeout import func_set_timeout
-from utils.code_utils import extract_code, replace_upload_fname
-from utils.data_utils import load_jsonl, save_jsonl
-
-pre_load = """
-import os
-if 'upload_file' not in os.getcwd():
-    os.chdir("./upload_file/")
-
-import seaborn as sns
-
-import matplotlib
-# matplotlib.use('Agg')
-import matplotlib.pyplot as plt
-plt.ion()
-
-import numpy as np
-import pandas as pd
-from sympy import Eq, symbols, solve
-import re
-import json
-import math
-"""
-
-tags_config = {
-    'visualization': {
-        'timelimit': True,
-        'extract_first_code': True,
-    },
-    'math': {
-        'timelimit': True,
-        'extract_first_code': False,
-    },
-    'general': {
-        'timelimit': False,
-        'extract_first_code': True,
-    }
-}
-
-code_executability = {'math': None, 'visualization': None, 'general': None}
-
-
-@func_set_timeout(10)
-def exec_limit_time(text):
-    exec(text, locals())
-
-
-def exec_code(text, timelimit=False):
-    if timelimit:
-        exec_limit_time(text)
-    else:
-        exec(text, locals())
-
-
-def postprocess_code(gen_code, line):
-    if '<|im_start|>' in line['query']:
-        first_action_code = get_action_input_code(line['query'])
-        gen_code = first_action_code + gen_code
-
-    upload_fname_list = line[
-        'input_file_path'] if line and 'input_file_path' in line else []
-    gen_code = replace_upload_fname(gen_code, upload_fname_list)
-
-    if 'def solution()' in gen_code:
-        gen_code += '\nsolution()\n'
-
-    if 'plt.show()' in gen_code:
-        gen_code += "\nplt.pause(1)\nplt.close('all')\n"
-
-    if 'sns.' in gen_code and 'plot' in gen_code:
-        gen_code += "\nplt.close('all')\n"
-
-    gen_code = pre_load + gen_code
-    return gen_code
-
-
-def get_action_input_code(text,
-                          model_name='qwen-14b-chat',
-                          extract_first_code=False):
-    action_input_list = []
-    tmp = text
-    react_parser = get_react_parser(model_name)
-    while True:
-        action_input = react_parser.get_first_action_input(tmp)
-        if not action_input:
-            break
-        action_input_list.append(action_input)
-        tmp = tmp.split(action_input)[1]
-        if not tmp or extract_first_code:
-            break
-
-    code = ''
-    for action_input in action_input_list:
-        code = code + '# concat\n' + extract_code(action_input) + '\n'
-    return code
-
-
-def eval_code_execution_rate(output_fname,
-                             tag='all_ci',
-                             model_name='qwen-14b-chat',
-                             timelimit=False,
-                             extract_first_code=False):
-    data_list = load_jsonl(output_fname)
-    pip_package = []
-
-    for line_id, line in enumerate(data_list):
-        line['idx'] = line_id
-        tags_list = line['tags'].split(',')
-        if tag not in tags_list:
-            continue
-
-        # update args
-        for cur_tag in tags_list:
-            if cur_tag != 'all_ci':
-                timelimit = tags_config[cur_tag]['timelimit']
-                extract_first_code = tags_config[cur_tag]['extract_first_code']
-
-        line['executable_code'] = False
-        line['missing_code'] = False
-        line['code_error_info'] = ''
-
-        # get Action Input code from response
-        gen_code = get_action_input_code(line['gen'],
-                                         model_name=model_name,
-                                         extract_first_code=extract_first_code)
-
-        if not gen_code:
-            line['missing_code'] = True
-            line['code'] = ''
-            line['code_error_info'] = 'missing code'
-            continue
-
-        line['code'] = gen_code
-        gen_code = postprocess_code(gen_code, line)
-
-        while True:
-            try:
-                exec_code(gen_code, timelimit=timelimit)
-                line['executable_code'] = True
-                break
-            except func_timeout.exceptions.FunctionTimedOut as ex:
-                line['code_error_info'] = str(ex)
-                break
-            except (ImportError, ModuleNotFoundError) as ex:
-                try:
-                    packege = str(ex).split("'")[1].strip()
-                except Exception:
-                    packege = ''
-                if packege and packege not in pip_package:  # install package
-                    pip_package.append(packege)
-                    os.system('pip install ' + packege)
-                    logging.info(f'Automatic installation: {packege}')
-                else:
-                    line['code_error_info'] = str(ex)
-                    break
-            except Exception as ex:
-                line['code_error_info'] = str(ex)
-                break
-
-        # double check
-        observation = get_react_parser(model_name).get_first_observation(
-            line['gen'])
-        if line['executable_code'] and ('error:' in observation):
-            logging.warning(
-                'The code executes correctly, but it has an error in IPython!')
-            logging.warning(f'Code:\n{gen_code}')
-            logging.warning(f'IPython error info:\n{observation}')
-            logging.info('=' * 60)
-        elif not line['executable_code'] and not ('error:' in observation):
-            logging.warning(
-                'The code has an execution error, but it runs correctly in IPython!'
-            )
-            logging.warning(f'Code:\n{gen_code}')
-            logging.warning(f"Exec error info:\n{line['code_error_info']}")
-            logging.warning(f'IPython observation:\n{observation}')
-            logging.info('=' * 60)
-
-    # save error data
-    error_data_list = [
-        item for item in data_list
-        if not item['executable_code'] or item['missing_code']
-    ]
-    error_data_output_fname = os.path.splitext(
-        output_fname)[0] + '_exec_error.jsonl'
-    save_jsonl(error_data_list, error_data_output_fname)
-
-    log_result(data_list)
-
-    return code_executability
-
-
-def log_result(data_list, verbose=True):
-    if verbose:
-        logging.info('*' * 60)
-        logging.info('{:^60}'.format('Detail'))
-        logging.info('*' * 60)
-        for line_id, line in enumerate(data_list):
-            logging.info(f'Question {line_id}'.center(60, '='))
-            logging.info(line['query'])
-
-            logging.info(f'Generated {line_id}'.center(60, '-'))
-            logging.info('\n' + line['gen'])
-
-            logging.info(f'Code {line_id}'.center(60, '-'))
-            logging.info('\n' + line['code'])
-
-            logging.info(f'Exec Result {line_id}'.center(60, '-'))
-            prefix_info = 'Exec Success' if line[
-                'executable_code'] else 'Exec Error: '
-            exec_info = prefix_info + line['code_error_info']
-            logging.info(exec_info)
-
-    logging.info('=' * 60)
-    logging.info('{:^60}'.format('Code Execuation Rate'))
-    logging.info('=' * 60)
-    involved_tags = []
-    for line in data_list:
-        involved_tags += line['tags'].split(',')
-    involved_tags = list(set(involved_tags))
-
-    for key in involved_tags:
-        logging.info(f'task: {key}'.center(60, '='))
-        key_item_list = [item for item in data_list if key in item['tags']]
-        all_count = len(key_item_list)
-        missing_code_count = len(
-            [item for item in key_item_list if item['missing_code']])
-        executable_code_count = len(
-            [item for item in key_item_list if item['executable_code']])
-
-        logging.info(f'All Test: {all_count}')
-        logging.info(f'Missing Code: {missing_code_count}')
-        logging.info(f'Predict Exec Success: {executable_code_count}')
-        logging.info('Codes available && Execution Rate: {:.2f}'.format(
-            executable_code_count / (all_count - missing_code_count) * 100))
-        logging.info('Execution Rate: {:.2f}'.format(executable_code_count /
-                                                     all_count * 100))
-        logging.info('Non-executable rate: {:.2f}'.format(
-            (all_count - missing_code_count - executable_code_count) /
-            all_count * 100))
-        logging.info('Missing code rate: {:.2f}'.format(missing_code_count /
-                                                        all_count * 100))
-
-        if key != 'all_ci':
-            code_executability[key] = executable_code_count / all_count * 100
-
-        if verbose:
-            logging.info('Error List: ')
-            error_list = [(item['idx'], item['code_error_info'])
-                          for item in key_item_list if item['code_error_info']]
-            error_list.sort(key=lambda x: x[1])
-            for x in error_list:
-                logging.info(x)

benchmark/metrics/gsm8k.py

@@ -1,54 +0,0 @@
-import logging
-import os
-import re
-
-import numpy as np
-from utils.data_utils import load_jsonl, save_jsonl
-
-INVALID_ANS = '[invalid]'
-
-
-def extract_answer(completion):
-
-    def _get_last_digit(s):
-        _PAT_LAST_DIGIT = re.compile(
-            r'(?<=(\s|[\$%#{]))([+-])?(?=(\S))(0|([1-9](\d*|\d{0,2}(,\d{3})*)))?(\.\d*[1-9])?(?=(\s|[.,}]|$))'
-        )
-        match = list(_PAT_LAST_DIGIT.finditer(s))
-        if match:
-            last_digit = match[-1].group().replace(',', '').replace('+', '')
-        else:
-            last_digit = None
-            logging.warning(f'No digits found in {s!r}')
-        return last_digit
-
-    job_gen = completion.strip('.').replace('\n', '\\n')
-    last_digit = _get_last_digit(job_gen)
-    if last_digit:
-        return eval(last_digit)
-    else:
-        return INVALID_ANS
-
-
-def is_correct(completion, answer):
-    gold = extract_answer(answer)
-    assert gold != INVALID_ANS, 'No ground truth answer found in the document.'
-    return extract_answer(completion) == gold
-
-
-def eval_gsm8k_acc(output_fname):
-    data_list = load_jsonl(output_fname)
-    acc_res = [item['acc'] for item in data_list]
-    logging.info('=' * 60)
-    logging.info('{:^60}'.format('Math Acc.'))
-    logging.info('=' * 60)
-    logging.info('Total num={:.2f}'.format(len(acc_res)))
-    logging.info('Right num={:.2f}'.format(np.sum(acc_res)))
-    logging.info('Zero-shot Acc={:.2f}'.format(np.mean(acc_res) * 100))
-
-    error_data_list = [item for item in data_list if not item['acc']]
-    error_data_output_fname = os.path.splitext(
-        output_fname)[0] + '_gsm8k_error.jsonl'
-    save_jsonl(error_data_list, error_data_output_fname)
-
-    return {'math': np.mean(acc_res) * 100}

benchmark/metrics/visualization.py

@@ -1,179 +0,0 @@
-import logging
-import os
-import re
-import base64
-import torch
-from config import get_model, get_react_parser
-from utils.data_utils import load_jsonl, save_jsonl
-
-torch.manual_seed(1234)
-
-EVAL_VISUAL_PROMPT_ZH = """请判断图片是否与下面的[问题]一致，如果一致则回复“right”，不一致则回复“wrong”。
-[问题]：{query}
-"""
-
-EVAL_VISUAL_PROMPT_EN = """Please judge whether the image is consistent with the [Question] below, if it is consistent then reply "right", if not then reply "wrong".
-[Question]: {query}
-"""
-
-visualization_code_correctness = {
-    'visualization-hard': None,
-    'visualization-easy': None,
-}
-
-
-def encode_image(image_path):
-    with open(image_path, "rb") as image_file:
-        a = base64.b64encode(image_file.read()).decode('utf-8')
-    return a
-
-
-def judger_model_inference(judger_model_name, judger_model, imgs=[], prompt=''):
-    output = ""
-    if judger_model_name == 'gpt-4-vision-preview':
-        logging.warning("This is an example of `gpt-4-vision-preview`. "
-                        "Please set the API key and use according to your actual situation.")
-        from openai import OpenAI
-        client = OpenAI()
-        content_list = []
-        content_list.append({"type": "text", "text": prompt})
-        input_images = []
-        for img in imgs:
-            if 'http' not in img:
-                base64_image = encode_image(img)
-                img = f"data:image/jpeg;base64,{base64_image}"
-            input_images.append({"type": "image_url", 'image_url': img})
-        content_list.extend(input_images)
-        response = client.chat.completions.create(
-            model="gpt-4-vision-preview",
-            messages=[
-                {
-                    "role": "user",
-                    "content": content_list,
-                }
-            ],
-            max_tokens=300,
-        )
-        output = response.choices[0]
-    elif judger_model_name in ['qwen-vl-plus', 'qwen-vl-chat']:
-        inputs = []
-        for img in imgs:
-            if 'http' not in img and judger_model_name == 'qwen-vl-plus':
-                img = "file://" + img
-            inputs.append({'image': img})
-        inputs.append({'text': prompt})
-
-        logging.info('Eval'.center(60, '-'))
-        logging.info(inputs)
-        output = judger_model.generate(inputs)
-    logging.info(output)
-    logging.info('=' * 60)
-    return output
-
-
-def extract_images(text):
-    regex = re.compile(r'!\[fig-(.+)\]\((.+)\)')
-    results = re.findall(regex, text)
-    images = []
-    for res in results:
-        assert len(res) == 2
-        if os.path.exists(res[1]):
-            images.append(res[1])
-    return images
-
-
-def check_images_observation(text, images, model_name):
-    start_flag = get_react_parser(model_name).observation
-    for image in images:
-        logging.info('Image'.center(60, '-'))
-        logging.info(image)
-
-        end_idx = text.find(image)
-        tmp_text = text[:end_idx + len(image)]
-        start_idx = tmp_text.rfind(start_flag)
-        check_text = tmp_text[start_idx + len(start_flag):]
-
-        logging.info('Observation'.center(60, '-'))
-        logging.info(check_text)
-
-        # As long as there exists correctly executed observation, we consider `True`
-        if 'error:' not in check_text and 'Traceback' not in check_text:
-            return True
-    return False
-
-
-eval_visual_prompt = {'zh': EVAL_VISUAL_PROMPT_ZH, 'en': EVAL_VISUAL_PROMPT_EN}
-
-
-def eval_visualization_acc(output_fname, model_name, judger_model_name='gpt-4-vision-preview'):
-    if judger_model_name == 'gpt-4-vision-preview':
-        judger_model = None
-    elif judger_model_name in ['qwen-vl-chat', 'qwen-vl-plus']:
-        if judger_model_name == 'qwen-vl-chat':
-            logging.warning('In this benchmark of version 20231206, `Qwen-vl-chat` is no longer used as the '
-                            'evaluation model for `Visualization` task.. If you insist on using it, '
-                            'the evaluation results might differ from the official results.')
-        judger_model = get_model(judger_model_name)
-    else:
-        raise Exception("Not supported judger model.")
-
-    one_action, one_action_right = 0, 0
-    zero_action, zero_action_right = 0, 0
-
-    data_list = load_jsonl(output_fname)
-    for item in data_list:
-        if 'visualization' not in item['tags']:
-            continue
-
-        item['vis_acc'] = False
-        if '<|im_end|>' in item['query']:
-            one_action += 1
-            prompt = item['query'].split('<|im_end|>')[0]
-        else:
-            zero_action += 1
-            prompt = item['query']
-
-        images = extract_images(item['gen'])
-
-        if images and check_images_observation(item['gen'], images,
-                                               model_name):
-            input_prompt = eval_visual_prompt[item.get('lang', 'en')]
-            format_prompt = input_prompt.format(query=prompt)
-            output = judger_model_inference(judger_model_name, judger_model, images, format_prompt)
-            if 'right' in output.lower():
-                item['vis_acc'] = True
-                if '<|im_end|>' in item['query']:
-                    one_action_right += 1
-                else:
-                    zero_action_right += 1
-
-    logging.info('*' * 60)
-    logging.info('{:^60}'.format('Visualization Acc.'))
-    logging.info('*' * 60)
-    logging.info(
-        'Visualization-Hard count={}, Visualization-Hard right count={}, Visualization-Hard acc={:.2f}'
-        .format(zero_action, zero_action_right,
-                zero_action_right / zero_action * 100))
-    logging.info(
-        'Visualization-Easy count={}, Visualization-Easy right count={}, Visualization-Easy acc={:.2f}'
-        .format(one_action, one_action_right,
-                one_action_right / one_action * 100))
-    logging.info('all count={}, all right={}, all acc={:.2f}'.format(
-        zero_action + one_action, zero_action_right + one_action_right,
-        (zero_action_right + one_action_right) / (zero_action + one_action) *
-        100))
-
-    visualization_code_correctness[
-        'visualization-hard'] = zero_action_right / zero_action * 100
-    visualization_code_correctness[
-        'visualization-easy'] = one_action_right / one_action * 100
-
-    error_data_list = [
-        item for item in data_list
-        if 'visualization' in item['tags'] and not item['vis_acc']
-    ]
-    error_data_output_fname = os.path.splitext(
-        output_fname)[0] + '_vis_error.jsonl'
-    save_jsonl(error_data_list, error_data_output_fname)
-
-    return visualization_code_correctness

benchmark/models/__init__.py

@@ -1,4 +0,0 @@
-from models.base import HFModel  # noqa
-from models.llm import LLM  # noqa
-from models.qwen import Qwen, QwenVL  # noqa
-from models.dashscope import QwenDashscopeVLModel

benchmark/models/base.py

@@ -1,17 +0,0 @@
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from transformers.generation import GenerationConfig
-
-
-class HFModel(object):
-
-    def __init__(self, model_path):
-        self.tokenizer = AutoTokenizer.from_pretrained(model_path,
-                                                       trust_remote_code=True)
-        self.model = AutoModelForCausalLM.from_pretrained(
-            model_path,
-            trust_remote_code=True,
-            device_map='auto',
-            low_cpu_mem_usage=True).eval()
-        self.model.generation_config = GenerationConfig.from_pretrained(
-            model_path, trust_remote_code=True)
-        self.model.generation_config.do_sample = False

benchmark/models/dashscope.py

@@ -1,40 +0,0 @@
-import logging
-from http import HTTPStatus
-import time
-import dashscope
-
-
-class QwenDashscopeVLModel(object):
-    def __init__(self, model, api_key):
-        self.model = model
-        dashscope.api_key = api_key.strip() or os.getenv('DASHSCOPE_API_KEY', default='')
-        assert dashscope.api_key, 'DASHSCOPE_API_KEY is required.'
-
-    def generate(self, prompt, stop_words=[]):
-        if isinstance(prompt, str):
-            prompt = [{'text': prompt}]
-
-        MAX_TRY = 3
-        count = 0
-        while count < MAX_TRY:
-            response = dashscope.MultiModalConversation.call(
-                self.model,
-                messages=[{'role': 'user', 'content': prompt}],
-                top_p=0.01,
-                top_k=1,
-            )
-            if response.status_code == HTTPStatus.OK:
-                output = response.output.choices[0].message.content[0]['text']
-                for stop_str in stop_words:
-                    idx = output.find(stop_str)
-                    if idx != -1:
-                        output = output[: idx + len(stop_str)]
-                return output
-            else:
-                err = 'Error code: %s, error message: %s' % (
-                    response.code,
-                    response.message,
-                )
-                logging.error(err)
-                count += 1
-            time.sleep(1)

benchmark/models/llm.py

@@ -1,26 +0,0 @@
-import torch
-from models.base import HFModel
-
-
-class LLM(HFModel):
-
-    def __init__(self, model_path):
-        super().__init__(model_path)
-
-    def generate(self, input_text, stop_words=[], max_new_tokens=512):
-        if isinstance(input_text, str):
-            input_text = [input_text]
-
-        input_ids = self.tokenizer(input_text)['input_ids']
-        input_ids = torch.tensor(input_ids, device=self.model.device)
-        gen_kwargs = {'max_new_tokens': max_new_tokens, 'do_sample': False}
-        outputs = self.model.generate(input_ids, **gen_kwargs)
-        s = outputs[0][input_ids.shape[1]:]
-        output = self.tokenizer.decode(s, skip_special_tokens=True)
-
-        for stop_str in stop_words:
-            idx = output.find(stop_str)
-            if idx != -1:
-                output = output[:idx + len(stop_str)]
-
-        return output

benchmark/models/qwen.py

@@ -1,36 +0,0 @@
-import torch
-from models.base import HFModel
-
-
-class Qwen(HFModel):
-
-    def __init__(self, model_path):
-        super().__init__(model_path)
-
-    def generate(self, input_text, stop_words=[]):
-        im_end = '<|im_end|>'
-        if im_end not in stop_words:
-            stop_words = stop_words + [im_end]
-        stop_words_ids = [self.tokenizer.encode(w) for w in stop_words]
-
-        input_ids = torch.tensor([self.tokenizer.encode(input_text)
-                                  ]).to(self.model.device)
-        output = self.model.generate(input_ids, stop_words_ids=stop_words_ids)
-        output = output.tolist()[0]
-        output = self.tokenizer.decode(output, errors='ignore')
-        assert output.startswith(input_text)
-        output = output[len(input_text):].replace('<|endoftext|>',
-                                                  '').replace(im_end, '')
-
-        return output
-
-
-class QwenVL(HFModel):
-    def __init__(self, model_path):
-        super().__init__(model_path)
-
-    def generate(self, inputs: list):
-        query = self.tokenizer.from_list_format(inputs)
-        response, _ = self.model.chat(self.tokenizer, query=query, history=None)
-
-        return response

benchmark/parser/__init__.py

@@ -1,2 +0,0 @@
-from parser.internlm_parser import InternLMReActParser  # noqa
-from parser.react_parser import ReActParser  # noqa

benchmark/parser/internlm_parser.py

@@ -1,11 +0,0 @@
-from parser.react_parser import ReActParser
-
-
-class InternLMReActParser(ReActParser):
-
-    def __init__(self):
-        self.action = '\nAction:'
-        self.action_input = '\nActionInput:'
-        self.action_input_stop = '<eoa>'
-        self.observation = '<|System|>:Response:'
-        self.observation_stop = '<TOKENS_UNUSED_2>\n<|Bot|>:'

benchmark/parser/react_parser.py

@@ -1,46 +0,0 @@
-class ReActParser(object):
-
-    def __init__(self):
-        self.action = '\nAction:'
-        self.action_input = '\nAction Input:'
-        self.action_input_stop = '\nObservation:'
-        self.observation = '\nObservation:'
-        self.observation_stop = '\nThought:'
-
-    def parse_latest_plugin_call(self, text):
-        action = self.action
-        action_input = self.action_input
-        observation = self.action_input_stop
-        plugin_name, plugin_args = '', ''
-        i = text.rfind(action)
-        j = text.rfind(action_input)
-        k = text.rfind(observation)
-        if 0 <= i < j:  # If the text has `Action` and `Action input`,
-            if k < j:  # but does not contain `Observation`,
-                # then it is likely that `Observation` is ommited by the LLM,
-                # because the output text may have discarded the stop word.
-                text = text.rstrip() + observation  # Add it back.
-            k = text.rfind(observation)
-            plugin_name = text[i + len(action):j].strip()
-            plugin_args = text[j + len(action_input):k].strip()
-            text = text[:k]
-        return plugin_name, plugin_args, text
-
-    def _extract_first_target(self, text, start_flag, end_flag):
-        target = ''
-        i = text.find(start_flag)
-        if i != -1:
-            j = text.find(end_flag, i)
-            if j != -1:
-                target = text[i + len(start_flag):j].strip()
-            else:
-                target = text[i + len(start_flag):].strip()
-        return target
-
-    def get_first_observation(self, text):
-        return self._extract_first_target(text, self.observation,
-                                          self.observation_stop)
-
-    def get_first_action_input(self, text):
-        return self._extract_first_target(text, self.action_input,
-                                          self.action_input_stop)

benchmark/prompt/__init__.py

@@ -1,4 +0,0 @@
-from prompt.internlm_react import InternLMReAct  # noqa
-from prompt.llama_react import LlamaReAct  # noqa
-from prompt.qwen_react import QwenReAct  # noqa
-from prompt.react import ReAct  # noqa

benchmark/prompt/internlm_react.py

@@ -1,103 +0,0 @@
-from prompt.react import ReAct
-
-INTERNLM_TOOL_DESCRIPTION = """用来执行Python代码。代码必须是一个函数，
-函数名必须得是 'solution'，代码对应你的思考过程。代码实例格式如下：
-```python
-# import 依赖包
-import xxx
-def solution():
-    # 初始化一些变量
-    variable_names_with_real_meaning = xxx
-    # 步骤一
-    mid_variable = func(variable_names_with_real_meaning)
-    # 步骤 x
-    mid_variable = func(mid_variable)
-    # 最后结果
-    final_answer =  func(mid_variable)
-    return final_answer
-```"""
-
-INTERNLM_TOOL = {'PythonInterpreter': INTERNLM_TOOL_DESCRIPTION}
-
-INTERNLM_REACT_PROMPT_ZH = """<|System|>:你是一个可以调用外部工具的助手，可以使用的工具包括：
-{tools_text}
-如果使用工具请遵循以下格式回复：
-```
-Thought:思考你当前步骤需要解决什么问题，是否需要使用工具
-Action:工具名称，你的工具必须从 [{tools_name_text}] 选择
-ActionInput:工具输入参数
-```
-工具返回按照以下格式回复：
-```
-Response:调用工具后的结果
-```
-如果你已经知道了答案，或者你不需要工具，请遵循以下格式回复
-```
-Thought:给出最终答案的思考过程
-FinalAnswer:最终答案
-```
-开始!<TOKENS_UNUSED_2>
-<|User|>:{query}<eoh>
-<|Bot|>:"""
-
-INTERNLM_REACT_PROMPT_EN = """<|System|>:You are a assistant who can utilize external tools.
-{tools_text}
-To use a tool, please use the following format:
-```
-Thought: Think what you need to solve, do you need to use tools?
-Action: the tool name, should be one of [{tools_name_text}]
-ActionInput: the input to the action
-```
-The response after utilizing tools should using the following format:
-```
-Response: the results after call the tool.
-``
-If you already know the answer, or you do not need to use tools,
-please using the following format to reply:
-```
-Thought: the thought process to get the final answer
-FinalAnswer: final answer
-```
-Begin!<TOKENS_UNUSED_2>
-<|User|>:{query}<eoh>
-<|Bot|>:"""
-
-
-class InternLMReAct(ReAct):
-
-    def __init__(self, query, lang='en', upload_file_paths=[]):
-        super().__init__(query, lang, upload_file_paths)
-        self.react_template = INTERNLM_REACT_PROMPT_ZH if self.lang == 'zh' else INTERNLM_REACT_PROMPT_EN
-
-    def build_prompt(self):
-        planning_prompt = super().build_prompt()
-        if '<|im_end|>' in self.query and planning_prompt.endswith(
-                '<eoh>\n<|Bot|>:'):
-            planning_prompt = planning_prompt[:-len('<eoh>\n<|Bot|>:')]
-
-        if '<|im_end|>' in self.query:
-            planning_prompt = planning_prompt.replace(
-                '<|im_end|>\n<|im_start|>assistant\n',
-                '<eoh>\n<|Bot|>:').replace(
-                    'Observation:', '<eoa>\n<|System|>:Response:').replace(
-                        '\nAction Input',
-                        '\nActionInput').replace('code_interpreter',
-                                                 'PythonInterpreter')
-            assert planning_prompt.endswith('Thought:')
-            planning_prompt = planning_prompt[:-len(
-                'Thought:')] + '<TOKENS_UNUSED_2>\n<|Bot|>:'
-
-        self.prompt = planning_prompt
-        return planning_prompt
-
-    def _build_tools_text(self):
-        return INTERNLM_TOOL
-
-    def _build_tools_name_text(self):
-        return list(INTERNLM_TOOL.keys())
-
-    def build_observation(self, observation):
-        return f'<eoa>\n<|System|>:Response:{observation}\n<TOKENS_UNUSED_2>\n<|Bot|>:'
-
-    def get_stop_words_list(self):
-        return ['<eoa>']

benchmark/prompt/llama_react.py

@@ -1,20 +0,0 @@
-from prompt.react import ReAct
-
-
-class LlamaReAct(ReAct):
-
-    def __init__(self, query, lang='en', upload_file_paths=[]):
-        super().__init__(query, lang, upload_file_paths)
-
-    def build_prompt(self):
-        planning_prompt = super().build_prompt()
-        planning_prompt = '[INST] ' + planning_prompt + ' [/INST]'
-
-        if '<|im_end|>' in self.query:
-            planning_prompt = planning_prompt.replace(
-                '<|im_end|>\n<|im_start|>assistant', ' [/INST] ')
-            assert planning_prompt.endswith(' [/INST]')
-            planning_prompt = planning_prompt[:-len(' [/INST]')]
-
-        self.prompt = planning_prompt
-        return planning_prompt

benchmark/prompt/qwen_react.py

@@ -1,80 +0,0 @@
-import json
-import os
-
-from prompt.react import ReAct
-
-QWEN_TOOLS_LIST = [
-    {
-        'name_for_human': '代码解释器',
-        'name_for_model': 'code_interpreter',
-        'description_for_model': '代码解释器，可用于执行Python代码。',
-        'parameters': [{
-            'name': 'code',
-            'type': 'string',
-            'description': '待执行的代码'
-        }],
-        'args_format': 'code'
-    },
-]
-
-TOOL_DESC = """{name_for_model}: Call this tool to interact with the {name_for_human} API. What is the {name_for_human} API useful for? {description_for_model} Parameters: {parameters}"""
-
-
-class QwenReAct(ReAct):
-
-    def __init__(self, query, lang='en', upload_file_paths=[]):
-        super().__init__(query, lang, upload_file_paths)
-
-        self.upload_file_paths = [
-            f'{os.path.basename(fname)}' for fname in upload_file_paths
-        ]
-        self.list_of_plugin_info = QWEN_TOOLS_LIST
-        self.fname_template = {
-            'zh': '[上传文件{fname_str}]',
-            'en': '[Upload file {fname_str}]',
-            'en_multi': '[Upload file {fname_str}]'
-        }
-
-    def build_prompt(self):
-        im_start = '<|im_start|>'
-        im_end = '<|im_end|>'
-        prompt = f'{im_start}system\nYou are a helpful assistant.{im_end}'
-
-        query = super().build_prompt()
-
-        query = query.lstrip('\n').rstrip()
-        prompt += f'\n{im_start}user\n{query}{im_end}'
-        if f'{im_start}assistant' not in query:
-            prompt += f'\n{im_start}assistant\n{im_end}'
-            assert prompt.endswith(f'\n{im_start}assistant\n{im_end}')
-
-        prompt = prompt[:-len(f'{im_end}')]
-        self.prompt = prompt
-        return prompt
-
-    def _build_tools_text(self):
-        # tool info
-        tools_text = []
-        for plugin_info in self.list_of_plugin_info:
-            tool = TOOL_DESC.format(
-                name_for_model=plugin_info['name_for_model'],
-                name_for_human=plugin_info['name_for_human'],
-                description_for_model=plugin_info['description_for_model'],
-                parameters=json.dumps(plugin_info['parameters'],
-                                      ensure_ascii=False),
-            )
-            if plugin_info.get('args_format', 'json') == 'json':
-                tool += ' Format the arguments as a JSON object.'
-            elif plugin_info['args_format'] == 'code':
-                tool += ' Enclose the code within triple backticks (`) at the beginning and end of the code.'
-            else:
-                raise NotImplementedError
-            tools_text.append(tool)
-        tools_text = '\n\n'.join(tools_text)
-        return tools_text
-
-    def _build_tools_name_text(self):
-        return ', '.join([
-            plugin_info['name_for_model']
-            for plugin_info in self.list_of_plugin_info
-        ])

benchmark/prompt/react.py

@@ -1,87 +0,0 @@
-import os
-
-tools_text = """code_interpreter: Call this tool to interact with the Code Interpreter API.
-What is the Code Interpreter API useful for?
-Code Interpreter is used to execute Python code to deal with the following tasks:
-1. Solving mathematical problems, both quantitative and qualitative
-2. Doing data analysis and visualization
-3. Converting files between formats
-Parameters:
-```py
-code
-```
-Enclose the code within triple backticks (```) at the beginning and end of the code.
-"""
-
-REACT_PROMPT = """Answer the following questions as best you can. You have access to the following tools:
-
-{tools_text}
-
-Use the following format:
-
-Question: the input question you must answer
-Thought: you should always think about what to do
-Action: the action to take, should be one of [{tools_name_text}]
-Action Input: the input to the action
-Observation: the result of the action
-... (this Thought/Action/Action Input/Observation can be repeated zero or more times)
-Thought: I now know the final answer
-Final Answer: the final answer to the original input question
-
-Begin!
-
-Question: {query}"""
-
-fname_template = {
-    'zh': '文件{fname_str}，',
-    'en_multi': 'Files {fname_str}. ',
-    'en': 'File {fname_str}. ',
-}
-
-
-class ReAct(object):
-
-    def __init__(self, query, lang='en', upload_file_paths=[]):
-        self.query = query
-        self.lang = lang
-        self.upload_file_paths = [
-            f'`{os.path.basename(fname)}`' for fname in upload_file_paths
-        ]
-
-        self.fname_template = fname_template
-        self.react_template = REACT_PROMPT
-        self.prompt = ''
-
-    def build_prompt(self):
-        query = self._format_upload_fname() + self.query
-        tools_text = self._build_tools_text()
-        tools_name_text = self._build_tools_name_text()
-        planning_prompt = self.react_template.format(
-            query=query,
-            tools_text=tools_text,
-            tools_name_text=tools_name_text)
-
-        self.prompt = planning_prompt
-        return planning_prompt
-
-    def _format_upload_fname(self):
-        prefix = ''
-        if self.upload_file_paths:
-            fname_str = ', '.join(self.upload_file_paths)
-            lang_key = 'en_multi' if self.lang == 'en' and len(
-                self.upload_file_paths) > 1 else self.lang
-            fname_template = self.fname_template[lang_key]
-            prefix = fname_template.format(fname_str=fname_str)
-        return prefix
-
-    def _build_tools_text(self):
-        return tools_text
-
-    def _build_tools_name_text(self):
-        return 'code_interpreter'
-
-    def build_observation(self, observation):
-        return f'\nObservation: {observation}\nThought:'
-
-    def get_stop_words_list(self):
-        return ['Observation:', 'Observation:\n']

benchmark/requirements.txt

@@ -1,13 +0,0 @@
-accelerate>=0.20.3
-func_timeout
-json5
-matplotlib
-numpy
-pandas
-PrettyTable
-scipy
-seaborn
-sympy
-transformers==4.33.1
-transformers_stream_generator
-openai

benchmark/utils/code_utils.py

@@ -1,31 +0,0 @@
-import os
-import re
-
-import json5
-
-
-def replace_upload_fname(text, upload_fname_list):
-    for full_input_fname in upload_fname_list:
-        if full_input_fname not in text and os.path.basename(
-                full_input_fname) in text:
-            text = text.replace(os.path.basename(full_input_fname),
-                                full_input_fname)
-    return text
-
-
-def extract_code(text):
-    # Match triple backtick blocks first
-    triple_match = re.search(r'```[^\n]*\n(.+?)```', text, re.DOTALL)
-    # Match single backtick blocks second
-    single_match = re.search(r'`([^`]*)`', text, re.DOTALL)
-    if triple_match:
-        text = triple_match.group(1)
-    elif single_match:
-        text = single_match.group(1)
-    else:
-        try:
-            text = json5.loads(text)['code']
-        except Exception:
-            pass
-    # If no code blocks found, return original text
-    return text

benchmark/utils/data_utils.py

@@ -1,28 +0,0 @@
-import json
-import logging
-
-from tqdm import tqdm
-
-
-def load_jsonl(path):
-    data = []
-    with open(path, 'r', encoding='utf8') as f:
-        for idx, line in enumerate(f, start=1):
-            try:
-                data.append(json.loads(line))
-            except Exception as e:
-                logging.info(line)
-                logging.warning(f'Error at line {idx}: {e}')
-                continue
-    return data
-
-
-def save_jsonl(data, path, progress=False, enabled=True):
-    if not enabled:
-        return
-    with open(path, 'w', encoding='utf-8') as f:
-        if progress:
-            data = tqdm(data)
-        for item in data:
-            line = json.dumps(item, ensure_ascii=False)
-            print(line, file=f)

setup.py

@@ -1,16 +0,0 @@
-from setuptools import find_packages, setup
-
-
-def read_requirements():
-    with open('requirements.txt') as req:
-        content = req.read()
-        requirements = content.split('\n')
-    return requirements
-
-
-setup(
-    name='qwen_agent',
-    version='0.0.1',
-    packages=find_packages(),
-    install_requires=read_requirements(),
-)