zhiminy/EvalEng huggingface dataset:
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Milvus error
Root cause: Validation Gap

MilvusException: <MilvusException: (code=1100, message=nlist out of range: [1, 65536]: invalid parameter[expected=valid index params][actual=invalid index params])>

@Aaryaveerkrishna23 Hello. Can you provide a full trace of the bug?

> [@Aaryaveerkrishna23](https://github.com/Aaryaveerkrishna23) Hello. Can you provide a full trace of the bug?

Hi , I resolved it thanks :)
I would like to know how we can add a layer of observability in auto RAG, such as using Langfuse, for example, to monitor the system in development, pre-production, and production environments , to check what chunks are being retrieved , what was the final prompt after data augmentation etc .

> [@Aaryaveerkrishna23](https://github.com/Aaryaveerkrishna23) Hello. Can you provide a full trace of the bug?

Just catching up , any updates on the observability part ?

@Aaryaveerkrishna23 Hi sorry for the late reply! 
It will be great to implement some great observability tool. 
From now, you can check the whole results in the each .parquet files in the result of AutoRAG.
But yes, it is quite hard to check it out manually 👎 

So we are happy to implement observability tool. 

https://github.com/Marker-Inc-Korea/AutoRAG/issues/990
I made an issue for it. But just let you know, there are so many issues so it can be take some time. 
Thanks for understanding:)

Since we resolve the error, I am closing the issue. 

I totally understand and thanks for considering it .
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Display AutoRAG Newsletter (New Features, Security Updates, Changelog) on CLI
GitHub labels: enhancement

**Is your feature request related to a problem? Please describe.**
Currently, users must access a separate platform or website to view the latest updates on AutoRAG, including new features, security alerts, and changelogs. This extra step can be inconvenient and may result in missed updates.


**Describe the solution you'd like**
Implement a command in the CLI that allows users to view the latest AutoRAG newsletter, covering new features, security updates, and changelogs. This would provide users with quick, in-terminal access to important updates and eliminate the need to check external sources.

**Describe alternatives you've considered**
Sending newsletters via email to subscribed users.
Creating a dashboard widget that displays updates within the GUI.
Displaying critical updates only on the CLI (e.g., security alerts) and reserving feature updates for other platforms.

**Additional context**
This feature would streamline user experience by consolidating all relevant updates into the CLI, making it easy for users to stay informed without leaving their workflow.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [API] [BUG] daemonic processes are not allowed to have children
Root cause: Resource Mishandling

When try to run parsing, this error occured. 

```
daemonic processes are not allowed to have children
```

https://github.com/pgjones/hypercorn/issues/191

Tried 
- Hypercorn : Daemon cannot have children issue
- Daphne : Task is not finished
- Uvicorn : nest_asyncio error (will not work with AutoRAG)

Wow. Just... wow

Just quart.run worked well to me. 

uvicorn app:app --host 0.0.0.0 --port 5001 --reload --loop asyncio


This worked!! YEAHHHH 
uvicorn is the GOD

close at #966
Repository: Marker-Inc-Korea/AutoRAG
Issue: 🐛 Fix WORK_DIR path resolution in development environment (API)
GitHub labels: duplicate

API

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Full Error log**
If applicable, add full error log to help explain your problem.

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
Add any other context about the problem here.


duplicate with #969
Repository: Marker-Inc-Korea/AutoRAG
Issue: 🐛 Fix WORK_DIR path resolution in development environment (api/)
Root cause: Architectural Constraint

**Describe the bug**
A clear and concise description of what the bug is.

https://github.com/Marker-Inc-Korea/AutoRAG/blob/81a2f63e26f1ed7c4d292c340f4a82fa216a6263/api/app.py#L91


**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Full Error log**
If applicable, add full error log to help explain your problem.

**Code that bug is happened**
If applicable, add the code that bug is happened.
(Especially, your AutoRAG YAML file or python codes that you wrote)

**Desktop (please complete the following information):**
 - OS: [e.g. Windows, Linux, MacOS]
 - Python version [e.g. 3.10]

**Additional context**
Add any other context about the problem here.


I am on it!

close at #971 and #968
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] azure AOAI hang to the final state
Root cause: Validation Gap

![Image](https://github.com/user-attachments/assets/7087ed36-da6a-4937-b557-0c212e72656a)

not only AOAI but also for ollama

Hi @hellangleZ Is there any bug or log?
Plus, what environment are you using?

@hellangleZ  hi guys could u share ur config.yaml and demo.py？
thanks!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Pydantic Validation Error at MockEmbedding
Root cause: External Dependency Breakage

**Describe the bug**
The Pydantic 2.10.0 has a error at the latest LlamaIndex version.

**Desktop (please complete the following information):**
Python 3.10
Pydantic 2.10.0
LlamaIndex 0.12.1
Repository: Marker-Inc-Korea/AutoRAG
Issue: [API] Add endpoint for env variable management
Root cause: Unimplemented Feature Gap

GET /env => return all env variable key

DELETE /env/<string:key> => delete the certain key as a env variable.


We will manage with '.env' file comes with app.py file. 
Load it when the api-key needed endpoint starting. (parse, chunk, qa creation, validation, evaluate, chat)

Is It work with .env.local file?

@hongsw It works with `.env` file at the api folder. API key only for running AutoRAG.

close at #975
Repository: Marker-Inc-Korea/AutoRAG
Issue: [API] fix validate and evaluation api config, set_trial_config
Root cause: Architectural Constraint

https://github.com/Auto-RAG/autorag-frontend/issues/3

`/projects/<string:project_id>/trials/<string:trial_id>/validate`
`/projects/<string:project_id>/trials/<string:trial_id>/evaluate`


Issue
Managing nested data with CSVs is inefficient, and frequent dumps are cumbersome.

Solution
Switch to SQLite with asynchronous execution managed by Celery for safe task handling.

Plan
Test project creation, trials (1, 2, 3), validation, and execution in the new setup.



draft

close at #1016
Repository: Marker-Inc-Korea/AutoRAG
Issue: [BUG] Ollama Embeddings do not working
Root cause: Resource Mishandling

**Describe the bug**
The Ollama Embeddings is not working. 

**To Reproduce**
Steps to reproduce the behavior:
1. Use Ollama Embedding as embedding model
2. Try to use it at vectordb retrieval module

The `aget_text_embedding_batch` has `asyncio.gather` itself, and it has conflict with `process_batch` function. It likely try to use different event loop.

**Full Error log**
[main_log.txt](https://github.com/user-attachments/files/17879756/main_log.txt)
Repository: Marker-Inc-Korea/AutoRAG
Issue: [API] fix Korean (UTF-8) Project Title
Root cause: Validation Gap

<!-- 새 문제의 본문을 편집한 다음 편집기의 오른쪽 상단에 있는 ✓ "문제 만들기" 버튼을 클릭합니다. 첫 번째 줄은 문제 제목이 됩니다. 담당자와 레이블은 빈 줄 뒤에 옵니다. 문제의 본문을 시작하기 전에 빈 줄을 남겨 두세요. -->

Plus the API server cannot get the korean title of file names also

I fixed this 

@vkehfdl1 which branch or commit?


@hongsw Ah sorry I misunderstood the issue.
It is "project title", not "file name"
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Add Observability tool like Langfuse
Root cause: Unimplemented Feature Gap

@Aaryaveerkrishna23

I would like to know how we can add a layer of observability in auto RAG, such as using Langfuse, for example, to monitor the system in development, pre-production, and production environments , to check what chunks are being retrieved , what was the final prompt after data augmentation etc .

---
Great idea!
Repository: Marker-Inc-Korea/AutoRAG
Issue: [API] Start chunk in trial_task.py
Root cause: Architectural Constraint

- [x] Celery integration completed.  
- [x] SQLiteProjectDB migration completed.  
- [x] `trial_task.py` is to be determined (TBD); assistance needed from @vkehfdl1.  
- [x] Frontend migration completed, including changes to the `waitForTask` function.
- [ ] The filepath configuration schema needs to be updated from `(raw_data, parse, chunk, qa)` to `(files, knowledges, qa_docs)`.
<img width="842" alt="image" src="https://github.com/user-attachments/assets/1c96f6f1-0d05-45fb-ab98-cd126e8dc9db">
Figure 1: Files are initially managed in the files folder. In the knowledges folder, selected files are processed, parsed, and chunked for further use. QA documents are generated and stored in the qa_docs folder. Lastly, in the trials folder, specific knowledge and QA documents are selected to perform evaluations and generate reports.

_Originally posted by @hongsw in https://github.com/Marker-Inc-Korea/AutoRAG/issues/987#issuecomment-2496144117_
            

resolved by me
Repository: Marker-Inc-Korea/AutoRAG
Issue: [API] The filepath configuration schema needs to be updated from (raw_data, parse, chunk, qa) to (files, knowledges, qa_docs).
Root cause: Architectural Constraint

- [x] Celery integration completed.  
- [x] SQLiteProjectDB migration completed.  
- [x] `trial_task.py` is to be determined (TBD); assistance needed from @vkehfdl1.  
- [x] Frontend migration completed, including changes to the `waitForTask` function.
- [ ] The filepath configuration schema needs to be updated from `(raw_data, parse, chunk, qa)` to `(files, knowledges, qa_docs)`.
<img width="842" alt="image" src="https://github.com/user-attachments/assets/1c96f6f1-0d05-45fb-ab98-cd126e8dc9db">
Figure 1: Files are initially managed in the files folder. In the knowledges folder, selected files are processed, parsed, and chunked for further use. QA documents are generated and stored in the qa_docs folder. Lastly, in the trials folder, specific knowledge and QA documents are selected to perform evaluations and generate reports.

_Originally posted by @hongsw in https://github.com/Marker-Inc-Korea/AutoRAG/issues/987#issuecomment-2496144117_
            

close at #1011
Repository: Marker-Inc-Korea/AutoRAG
Issue: [API] change api port from 5000 to 8000
GitHub labels: enhancement

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


close at #1007
Repository: OpenLMLab/GAOKAO-Bench
Issue: 数学I小错误
Root cause: Validation Gap

https://github.com/OpenLMLab/GAOKAO-Bench/blob/main/data/Multiple-choice_Questions/2010-2022_Math_I_MCQs.json#LL18C29-L18C184
2010年新课标数学I第二题
(5 分) 已知复数 $z=\\frac{\\sqrt{3}+i}{(1-\\sqrt{3} i)^{2}}, \\bar{z}$ 是 $z$ 的共轭复数, 则 $z=\\bar{z}=(\\quad)$\nA. $\\frac{1}{4}$\nB. $\\frac{1}{2}$\nC. 1\nD. 2\n"

应该是z \\cdot \\bar{z}

(5 分) 已知复数 $z=\\frac{\\sqrt{3}+i}{(1-\\sqrt{3} i)^{2}}, \\bar{z}$ 是 $z$ 的共轭复数, 则 $z \\cdot \\bar{z}=(\\quad)$\nA. $\\frac{1}{4}$\nB. $\\frac{1}{2}$\nC. 1\nD. 2\n"

@zzzyyi @luchyy 

Thanks for your correction! We have modified it :)
Repository: OpenLMLab/GAOKAO-Bench
Issue: 数学题格式

数学题的公式里面是用\\来表示符号的，如下面例子所示。这不是latex的标准语法，背后有什么特殊原因吗?
```
1. （5 分) 已知集合 $A=\\{x \\in R|| x \\mid \\leqslant 2\\}\\}, B=\\{x \\in Z \\mid \\sqrt{x} \\leqslant 4\\}$, 则 $A \\cap B=(\\quad$ ）\nA. $(0,2)$\nB. $[0,2]$\nC. $\\{0,2\\}$\nD. $\\{0,1,2\\}$\n
```

数据准备的时候没注意到json.dump不会翻译转义符，之后会处理

因为python里只能存储反转义的latex公式，所以\会被反转义成\\
Repository: OpenLMLab/GAOKAO-Bench
Issue: 数据收集过程

感谢您的工作，为评估大模型提供了一个很棒的基准！

想请问一下高考题是如何收集的呢？我看到网上大部分的网站都是付费下载，并且是扫描版的格式，请问您都是手动下载并且人工做的数据处理吗？

您好，高考历年真题在网络上有免费的下载源，之后使用python脚本抽取题目
Repository: OpenLMLab/GAOKAO-Bench
Issue: Solution of "Exception: list index out of range" when run `python choice_bench.py`
Root cause: Validation Gap

I meet this and spent 10mins to fig it out. now I will show you the reason and solution.  

### Issue
when you following the README.md and run `python choice_bench.py`, you may meet following error and the program loop endlessly
![image](https://github.com/OpenLMLab/GAOKAO-Bench/assets/15920086/9cee3931-940f-4569-b107-26297d618c1a)

### Reason
the regex pattern ends with a "\n", in line 16 of "GAOKAO-Bench/Bench/bench_function.py".
so if you don't add a "\n" in "GAOKAO-Bench/data/my_openai_apikey.txt", then the regex match failed.

### Solution
just add a "\n" (press Enter of your keyboard) at the end of "GAOKAO-Bench/data/my_openai_apikey.txt"


Thank you for your contribution!
Repository: OpenLMLab/GAOKAO-Bench
Issue: json. decoder.JSONDecodeError: Invalid escape: 
Root cause: Validation Gap

error:
![32ef96412c37dc4ce5507b792a97de9](https://github.com/OpenLMLab/GAOKAO-Bench/assets/73881472/2f300ea0-b792-4d89-9fa2-90659e9c62d5)
solution:
clearly, escape character, the error position is in 2010-2022_Math_I_MCQs.json related with the question "question": "2. (5 分) 已知复数 $z=\\frac{\\sqrt{3}+i}{(1-\\sqrt{3} i)^{2}}, \\bar{z}$ 是 $z$ 的共轭复数, 则 $z=\\cdot\bar{z}=(\\quad)$\nA. $\\frac{1}{4}$\nB. $\\frac{1}{2}$\nC. 1\nD. 2\n"
delete it or replace it to "question": "2. (5 分) 已知复数 $z=\\frac{\\sqrt{3}+i}{(1-\\sqrt{3} i)^{2}}, \\bar{z}$ 是 $z$ 的共轭复数, 则 $z=\\\cdot\bar{z}=(\\quad)$\nA. $\\frac{1}{4}$\nB. $\\frac{1}{2}$\nC. 1\nD. 2\n"
or \cdot -> \\\cdot

Same error.

同样的问题，在json处理的过程中因同样的报错发现。希望能够修正这个数据的错误：\c -> \\\c
/GAOKAO-Bench/data/Multiple-choice_Questions/2010-2022_Math_I_MCQs.json  line 18  column 114
![数据错误1 少了反斜杠](https://github.com/OpenLMLab/GAOKAO-Bench/assets/66634899/ff823f32-1e1f-49ad-ab0b-b0338ad6daee)


Thank you for your report! We've fixed the error in [this line](https://github.com/OpenLMLab/GAOKAO-Bench/blob/e4c84266523df2b4152893a8ca6c7cc72fecadcf/data/Multiple-choice_Questions/2010-2022_Math_I_MCQs.json#L18).
Repository: OpenLMLab/GAOKAO-Bench
Issue: Typos in the dataset.
Root cause: Validation Gap

When I parsed the options in the MCQ dataset, I found the following typos. These typos didn't follow the correct format. Could you please fix it?

- [Option C](https://github.com/OpenLMLab/GAOKAO-Bench/blob/main/data/Multiple-choice_Questions/2010-2022_Chemistry_MCQs.json#L1228) missing in index 111 in 2010-2022_Chemistry_MCQs.json
![image](https://github.com/OpenLMLab/GAOKAO-Bench/assets/70520919/50aedf7e-3a7f-4bdb-b39d-b3a0a5b06537)

- [Option B](https://github.com/OpenLMLab/GAOKAO-Bench/blob/main/data/Multiple-choice_Questions/2012-2022_English_Cloze_Test.json#L247) missing "." after it in index 16 in 2012-2022_English_Cloze_Test.json
![image](https://github.com/OpenLMLab/GAOKAO-Bench/assets/70520919/44936813-859e-4316-9675-370a12872224)

- [Option G](https://github.com/OpenLMLab/GAOKAO-Bench/blob/main/data/Multiple-choice_Questions/2012-2022_English_Cloze_Test.json#L352) missing "." after it in index 23 in 2012-2022_English_Cloze_Test.json
![image](https://github.com/OpenLMLab/GAOKAO-Bench/assets/70520919/907e7889-3472-4219-8e69-c06d8234db09)

Thank you for your correction! We've modified the format that you mentioned above:)
Repository: OpenLMLab/GAOKAO-Bench
Issue: 关于测试题目格式
Root cause: Documentation Deficiency

请教个问题，除了gpt-3.5-turbo，其它开源模型在选择题上的得分情况都普通只有100多分，是不是与题目的格式有关系?对于multi_question_choice的题目，比如2010-2022_English_Fill_in_Blanks，每个题目里面有多个题，需要模型输出多个选项，但模型可能就无法理解，或者理解了，但是没有按照规定的格式输出。这种情况会都会导致评分偏低对吗？而对于single_choice的题目，大部分模型应该都可支持。所以分数的高低展示的不仅仅是模型知识问答的能力，也反应了模型对于数据集格式的理解程度？

可以这样理解，一方面评测模型的知识能力，另一方面是模型遵循指令的能力，即是否能按照指定的格式进行回答。
Repository: OpenLMLab/GAOKAO-Bench
Issue: 请问1000道主观题都是完全的人工评分吗？涉及到用gpt4来评分吗？
Root cause: Documentation Deficiency


目前公布的主观题都是完全的人工评分。

那人工评分的榜单在哪里呢？

img/score_rate_subjective.png 这是gpt-3.5-turbo的主观题得分率，后续会有其他模型的人工评测结果。

我们更新了新的评测结果:)
Repository: OpenLMLab/GAOKAO-Bench
Issue: 按照README文档运行Openai Api简单实例，运行choice_bench.py后输出2010-2022_Math_II_MCQs
single_choice
mkdir: cannot create directory ‘../data/Multiple-choice_Questions/gpt-3.5-turbo_2010-2022_Math_II_MCQs’: File exists
  0%|                    | 0/44 [00:00<?, ?it/s]Exception: Cannot choose from an empty sequence
  0%|                    | 0/44 [00:00<?, ?it/s]Exception: Cannot choose from an empty sequence
  0%|                    | 0/42 [00:00<?, ?it/s]Exception: Cannot choose from an empty sequence
  0%|                    | 0/44 [00:00<?, ?it/s]Exception: Cannot choose from an empty sequence
  0%|                    | 0/44 [00:00<?, ?it/s]Exception: Cannot choose from an empty sequence
Exception: Cannot choose from an empty sequence
Root cause: Validation Gap
Repository: OpenLMLab/GAOKAO-Bench
Issue: 有GPT-4的具体实验结果吗
Root cause: Documentation Deficiency

有GPT-4的具体实验结果吗？只看到一个总分

即将更新的论文当中有这部分内容，感谢您的关注:)

我们已经更新新的实验结果:)
Repository: OpenLMLab/GAOKAO-Bench
Issue: Missing Question
Root cause: Documentation Deficiency

乔治·华盛顿在 1787 年 3 月写给詹姆斯·麦迪逊的一封信中写道：“对现行制度进行彻底改革是必不可少的，任何有判断能力的人都不会否认——我用双手和真心希望这项工作能够得到全面的探讨。习俗。” “彻底改革”指的是什么？\nA. 消除联邦制度的缺陷。B. 建立三权制衡的共和制度。  \n C. 废除君主立宪制。D. 改变松散的联邦制度。 \n",
回答:D
年:2013

Do you mean this [question]( https://github.com/OpenLMLab/GAOKAO-Bench/blob/df2e9667f39145848fb4466c50fe5210d54d0c8c/Data/Objective_Questions/2010-2022_History_MCQs.json#L554C9-L564C11)?
Repository: OpenLMLab/GAOKAO-Bench
Issue: The evaluation result can't match the paper result
Root cause: External Dependency Breakage

We evaluated your `objective_bench.py` on `2010-2022_Physics_MCQs` dataset with openai api model `gpt-3.5-turbo` with `temperature=0.3`, `max_token=4096`, and we got the result `27.343%` , unable to match your result `41.1%` . We assume that answering completely correct = 6 points, partially correct = 3 points, and one error = 0 points. We calculated the total score gained $m$ and total score of problems $n$ and used $m/n$ as the result.

Also, we tested `gpt-3.5-turbo` on your dataset `2010-2022_Physics_MCQs` with our own script, and tried on different tempertures from 0 to 1 . We still got results near `28%`, a lot smaller than `41%`.

1. Do you use the same prompt we provide in the [Obj_Prompt.json](https://github.com/OpenLMLab/GAOKAO-Bench/blob/df2e9667f39145848fb4466c50fe5210d54d0c8c/Bench/Obj_Prompt.json)?
2. The results of gpt-3.5-turbo in the paper comes from the 0301 checkpoint and we conducted the experiments in June, 2023. The model you used may be different from the one we used.
3. We evaluate the gpt-3.5-turbo on 2024.03.13, which refers to the gpt-3.5-turbo-0125 checkpoint according to the OpenAI. We use `temperature=0.3` and `max_token=4096`, and the scoring rate is 37.5%. 



I re-evaluated the physics dataset with your script today, just as I did before, and the result was `50%`. It seems that because the size of the physics dataset is `64`, the result is more flexible than I had thought, so it's reasonable to get different results. Thank you for your reply!
Repository: OpenLMLab/GAOKAO-Bench
Issue: 是否支持其他开源模型本地调用，不是以apikey方式
Root cause: Documentation Deficiency


支持的，不过您需要将模型的推理脚本按照https://github.com/OpenLMLab/GAOKAO-Bench/blob/main/Models/openai_gpt4.py 中的方式进行封装。如果需要更多细节可以联系我～

好的，多谢




707027845
***@***.***



&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "OpenLMLab/GAOKAO-Bench"                                                                                    ***@***.***&gt;;
发送时间:&nbsp;2024年7月14日(星期天) 下午4:16
***@***.***&gt;;
***@***.******@***.***&gt;;
主题:&nbsp;Re: [OpenLMLab/GAOKAO-Bench] 是否支持其他开源模型本地调用，不是以apikey方式 (Issue #35)





 
支持的，不过您需要将模型的推理脚本按照https://github.com/OpenLMLab/GAOKAO-Bench/blob/main/Models/openai_gpt4.py 中的方式进行封装。如果需要更多细节可以联系我～
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

温度设置是0.3，可不可以告诉一下，实验中top_k设置的值是多少




707027845
***@***.***



&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "OpenLMLab/GAOKAO-Bench"                                                                                    ***@***.***&gt;;
发送时间:&nbsp;2024年7月14日(星期天) 下午4:16
***@***.***&gt;;
***@***.******@***.***&gt;;
主题:&nbsp;Re: [OpenLMLab/GAOKAO-Bench] 是否支持其他开源模型本地调用，不是以apikey方式 (Issue #35)





 
支持的，不过您需要将模型的推理脚本按照https://github.com/OpenLMLab/GAOKAO-Bench/blob/main/Models/openai_gpt4.py 中的方式进行封装。如果需要更多细节可以联系我～
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

可以分享一下，除了温度其他的值的设置吗，top_p，top_k




707027845
***@***.***



&nbsp;




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "OpenLMLab/GAOKAO-Bench"                                                                                    ***@***.***&gt;;
发送时间:&nbsp;2024年7月14日(星期天) 下午4:16
***@***.***&gt;;
***@***.******@***.***&gt;;
主题:&nbsp;Re: [OpenLMLab/GAOKAO-Bench] 是否支持其他开源模型本地调用，不是以apikey方式 (Issue #35)





 
支持的，不过您需要将模型的推理脚本按照https://github.com/OpenLMLab/GAOKAO-Bench/blob/main/Models/openai_gpt4.py 中的方式进行封装。如果需要更多细节可以联系我～
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;