zhiminy/EvalEng huggingface dataset:
Repository: OpenLMLab/GAOKAO-Bench
Issue: A question about the paper

In the paper, it mentions setting up 0-shot to mitigate the issue of data leakage. I don't quite understand the logic behind this. Could you explain it?

Thank you for your question, Elon:).
Zero-shot means model is evaluated without seeing any task-specific examples. This can reduce the risk of overfitting and evaluate the model's generalization ability. On the other hand, for models that may have encountered GAOKAO questions during training, using few-shot settings can make them memorize this type of data, resulting in unfair results. Therefore, we believe using zero-shot settings can mitigate benchmark leakage.
Hope it can help you~

Hahaha:), you mean that few-shot might "make it recall similar data received during training, such as questions from the same exam paper" (my understanding)? As you mentioned, if the model received similar GAOKAO content during training, then I believe the model's accuracy should have little to do with the shot setting (I have done experiments to verify this, even a 7b model in 0-shot can achieve 90+ accuracy). Currently, the results of models in few-shot and zero-shot are generally not much different (many models now tend to perform worse in few-shot). My understanding is that the shot setting is to better help the model understand the task of multiple-choice questions, guiding it on how to answer the question? (I see it seems to have first appeared in MMLU and achieved good results with GPT-3, but now it seems to be counterproductive). If there are concerns about data leakage, can we choose some multiple-choice questions unrelated to GAOKAO content as shots? These are just my immature thoughts, and I would appreciate your guidance.








For base model, few-shot prompt can help it understand the task, as you mentioned. On the other hand, chat model can understand the task without additional examples, so zero-shot is available. So if you want to evaluate base model and avoid benchmark leakage, choose unrelated examples as shots is considerable, from my perspective. Hope it can help:)

Thank you so much for your thoughtful explanation! I appreciate your insights into the use of zero-shot settings to mitigate benchmark leakage and enhance the evaluation of a model's generalization ability. Your suggestion to use unrelated examples as shots to avoid data leakage when evaluating base models is also very intriguing and seems like a practical approach. I will consider this strategy moving forward in my experiments.

I look forward to further discussions and learning more from your expertise!
Repository: OpenLMLab/GAOKAO-Bench
Issue: 分数不一致
Root cause: Documentation Deficiency

Results/merge_score.json结果里（"Liberal-Arts_Total_score": 548.3,"Science_Total_score": 466.1）给出的gpt_4_0314计算的高考总分和models-score图表里的文理科总分485和447 为什么不一致呢 ？                                                                                                                                                      



你好，这是因为我们的models-score图表中的文理科总分的主观题部分是人工评测的；而Results/merge_score.json中的总分的主观题部分是gpt评测的，因此两者不一致。merge_score.json的sub_correction_type和sub_teacher_model_name字段展示了主观题的评测者为gpt-4-1106-preview。
感谢您的关注～
Repository: OpenLMLab/GAOKAO-Bench
Issue: 2024高考评测集
Root cause: Unimplemented Feature Gap

hello，2024年的高考题集打算什么时候集成呢

2024年的选择题已经更新在[GAOKAO-Bench-Updates](https://github.com/OpenLMLab/GAOKAO-Bench-Updates/tree/main/Data/GAOKAO-Bench-2024)中了～
Repository: OpenLMLab/GAOKAO-Bench
Issue: gpt-3.5-turbo 高考得分 在哪？
Root cause: Documentation Deficiency

readme里面的[测试结果](https://github.com/OpenLMLab/GAOKAO-Bench#%E6%B5%8B%E8%AF%95%E7%BB%93%E6%9E%9C)是空的

已经更新新的评测结果:)
Repository: Picovoice/speech-to-text-benchmark
Issue: Text Enhancement

Hi!

I've tried Cheetah briefly in the past and I could only get capitalized letters and spaces for the transcription output. The cloud providers featured on this benchmark offer a transcription with punctuation and appropriate letter capitalization.

Is it still the case? If so, maybe add this information to the README?

Thanks.

Thank you @andrestone no we don't do punctuation and capitalisation, yet. but it is coming this year. the benchmark only goes through WER. other features are not compared here.
Repository: Picovoice/speech-to-text-benchmark
Issue: Consider adding whisper.cpp and faster-whisper to comparison
Root cause: Unimplemented Feature Gap

Both run on CPU.

[whisper.cpp](https://github.com/ggerganov/whisper.cpp) is a port of Whisper in C++, it's now probably one of the most popular open-source speech-to-text models running offline on CPU. With the smallest `tiny` model (75 MB) it's 5-6 times slower than Leopard in my tests, though.

[faster-whisper](https://github.com/guillaumekln/faster-whisper) is a reimplementation of Whisper model using [CTranslate2](https://github.com/OpenNMT/CTranslate2/). It's not as popular yet, but it's 5 times faster than whisper.cpp on the same model, which makes it comparable to Leopard in terms of speed (even though it used 75 MB `tiny` model), in my tests.

@shvchk please feel free to submit a PR for any new engines you like. we are happy to look into them.
Repository: Picovoice/speech-to-text-benchmark
Issue: MacBook Pro M3 Crashes 
Root cause: Resource Mishandling

When i try 

```python3 benchmark.py \
--engine WHISPER_LARGE \
--dataset LIBRI_SPEECH_TEST_OTHER \
--dataset-folder /Users/aedell/Documents/GitHub/speech-to-text-benchmark/datasets/LibriSpeech/test-other  \
```

my Macbook Pro's memory spikes and then the machine locks up. According to ChatGPT, the issue is; 

**Concurrent Execution**: The script uses ProcessPoolExecutor for parallel processing. If the dataset is large and the number of processes (num_workers) is not optimally set, this could lead to high memory usage as each process might consume a significant amount of memory.

**Dataset and Engine Initialization**: Depending on how the Dataset and Engine classes are implemented (not visible in the provided script), loading large datasets or initializing multiple instances of speech-to-text engines could consume a lot of memory.

**Memory Leaks**: If there are any memory leaks within the Engine or Dataset classes (e.g., not properly releasing resources), repeated calls in a loop could exacerbate memory consumption over time.

Any suggestions? 

Do you get the same issue when running other engines or datasets? Or is it only this combination of arguments?

I tried it with LibriSpeech/test-clean and got the same result. Here is a screengrab of my activity monitor before the crash. 
<img width="843" alt="Screenshot 2024-02-16 at 7 50 18 AM" src="https://github.com/Picovoice/speech-to-text-benchmark/assets/11819814/9837df26-c231-4368-959d-fbb45135a3ec">


However, when I try with WHISPER_SMALL I don't have the issue. 

I see. Whisper large is a big model and takes a lot of RAM to run. By default we run a thread for each CPU count, and it has a copy of the model on each thread. Try reducing the amount of threads used in the benchmark using the `--num-workers` argument? Try something like 7 or 8?  I imagine then it will run fine.

Closing due to inactivity
Repository: Picovoice/speech-to-text-benchmark
Issue: [CONTRIBUTION] Speech Dataset Generator
Root cause: Others

Hi everyone!

My name is David Martin Rius and I have just published this project on GitHub: https://github.com/davidmartinrius/speech-dataset-generator/

Now you can **create datasets automatically** with any audio or lists of audios. 

I hope you find it useful.

## Here are the key functionalities of the project:

1. **Dataset Generation:** Creation of **multilingual**  datasets with Mean Opinion Score (MOS).

2. **Silence Removal:** It includes a feature to remove silences from audio files, enhancing the overall quality.

3. **Sound Quality Improvement:** It improves the quality of the audio when needed.

4. **Audio Segmentation:** It can segment audio files within specified second ranges.

5. **Transcription:** The project transcribes the segmented audio, providing a textual representation.

6. **Gender Identification:** It identifies the gender of each speaker in the audio.

7. **Pyannote Embeddings:** Utilizes pyannote embeddings for speaker detection across multiple audio files.

8. **Automatic Speaker Naming:** Automatically assigns names to speakers detected in multiple audios.

9. **Multiple Speaker Detection:** Capable of detecting multiple speakers within each audio file.

10. **Store speaker embeddings:** The speakers are detected and stored in a Chroma database, so you do not need to assign a speaker name.

11. **Syllabic and words-per-minute metrics**

Feel free to explore the project at https://github.com/davidmartinrius/speech-dataset-generator

David Martin Rius

closed as it is not related to the repository's purpose.
Repository: Picovoice/speech-to-text-benchmark
Issue: feature request: compare with more ASR engines
Root cause: Unimplemented Feature Gap

Hi all, just to say thank you for the benchmark. 
I propose to add more speech recognition engine to you tests.  Here below some engines to add in your benchmark:

1. WIT.ai 
official API doc: https://wit.ai/docs/http/20170307#post__speech_link
https://www.liip.ch/en/blog/speech-recognition-with-wit-ai

2. IBM Watson speech to text
official API doc: https://www.ibm.com/watson/services/speech-to-text/
https://www.pragnakalp.com/speech-recognition-speech-to-text-python-using-google-api-wit-ai-ibm-cmusphinx/

3. Microsoft Cognitive Service Speech To text
official API doc: https://azure.microsoft.com/en-us/services/cognitive-services/speech-to-text/

4. Kaldi
official API doc: https://kaldi-asr.org/doc/about.html

thanks again
giorgio

Thanks for the comment. (4) is hard as Kaldi is basically a toolkit rather than a ready to go ASR. Hence the performance depends on how one trains it. Makes sense? (1-3) looks like good candidates. Though I expect the performance to be similar to Amazon and Google. If you get to integrate them into the framework and they don't cost much money to run (5 hours of LibriSpeech dataset) happy to merge a PR :)

Thanks Alireza for your feedback. My expectation/hypothesis is that mentioned ASRs give worst WER compared with Google Speech. I'm working with the Wit.ai now. I'll try to push a PR.

BTW, please mark this open issue as a feature request/other or close it if you prefer. 

Thanks
giorgio 

Thanks a lot. I will close the issue for now. If you add an engine to this please submit a PR and happy to review/merge.
Repository: Picovoice/speech-to-text-benchmark
Issue: Compare against Mozilla DeepSpeech TFLite and update to current DeepSpeech?
Root cause: Unimplemented Feature Gap

Heads up, [Mozilla DeepSpeech v0.7.0](https://github.com/mozilla/DeepSpeech/releases/tag/v0.7.0) has been out for a bit with a word error rate of 5.97%.

There is also a Tensorflow Lite model weighing in at 45MB, not sure if this was overlooked earlier, but it has been around since [January 2019](https://github.com/mozilla/DeepSpeech/releases/tag/v0.4.1). This TFLite model is likely a better thing to compare Picovoice Leopard & Cheetah against if the focus is on low resource devices.

I hope your team is faring well in spite of this pandemic!

We update this repo with cheetah/leopard release cycles. the version of deepspeech used is depicted in the result table to avoid confusion. we will use the latest available in the time of the next release. it seems like the tflite models are available for download in 0.7 (and hopefully onward). this would be good useful addition (in older verson you had to export yourself).
Repository: Picovoice/speech-to-text-benchmark
Issue: DeepSpeech benchmark statistics are very outdated
Root cause: Documentation Deficiency

The statistics for Mozilla's DeepSpeech are outdated even compared to version 0.6 (Please see https://hacks.mozilla.org/2019/12/deepspeech-0-6-mozillas-speech-to-text-engine/)

I agree and we will update this when we release a new version of our engine. we have the version of DS used in the table to remove any confusion ...
Repository: UKGovernmentBEIS/inspect_ai
Issue: Typo in README

In the README.md file the following:

> Note that you'll be promoted to install these

Should read:

> Note that you'll be prompted to install these

Thanks, fixed with https://github.com/UKGovernmentBEIS/inspect_ai/commit/671a239f13fe2706f89ffccaf5ac5eef86958fc3
Repository: UKGovernmentBEIS/inspect_ai
Issue: O1 api doesn't expect the `parallel_tool_calls` parameter
Root cause: Validation Gap

When running an o1 model in a sweep with other models I get the following:
```
Task Failed: Error code: 400 - {'error': {'message': "Unsupported parameter: 'parallel_tool_calls' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'parallel_tool_calls', 'code': 'unsupported_parameter'}}
```

It seems like o1 model(s?) unlike other models does not expect the `parallel_tool_calls` parameter

I suggest somewhere around [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/b8bfb9b67f3695a926d7ebe21bb427a4f50b30a5/src/inspect_ai/model/_providers/openai.py#L288) we should check this isn't an o1 model. If it is `parallell_tool_calls` should be `None`

resolved here https://github.com/UKGovernmentBEIS/inspect_ai/pull/1025
Repository: UKGovernmentBEIS/inspect_ai
Issue: What's the easiest way to evaluate a given input, output and expected output?
Root cause: Documentation Deficiency

I have a dataset that has an input, output and expected output fields. If I want to evaluate for factual accuracy(is input = expected output?), profanity(does input contain profanity) etc., what's the easiest way to do this?

You can use the `match()` or `includes()` scorer for simple text matching/comparison. You can also consider the `exact()` or `f1()` scorers.

in terms of looking for profanity, you might want to either write your own scorer that does this explicitly by looking for words (or perhaps uses a model). Depending on your use case the model-based one might call a moderation model (e.g. https://platform.openai.com/docs/api-reference/moderations).

Thanks for the response. This is helpful. Maybe I will clarify my question. I would like to use LLM as a judge (self_critique()) to set up evals on a dataset that already has the generated output and expected output. I do not want to generate responses again using inspect. Is there a way to do this?

Yes, you don't need to ever call `generate()` in an eval, you can setup your own solver chain that primes self_critique()
Repository: UKGovernmentBEIS/inspect_ai
Issue: How to work with limited batch size
Root cause: Documentation Deficiency

I want to run evaluations on models in one gput,
how can i avoid `torch.cuda.OutOfMemoryError: CUDA out of memory.` 

i couldn't find batch size limiting approaches.



This is for `hf/meta-llama/Meta-Llama-3-8B` model

@dragonstyle

try setting `max_connections=1` which will limit the number of samples that will sent to the GPU at a time. See also https://inspect.ai-safety-institute.org.uk/models.html#sec-hugging-face-transformers

Another note, you may want to give the new vLLM provider a try for local HF models (it was just merged a few days ago): https://inspect.ai-safety-institute.org.uk/models.html#sec-vllm


vllm worked thank you
Repository: UKGovernmentBEIS/inspect_ai
Issue: OpenAI Parallel Tool Calls not configurable
Root cause: Unimplemented Feature Gap

According to [the OpenAI docs](https://platform.openai.com/docs/api-reference/chat/create#chat-create-parallel_tool_calls), parallel tool calls (multiple function calls per assistant turn) are enabled by default. Currently, there is no way to disable/configure this which may cause issues/wasted tokens.

Looking at this problem from afar, this will likely be a recurring theme. I feel like there should be something akin to `generation_kwargs` within the `GenerateConfig` class that would allow passing arguments that are specific to a single provider flexibly and enable users to work around constantly changing provider API specs without requiring explicit support in inspect.
Open to suggestions on this and I see that this is not how it was handled in the past. But with a growing number of providers will it remain the ambition to support each of their arguments?

I agree that it would be nice to pass arbitrary stuff through to `GenerateConfig`. For this one though it's going to be common enough that adding a formal parameter seemed like the way to go. It looks like parallel tool calling occurs by default for OpenAI, Anthropic, and Mistral (it does not occur for Google). So far OpenAI is the only provider that lets you turn this off, which you can now do with the `parallel_tool_calls` `GenerateConfig` option.

PR here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/122
Repository: UKGovernmentBEIS/inspect_ai
Issue: Vertex AI safety attributes
Root cause: Unimplemented Feature Gap

Hi, 

Thank you for the useful tool. I'd like to be able to configure the [Vertex AI safety attributes](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/configure-safety-attributes#configurable-filters) through inspectAI.

Is this something you're ok with providing support for? If so, I can make a PR with the feature.

Cheers,
Vas 

This is now implemented (https://github.com/UKGovernmentBEIS/inspect_ai/pull/120)

Docs here: https://inspect.ai-safety-institute.org.uk/models.html#google
Repository: UKGovernmentBEIS/inspect_ai
Issue: Generation with HuggingFace Models
Root cause: Validation Gap

It looks like generation with `hf/openai-community/gpt2`  is broken. Commenting out `chat = self.hf_chat(input, tools)` in `hf.py`  and changing the argument to `GenerateInput`, and then things go through without an error.

@mattmdjaga Could you help out with assessing the right fix here? @lennart-finke We definitely need to keep the chat templates for models that support them. I'm not sure exactly what the right heuristic is for when to not use as well as what the right argument to `GenerateInput` would be to make it work in these cases.

I’ll take a look at it later today, I’m guessing this is for a use case with no tools? 

Right, no tools (gpt2).

Addressed with #1108 . I think this makes the most sense, we check if chat_template is None (seems like by default all HF models have it but None means it's not present), and if it's not present we just extract the role and content in a loop.

Thanks @mattmdjaga! Fix merged to `main`
Repository: UKGovernmentBEIS/inspect_ai
Issue: Docker timeout breaks long-running commands
Root cause: Configuration Error

Hi all!

The Docker Compose executor retries after a timeout:
https://github.com/UKGovernmentBEIS/inspect_ai/blob/326285f7c0b3f93e127a7296da3b6256a60f1d6a/src/inspect_ai/util/_sandbox/docker/compose.py#L307
This can cause problems for long-running commands, especially ones that aren't idempotent.

We ran into this when debugging why our setup steps seemed to be run twice in certain situations. We narrowed it down to the timeout here:
https://github.com/UKGovernmentBEIS/inspect_ai/blob/326285f7c0b3f93e127a7296da3b6256a60f1d6a/src/inspect_ai/util/_sandbox/context.py#L199
Our setup script installs some packages, so takes a little while. Then midway through, the script is retried. (At least it doesn't cause a doom spiral of retries, because the first script shortly finishes with the packages, enabling the second invocation of the script to complete quickly.) And it turned out it only happened in high-load situations, where we were running enough instances of the task in parallel that the setup script took just a little too long. This was a fun one to debug 😭

I guess the immediate solution here is just to increase the timeout for setup steps. But it would be great if the underlying issue could be fixed here. It'd probably be tough to narrow down the Docker bug mentioned in the comment, but from a usability perspective I think it'd be better to abort if a timeout occurs rather than run the risk of running the same commands twice. (And maybe that behaviour could be overridden by a flag `--retry_docker_compose` for people who are sure retrying _is_ ok for their eval?)

Thanks! Have a great week :)

Okay, thank you for running this down! I think setup scripts are indeed a special case. I think we should substantially increase the default timeout to reflect systems under load and/or software installation/configuration that might occur in setup scripts.

Here is a PR that implements this: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1107. 

This increases the timeout + doesn't retry for setup script. LMK if this look like it will resolve your issue as well as any other considerations we should be mindful of.







Should be resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1107

Thank you for the very quick fix J.J.!
Repository: UKGovernmentBEIS/inspect_ai
Issue: How to work with logprob_based_scorer for openai models
Root cause: Documentation Deficiency

i wanted to work on logprob_based_scorer rather than compare outputs for openai models. 
can you help me do that.



thank you for the awesome repo.

@jjallaire-aisi
apologies for the @

You just need to add logprobs options to the GenerateConfig and then write a custom scorer: https://inspect.ai-safety-institute.org.uk/scorers.html#custom-scorers

the logprobs will be in the `TaskState.output` (object of class `ModelOutput`).
Repository: UKGovernmentBEIS/inspect_ai
Issue: Groq model names not recognized
Root cause: Environment Incompatibility

``When I run an evaluation with Groq models, I get the following error:

`ValueError: Model name llama3-8b-8192 from groq not recognized.`

I have specified `INSPECT_EVAL_MODEL=groq/llama3-8b-8192` and `GROQ_API_KEY` in `.env`.

In `model/_model.py`, I added a print statement on line 482: 

```
modelapi_types = registry_find(match_modelapi_type) 
print(modelapi_types)`.

```

When running inspect eval, the print output I get before the error is: 

```
['inspect_ai', 'openai']
['inspect_ai', 'anthropic']
['inspect_ai', 'google']
['inspect_ai', 'hf']
['inspect_ai', 'cf']
['inspect_ai', 'mistral']
['inspect_ai', 'together']
['inspect_ai', 'ollama']
['inspect_ai', 'azureai']
['inspect_ai', 'bedrock']
['inspect_ai', 'mockllm']
[]
```

So it seems that groq models aren't yet set up?

Ah, in my local `inspect_ai` folder, `model/_providers/` doesn't contain groq.py, unlike [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_providers/groq.py). 

I'm confused because I've updated the package.

The groq stuff is only in the development version right now (going to PyPI soon though!). You can install the dev version with:

```python
pip install git+https://github.com/ukgovernmentbeis/inspect_ai.git
```


Ah got it, thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: renamed logs don't open with 'inspect view'
Root cause: Documentation Deficiency

I renamed my logs to better keep track of them, but they don't show up when running 'inspect view'. Is that a feature or a bug?

Can you say more about how you renamed them (e.g. a before and after)? What version of Inspect are you using?

(just FYI I'm going to transfer this issue over to the inspect_ai repo)

Example of working name: "2025-01-15T16-26-24+00-00_random-characters.json" (.eval is the default format but I use the --log-format=json flag) but if i remove the date stamp it doesn't work with inspect view. I cloned inspect yday so it was the latest version.
sorry for slow reply, didn't get a notification.

Charles will know more definitively, but we do enforce the timestamp as part of the name of `.json` files -- this is so that when we are enumerating log files we don't inadvertently try to load other `.json` files which aren't logs. So that is a "feature".

Note that `.eval` log files (the higher performance binary format which is now the default) AFAIK don't have these restrictions b/c they have a more distinguished file extension.