zhiminy/EvalEng huggingface dataset:
Repository: confident-ai/deepeval
Issue: [python] Zero test case in benchmark HellaSwag Task of POLISHING_FURNITURE
Root cause: Validation Gap

**Describe the bug**
Empty test case in HellaSwag Task of POLISHING_FURNITURE which resulted in zero division error

```
Processing Polishing furniture: 0it [00:00, ?it/s]

---------------------------------------------------------------------------

ZeroDivisionError                         Traceback (most recent call last)

[<ipython-input-28-57a71af8524d>](https://localhost:8080/#) in <cell line: 10>()
      8 )
      9 
---> 10 benchmark.evaluate(model=CustomModel(model=model, tokenizer=tokenizer))
     11 print(benchmark.overall_score)

[/usr/local/lib/python3.10/dist-packages/deepeval/benchmarks/hellaswag/hellaswag.py](https://localhost:8080/#) in evaluate(self, model, batch_size)
     77                     )
     78 
---> 79             task_accuracy = task_correct_predictions / task_total_predictions
     80             print(
     81                 f"HellaSwag Task Accuracy (task={task.value}): {task_accuracy}"

ZeroDivisionError: division by zero
```

**To Reproduce**
```python
from deepeval.benchmarks import HellaSwag
from deepeval.benchmarks.tasks import HellaSwagTask

# Define benchmark with specific tasks and shots
benchmark = HellaSwag(
    n_shots=0
)

benchmark.evaluate(model=CustomModel(model=model, tokenizer=tokenizer))
print(benchmark.overall_score)
```

**Expected behavior**
 Add POLISHING_FURNITURE task properly or skip the evaluation when the task is included in the tasks list

**Desktop (please complete the following information):**
 - OS: Windows 10, Google Colab
 - Browser: Firefox
 - Version 1.1.6
 


This should be fixed soon. In the HF datasets, the task is currently labeled as "Polishing furniture."
Repository: dezoito/ollama-grid-search
Issue: Experiment started timestamp updates on re-render

The timestamp for Experiment started uses new Date() in the render function which incorrectly updates the start time on re-renders.

The start time should not update on re-renders.

Recreate this issue by starting an experiment, waiting a few seconds, then pressing one of the Expand buttons which updates state and re-renders the date.

Thank you!
Adding that to the list of changes for the next release.

Fixed as of v0.4.0
Repository: dezoito/ollama-grid-search
Issue: Enable/disable parameters and their dependancies
Root cause: Unimplemented Feature Gap

Suggestion for UX:

1. Allow parameters to be explicitly disabled (which results in using their default values, regardless of the input value).
2. If there are dependant parameters (e.g.: Mirostat must be enabled for Mirostat-Tau and Mirostat-Eta), the dependant parameters respect the parent's status.
~~3. Promote System Prompt from Settings Modal to Form Grid (it is the only parameter in Settings). By default, disable system prompt.~~

Disabled params will appear gray and only display on one line:

```
<title> <tooltip> <toggle>
```

Enabling params will appear normal and display on three lines:

```
<title> <tooltip> <toggle>
<input>
<explanation>
```


Thank you for the suggestions.

I'll definitely consider #3 first, as it's the less messy to implement.
To disable the system prompt all you have to do is to leave it empty.

If the suggestions are ok, we can leave the issue open until someone can work on it

Done.

@calebsheridan 

System prompt promoted to main form as of v0.4.0 (plus the changes you submitted and other minor updates)

Thank you for the input and PRs.
Repository: dezoito/ollama-grid-search
Issue: feat: Hide model names
Root cause: Unimplemented Feature Gap

The option to hide model names can help eliminate personal bias especially when comparing different models. Also, is there any plan to use this to make elo comparisons like a locally running personal https://chat.lmsys.org?

I like the idea of allowing model names to be hidden - at least in the immediate results view.

On the ELO comparisons, could you please expand?

How do you see that working in terms of the UI?

Would you track evals based on the the model's name only, or also for the combination of parameters used in each experiment iteration? 

This would require significant effort, so I'm mostly writing this out to show how I envision it:

Each configuration of model and parameters would be treated as a different competitor. 

They are all given an initial elo. As you pick one configuration and model over another their elo's are adjusted after each vote. At the end a leaderboard would be displayed showing the end elo of each model and configuration. 

This would work best if multiple prompts could be fed in similar to how the https://chat.lmsys.org arena works. For instance, I would test between 3 models each with 2 different parameters each for 6 competitors. I write out a list of prompts, the number required should increase as the competitors increase. The program will ask me to vote between the best response of two random competitors. Then, it repeats this process for all my prompts randomly selecting two competitors every time. Given 20-30 prompts a clear trend should appear in the elo of the 6 competitors, illustrating which model and parameters provides the preferred response as well as how far apart each model and parameter is.

@notasquid1938 ,

Thank you for the clarification.

Although I can see some similarities, I agree that it would take considerable effort (especially since we don't have a database layer in place yet), and diverge from what I had in mind when I started the project.

It's an interesting idea and I'd be happy if someone could make a fork and work on it.

> The option to hide model names can help eliminate personal bias especially when comparing different models. 

@notasquid1938 

Added in [v0.5.0](https://github.com/dezoito/ollama-grid-search/releases/tag/v0.5.0)
Repository: dezoito/ollama-grid-search
Issue: Latest release fails to run with: “ollama-grid-search” is damaged and can’t be opened
Root cause: Environment Incompatibility

1. Download the latest aarch64 dmg from releases.
2. Copy to /Applications
3. Open the application
4. Get error

<img width="261" alt="image" src="https://github.com/dezoito/ollama-grid-search/assets/862951/9dab895b-8825-463f-8bd9-aab2e925c68e">


A workaround seems to be to remove all extended attributes from the App:

```
xattr -rc ollama-grid-search.app
```

Can you try the x64 release?

https://github.com/dezoito/ollama-grid-search/releases/download/v0.4.1/ollama-grid-search_0.4.1_x64.dmg

Unfortunately, I merely use the GitHub action provided by the framework, and have no way to address this 

The aarch64 release showed up as a surprise this weekend too! 

The x64 release works (but have to right click -> open, accept its unsigned etc...) - but it's an intel binary, not an ARM (aarch64) binary so it runs in emulation.

<img width="863" alt="image" src="https://github.com/dezoito/ollama-grid-search/assets/862951/0255fc1e-811f-4f99-8ceb-b8fb546c0843">


Any other Apple users having issues with the aarch64 dmg?

More importantly, anyone who knows what they are doing can suggest a way to fix it? :)

@dezoito this _might_ fix it - https://github.com/dezoito/ollama-grid-search/pull/19

Awesome work, sir!

Thank you!

Hi I am still facing this issue, even with the latest release. Any reason why?

> Hi I am still facing this issue, even with the latest release. Any reason why?

Hard to tell without specs (or me even having an ARC Mac)... I can only suggest you try a complete removal of the previous version before installing the latest, or try the x64 installer and run it in emulation.
Repository: dezoito/ollama-grid-search
Issue: error:llama runner process no longer running: 1 error:failed to create context with model
Root cause: External Dependency Breakage


Facing the below error after I install and try for the first time. Are there are any other pre-requisites?

Windows
Models: Mixtral and Openhermes

`String error: An error occurred with ollama-rs: {"error":"llama runner process no longer running: 1 error:failed to create context with model 'C:\\Users\\<Username>\\.ollama\\models\\blobs\\sha256-671d85d606174999e3a99eb20534a120d0564ad2db3c38f04b7932beb021a687'"}`


According to the error message, it seems that Ollama was not properly running on your machine. 

Ollama works fine on PowerShell. And also through Python code in VSCode



Was there any change to the server URL config? Does it match the one you have working with Python? 


I tried the url in browser and it says ollama is running.

Yes, my code in python still works

I've searched on the Ollama repository and it seems that it is having issues with Windows (hopefully solved in the next release). 

Please see:

https://github.com/ollama/ollama/issues/3774


https://github.com/ollama/ollama/pull/3850
Repository: dezoito/ollama-grid-search
Issue: A/B test system prompts
Root cause: Unimplemented Feature Gap

Change data structure to allow many different system prompts in the grid search (similar scope at #18).

The code to manage multiple prompts has been updated so it **should** be easier to implement this now, by using the same existing logic.

I'm marking this as a "good first issue" in case any brave souls want to tackle it.

Closing due to inactivity.
Repository: dezoito/ollama-grid-search
Issue: Styling improvements for prompt display

Currently prompts are displayed as text with whitespace preserved in the inference parameters section and experiment dialog. This has a drawback that long prompts do not fit well and push other important content down.

Potential solutions:
- Use collapsible sections to display any text prompts. Truncate prompt content until it is clicked.
- Use titles or uuids to display text prompts (see #20). Allow library to be opened by clicking on the titles so the prompt can be reviewed.



> Use collapsible sections to display any text prompts. Truncate prompt content until it is clicked.

I'm all for this option, @calebsheridan, since the experiments may refer to prompts that have been deleted or updated since they were run.

For context, I see this view as a human readable log (not saying it can't have some sort of interactivity, just that the data displayed should be a "frozen" snapshot of what happened.







> Use collapsible sections to display any text prompts. Truncate prompt content until it is clicked.

Done in v0.5.1
Repository: dezoito/ollama-grid-search
Issue: Limited Concurrency
Root cause: Validation Gap

As of v. 0.1.33, Ollama supports request concurrency.

To reflect that, a `Concurrent Inferences` input was added to the settings panel, where users can set the number of concurrent calls sent to the server (up to a max of 5, for now).

Due to an issue with the implementation of [`atomWithLocalStorage`](https://github.com/dezoito/ollama-grid-search/blob/661bb8bd396d7e1f9e83aaf8ac3fb13bb8dc0414/src/Atoms.ts#L15), however, this field will be empty, instead of displaying the default value of 1, **until the user sets a value and updates the form**.

Help wanted:
Due to time constraints, I have been unable to implement a version of Jotai's `atomWithLocalStorage` that **recursively** checks if the nested keys for the `config` atom are stored in localStorage, returning their default value instead of an empty value if that key does not exist.

Any help implementing that would be much appreciated.

Added in [v0.5.0](https://github.com/dezoito/ollama-grid-search/releases/tag/v0.5.0)
Repository: dezoito/ollama-grid-search
Issue: Choose other parameters?
Root cause: Unimplemented Feature Gap

This is a very helpful tool! Is there a way to experiment with other values other than temp,repeat penalty, top_k and top_p? 

Such as `repeat_last_n` -- default value of which seems not to work very well when handling conversation history.

Thank you for trying it and providing feedback!

I'll definitely add this to the next iteration.
(As a workaround until then, you should be able to set this value in the settings modal). 

Implemented on v0.0.2.

@boxabirds, I'm closing this for now but please let me know if this works for you.

Just as an FYI, v 0.2.1 supports user input for adicional parameters: tfs_z, mirostat, mirostat_eta and mirostat_tau.
Repository: dezoito/ollama-grid-search
Issue: Install ollama-grid-search as a module
Root cause: Unimplemented Feature Gap

I would like to install ollama-grid-search as a module in my main application. Could you please provide guidance or steps on how to achieve this?

I don't think so.

Can we use a command line to run the program and get the result, or we can only install it as an application and use it?


I like the idea, but for now it's only a standalone application.. if you call it from a command line you can see the "results" logged to the terminal (mostly left for debugging purposes), but there's no way to set the input arguments.

If you know Rust and you want something to get you started in that direction, please take a look at the old cli version:

https://github.com/dezoito/ollama-grid-search/tree/main/old

Alright, thanks for answering!
Repository: dezoito/ollama-grid-search
Issue: AppImage shows a blank screen then closes while the deb package works fine

Hey, there.

Thank you for developing and maintaining this nice app.

This is more a heads up than a bug report, since it seems that the problem only happens with the AppImage. After downloading the file, making it executable and trying to run the AppImage, it starts, shows a black window for a short time then closes (the same happened with the previous version - 0.5.0 - as well by the way):

```
$ ./ollama-grid-search_0.5.1_amd64.AppImage'
Cannot get default EGL display: EGL_SUCCESS
Cannot create EGL context: invalid display (last error: EGL_SUCCESS)
[1]    431463 segmentation fault (core dumped)  './ollama-grid-search_0.5.1_amd64.AppImage'
```

I am running Arch/Garuda, with GNOME on Wayland. I then used `debtap` to convert the deb package to an Arch package, as it seems this looks like a "self contained app" (Tauri or something like this?) and it worked just fine.

The closest thing I have found online to this bug is [this](https://github.com/tauri-apps/tauri/issues/4315), but using the suggested environment variable did not work for me. Also, using `G_MESSAGES_DEBUG=all` did not helped much, at least for me. I hope this helps debugging the issue somehow.

Thanks again. Regards.

Thank you for the heads up.

This is more likely a Tauri issue, based on these results:

https://github.com/tauri-apps/tauri/issues/4315
https://github.com/tauri-apps/tauri/discussions/7879 


Adding  `WEBKIT_DISABLE_COMPOSITING_MODE=1` seemed to fix it in some of the comments above.

> Adding `WEBKIT_DISABLE_COMPOSITING_MODE=1` seemed to fix it in some of the comments above.

> >  but using the suggested environment variable did not work for me.

And it happens only with the AppImage. Anyway, thanks for your reply. Regards.
Repository: dezoito/ollama-grid-search
Issue: Feat/idea: A/B testesting diff / horizontal layout
Root cause: Unimplemented Feature Gap

It might be nice to be able to:

1. Layout the generated output horiztonally / in a grid. This would make it easier to compare results.
2. Optionally use diff style highlighting to show differences in the output.

Here's a crappy mock I whipped up:

![ogs-ab-diff](https://github.com/user-attachments/assets/1eee331d-6ea2-41dc-a8f2-4ff5773de119)

Just an idea - feel free to close if not desirable.

I like this idea.

Some possible issues to brainstorm (and I am interested in inputs from anyone that likes it as well).

1) Since a user can generate 100s of results in a single experiment, and that might make scrolling horizontally uncomfortable, and cause issues with the diffing, what if we let the user generate the results then select two of them to compare (like when you compare two different products on Amazon)?

2) If we were to display more than two results side by side, should we diff from the first result or from the previous result?

I think diffing the inference parameters is a great idea, BTW!

Diffing requested on  #62
Repository: dezoito/ollama-grid-search
Issue: Crowdstrike finding: Malicious file nsis_tauri_utils.dll
Root cause: External Dependency Breakage

First of all: Thanks for the tool, I really like it a lot.

But let me get this feedback: I was just contacted by our administrators that the setup `ollama-grid-search_0.5.1_x64-setup.exe` caused a Crowdstrike finding: 

> Filename: nsis_tauri_utils.dll
> Finding: Win/malicious_confidence_70%

I found that this is from an  [open source NSIS plugin](https://github.com/tauri-apps/nsis-tauri-utils). They already have a thread about anti virus findings:
 - https://github.com/tauri-apps/nsis-tauri-utils/issues/37

It seems their newer version 0.4.1 passes the anti virus scans. It might be a good idea to check if you can rebuild your setup with this newer version. For me personally, this is not urgent - my app is running fine. Thanks once again.


Hey there!

Thank you for the heads up (and the kind words)... This seems to be an issue with one of the Tauri plugins  used to build the releases and, unfortunately, there is nothing I can do on my end.

I checked the link you posted and I can see that FabianLars is already aware of the issue.
I'm not sure if I'll need to update any packages here (the entire build pipeline runs on a GitHub action), but if so I'll do it and publish a new release.
Repository: dezoito/ollama-grid-search
Issue: Enable Link-Time Optimization (LTO) for the Tauri part

Hi!

I noticed that in the `Cargo.toml` file Link-Time Optimization (LTO) for the project is not enabled. I suggest switching it on since it will reduce the binary size (always a good thing to have) and will likely improve the application's performance a bit.

I suggest enabling LTO only for the Release builds so as not to sacrifice the developers' experience while working on the project since LTO consumes an additional amount of time to finish the compilation routine. If you think that a regular Release build should not be affected by such a change as well, then I suggest adding an additional `dist` or `release-lto` profile where additionally to regular `release` optimizations LTO will also be added. Such a change simplifies life for maintainers and others interested in the project persons who want to build the most performant version of the application. Using ThinLTO should also help to reduce the build-time overhead with LTO. E.g., check `cargo-outdated` Release [profile](https://github.com/kbknapp/cargo-outdated/blob/master/Cargo.toml#L48).

Basically, it can be enabled with the following lines:

```
[profile.release]
lto = true
```

According to the official Tauri [documentation](https://v1.tauri.app/v1/guides/building/app-size/#rust-build-time-optimizations), you may also be interested in enabling other optimizations.

Thank you.

Thank you for the suggestion.

I'll add this to the next release and try to get some metrics on how it affects the build.

Done. 
Thank you!
Repository: dezoito/ollama-grid-search
Issue: File ollama-grid-search_0.6.2_aarch64.dmg is damaged
Root cause: Environment Incompatibility

Upon downloading the .dmg installation file, I am getting an error:

 **"ollama-grid-search is damaged and can't be opened. You should eject the disk image."**
 
 I am running on a M2 MacBook.

Thank you for the heads up.

I just re-generated that release file:  [ollama-grid-search_0.6.2_aarch64.dmg ](https://github.com/dezoito/ollama-grid-search/releases/download/v0.6.2/ollama-grid-search_0.6.2_x64.dmg)

Can you try again?

If that doesn't work, can you try the 0.6.1 version?

https://github.com/dezoito/ollama-grid-search/releases/download/v0.6.1/ollama-grid-search_0.6.1_aarch64.dmg.

0.6.1 was stable, and the updates were minimal (a single .tsx file), so I assume there was some glitch with the github action during the last release (or maybe something else was changed with the MacOS image).

If none of these works, please let me know so I can pester someone who knows this better than I do :)






Closed do to new release.
Repository: dezoito/ollama-grid-search
Issue: OGS turns off logging - is there a way to fix this?
Root cause: Documentation Deficiency

Looking through the logs at server.log I find this:

```time=2024-04-16T19:42:38.623-07:00 level=INFO source=server.go:264 msg="starting llama server" cmd="C:\\Users\\J\\AppData\\Local\\Temp\\ollama2950239959\\runners\\cuda_v11.3\\ollama_llama_server.exe --model C:\\Users\\J\\.ollama\\models\\blobs\\sha256-aa44bfe80b3d9cc9dbb32cb700e29fd3918f24d80693f27dd7dfda5cb68476d5 --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 0 --port 62547"```

Looks like OGS is adding a `--log-disable` option - I would like logging on. How do I fix this?

That's set on your server.

AFAIK, clients have no way to set that,as that parameter is not even available in the REST API. 

Someone pointed me to this

https://github.com/ollama/ollama/blob/8d1995c625e7f2ed2ff98eb099e1bd8d7e6e133e/llm/server.go#L204-L208

looks like this can be done through the environmental variable OLLAMA_DEBUG.

You probably know this, but just for the sake of someone searching for a similar issue, this is something set on the server side.
Repository: dezoito/ollama-grid-search
Issue: Feat Request: Diff Parameters that changed at each inference
Root cause: Unimplemented Feature Gap

When generating inferences for multiple parameters, use a diff to highlight the parameters that changed for each run at the "Inference Parameters" section:

<img width="826" alt="image" src="https://github.com/user-attachments/assets/be7c3696-17a0-4598-9304-357f33adf318" />
(Stealing @sammcj 's idea and picture here ;) )
Repository: dezoito/ollama-grid-search
Issue: Is it possible to add "Vision" model to the A/B vision test ? with images
Root cause: Unimplemented Feature Gap

Is it possible to add "Vision" model to the grid with an image upload to analysis ?

Hello @Elgokoo ,

I think this would be a great feature, but have no availability in the foreseeable future.

If someone submits a PR, I'll gladly look into it... I believe ollama-rs already works with vision models too? 

Thanks for your feedback,

I'm a "Not coder" user, so I don't know how to help to add Vision PR with https://github.com/pepperoni21/ollama-rs !!!

I wish someone can help because it will make your software the best A/B tester, better than Msty and the webui Arena
Repository: dezoito/ollama-grid-search
Issue: cudaMalloc failed: out of memory only when using ollama-grid-search
Root cause: External Dependency Breakage

I can run ollama with these two models from the command line but not inside of ollama-grid-search:
deepseek-r1:70b
mistral-large:123b

They run slowly on ollama directly as only part of the model will fit in the rtx3090 24 GB.
When run with ollama-grid-search I get the out of memory error.

String error: An error occurred with ollama-rs: {"error":"llama runner process has terminated: error loading model: unable to allocate CUDA0 buffer\nllama_load_model_from_file: failed to load model"}

Is there some parameter I can pass via ollama-grid-search so these large models will run albeit slow?

 

That is being caused by a depency called ollama-rs (the Rust library that interfaces with Ollama).

I'll check if there's an update available and will try to merge it as soon as possible.

Meanwhile, in settings, try adding the optional parameter `num_gpu`, setting it to zero. 

I just updated ollama-rs to its newest version and was able to run deepseek-r1:7b on my machine.

Can you try upgrating to version 0.9.1?