Repository: dezoito/ollama-grid-search
Issue: [Suggestion] Preload model option
Root cause: Unimplemented Feature Gap

Would be possible to add an option to preload the model before running the prompt?

I use it to compare the inference speed of different quantizations but the first run is always skewed by the model loading.

@mann1x , 

During development, I had an issue where I was getting only a partial response from Ollama in generations, even when those were marked as `done`.

I looked to me like it was a bug in Ollama itself, and not in ollama_rs:

https://github.com/pepperoni21/ollama-rs/issues/27#issuecomment-1962950931

The workaround was setting the `keep_alive` parameter to `0` on all requests.. the unfortunate side effect is that it forces Ollama to constantly reload models.

Ollama also does not have an option to preload models (which is actually a good idea, I think), so the only way I could think of doing that from a client would be to send a quick generation request (something like a "2+2 = ?" prompt) before every model iteration, which I'm not sure I like ( I would also have to add this as an option in the settings panel).

Of course, that would only work if the previous issue with the partial responses was fixed (I'm in process of pushing a minor release and will test that.

---

I just had a different idea that might help, though.

On a complete response, Ollama returns the following stats:

```
            total_duration: 6024419654,
            prompt_eval_count: 8,
            prompt_eval_duration: 1395994000,
            eval_count: 25,
            eval_duration: 4627294000,

```
I could add a `inference_duration` variable to the output calculated by adding `prompt_eval_duration` and `eval_duration`. I think this would not take into account the time it takes Ollama to load and setup the model, and would display exactly what you are trying to measure (correct me if I'm wrong here, please).

That would be much simpler to implement and would not depend on any changes in Ollama.

Would that work for you?




@dezoito 
if it's measured after the model is loaded yes, indeed.

Anyway you can preload the model with an empty generate API call:

`curl http://localhost:11434/api/generate -d '{"model": "mistral", "prompt": ""}'`

Directly from the developers

Oh cool!

GIve me a few days so I can add the time option and we can test it out.

Cool thanks!

I don't want to abuse of your availability... but I'll ask just in case as suggestions for the future.
When I compare quantizations I test with the same settings, temperature at 0.
Would be nice to actually have the number of iterations for each setting.
This way I could have directly the results in one json for the 3/5 runs I usually do to get an average.

Then maybe this is too complex and very specific but even better would be to have directly inside a "benchmark result overview".
A table with all the models tested with the average of t/s for the iterations.
Perfect would be small graph but that's really a lot :)


> Would be nice to actually have the number of iterations for each setting.

I agree, and  was thinking about adding that because it would be useful for A/B testing.

I'll add this to my "todo" list, and keep the other suggestions - which I really appreciate - in mind.

Love the idea of adding graphs and charts, but first I think I want to implement some way to store tests and results in a DB.. that would make it much easier to do this kind of processing, reporting, or adding features like filters, where I could show results only for "model x", or for "temperature=y", for example.

> I'll add this to my "todo" list, and keep the other suggestions - which I really appreciate - in mind.

Fantastic!
Yes I agree without a db support it gets quickly unmanageable.
Thanks for working on it

Quick update.

I was doing a little source code dumpster diving and, apparently, Ollama calculates (what I assume is) the time to load the model:

https://github.com/ollama/ollama/blob/0efb7931c7bc5c140c019238aa88b9a41d382e61/api/types.go#L105

```go
type Metrics struct {
	TotalDuration      time.Duration `json:"total_duration,omitempty"`
	LoadDuration       time.Duration `json:"load_duration,omitempty"`
	PromptEvalCount    int           `json:"prompt_eval_count,omitempty"`
	PromptEvalDuration time.Duration `json:"prompt_eval_duration,omitempty"`
	EvalCount          int           `json:"eval_count,omitempty"`
	EvalDuration       time.Duration `json:"eval_duration,omitempty"`
}
```

Unfortunately, this is not returned in the generation responses.

I was unable to get a definitive answer on what each timing metric means, so I decided to go ahead and calculate the `inference time` by adding `prompt_eval_time` and `eval_time`.

Running a single inference on `dolphin-mistral:7b`, here's the output presented in the UI:

```
Created at: Thu, 11 Apr 2024 18:37:25 GMT
Prompt Eval Count: 37 tokens
Prompt Eval Duration: 0h, 0m, 3s
Eval Count: 39 tokens
Eval Duration: 0h, 0m, 5s
Inference Duration (prompt + eval): 0h, 0m, 9s
Total Duration: 0h, 0m, 10s
Throughput (tokens/total_duration): 3.75 tokens/s
```

This is without preloading the model, so it looks like it's taking approximately 1s to do so in my abysmal GPU, and 9s in inference time.

I had tried the same experiment with a larger model (dolphin-mixtral:8x7b) and, predictably, the time it took to load the model was much higher (close to 3 minutes, for 25s in inference):

```
Created at: Thu, 11 Apr 2024 18:06:22 GMT
Prompt Eval Count: 35 tokens
Prompt Eval Duration: 0h, 0m, 8s
Eval Count: 18 tokens
Eval Duration: 0h, 0m, 16s
Inference Duration (prompt + eval): 0h, 0m, 25s
Total Duration: 0h, 3m, 21s
Throughput (tokens/total_duration): 0.7 tokens/s
```

Also, pre-loading the model didn't work... not because it can't be done (it as actually simple to implement), but because it would cause that same issue with incomplete responses and no metrics.

So, unless you spot something glaringly wrong with my assumptions, I think I am going to use these metrics for the next release.

What do you think?



> Unfortunately, this is not returned in the generation responses.

Yes it's the load time...
Maybe there's a way to access it, not sure.

obm is using typescript and seems it can access it:
https://github.com/technovangelist/obm/blob/1cf3c956ad2e629bce409eadd69483b8c6fb2690/main.ts#L205C54-L205C67

> Also, pre-loading the model didn't work... not because it can't be done (it as actually simple to implement), but because it would cause that same issue with incomplete responses and no metrics.

This is really weird, it shouldn't happen... something must be wrong.
I'll try to make some tests.

> So, unless you spot something glaringly wrong with my assumptions, I think I am going to use these metrics for the next release.
> 
> What do you think?

I think these metrics are more than enough.
I'd just abstract the loading time from the throughput doing tokens/prompt_time+eval_time.


I just checked that out... and `load_duration` is returned only for the "chat" endpoint, but not for "generate" :/

BTW:
> Would be nice to actually have the number of iterations for each setting.

Done... I just need to test this a little more and will probably make a new release tomorrow.


## [Version 0.3.0] - 2024-04-12

### Added

- Allow multiple generations for each combination of parameters.
- Improved metrics are displayed after inference results
- Allows inspecting past experiments without the need to download the corresponding file.
- Displays current version number to the settings interface.
- Displays the version of Ollama running on the selected server

@mann1x , I'm closing this, but please try a clean install and let me know if you have any issues or any other suggestions.

Thank you for the discussion!

> @mann1x , I'm closing this, but please try a clean install and let me know if you have any issues or any other suggestions.

Thanks a lot!
Will test it soon, I have my GPU busy with a model lasering :)
Just in time to check how the new models are performing!

I've seen the work you are doing with these models and find it really impressive, by the way.

Keep up the great work!