Maximum `--max-context` setting on single 5090
Ninfer seems to offer no smart fitting mechanism llama cpp offers, so I quickly ran some tests to figure out the maximum context window one can set when running this model.
The results:
| MTP disabled | --spec mtp --draft-tokens 3 --lm-head-draft |
|
|---|---|---|
--max-context |
~172096 | ~160768 |
I set headroom for the tests, so strictly speaking this is not really the maximum one can set.
Sharing this for whom may read.
--max-context is for single request , --kv-capacity auto for concurrency, eveything is IN the doc
262144 capacity with fp8 kv is reachable
--max-contextis for single request ,--kv-capacity autofor concurrency
Yeah I got that afterwards. My original description is pretty misleading, edited that out.
What I wanted to say is that without the --fitmechanism it's a bit tricky to figure out the context ceiling for a parameter set.
Would be really helpful if that could happen, or if a PR could help making things faster I'd love to contribute as reference.
Thanks for your hard work, it's an awesome project.