I cant run 70b models on my machine but using Q2_K quants it works. Im generally new this whole open source AI stuff but from my understanding IQ offers a bit higher quality but with a inference speed tradeoff.
· Sign up or log in to comment
