INT4 seems to yield poor output quality. Could we perhaps try quantizing it to W4A8 format?
#1
by someshijun - opened
INT4 seems to yield poor output quality. Could we perhaps try quantizing it to W4A8 format?
INT4 seems to yield poor output quality. Could we perhaps try quantizing it to W4A8 format?