MaziyarPanahi

/

WizardLM-2-8x22B-GGUF

like 106

This looks really cool! For IQ3_XS! Wow! Thanks for sharing, appreciate it. @mirek190

Log start
main: build = 2679 (7593639c)
main: built with Apple clang version 15.0.0 (clang-1500.0.40.1) for arm64-apple-darwin23.4.0
main: seed  = 1713219739
llama_model_load: error loading model: invalid split file: /Volumes/SSD2/llama.cpp/models/WizardLM-2-8x22B.IQ3_XS.gguf
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/Volumes/SSD2/llama.cpp/models/WizardLM-2-8x22B.IQ3_XS.gguf'
main: error: unable to load model

@mantafloppy you have used cat to put the splits together. Please use the splits as they are following the README and do not merge them please:

llama.cpp/main -m WizardLM-2-8x22B.Q2_K-00001-of-00005.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 1024 -e

@MaziyarPanahi
I did use cat

First time i see a "sharded model".

Thx for the quick reply, will try.

Q2_k answer to the same question below:

Let's start by calculating the number of apples you have after all the transactions:

1. You start with 10 apples.
2. You lose 4 apples, so you have 10 - 4 = 6 apples.
3. Three birds drop 6 apples each, so you gain 3 * 6 = 18 apples.
4. After the bird incident, you have 6 + 18 = 24 apples.

Now let's calculate the gold coins you have:

1. You find 3 gold coins in the river.
2. You gain another gold coin, so you have 3 + 1 = 4 gold coins.
3. You win 6 gold coins in an online game and have to share them equally with your 2 teammates. So you get 6 / 3 = 2 gold coins from the game.
4. Now you have 4 + 2 = 6 gold coins.

You decide to buy apples with all your coins:

1. The price of an apple is 0.5 coins.
2. With 6 gold coins, you can buy 6 / 0.5 = 12 apples.

Now we add the apples you bought with the coins to the apples you have:

12 (bought with coins) + 24 (current total) = 36 apples.

So, after all the transactions, you have 36 apples.

Regarding the location of the river, the information provided does not specify where the river is located. It only mentions that the river runs near a big city, which suggests that the river could be in a rural or suburban area near an urban center. The city's significance is related to what you can spend the coins on, implying it has a market or shops where the coins are accepted as currency. However, the exact location of the river is not provided in the scenario you've described.

I'm also getting the impression that it's a very smart model, but the 3_K_M version sometimes replies weird things like "the confirmation statement was received for filing on 3/3/20/2" and "the confirmation statement and does not specifically those allocated in the year 20222."

Basically mixing numbers up or missing a few letters.

Anyone else has this problem?

I'm also getting the impression that it's a very smart model, but the 3_K_M version sometimes replies weird things like "the confirmation statement was received for filing on 3/3/20/2" and "the confirmation statement and does not specifically those allocated in the year 20222."

Basically mixing numbers up or missing a few letters.

Anyone else has this problem?

I will make an imatrxi today for IQ-1 and I was told it also helps for smaller quants. I will redo the Q3_K_M if this is not a prompt template issue.

I will make an imatrxi today for IQ-1 and I was told it also helps for smaller quants. I will redo the Q3_K_M if this is not a prompt template issue.

Sounds great! I tried IQ3_XS too yesterday and it seemed to have a similar issue. I'm using LM Studio 0.2.19 (llama.cpp commit hash 400d5d722d7edf7de0cf24a18c42b183c65047d2) with the built-in Vicuna 1.5 16K preset.

Impressive that it handles that question so well when quantized. Command-R-Plus can do it full size, but EXL2 4.25BPW it gets it slightly wrong.

Impressive that it handles that question so well when quantized. Command-R-Plus can do it full size, but EXL2 4.25BPW it gets it slightly wrong.

I myself never had a model that could handle questions like that with such correctness and coherence! Never tried Command R+, I know it's an impressive model as well.

An actually usable Q2!? Looks like it is time for plebs like me to try big models, thanks for the info!

Q1_M answer to the same question below (good answer but little error in step 6):

Let's start by calculating the number of apples you have after all the transactions:

1. You start with 10 apples.
2. You find 3 gold coins, which you will use to buy more apples, but we'll calculate this later.
3. You lose 4 apples, so you have 10 - 4 = 6 apples.
4. You gain another gold coin, so you have 3 + 1 = 4 gold coins.
5. Three birds run into your path and drop 6 apples each. You gain 3 x 6 = 18 apples.
6. You win 6 gold coins playing an online game, which you have to share equally with your 2 teammates. You get to keep 6 / (2 + 1 + 1) = 2 gold coins.
7. You buy apples with all your coins. You have 4 gold coins from before plus 2 from the game, totaling 6 gold coins.
8. The price of an apple is 0.5 coins, so with 6 gold coins, you can buy 6 / 0.5 = 12 apples.

Adding the apples you have gained and bought, and subtracting the ones you lost, you have:

6 (initial) + 18 (from birds) + 12 (bought with gold coins) = 36 apples.

As for the location of the river, it's hard to determine without more context. The river runs near a big city, which could be any large city with a river nearby. Since you're using the gold coins to buy apples, which could imply spending on something related to the city, it might be a place where the economy allows for such transactions, possibly a metaphor for a city known for trade or commerce. However, without specific details or clues, pinpointing the exact city is impossible. It could be any large city known for trade and commerce, such as New York City, London, Tokyo, or any other city with a river and economic system that would allow for the transactions described.

Wow! I usually expect a lot of error and short responses from IQ-1 models! Kudos on the gangs at Llama.cpp for the imatrix and improving the quants every day, and congrats to the @WizardLM team for making this model! (I really hope they are not being punished unfairly for missing one step enforced by Microsoft)

When playing around with the settings, the AI almost gave the right answer in French! The reasoning is all good, but it decided that 24 apples was enough and didn't bother to calculate 24 + 12. This model is quite impressive.

Q1_M :
Une conversation entre un utilisateur curieux et une assistance d'intelligence artificielle. L'assistance donne des réponses détaillées, utiles et polies aux questions de l'utilisateur.

USER: Je possède 10 pommes. Je trouve 3 pièces d'or au fond d'une rivière. La rivière passe près d'une grande ville où je peux dépenser ces pièces. Je perds ensuite 4 pommes mais j'obtiens une autre pièce d'or. Trois oiseaux se précipitent sur mon chemin et me font tomber 6 pommes chacun. Je joue à un jeu en ligne et gagne 6 pièces d'or supplémentaires, mais je dois les partager équitablement avec mes 2 coéquipiers. Je décide d'utiliser toutes mes pièces pour acheter des pommes. Chaque pomme coûte 0,5 pièce. Combien de pommes ai-je maintenant ? Et où se trouve cette rivière ?

ASSISTANT: Commençons par résumer les informations données :

1. Vous avez commencé avec 10 pommes.
2. Vous avez trouvé 3 pièces d'or au fond d'une rivière.
3. Vous avez perdu 4 pommes, mais vous avez obtenu une pièce d'or supplémentaire.
4. Trois oiseaux vous ont fait tomber 6 pommes chacun, donc vous avez gagné 18 pommes supplémentaires (6 pommes x 3 oiseaux).
5. Vous avez gagné 6 pièces d'or supplémentaires lors d'un jeu en ligne, que vous devez partager équitablement avec vos 2 coéquipiers, donc vous avez 2 pièces d'or supplémentaires pour vous (6 pièces d'or divisées par 3 personnes).
6. Vous souhaitez utiliser toutes vos pièces pour acheter des pommes à 0,5 pièce chacune.

Pour calculer le nombre de pommes que vous pouvez acheter avec vos pièces d'or :

1. Pièces initiales (3) + pièce supplémentaire (1) + pièces du jeu en ligne (2) = 6 pièces d'or.
2. Vous avez perdu 4 pommes, donc vous avez 10 - 4 = 6 pommes.
3. Les oiseaux vous ont fait tomber 18 pommes.
4. Total de pommes = 6 (initiales) + 18 (des oiseaux) = 24 pommes.
5. Pièces disponibles pour acheter des pommes = 6 pièces d'or.

Chaque pomme coûte 0,5 pièce d'or, donc vous pouvez acheter :

6 pièces d'or / 0,5 pièce d'or par pomme = 12 pommes.

Mais vous avez déjà 24 pommes, donc vous n'avez pas besoin d'en acheter plus. Vous avez suffisamment de pommes pour satisfaire vos besoins selon le scénario décrit.

Quant à la localisation de la rivière, elle pourrait être fictive ou métaphorique, car elle n'est pas spécifiée géographiquement. Elle pourrait représenter une opportunité de trouver quelque chose de valeur dans un endroit inattendu, comme dans les histoires de chasse au trésor ou de découverte fortuite. Pour une localisation réelle, nous aurions besoin de plus d'informations sur le contexte de l'histoire ou sur les indices géographiques qui pourraient la relier à une ville ou une région spécifique connue pour ses rivières.

that actually impressive q1 even works .and give some output ;)

I am just at shock the Q1 can be capable of such responses let a alone in French! Wow!

Log start
main: build = 2679 (7593639c)
main: built with Apple clang version 15.0.0 (clang-1500.0.40.1) for arm64-apple-darwin23.4.0
main: seed  = 1713219739
llama_model_load: error loading model: invalid split file: /Volumes/SSD2/llama.cpp/models/WizardLM-2-8x22B.IQ3_XS.gguf
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/Volumes/SSD2/llama.cpp/models/WizardLM-2-8x22B.IQ3_XS.gguf'
main: error: unable to load model

i get the same error, i did not understand the 'cat' approach @MaziyarPanahi is suggesting. @MaziyarPanahi would you care to help me solve this please? Thanks for both the model gguf and for helping me in advance :)

Log start
main: build = 2679 (7593639c)
main: built with Apple clang version 15.0.0 (clang-1500.0.40.1) for arm64-apple-darwin23.4.0
main: seed  = 1713219739
llama_model_load: error loading model: invalid split file: /Volumes/SSD2/llama.cpp/models/WizardLM-2-8x22B.IQ3_XS.gguf
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/Volumes/SSD2/llama.cpp/models/WizardLM-2-8x22B.IQ3_XS.gguf'
main: error: unable to load model

i get the same error, i did not understand the 'cat' approach @MaziyarPanahi is suggesting. @MaziyarPanahi would you care to help me solve this please? Thanks for both the model gguf and for helping me in advance

There is no cat approach, this is a new way of splitting GGUF models. It is natively supported, all you need is to use the 00001 and it loads the rest automatically: https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF#load-sharded-model