mlc-ai
/

NeuralHermes-2.5-Mistral-7B-q3f16_1-MLC

Model card Files Files and versions Community

ruihanglai commited on Apr 15

Commit

6f927d6

•

1 Parent(s): 1bc3393

Update mlc-chat-config.json

Files changed (1) hide show

mlc-chat-config.json +2 -1

mlc-chat-config.json CHANGED Viewed

@@ -14,7 +14,8 @@
     "sliding_window_size": 1024,
     "prefill_chunk_size": 128,
     "attention_sink_size": 4,
-    "tensor_parallel_shards": 1
   },
   "vocab_size": 32002,
   "context_window_size": -1,

     "sliding_window_size": 1024,
     "prefill_chunk_size": 128,
     "attention_sink_size": 4,
+    "tensor_parallel_shards": 1,
+    "max_batch_size": 80
   },
   "vocab_size": 32002,
   "context_window_size": -1,