Reproducible adjacent-record isolation regression with Qwen3.8-27B-4bit-MLX-TextOnly on M1 Pro 32 GB
Hi — I’m evaluating Qwen3.8-27B for a local evidence-analysis workflow on Apple Silicon.
The workload is not general chat. It involves synthesizing bounded evidence, keeping adjacent records separate, following constrained instructions, and producing grounded analytical outputs without treating supplied evidence as missing.
I’m testing both:
- practical operation on a 32 GB M1 Pro; and
- semantic behavior on a frozen local qualification battery.
I found a reproducible adjacent-record isolation failure with this exact artifact/runtime tuple and wanted to share the evidence in case it is useful.
Exact model tested
Model:
lukaskremla/Qwen3.8-27B-4bit-MLX-TextOnly
Pinned revision:
e4833b85af3d88155fbfa24b0680c0e4cb90dde5
Artifact-local tokenizer:
tokenizer.json
SHA-256:87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4tokenizer_config.json
SHA-256:b2bed5e033438f09f22b0ce9522115b4807d9bbcc3b82bf23372cb74d93ed081
Quantization:
- MLX weight-only 4-bit
- bits=4
- group_size=64
- mode=affine
The artifact is text-only; the vision tower is stripped.
Hardware
MacBook Pro 16-inch (2021)
- Apple M1 Pro
- 10-core CPU
- 16-core GPU
- 32 GB unified memory
- arm64
- macOS 27.0, build 26A428
No other large model was intentionally resident during the bounded tests.
Runtime
oMLX:
0.6.3rc3
commit:
8662ad0998985a73dc04f2542cf8afb22600fa4f
Supporting stack:
- MLX / mlx-metal:
0.32.0 - mlx-lm:
0.31.3 - mlx-vlm:
0.6.3 - tokenizers:
0.22.2 - Python:
3.12.7
Relevant configuration:
- context: 4096
- concurrency: 1
- temperature: 0
- top_p: 1.0
- top_k: 0
- min_p: 0
- seed: 0
- thinking disabled
- max output: 180 tokens
What I observed
The model successfully loaded and ran on the 32 GB M1 Pro.
Typical load time was about 5.9–6.3 seconds.
In the original bounded qualification run:
- ready state was reached
- inference completed
- unload/recovery completed cleanly
- memory pressure reached warning transiently but never critical
- no swap growth occurred in that original run
The main issue was semantic.
Using a frozen local 24-assertion qualification battery:
- this MLX candidate passed 21/24
- a pinned GSQ-RCO / llama.cpp comparison path passed 24/24
All three failures were the same adjacent-record isolation case.
The test structure was approximately:
Question:
When was Arden Vale's foundation event?
Evidence:
Foundation Table
Subject | Event | Recorded date
Arden Vale | foundation | 2031-04-12
Arden Vale | review | 2031-04-21
Arlen Vale | foundation | 2032-04-12
Expected answer:
Arden Vale's foundation event was on 2031-04-12.
The MLX candidate returned exactly:
INSUFFICIENT_EVIDENCE
This occurred 3/3 times.
The failure is not selection of the adjacent distractor. The requested row is directly present, but the model treats the evidence as insufficient.
Follow-up isolation
I initially wondered whether oMLX's VLM-MTP fallback path might be contributing.
The original MTP-configured attempt could not construct the VLM target because the text-only artifact lacks the vision tower, so oMLX correctly fell back to its batched LLM engine.
I therefore ran a second bounded isolation with explicit autoregressive LLM configuration:
model_type_override=llm
mtp_enabled=false
vlm_mtp_enabled=false
This second experiment performed exactly:
- one model load
- three requests
- only the failed adjacent-record case
No VLM construction or MTP path was involved.
Result:
INSUFFICIENT_EVIDENCE
again, 3/3.
All three outputs were byte-identical.
This appears to rule out the earlier VLM-to-LLM fallback path as the cause.
What has been ruled out so far
The retained diagnostics do not support:
- prompt/message drift
- profile drift
- template-file drift
- missing requested evidence
- gross truncation
- ordinary sampling variance
- MTP draft influence
- VLM-to-LLM fallback involvement
The failure remains reproducible under explicit AR execution.
What is NOT yet isolated
I do not think the current evidence supports blaming the quantization or oMLX specifically.
The remaining causal boundary includes some combination of:
- artifact construction / 4-bit weight representation
- tokenizer behavior
- cross-runtime effective tokenization differences
- oMLX orchestration or generation behavior
- MLX / mlx-lm execution behavior
So I am reporting this as a reproducible behavior of the exact artifact/runtime tuple, not as a conclusion that the model conversion itself is defective.
Comparison control
The comparison path was:
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
revision:
21e67c72f41299a8285cb047a58790bac8c38717
artifact:
Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf
with:
llama.cpp b10917
That path passed the frozen qualification battery 24/24 and answered the adjacent-record case correctly 3/3.
This was historical control evidence using the same frozen benchmark messages and scoring methodology, not a simultaneous runtime comparison.
One additional operational observation
The explicit-AR isolation run remained below the configured reserve and never entered critical memory pressure, but it caused:
468,647,936 bytes
approximately:
446.94 MiB / 0.436 GiB
of additional backing/swap allocation.
That backing remained allocated through the recorded unload/recovery window, although:
- the model unloaded successfully
- owned processes exited
- pressure returned to normal
- the machine returned to idle
I am treating that as a separate operational observation, not evidence that it caused the semantic behavior.
Question
Would you be interested in reproducing the adjacent-record case directly against this pinned artifact using mlx-lm or another MLX execution path outside oMLX?
That seems like the most useful next step because it could help separate:
artifact/tokenizer behavior
from
oMLX/runtime behavior.
I have retained hashes, exact package versions, token IDs, rendered-prompt hashes, and bounded-run evidence.
I have not posted the complete rendered prompt because the surrounding system instruction is from a privately owned test profile. I can work on producing and verifying a fully public minimal reproduction fixture if that would be useful.
Thanks — I’m mainly sharing this because the failure is deterministic and I thought the controlled evidence might be useful for improving or characterizing the artifact.
This is simply a mlx-lm produced 4 bit quantization. I did not much extra work besides creating the repo, labeling it, and running the quants on my hardware and uploading them.
For starters - this is heavily AI generated, and most of the text makes no sense/is purely hallucinated. I'd like to ask you, on behalf of the entire internet community (from huggingface, open-source code, to literally any other online communities) don't do what you just did. Posting long AI-generated text that was not reviewed by a human, that is full of hallucinations, and that I have a sneaking suspicion you don't even fully understand only wastes people's time and contributes to the AI-slop fatigue felt all around the online space.
Secondly - you're comparing a non uniform GGUF quant with mlx-lm produced 4 bit quants. Therefore saying that the evidence doesn't support blaming the quantization can't be based on anything.
This issue could literally be anything from inference backend bugs, incompatibilities, mis-configurations, differences in the quants, incorrect rendering of the template - anything.
It is out of the scope of this repo to debug this, because as I said, I only ran an existing mlx library to create the quants, I labeled them, and uploaded them as a whole collection of text-only and VLM capable variants + separate MTP heads. The libraries used for the quantizations are out of my control so I couldn't really do anything if there truly was wrong with the uploaded artifacts. The sole purpose of this is to make it easy for people with MLX-capable machines to download and run Qwen 3.8 27B quants or experiment with them without having to worry about fiddling with the original full-precision weights wasting time and disk space.
It's okay if you don't understand what is going wrong when the LLM is not doing what you want, it is okay if you use bigger AI models to debug it (though be careful, since they can and do hallucinate a lot as your LLM likely did in the original message). But please don't take AI output 1:1 and paste it online if you don't understand it.
For your issue... you could try experimenting with different inference backends (mlx-vlm or mlx-serve), or quants (uploaded be me in the collection or anything posted by anyone else) or sticking with the GGUF if it works...