I have aphantasia, but I don't doubt that people actually do visualize things, for two reasons:

1. Dreams. I have "after-images" of dreams for brief moments after I wake up, but then rapidly fade, as do memories of what happened in the dreams unless I quickly make sure to fix memories of specific details in memory by thinking about them after I wake up. And so I have memories of having the dreams and seeing images in them, though I can't recall the dreams themselves.

2. I've meditated for years, and once I had an experience where I suddenly could see clearly, as if I was on a movie-set. Everything was clear. I could turn around and see what was around me. I was lucid - I could have fallen asleep for a moment, it's possible, but I was controlling the "camera", and I was fully aware it was not real, which sets it apart from most dreams for me. It was also a far clearer image than I recall from dreams. I call it a "movie set" because it was clear to me it was not real and it felt like a "replay" of events happening around me that I was not taking part in, which also does not match how dreams feel to me.

3. I occasionally see very brief, fuzzy flashes. E.g. in writing #1 above, I for a fraction of a second saw a flash of my grandparents cabin, which was the subject of a weird recurring nightmare I used to have where on one side of the cabin it was sunny, while on the other side there was a torrential downpour that eventually became a wall of green water. I can describe the cabin in great detail, but I can only see that very brief flash of an image, even that inconsistently.

So, I know my brain can create visuals of different fidelity under different circumstances.

At the same time, when I "visualise" things, the "visualisations" are entirely spatial, not visual, and I can reproduce abstract drawings with a level of precision that I can't copy if I sit down in front of something and try to draw it. I remember a particular time in school (as I type that, I can recall the layout of the class-room, down to where I sat, how we were seated in groups of four, what the desks and chairs looked like, but I can't see it) where we were given the task of doing two drawings of our shoes: For one we were to draw purely from memory. For the other we were to put a shoe in front of us and copy it.

My drawing from memory was very precise. I remembered almost exactly what my shoe looked like, and drew a stylised and precise representation of the concept of that shoe that incorporated most details of it. My drawing from having it front of me was far more fuzzy and impressionist, with lots of imprecision and imperfections that were not there in the stylised drawing from memory. But it looked more like the real-life shoe.

My point being that I see no reason for you to assume that how people draw things are linked to their ability to visualise the thing. I can draw things I can't visualise in ways that are many ways better than how I can draw things sitting on my desk in front of me, but in some ways worse. I think it is in part because the lack of visual recall forces me to compensate by learning the details of something.

E.g. another comment asked about number of windows in their house. Well, I can't see my house, but I can "spatially walk around it" and count, but if I do I quickly realise that if I could visualise the house it'd likely be fuzzy in parts because I need to think about certain rooms because I mostly see the house from the inside and in some rooms the curtains towards the road are usually drawn shut. I see no reason why someone would accurately be able to recall or visualise most objects, because most of the time we don't really look at objects properly. People superficially pay attention to just some details and never bother to actually fully take in enough detail to ever remember an object the way it actually looks.

So when you say other people think they have a visualisation but do not, I think you're drawing the wrong conclusion: People don't have an accurate memory, as you point out. And whether or not it is limited that way for everyone, I think it's reasonable to assume that visualisation is limited by recall and imagination, and that you can't "test" whether or not people can produce visualisations by judging their ability to recall an object precisely.