Yet linguistically, it really doesn't hold up. Words are built up from phonemes, which undergo some alterations depending on their order and the context to become the phones (as in sounds, not telephones) which are actually pronounced. But this allophonic variation is never dependent on the spelling of the word†, and the word "debt" is in all dictionaries transcribed with three phonemes: /dɛt/. The set of phonemes in the English language is also limited and the consonants clearly demarcated, a hypothetical ghostly "b" phoneme would not go unnoticed.

You describe the popular version of a linguistic theory, not established fact. There are facts corroborating it, and also some observations (such as the multitude of variations in basically unwritten dialects) that speak against this simple view (that I for one really want to be true, having invested years of my life into, but that doesn't make it any more valid).

An Achilles' heel of your argument is that it is generally very, very hard for people using a more-or-less phonetic writing system (such as an alphabet or a syllabary) to discuss speech sounds without referring to orthography, without mixing up speech sounds and written signs, and without hallucinating subtle differences (or their absence) in speech perception where there are differences (or the lack thereof) in the spelling.

As an example, take final obstruent devoicing as observed in Standard German[1]. The majority of the phonetic / phonological literature of the 20th c seems to leave no doubt that a final systematic -d- (as in 'Rad' [ra:t] (wheel; bicycle)) is phonetically indistinguishable from final systematic -t- (as in 'Rat' [ra:t] (counsel)). Yet there is a minority view (held among others by a well-respected language teacher of mine) who insist there is a minor auditory difference. We can test against that by presenting material that has been so cut and sliced as to erase the meaningful context of utterances and then ask people to vote for the one (Rad) or the other (Rat) based solely on auditory perception. This however is complicated by the fact that not all people speak pure standard high German (in fact—nobody does, of course), so what people do get to hear will always be a blend of dozens or thousands different dialects, idiolects, and mannerisms. Listeners are also faced with the task to accept that, depending on the speaker, 'Rad' may get uttered as [rat] (with a short [a], somewhat similar to En. 'rut'), and while the same may happen to 'Rat', that pronunciation seems to have a slightly differing geographical distribution than the short version of 'Rad'. So when I'm from Bremen, I will probably say 'Ich fahr mit dem [rat] zum [ra:t]haus' (I take the bike to city hall), whereas someone from Westphalia or Wuppertal may prefer short vowels in both words. The effect might well be that the listener's 'phonemes'—more generally, their internal systematic understanding of speech sounds—are much more tied to probabilistic aspects and the history of their personal experience than the clean, orderly phonological models presented in textbooks may lead one to believe.[2][3]

[1] https://de.wikipedia.org/wiki/Auslautverhärtung
[2] see doubts about the reality of the phoneme expressed by BE Vidos Handbuch romanische Sprachwissenschaft https://www.amazon.de/Handbuch-romanischen-Sprachwissenschaf...
[3] https://languagelog.ldc.upenn.edu/nll/?p=47503 and a number of other threads on the reality of linguistic models