Ok, I think I'm getting some of your idea more clearly. Essentially, the observation is that we probably can't consistently replace the words in a novel with other words without preserving the meaning of the novel (consistently meaning each word is always replaced with the same other word).

I think the biggest problem with this argument is the assumption that '[the structure in a natural language] maps to these real world relationships in natural ways'. One thing we know for sure is that human language maps to internal concepts of the human mind, and that it doesn't map directly to the real world at all. This is not necessarily a barrier to translation between human languages, but I think it makes the applicability of this idea to translations between human and alien languages almost certainly null.

Perhaps the most obvious aspect of this is any word related directly to the internal world - emotions, perceptions (colors, tastes, textures etc) - there is no hope of translating these between organisms with different biologies.

However, essentially any human word, at least outside the sciences, falls in this category. At the most basic level, what you perceive as an object is a somewhat arbitrary modeling of the world specific to our biology and our size and time scale. To a being that perceived time much slower than us, many things that we see as static and solid may appear as more liquid and blurry. A significantly smaller or larger creature may see or miss many details of the human world and thus be unable to comprehend some of our concepts.

Another obstacle is that many objects are defined exclusively in terms of their uses in human culture and customs - there is no way to tell the difference between a sword, a scalpel, a knife, a machete etc unless you have an understanding of many particulars of some specific human society. Even the concept of 'cutting object' is dependent on some human-specific perceptions - for example, we perceive a knife as cutting bread, but we don't perceive a spoon as cutting the water when we take a spoonful from a soup, though it is also an object with a thin metal edge separating a mass of one substance into two separate masses (coincidentally, also doing so for consumption).

And finally, even the way we conceive mathematics may be strongly related to our biology (given that virtually all human beings are capable of learning at least arithmetic, and not a single animal is able to learn even counting), possibly also related to the structure of our language. Perhaps an alien mind has come up with a completely different approach to mathematics that we can't even fathom (though there would certainly be an isomorphism between their formulation of maths and ours, neither of our species may be capable of finding it).

And finally, there are simply so many words and concepts that are related to specific organisms in our natural environment, that you simply can't translate without some amount of firsthand experience. I could talk about the texture of silk for a long while, and you may be able to understand roughly what I'm describing, but you certainly won't be able to understand exactly what a silkworm is unless you've perceived one directly in some way that is specific to your species, even though you probably could understand I'm talking about some kind of other life form, it's rough size and some other details.