I think what you're getting at is two things. You have a dataset. This consists of some set of observations, each of which contains features that have been observed. These may be things like education level, age, income, favorite color, gender, and height. Three of these variables are continuous. One is ordinal. One is nominal. One is binary. Can you do any kind of meaningful matrix operations with this data?

Traditionally, in regression analysis, you'd use dummy variables for the nominal/binary data, turning all of them into binary {0, 1}. This is GF(2), a bonified vector space. Education level you've got some choices. You can also encode these as binary, potentially losing some information given they're ordered. But if you choose to map them to integers or real numbers, what is the justification for the distance function you're defining? Is MD > PhD? How much greater is PhD than BA than high school diploma?

If you stick to one-hot encoding, there's justification in performing regression analysis still, because you end up in a case where most of the values are 0 and drop out and you get multiple models, one for each case, so the matrix operations being performed to generate the model only operate on the remaining real-valued vectors, and the one case that is 1 becomes the constant term. This is just basic ANCOVA.

You have another issue, though, right? Are "height" and "income" and "age" really from the same vector space? Obviously not. Some modelers choose to ignore this. Some will assume they're all normally distributed and tranform the individual values to a Z-score or something and then they're all from the same vector space. But this still may be dubious. For one thing, they're not actually normally distributed. Height and age have a strictly zero lower-bound. They have some upper-bound, whatever it may be. Income is severely right-skewed. But they're probably all at least approximately normal over some restricted range of the most common values.

We're starting to see the issue. Modeling in this way might work reasonably well within some restricted range of common values, but extreme cases are left out. We need some other techniques that don't rely on matrix math, because once we have to admit all of our elements from not from the same field, we admit we don't really have a matrix, even if a computer will gladly let us pretend anything we can encode as a number is actually a number.

I think your question is how can we encode data such that it all ends up in the same vector space? It's easy if your data is imagery. Color channel intensity of each pixel is all real-valued, possibly within different ranges, but easily normed. If all you have is text, representing the corpus by an incidence matrix puts all of your elements in GF(2). Plenty of other word embeddings are well-justified. But when you start looking at longitudinal data from epidemology, the shit they do in econometrics, it starts to get less math and more black magic. By the time you're a data scientist at Netflix, I'm sure they have reason to believe whatever they're doing works, but it may be hard to reason about why it works.

I don't really know if there any field of mathematics studying ways of encoding data such that doing things like adding and multipying height by favorite color actually means anything, but it's an interesting question.