>This is what I would expect from the underlying mathematics. NaN roughly means the output of a function is undefined,

That's not true, NaN means "not a number". Take for example f(x) = 1/x. What is f(0)? Infinity? Well, you can't divide by zero. In fact, I tricked you. When you talk about a function in mathematics, you need more information that just the formula, you also need to know over what domain I am allowed to evaluate such a function. What I should have said was, define f: [1, 2] -> R s.t. f(x) = 1/x. Now you are not allowed to ask the question, "what is f(0)?" because I didn't include zero in the domain. You might then ask, "so how do you define f at zero?" That question mentions 'f', but the 'f' it is referring to is now different. Such an 'f' you are allowed to ask this question for would be the following:

f:[0, 2] -> R

f(x) = 1/x when x > 0

f(0) = 1

Now, when you ask me what f(0) is, I can tell you. It is 1. "That's cheating!" you might add. No it isn't. It is a different function with different properties. Just because the formula is the same does not mean you can evaluate it wherever you want.

Note: you are not allowed to divide by zero. The result would be meaningless if you could. And 'NaN' is only something the computer spits out when it encounters an operation that doesn't return a number. Usually, a computer will give you 'Inf' when you try to divide by zero numerically. Mathematically, you cannot do it. So don't try.

NaN means 'not a number', and '=' is an infix operator to compare numbers. You can't compare for equality on things that are not numbers.

You can compare infinities, however. "I thought you said dividing by zero was meaningless!" It is. I'm not going to divide by zero, but I can still talk about infinity. Take for example the following function:

f: (0, 1] -> R

f(x) = 1/x

f is defined on every number strictly greater then zero, but less than or equal to one. Zero is not in the domain, so I can't evaluate f there. However, I can still study what happens near zero. Take a really small number bigger than zero. The reciprocal of that is huge. Now take a smaller number, the reciprocal becomes even bigger. In fact, we can keep doing this: construct any positive sequence that converges to zero (but never attains it). Now evaluate f at each element in that sequence, you will notice that the result becomes larger and larger. We call the limit of this 'infinity', and we have notation for this. Notice I did NOT evaluate f at zero, I merely studied what happens as I approach zero. Infinity is not a real number, and I cannot have f ever evaluate to infinity, but I can talk about limiting properties. Computers do this too, but they have a much lower standard for doing so. Essentially, anything bigger than the maximum value of a double (about 10^300, I think?) is called 'Inf' on a computer.

Edit: I forgot to address the second paragraph of the parent comment.

>For example, it's probably more sensible to say that 2x/x != 3x/x for all values of x, rather than 2x/x != 3x/x for all x except 0, wherein both functions are undefined and therefore equal to each other.

This is not a sensible thing to do.

Let me elaborate. I'm going to ignore any cancellation issue and say it is definitely true that 2/x != 3/x for x != 0. Let's also say that each of these functions has domain (0, 1]. These two functions are not equal are x = 0 because you cannot evaluate them at x = 0. It is forbidden. However, behaviour near to zero is very similar. See my discussion above. In fact, you can make a stronger statement. You can say that as x approaches zero (from the positive side), in the limit, 2/x and 3/x are equal. That is, they have the same limit as you approach zero. This does not make them equal at zero, because I cannot evaluate them at zero. To make this point absolutely clear, I will add zero into the domain and define two different functions:

f:[0,1] -> R

f(x) = 2/x for x > 0

f(0) = 1

and

g:[0,1] -> R

g(x) = 3/x for x > 0

g(0) = 2

These functions have the same limit as you approach zero, but they are not equal at zero. I can evaluate them at zero this time, because I defined what it meant to do so. And indeed f(0) = 1 != 2 = g(0).