I live in rural Japan. I've been here for 6 of the last 8 years (worked 2 years in the UK in the middle). In my estimation, Japanese people are not particularly racist. I've lived in Canada, the US and the UK in my life. I've seen less examples of problematic racism here in Japan than in the other places I've lived. As others have pointed out, the things that you tend to see are mostly just curiosity or ignorance. For example, people will stare at you in the country side. Children will come up and talk to you for no reason except that they've never met a foreigner before. People are surprised if you can use chopsticks, speak Japanese and read/write Japanese (actually the last one will particularly surprise people). I've occasionally had women with children waiting at a bus stop leave when I arrived, presumably because they were afraid of waiting at a bus stop alone with a foreigner around. That last bit is nothing I haven't seen everywhere in the world.

There is a certain cache to being a foreigner in Japan, especially if you are white and of North American or European background. It's harder for other people like Brazilians, Koreans and Chinese in Japan. However, I have never seen racism to the level that I've seen in other places that I've lived. I think that's not necessarily because Japanese people are less racist. It's just that Japanese culture does not encourage people to speak their mind openly about such things. If people are racist, you just aren't going to know about it most of the time.

I have to say that Japan is an extremely hard place to live as a foreigner if you wish to hold on to your foreign values. Japan is also a place where there is a strong sense of moral right and wrong. This sense is often directly opposed to the moral sense that some foreigners have (especially Americans in my experience). Because of this, some people have an incredibly rough time here. Unable to cope with the idea that a different set of moral values is OK, some people come away with the idea that Japanese people/culture is bad. It is particularly easy to come away with the idea that Japanese culture is racist because of the inside/outside nature of society (either you are inside a group or outside; the rules are different depending on where you are; you can't just decide to join/leave a group; etc).

Anyway, living here is very nice if you can adopt a 100% Japanese lifestyle. Either that or if you can just not care about not fitting in. I've seen both work well (the latter is exceptionally difficult, though). The problem is that many foreigners think they can pick and choose from Japanese culture and their own home culture. This is usually extremely bad and if you ever go to Japan, I would try to avoid that.

Having said all that, immigration is dramatically easier here than any of the other countries I mentioned. To work you just need a job offer and a university degree. The cost is fairly low (I think the visa costs about $1000).

I don't know how realistic coming here for retirement is, unless you have a lot of money. You can sponsor your own visa as a business person if you start a company with $50k capital (which is quite low compared to other countries).

Land and housing is quite cheap where I live. You can easily buy a small (2 bedroom) house for $100k. Large north american style houses go for about $300k. Cost of living is quite low, but there are certain things that are very expensive. A good example is that fruit and vegetables are often twice the price that you'll be used to (or even more). However, food is a pretty small percentage of most people's budget. In contrast, eating out is very cheap here, if you eat Japanese food. It's very, very easy to get a huge meal of healthy food for about $5 in the country side. Even in Tokyo, you could easily eat out for $7-$10 per person (in fact, I did so this weekend). And there is no tipping.