Some people use brainstorming, others use mindstorming. I could not find the difference between the two words. 


Question: How can I replicate the color limitations of the NES with an HLSL pixel shader?
Answer: If you don't really care about texture memory usage (and the idea of blowing an insane amount of texture memory to achieve a retro look has a kind of perverse appeal) you could build a 256x256x256 3d texture mapping all RGB combinations to your selected palette.  Then in your shader it just becomes one line of code at the end:

return tex3d (paletteMap, color.rgb);


It may not even be necessary to go all the way to 256x256x256 - something like 64x64x64 may be sufficient - and you could even change palette mappings on the fly using this method (but at significant cost owing to a large dynamic texture update).