(You're making me remember things I haven't thought about in a long time :) my apologies if some details are off)

The simple definition of a shader is "a shader is a program that runs on the GPU instead of the CPU" (or at least is intended to run on the GPU instead of the CPU; if you write a shader that makes use of a feature your GPU doesn't support, Mesa will actually use LLVM to compile it to run on the CPU instead. Performance will be terrible, but at least the program will run). The compilation is only involved when shaders are involved, though on many graphics cards the driver will implement various apparently-non-shader things as shaders under the hood.

In the early days of OpenGL, it had what we now call a "fixed pipeline" where the GPU isn't very configurable, you feed it geometry and textures, configure a few knobs, and then let it run. You could feed it geometry (either polynomial curves or polygon vertices), then the first stage of the pipeline would approximate all the curves as polygon vertices. Then the second stage would take that, and you'd feed it some lighting information and feed it some textures, and would render each polygon to an individual fragment. Then the third stage would composite those fragments together in to the framebuffer. Which it would then presumably display on your screen. You didn't quite have to do things exactly that way, but it was a set of large-ish "blocks" that weren't terribly configurable.

And then very creative people would say "I wish I could change this little aspect of how it processes vertices" or "... of how it processes fragments". And so some hardware vendor would implement another knob on their GPU to configure a specific aspect of one of the stages in the pipeline, make up their own OpenGL extension to configure that knob, and if the OpenGL Architecture Review Board liked that extension, it might become a core part of the next minor release of OpenGL. So it progressed toward a "configurable pipeline".

Then they ended up with enough knobs that they said "Let's just let them write code that will run on the GPU to process the vertices and fragments, instead of adding more and more knobs". So in OpenGL 2.0, we got vertex shaders and fragment shaders. The OpenGL spec specified a C-like language, GLSL, which the graphics card driver would compile in to code that would run directly on the GPU. And so you could run your own "vertex shader" instead of using the vertex-processing behavior that was baked-in to the graphics card in OpenGL 1. You could write the original baked-in behavior as a shader; and so that's what is happening under-the-hood in the drivers for most modern graphics cards. So now we have a "programmable pipeline". At this point they were called shaders because largely what they allowed you to accomplish was, well, fancy shading.

And then very creative people wanted to be able to specify the behavior at more parts of the pipeline, and so OpenGL grew geometry shaders and pixel shaders and whatnot. And so that's the general direction that OpenGL is going, carving out more and more pieces of what used to be "part of" OpenGL and letting you replace it with your own shader code. And then we even got compute shaders that don't even have anything to do with graphics, but let you run general-purpose computation on the GPU instead of the CPU. So the word "shader" a bit of a misnomer these days.

I got a lot out of CS 334 with Voicu Popescu at Purdue, but I'm not sure if much course material from that is online. Also, I find the OpenGL specs to be surprisingly approachable, but I spend a lot of time reading specs so maybe that's just me.