Question: do panniers slow you down?
Answer: What kind of rack panniers do you have? They're all going to be slightly heavier than a pack, but not enough to slow you down that much. I find a backpack more tiring. It's weight that you're having to support on your back instead of letting the bike do the work.