Original: The nonlinear layer in Rijndael uses an 8x8 S-box whereas Serpent uses eight different 4x4 S-boxes.
Simplified: The nonlinear layer in Rijndael uses an 8x8 S-box. The nonlinear layer in Serpent uses 8 different 4x4 S-boxes.