  We develop a systematic and unified approach to estimate all possible
secondary (i.e. non-primordial) nonlinear effects to the cosmic microwave
background (CMB) polarization, named curve-of-sight integration approach. In
this approach, the Boltzmann equation for polarized photons is rewritten in a
line-of-sight integral along an exact geodesic in the perturbed universe,
rather than a geodesic in the background universe used in the linear-order CMB
calculation. This approach resolves the difficulty to solve the Boltzmann
hierarchy with the nonlinear gravitational effects in the photon free-streaming
regime and thus unifies the standard remapping approach for CMB lensing into
the direct approach solving the Boltzmann equation for the nonlinear
collisional effects. In this paper, we derive formulae that: (i) include all
the nonlinear effects; (ii) can treat extended sources such as the
contributions after the reionization. It offers a solid framework to discuss
possible systematics in the standard estimation of CMB lensing by the remapping
approach. As an explicit demonstration, we estimate the secondary B-mode power
spectrum induced by all foreground gravitational effects: lensing, redshift,
time-delay, emission-angle, and polarization-rotation effects. We define these
effects properly so that they do not have any overlap, also without overlooking
any effect. Then, we show that these effects only give corrections of the order
of 0.001-0.01% to the standard lensing-induced B-mode power spectrum in the
concordance $\Lambda$ cold dark matter model. Our result confirms the
reliability of using the remapping approach in upcoming CMB experiments aiming
to detect the primordial gravitational waves with the tensor-to-scalar ratio of
$r \sim 10^{-3}$.
