  Nonlinear effects have been considered as the major limitations in coherent
optical (CO) fiber transmission system. DSP based CO receiver with digital
backpropagation (DBP) method has recently facilitated the compensation of fiber
nonlinear impairments as well as compensating dispersion of optical fiber. In
this paper, a comprehensive design is presented for circular-polarization
division multiplexed (CPDM) 8-quadrature amplitude modulation (8-QAM) with DSP
based CO receiver for the fiber optic communication (FOC) system to investigate
the impact of nonlinearities. We have examined the performance of nonlinearity
compensating DSP based CO receiver with DBP method and demonstrated that it can
be effectively employed to mitigate the intrachannel nonlinearities in CPDM
8-QAM FOC system over 800-km SSMF link. By effectively compensating the fiber
nonlinearities, we analyze the performance of bit error rate (BER), optical
signal to noise ratio (OSNR), optimum launch power, and investigate the link
margin by evaluating OSNR margin for a specific launch power. Moreover, a CPDM
technique helps as a very suitable means of maximizing the link capacity as
well as enhancing the spectral-efficiency (SE) of the FOC system.
