  In this paper, we propose a scheme that utilizes the optimization ability of
artificial intelligence (AI) for optimal transceiver-joint equalization in
compensating for the optical filtering impairments caused by wavelength
selective switches (WSS). In contrast to adding or replacing a certain module
of existing digital signal processing (DSP), we exploit the similarity between
a communication system and a neural network (NN). By mapping a communication
system to an NN, in which the equalization modules correspond to the
convolutional layers and other modules can been regarded as static layers, the
optimal transceiver-joint equalization coefficients can be obtained. In
particular, the DSP structure of the communication system is not changed.
Extensive numerical simulations are performed to validate the performance of
the proposed method. For a 65 GBaud 16QAM signal, it can achieve a 0.76 dB gain
when the number of WSSs is 16 with a -6 dB bandwidth of 73 GHz.
