Advanced Convolutional Neural Networks for Nonlinearity Mitigation in Long-Haul WDM Transmission Systems

Oleg Sidelnikov, Alexey Redyuk, Stylianos Sygletos, Mikhail Fedoruk, Sergei K. Turitsyn

Research output: Contribution to journalArticlepeer-review


Practical implementation of digital signal processing for mitigation of transmission impairments in optical communication systems requires reduction of the complexity of the underlying algorithms. Here, we investigate the application of convolutional neural networks for compensating nonlinear signal distortions in a 3200~km fiber-optic 11x400-Gb/s WDM PDM-16QAM transmission link with a focus on the optimization of the corresponding algorithmic complexity. We propose a design that includes original initialisation of the weights of the layers by a filter predefined through the training a single-layer convolutional neural network. Furthermore, we use an enhanced activation function that takes into account nonlinear interactions between neighbouring symbols. To increase learning efficiency, we apply a layer-wise training scheme followed by joint optimization of all weights applying additional training to all of them together in the large multi-layer network. We examine application of the proposed convolutional neural network for the nonlinearity compensation using only one sample per symbol and evaluate complexity and performance of the proposed technique.
Original languageEnglish
Article number9324921
Pages (from-to)2397-2406
Number of pages10
JournalJournal of Lightwave Technology
Issue number8
Early online date21 Jan 2021
Publication statusPublished - 15 Apr 2021

Bibliographical note

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see

Funding: The work was supported by the Russian Science Foundation (Grant No. 17-72-30006). SS and SKT acknowledges support by the EPSRC Programme grant TRANSNET (EP/R035342/1).


  • Convolutional neural networks
  • nonlinearity mitigation in fiber-optic links


Dive into the research topics of 'Advanced Convolutional Neural Networks for Nonlinearity Mitigation in Long-Haul WDM Transmission Systems'. Together they form a unique fingerprint.

Cite this