Performance investigation of geometric constellation shaping-based coherent WDM optical fiber communication system supported by deep-learning autoencoder

• Published in: Results in optics Vol. 15; p. 100629
• Main Authors: Abbass, Ayam M., Fyath, Raad Sami
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01-05-2024; Elsevier
• Subjects: Autoencoder-based deep learning; Coherent optical communication; End-to-end deep learning; End-to-end deep learning; Coherent optical communication; Autoencoder-based deep learning
• ISSN: 2666-9501; 2666-9501
• DOI: 10.1016/j.rio.2024.100629

•Presenting a comprehensive simulation platform to design and simulate GCS to enhance the performance of WDM system using an advanced autoencoder (AE) technique.•Introducing detailed investigation of the effect DL parameters on the performance of the design AE: effect of batch size, number of hidden layers, and number of nodes.•Addressing the effect of both linear parameters and nonlinear interference noise of the OFC channel on the transmission performance of the GCS- coherent WDM systems.•Investigating the AE's behavior to identify the allowed ranges of the optimal launch power using two different procedures that achieve better system performance for both 2D and 4D constellations.
Geometric constellation shaping (GCS) has been proposed to enhance the performance of wavelength-division multiplexing (WDM) coherent optical fiber communication (OFC) systems. This paper presents a comprehensive simulation platform to design and simulate these systems' performance using an advanced autoencoder (AE) technique. The design of the AE is based on end-to-end deep learning and takes into account the system parameters, especially those related to the OFC channel. The system design uses identical AEs, one in each WDM channel, and only the AE of the central channel is trained to achieve the required performance target. The developed AE simulation platform is capable of assessing the bit error rate characteristics, signal-to-noise ratio, and mutual information for long-haul coherent OFC-WDM systems operating with various modulation formats. The simulation results indicate that using a multilayer perceptron neural network to design AE requires four hidden layers, sixteen nodes per hidden layer, and a “16x modulation order” batch size to achieve optimum system performance. The AE's behaviour is investigated to identify the allowed ranges of the optimal launch power using two different procedures that achieve better system performance for 2D and 4D constellations.