Deep Learning for Channel Estimation in RIS-NOMA-assisted THz Communications over Generalized Fading Channels

This paper investigates the channel estimation problem in a multiple-input single-output (MISO) reconfigurable intelligent surface (RIS) non-orthogonal multiple access (NOMA)-assisted terahertz (THz) communication system where users experience mobility and varying small-scale fading characterized by the generalized α − μ distribution. Reliable channel estimation becomes challenging when passive RIS elements and superimposed NOMA signals operate under high attenuation of THz band. We propose a novel deep learning framework, THz RIS-NOMA Channel Estimation (TRiNCE), for cascaded channel estimation. TRiNCE is designed as a GRU-based conditional Wasserstein Generative Adversarial Network with Gradient Penalty (cWGAN-GP). TRiNCE performance is evaluated under various α − μ fading conditions, the number of RIS elements, NOMA power allocation factors, and the number of BS antennas. The model achieves an R2 score of 0.85 under Rayleigh fading and up to 0.95 under milder α–μ conditions. It further demonstrates strong robustness, maintaining high estimation accuracy with less than 2% performance variation across different RIS sizes and NOMA power allocation factors, and only ∼ 7% degradation when the number of BS antennas increases from 1 to 5. Results show that TRiNCE not only provides accurate and reliable channel estimation across all tested network configurations but also significantly outperforms convolutional neural network (CNN), long short-term memory (LSTM), and gated recurrent unit (GRU) baseline models while requiring substantially fewer trainable parameters. This establishes TRiNCE as a computationally efficient and effective solution for channel estimation in RIS-NOMA-assisted THz communication systems.