State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory.
Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023