Developing Variational Generative Models using Predictive Coding

This paper proposes a new learning algorithm, Variational Generative Predictive Coding (VGPC), for generating new samples from a given dataset. VGPC learns using locally computed errors and uses in-parallel information propagation across the network. VGPC is based on Predictive Coding (PC) which is a computational theory for processing information in the brain. PC treats the brain as a generative model. The key concept of PC is that the neuronal activities in a given layer predict the neuronal activities in the next layer. The errors between the current neuronal activities of a layer and its corresponding predictions are used to infer better representations for a layer and the weights. The approach does not require a systematic feedforward and feedback propagation for prediction and learning as needed by methods trained using error-backpropagation. The representations and weights associated with any given layer are updated parallelly across the network using locally computed errors. We extend PC with an architecture for variational inference and incorporate the KL-divergence loss for learning in the bottleneck layer. The generation performance of VGPC, evaluated with MNIST, FashionMNIST and CelebA datasets, is compared with generative methods based on Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) using two metrics, called Fréchet Inception Distance (FID) and Inception Score (IS). The results indicate that VGPC achieves higher generation performance compared to GAN and VAE-based methods on MNIST and FashionMNIST datasets, with FID score reductions ranging from 47.0% to 81.0%. Additionally, VGPC offers the benefits of parallel and local learning.