Deep Predictive Coding with Bi-directional Propagation for Classification and Reconstruction

Predictive Coding (PC) has emerged as a prominent theory underlying information processing in the brain. The general concept for learning in PC is that each layer learns to predict the activities of neurons in the previous layer, which enables local computation of error as well as in-parallel learning across layers. Deep Bi-directional Predictive Coding (DBPC) is proposed here as a new learning algorithm that enables neural networks to simultaneously perform classification and reconstruction tasks using the same learned weights. Building on existing PC approaches, DBPC supports both feedforward and feedback propagation of information. Each layer in the network trained using DBPC learns to predict the activities of neurons in the previous and next layers, enabling the network to simultaneously perform classification and reconstruction tasks using feedforward and feedback propagation, respectively. DBPC also relies on locally available information for learning, thus enabling in-parallel learning across all layers in the network. DBPC enables the training of both fully connected networks and convolutional neural networks. The classification accuracies of DBPC on the MNIST, Fashion-MNIST, and CIFAR-10 datasets (99.58%, 92.42%, and 74.29%, respectively) exceed those of well-established PC-based benchmark approaches (including FIPC3 and iPC) and are competitive with state-of-the-art Error-Backpropagation-based methods (including ResNet and DenseNet) on MNIST, Fashion-MNIST, and EuroSAT datasets. Importantly, DBPC achieves these results using significantly smaller networks for MNIST, Fashion-MNIST, and CIFAR-10 datasets (0.425, 1.004, and 1.109 million parameters), and every representation estimated in DBPC can be used for the reconstruction of inputs. The significant benefit of DBPC is its ability to achieve this performance using locally available information and in-parallel learning mechanisms, which results in an efficient training protocol. Overall, we demonstrate that DBPC is a much more efficient approach for training networks that can perform both classification and reconstruction simultaneously.