The studio of research fellow Chen Luonan at the Hangzhou Institute of Advanced Studies, UCAS (hereinafter referred to as HIAS) effectively improved the global optimization and generalization capabilities of artificial neural networks through the organic combination of chaotic dynamics in the learning process of real brains and the famous BP algorithm in deep learning.The research paper entitled "Brain-inspired chaotic backpropagation for MLP" was published online in Neural Networks (impact factor: 9.657; ranked among the top 10% of journals in the field) on August 10, 2022, Beijing time.
Artificial neural networks have been widely used in all aspects of scientific and technical research. The results achieved in this area are remarkable, including the prediction of three-dimensional protein structures, face recognition, machine translation, self-driving, time series prediction and disease diagnosis.One of the main contributors to these achievements is the error backpropagation (BP) algorithm, which can concurrently update the weight of neural networks according to the error signal to achieve the purpose of learning. So far, almost all the state-of-the-art (SOTA) models in deep learning are learned using the BP algorithm. Although the BP algorithm has been very successful, this gradient dynamics-based learning approach makes models very susceptible to falling into a localminimum. In addition, experiments have shown that chaotic dynamics, which are very sensitive to initial values, is used in the learning process of a real brain. This nonlinear aperiodic dynamical behavior has been theoretically proved to have global search capability.Therefore, the introduction of real chaotic dynamics in the brain into the BP algorithm is likely to help the network models jump out of a local minimum, to improve their learning efficiency.
Specifically, the research is the first to theoretically and computationally demonstrate that the addition of chaotic loss in the cross-entropy form to the original BP loss function is required to introduce real chaotic dynamics into the learning process (Figure 1c).According to the subsequent tests on multiple benchmark datasets represented by Cifar10, the newly proposed chaotic backpropagation (CBP) algorithm is significantly superior to the traditional BP algorithm and its variants (such as SGDM and Adam) in terms of optimization and generalization properties when the calculated quantity is not significantly increased.
In conclusion, this research has introduced biologically significant chaotic dynamics into the learning process of artificial neural networks for the first time and significantly improved the learning efficiency of network models.As a result, new thought changes have taken place in the field of deep learning, and a successful case for studying the superiority of brain-like computing has been provided.
Figure 1. Schematic Diagram of CBP Algorithm
The first author of the paper is Tao Peng, an assistant research fellow from the School of Life and Health Sciences, HIAS, and the corresponding author is research fellow Chen Luonan.The research was funded by the National Natural Science Foundation of China and the Chinese Academy of Sciences.
Source | School of Life Science
Typesetter | Wang Peng and Wang Zhe
Executive Editor | Wang Xia
