Gradient Decomposition Methods for Training Neural Networks with Non-ideal Weights
Open AccessResistive switching (memristor) devices are an emerging hardware technology useful for the implementation of dense and efficient synaptic weights in artificial neural networks. Prior work showed 3-6 orders of magnitude improvement in speed and energy efficiency by comparison with purely digital electronic counterparts on inference tasks. However, there are practical challenges due to manufacturing and inherent physics of these devices. These challenges reduce the performance of network training when using the traditional Stochastic Gradient Descent (SGD) algorithm used in neural networks.This thesis aims to investigate low-bit precision, low-overhead alternative algorithms that tackle these issues and to test them with realistic resistive switching models. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average out non-idealities, stochastic rounding (to avoid vanishing weight updates), and decomposition methods to keep the memory overhead low during mini-batch training. Two decomposition methods are explored: streaming batch Principal Component Analysis (PCA) algorithm and the Non-negative Matrix Factorization (NMF) algorithm. Since the weight update has to be transferred to the memristor matrices efficiently, two weight update methods are also investigated. The first is rank-sum where the gradient is averaged externally to the array and then transferred weight by weight to each device. The second is rank-seq where rank-1 updates are applied to the array sequentially and approximate gradient recomposition happens at the array level. Rank-seq operations are preferred since they are theoretically faster than rank-sum operations and allow for memristor array level updates. This was tested for different ranks, batch sizes, and memristor models using a fully connected network trained on MNIST. Our results show that a network with ideal memristor weights can reach equivalent software accuracies when SGD is used for training. However, non-ideal memristor networks have poor performance when SGD is used, but accuracy recovery is possible using MBGD training. Convergence curves for different batch sizes show a consistent increase in performance with the increased batch size, despite device non-idealities. Nevertheless, the convergence results are highly sensitive to the learning rate. In addition, stochastic rounding for delta weight update outperforms regular truncation minimizing the quantization error vanishing gradient issue in limited precision weights. Streaming batch PCA and NMF can achieve near MBGD accuracy in non-ideal systems with only 3 ranks at significant memory savings. Moreover, higher batch sizes can be used to increase the accuracy of the low-rank approximation algorithms. Moreover, NMF rank- seq outperforms streaming batch PCA rank-seq at low-ranks making it more suitable for hardware implementations in memristor accelerators. In conclusion, the choice of gradient descent method and hyperparameters is critical for high accuracy training in non-ideal memristive weights. Various decomposition methods can be successfully used for MBGD gradient compression to avoid memory overhead. Nevertheless, algorithmic performance and efficiency should be co-optimized to support the transfer of gradient information to the memristor matrices at the level. Since these results were obtained with realistic device models, they open the way for further investigations on deeper neural networks and more complex datasets as well as experimental demonstration of efficient memristor-based machine learning accelerators.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.