Low Rank Approximations for Efficient DNN Training
Open AccessAs the market demand for artificial intelligence grows, deep neural networks (DNN) frameworks are constantly being improved to accommodate higher machine learning workloads as well as meet the need of achieving superior hardware resource utilization. To fully exploit the hardware availability, deep network frameworks have increasingly been natively supporting methods for model and data parallelism. However, this presents new challenges. The synchronization of the network models is time consuming due to the limited communication bandwidth and the large size of the models, sometimes even taking up more than 60% of the overall training time. The usual data reduction approaches to deal with this problem may have high local memory and computational overhead, requiring significant logic to be performed over the full set of estimated gradients and weights.This thesis aims to develop resource-efficient algorithms for neural network training and is focused on the low rank approximations needed for gradient memory-efficient update without significant loss in accuracy. We propose the streaming batch principal component analysis (SBPCA) algorithm which extracts the top k principal components of gradient matrices during training. We have shown using CIFAR10, CIFAR100 and ImageNet datasets that it is possible to reduce the memory and computational overhead while producing similar accuracy results with a regular Stochastic Gradient Descent (SGD) algorithm.We developed the SBPCA algorithm having in mind its applicability to hardware neuromorphic accelerators based on emerging non-volatile memory technologies. Such hardware promises to outperform state-of-the-art GPUs/TPUs by 100x since they efficiently store the weights of large networks. However, for training, device non-idealities lead to poor accuracy or even non-convergence. Our systematic investigations have shown that batch training can provide significant accuracy improvements approaching software-level accuracies. Low rank decomposition algorithms can reduce the large quantity of training and gradient data that needs to be stored during batch updates. This approach can play a crucial role in enabling the utilization of emerging hardware despite their imperfections, bringing these new technologies closer to deployment in advanced neural network training accelerators.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.