Accelerating Graph Neural Network Computation on CPUs and GPUs
Open AccessGraph Neural Networks (GNNs) are becoming popular because of their effectiveness in extracting structural information from graph data. Recent years have seen the revolutionizing results of GNNs in various applications, e.g., chemistry, social science, knowledge graph, recommendation system, and neuroscience. However, high-performance computation of GNNs is challenging. This dissertation strives to understand and accelerate the performance of GNN computation on multi-core CPUs and general-purpose GPUs by performance profiling, code generation, and efficient workload scheduling.This dissertation first presents a novel compiler-based software framework Gin optimized for GNN inference on CPUs (Chapter 3), which offers a user-friendly interface, via an intuitive programming model, for defining graph neural network models. Gin builds high-level dataflow graphs as intermediate representations, which are transformed into highly efficient codes and then compiled into binary inference kernels. Our evaluation shows that Gin significantly accelerates the inference on billion-edge graphs, beating three state-of-the-art GNN solutions as well as a traditional graph processing system Ligra.Chapter 4 describes TLPGNN, a lightweight two-level parallelism paradigm for GNN computation on single and multiple GPUs. First, we divide the GNN computation into two levels, i.e., vertex parallelism for the first level and feature parallelism for the second. Next, we employ a novel hybrid dynamic workload assignment to address the imbalanced workload distribution. Furthermore, we fuse the kernels to reduce the number of kernel launches and cache the frequently accessed data into registers to avoid unnecessary memory traffic. To scale TLPGNN to multi-GPU environments, we propose an edgeaware row-wise 1-D partition method to ensure a balanced workload distribution across different GPU devices. Together, TLPGNN is able to significantly outperform existing GNN computation systems. Evaluations of multiple-GPU TLPGNN also demonstrate that our solution achieves both linear scalability and a well-balanced workload distribution.Chapter 5 proposes JitSpMM, a just-in-time (JIT) assembly code generation framework to accelerated SpMM (Sparse Matrix-Matrix Multiplication) computation, which is an important component of GNN implementation, on multi-core CPU with SIMD extensions. Firstly, JitSpMM integrates the JIT assembly code generation technique into three widelyused workload division methods for SpMM, i.e., row-split, nnz-split and merge-split, to achieve balanced workload distribution among CPU threads. Next, with the availability of runtime information, JitSpMM employs a novel technique coarse-grain column merging to maximize instruction level parallelism by unrolling the performance-critical loop. Furthermore, JitSpMM intelligently allocates registers to cache frequently accessed data, minimizing memory access, and employs selected SIMD instructions to enhance arithmetic throughput. Together, JitSpMM is able to significantly outperform two AOT implementations for SpMM including Intel MKL by 3.8× and 1.4×, respectively, on average.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.