Memristive Neural Networks: Modeling, Prototyping, and Hardware-Software Co-Design
Open Access DepositedThe recent surge in the performance of artificial intelligence (AI) systems has been fueled by scaling neural network dimensionalities. As this scaling continues, conventional computing systems are becoming increasingly inefficient due to bottlenecks between processing and memory affecting computational speed and energy efficiency. Thus, there is a major need to investigate alternative in-memory computation systems. Emerging non-volatile memory devices are a great candidate to circumvent this problem as they combine processing and memory capabilities while being more power efficient. However, they exhibit non-idealities that hinder the performance of analog and mixed-signal systems based on these devices compared to their software counterparts. These non-idealities make write operations and efficient weight updates challenging, and new algorithms are required for efficiently bridging this performance gap in neural network training as well as inference workloads. Furthermore, system-level prototyping with such emerging devices is costly, and algorithmic investigations require hardware neural network modeling which often deviates from experimental reality. In this work, novel solutions are proposed to these key challenges in modeling, prototyping, and hardware-software co-design of memristive neural networks. Five key contributions can be highlighted as follows. This dissertation contributes to the development, testing, and demonstration of a mixed-signal hardware prototyping platform for device-level characterization and neural network-level benchmarking of two-terminal memristive devices [1]. It also proposes data-driven device models that mimic physical characteristics (current vs. voltage, conductance vs. pulse etc.) of two-terminal and three-terminal memristive devices efficiently and realistically [2], [3]. A neural network simulation framework is also developed to study device-network interactions and train hardware-aware neural networks that are robust to device and system non-idealities [4]. Statistical metrics are proposed and included for quantifying the quality of emerging memory device models and networks [2], [3], [4]. Lastly, the dissertation proposes algorithmic improvements and optimizations for training memristive neural networks [5] and a training-free approach for improved fault-tolerance in memristive networks when mapping pre-trained network solutions to non-ideal memristive hardware [6]. Studies are conducted on a variety of problems across numerous neural network architectures, including multi-layer perceptron networks (MLPs), convolutional neural networks (CNNs), long short-term memory networks (LSTMs), as well as recent transformers and large language models (LLMs). The primary focus is on resistive RAM (ReRAM) devices but other technologies such as ferroelectric field effect transistors (FeFETs) and magnetic tunnel junctions (MTJs) are also explored to highlight the broader applicability of the proposed work. These contributions are key towards a comprehensive end-to-end solution for the implementation and evaluation of memristive neural networks, bridging gaps between experimental prototyping and theoretical modeling. By addressing key challenges in device modeling, hardware-software co-design, and system-level integration, this work accelerates the development of memristive neural network accelerators, architectures, and algorithms. These advancements are particularly impactful for large-scale, energy-efficient machine learning and AI applications, where unique properties of memristive devices – such as high density, non-volatility, and analog computing capabilities – offer transformative potential. The methodologies and tools developed facilitate the design and optimization of individual memristor devices and memristor-based neural networks, paving the way for scaling these technologies to tackle the high computational demands of modern AI workloads.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.