Electronics Guide

NPUs Among Processor Types

The term "NPU" is used loosely across the industry, and its meaning depends on context. In consumer silicon it usually denotes a fixed-function inference block integrated into a system-on-chip alongside the CPU and GPU. In the data center the equivalent devices are marketed as tensor processing units, AI accelerators, or intelligence processing units. Despite the naming, all of these devices share the same design premise: replace instruction-level generality with dense arithmetic and tightly managed data movement.

A CPU spends most of its transistor budget on control logic, branch prediction, out-of-order scheduling, and cache coherence, all of which serve unpredictable code. A GPU dedicates far more area to arithmetic but retains a general programming model and a deep memory hierarchy, and modern GPUs embed matrix units of their own. An NPU goes further still, hard-wiring the dataflow of matrix multiplication so that a single instruction can launch thousands of multiply-accumulate operations and so that operands are reused many times before they return to memory. The result is exceptional efficiency on the operations the hardware anticipates, and poor efficiency, or outright incapability, on operations it does not.

That trade-off has practical consequences. An NPU that lacks a native operator falls back to the CPU or GPU, and the resulting round trips can erase the accelerator's advantage entirely. Layer coverage in the vendor's compiler, not peak arithmetic throughput, is frequently the factor that determines whether a given model runs well on a given NPU.

Tensor Processing Architectures

Tensor processing architectures form the computational foundation of modern NPUs, optimized for the multi-dimensional array operations central to neural networks. These architectures exploit the fact that deep learning workloads exhibit highly predictable access patterns and computation sequences, enabling aggressive specialization that would be impossible in a general-purpose processor.

The fundamental operation in most neural networks is the multiply-accumulate (MAC), in which input values are multiplied by learned weights and accumulated to produce outputs. Tensor processors pack thousands of MAC units into regular arrays with carefully designed data paths that keep those units supplied with operands. The difficulty lies less in providing raw arithmetic than in managing the data movement required to feed it. Moving a value from external DRAM costs on the order of a hundred times the energy of the multiply that consumes it, so the memory system, not the arithmetic array, sets the practical efficiency ceiling.

Modern tensor processors therefore employ hierarchical memory systems: register files inside each processing element, shared scratchpads or software-managed buffers per tile, a large on-chip SRAM, and finally external memory. The original Google TPU, for example, paired its arithmetic array with a 28 MiB software-managed on-chip buffer rather than a conventional cache, trading hardware flexibility for predictable latency. Data-center training accelerators supplement on-chip storage with stacked high-bandwidth memory (HBM), where each stack supplies on the order of a terabyte per second.

How data traverses this hierarchy is described by the accelerator's dataflow. Weight-stationary designs hold weights in place and stream activations past them, which suits layers that reuse a small weight set across many inputs. Output-stationary designs pin partial sums in place and stream both weights and activations, minimizing accumulator traffic. Row-stationary designs, popularized by the Eyeriss research accelerator, balance reuse across weights, activations, and partial sums simultaneously. The choice interacts strongly with layer shape: a dataflow that is near optimal for a large convolution may be badly underutilized on a depthwise-separable or attention layer, which is why many production NPUs support more than one mapping.

Numeric Precision and Quantization

Reduced precision is the single largest lever in NPU efficiency. The energy and area cost of a multiplier scales roughly with the square of operand width, so narrowing from 32-bit floating point to 8-bit integer reduces multiplier cost by more than an order of magnitude while also shrinking every buffer, bus, and memory transfer along the path. Neural networks tolerate this remarkably well, because inference is statistically robust to small perturbations in individual weights.

Eight-bit integer arithmetic is the dominant inference format. Quantization maps a floating-point tensor onto an integer range through a scale factor and, in asymmetric schemes, a zero point. Per-channel scales, in which each output channel receives its own scale factor, recover most of the accuracy lost by a single per-tensor scale. Post-training quantization applies these mappings to an already-trained model using a small calibration set, while quantization-aware training simulates the rounding during training and generally preserves accuracy better on sensitive models.

Training favors floating-point formats that preserve dynamic range. The bfloat16 format keeps the 8-bit exponent of IEEE single precision and truncates the mantissa to 7 bits, so it matches FP32 range and rarely requires loss scaling; IEEE FP16 offers more mantissa bits but a narrower exponent. Newer 8-bit floating-point formats, standardized through the Open Compute Project and supported in recent accelerator generations, provide the E4M3 and E5M2 encodings, the first favoring precision for weights and activations and the second favoring range for gradients.

Below 8 bits, plain fixed-point formats degrade quickly, and block-based formats have become the standard answer. Microscaling formats, published by the Open Compute Project in 2023 with backing from AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm, attach a shared exponent to a small block of elements, so a 4-bit element retains usable dynamic range. These formats now underpin much of the low-precision inference and, increasingly, training support in current NPUs. Binary and ternary weights remain viable in narrow, heavily constrained applications, but they demand architecture changes rather than simple retraining.

Systolic Array Designs

Systolic arrays are among the most efficient known structures for matrix multiplication, the operation that consumes the majority of computation in neural networks. Named for the rhythmic data flow that resembles the pumping of blood through the circulatory system, a systolic array is a regular grid of processing elements that pass operands to their neighbors in a coordinated wave.

In a typical arrangement, activations enter from one edge while weights enter from a perpendicular edge. Each processing element multiplies its inputs, adds the product to a running accumulator, and forwards operands to its neighbors. The structure achieves its efficiency through reuse: a value fetched once from memory is consumed by an entire row or column of processing elements as it propagates, and partial sums never leave the array until they are complete. Google's first-generation Tensor Processing Unit demonstrated the approach at scale with a 256 by 256 array of 8-bit MAC units, giving 65,536 MACs and a peak of 92 tera-operations per second. In the published data-center evaluation it ran production inference roughly 15 to 30 times faster than the contemporary server CPU and GPU it was compared against, at 30 to 80 times better performance per watt.

Systolic arrays excel at large, dense, regular computations and struggle elsewhere. Sparse networks, in which many weights are zero, waste array cycles unless the hardware can skip zero-valued operands. Layers smaller than the array leave processing elements idle, a mismatch that grows worse as arrays grow larger; a 256 by 256 array running a layer with 64 output channels utilizes a quarter of its columns at best. Contemporary implementations mitigate these effects by partitioning a large array into independently scheduled tiles, by supporting structured sparsity patterns that the hardware can compress predictably, and by fusing small layers so they fill the array together.

Dataflow Accelerators

Dataflow accelerators organize computation around the movement and transformation of data rather than around the sequential execution of instructions. Work proceeds whenever operands and operators are available, which exposes fine-grained parallelism and removes much of the instruction fetch and decode overhead that burdens conventional processors.

Spatial dataflow architectures map a neural network onto a physical array of processing elements, with results flowing directly between elements over an on-chip network. Because intermediate activations pass from producer to consumer without a round trip through memory, the approach suits deep pipelines of small layers particularly well. Graphcore built its Intelligence Processing Unit on this principle; its second-generation chip carries 1,472 independent processor tiles, each with its own local SRAM and hardware multithreading, connected by a high-bandwidth on-chip exchange. Holding the entire model in distributed on-chip memory avoids external memory traffic altogether, at the cost of a hard capacity limit that large models can exceed. SoftBank acquired Graphcore in 2024, and the company continues to develop the architecture as a subsidiary.

Reconfigurable dataflow accelerators adapt their interconnection patterns to match different network structures rather than executing every model on a fixed topology. The hardware is capable of near-optimal efficiency across diverse architectures, but the burden shifts decisively to the compiler, which must partition the graph, allocate on-chip memory, schedule communication, and respect capacity and bandwidth constraints simultaneously. Compilation quality, not silicon capability, is usually what separates a strong dataflow deployment from a disappointing one.

Reconfigurable AI Processors

Reconfigurable AI processors combine the efficiency of specialized accelerators with the flexibility of programmable hardware. Field-programmable gate arrays (FPGAs) and coarse-grained reconfigurable architectures (CGRAs) can be customized for particular models, approaching application-specific efficiency while remaining able to absorb new operators. That flexibility is valuable precisely because neural network architectures continue to change faster than silicon design cycles.

FPGA-based accelerators implement network layers as custom digital circuits, with data path width, numeric precision, and parallelism tailored to the model in hand. Modern devices include hardened digital signal processing blocks with native support for low-precision multiplication, on-die high-bandwidth memory, and hardened network interfaces, all of which suit inference pipelines. AMD, which acquired Xilinx in 2022, and Intel, whose programmable logic business now operates as Altera, both offer AI-oriented FPGA families with accompanying compilation tools. FPGAs are most compelling where volumes are too low to justify a custom chip, where latency must be deterministic, or where the accelerator must sit inline with a network or sensor interface.

CGRAs occupy a middle ground. Rather than reconfiguring at the bit level as FPGAs do, they reconfigure at the word level, connecting arrays of arithmetic-capable processing elements through a programmable interconnect. The coarser granularity yields far smaller configuration bitstreams, faster reconfiguration, and higher clock frequencies than bit-level fabrics, at the price of flexibility for non-arithmetic logic. Because neural network operators are word-oriented by nature, the granularity matches the workload well, and CGRA-like fabrics now appear inside several commercial AI accelerators and inside the AI engine tiles of recent adaptive FPGA families.

Neuromorphic Processors

Neuromorphic processors emulate the structure and dynamics of biological neural systems, communicating through discrete spikes rather than continuous values. Because a spiking neuron consumes energy only when it fires, and because activity in a well-designed spiking network is sparse, the approach promises large efficiency gains on workloads dominated by long idle periods punctuated by brief events.

The human brain performs remarkable perception and cognition on roughly 20 watts, a figure that motivates the field. The architectural principles drawn from it include event-driven computation, in which processing occurs only in response to change; sparse connectivity, in which each neuron reaches a small fraction of the others; co-located memory and computation; and local learning rules that adapt synapses without global gradient descent.

IBM's TrueNorth, introduced in 2014, packs 4,096 neurosynaptic cores onto a 28 nm chip, modeling roughly one million neurons and 256 million synapses while drawing on the order of 70 milliwatts on representative workloads. Intel's Loihi, introduced in 2017, distributed about 130,000 neurons across 128 neuromorphic cores and added on-chip programmable learning rules, allowing synaptic adaptation without host intervention. Loihi 2, announced in 2021, retains 128 neuromorphic cores but raises capacity to roughly one million neurons per chip and adds programmable neuron models and graded spikes. Intel's Hala Point research system, unveiled in 2024, assembles 1,152 Loihi 2 processors into a six-rack-unit chassis supporting about 1.15 billion neurons and 128 billion synapses within roughly 2,600 watts.

Neuromorphic systems require different algorithms than conventional deep learning. Spiking neural networks encode information in the timing and rate of discrete pulses, and the spike generation function is not differentiable, so training relies on surrogate gradients, conversion from trained rate-based networks, or local plasticity rules. The absence of a mature, portable software stack remains the principal barrier to adoption. The clearest wins to date are in always-on sensing, event-camera vision, sparse signal processing, and certain constraint-satisfaction and optimization problems, rather than in general-purpose deep learning.

Analog AI Accelerators

Analog AI accelerators perform neural network computations with continuous physical quantities rather than digital representations, exploiting device physics to implement multiply-accumulate operations directly. Eliminating explicit digital arithmetic and, more importantly, eliminating the movement of weights from memory to compute units, offers efficiency well beyond what digital design can reach.

The most developed analog approach uses crossbar arrays of non-volatile memory devices, where the conductance of each cell encodes a weight. Applying voltages to the rows and reading currents from the columns performs a full matrix-vector multiplication in a single step, with Ohm's law forming the products and Kirchhoff's current law summing them. The candidate devices include phase-change memory, resistive RAM, magnetoresistive RAM, and floating-gate flash cells, each with a different balance of endurance, retention, conductance range, and programming energy.

Working silicon exists. IBM's HERMES project chip, reported in Nature Electronics in 2023, integrates 64 analog in-memory computing cores in 14 nm CMOS with phase-change memory added in the back end of line, each core holding a 256 by 256 array of unit cells for more than 16 million devices in total. On 8-bit matrix-vector multiplications the chip reached 2.48 tera-operations per second per watt in its high-precision four-phase read mode and 9.76 in its single-phase mode, illustrating both the promise of the technique and the precision-versus-efficiency trade it forces.

Analog computation faces persistent challenges in precision and stability. Device-to-device variation, conductance drift over time, temperature sensitivity, and the cost of the analog-to-digital converters at the array periphery all limit effective resolution, and those converters often dominate the energy budget of a real design. Mitigations include programming with iterative write-and-verify, differential cell pairs that cancel common-mode error, hardware-aware training that anneals the model against measured device statistics, and hybrid partitioning that keeps sensitive layers in digital logic. Weight update is also expensive on most device technologies, which is why analog accelerators target inference rather than training.

Optical Neural Networks

Optical neural networks use light to perform the linear algebra of inference, exploiting the parallelism of optics and the absence of resistive loss in propagation. A passive optical element can apply a fixed linear transformation to an entire input vector at the speed of propagation, with the energy cost dominated by the sources and detectors rather than by the transformation itself. Wavelength, spatial mode, and polarization all provide additional parallel channels.

Several architectures have demonstrated the principle. Free-space diffractive systems encode data in light intensity or phase and implement learned weights as fabricated optical masks, computing a multilayer transformation as the beam passes through successive layers. Integrated photonic circuits build the same transformations on silicon from waveguides, phase shifters, and meshes of Mach-Zehnder interferometers, which are compact and compatible with CMOS fabrication. Coherent approaches use wavelength-division multiplexing and microring resonator banks to compute many products in parallel on a single waveguide, and programmable metasurfaces and spatial light modulators provide reconfigurable free-space alternatives.

The obstacles are at the boundaries rather than in the optics. Input data must be modulated onto light and results must be detected and digitized, and the energy and latency of those conversions can consume the advantage unless a large amount of computation happens between them. Nonlinear activation functions, essential to network expressiveness, generally require either an electronic round trip or optical nonlinearities that demand high power. Phase shifters drift with temperature and require calibration and control loops, and analog optical computation carries the same limited effective precision as its electronic analog counterpart. Optical approaches consequently look most promising for high-throughput, latency-critical inference on large linear layers, and for optical front ends that process sensor data before it is ever digitized.

Quantum Machine Learning Hardware

Quantum machine learning hardware seeks to exploit quantum mechanical phenomena for learning tasks. A register of qubits spans a state space that grows exponentially with qubit count, and certain linear algebra primitives admit quantum algorithms with better asymptotic scaling than the best known classical ones. Whether that translates into practical advantage on realistic data is an open research question, and several early claims of advantage have since been matched by improved classical algorithms.

Near-term work concentrates on variational algorithms that treat a parameterized quantum circuit as a trainable model, with a classical optimizer adjusting the parameters while the quantum processor evaluates the objective. Quantum kernel methods use the quantum state space as an implicit feature space for classification. Quantum sampling approaches use the processor to draw samples from distributions that are hard to sample classically, which is of interest for generative modeling. Variational training also encounters its own obstacles, notably barren plateaus, where gradients vanish exponentially with circuit width and make optimization intractable.

Hardware limitations remain the binding constraint. Physical qubit counts have passed one thousand in the largest systems, including IBM's 1,121-qubit Condor superconducting processor announced in 2023 and neutral-atom arrays of comparable size, but raw count is not the useful figure of merit. Coherence times and two-qubit gate errors bound the circuit depth that can be executed before noise dominates. Fault-tolerant operation requires error correction, and surface-code estimates imply roughly a thousand physical qubits per logical qubit at current error rates; error mitigation offers a partial near-term substitute at the cost of sampling overhead that grows steeply with circuit size. Loading classical data into quantum states is itself a bottleneck that can negate an algorithm's theoretical speedup. Quantum machine learning is therefore best understood as a research direction rather than a deployable accelerator class.

Edge AI Chips

Edge AI chips bring inference to devices at the network edge, enabling real-time results without a round trip to the cloud. Local inference is often chosen for reasons beyond latency: it keeps sensitive data on the device, removes dependence on connectivity, and eliminates recurring inference costs. These processors must balance capability against strict limits on power, thermal dissipation, cost, and silicon area. The treatment here stays at the level of accelerator architecture; Edge AI Processors is the canonical article for the edge tier, covering system integration, thermal and battery budgets, and the edge software stack in depth.

Power efficiency defines the category. Where a data-center accelerator may draw several hundred watts, edge parts operate from milliwatts to a few watts, which forces optimization at every level: 8-bit or narrower arithmetic, sparsity exploitation, aggressive operator fusion to keep intermediate tensors on chip, dynamic voltage and frequency scaling, and power gating of idle blocks. Many designs also add a tiny always-on block that watches for a wake word or a motion event and activates the main accelerator only when needed.

The landscape now spans three tiers. Personal-computing silicon integrates NPUs of tens of tera-operations per second: Microsoft's Copilot+ program set a 40 TOPS floor for on-device features, and current parts sit near or above it, including Qualcomm's Snapdragon X Elite at about 45 TOPS, Apple's M4 Neural Engine at roughly 38 TOPS, AMD's XDNA 2 engines at about 50 TOPS, and comparable Intel Core Ultra designs. Standalone accelerator modules such as Google's Coral Edge TPU, which delivers about 4 TOPS of 8-bit throughput within roughly 2 watts, serve embedded vision and robotics. At the smallest scale, microcontroller-class NPUs, many built on Arm's Ethos-U cores and shipped by vendors including STMicroelectronics and NXP, run keyword spotting, anomaly detection, and simple vision models within a few milliwatts.

Two cautions apply when reading these numbers. Advertised TOPS is a peak figure at the narrowest supported precision, and sustained throughput on a real model is commonly a fraction of it. Memory bandwidth, not arithmetic, limits most edge deployments of transformer-based models, because generating each token requires reading the entire weight set at least once.

Brain-Inspired Computing Systems

Brain-inspired computing draws architectural lessons from biology beyond the spiking neurons of neuromorphic processors. Massive parallelism, hierarchical processing, selective attention, integration of memory with computation, and continuous learning all suggest paths toward more capable and more efficient machines, and several have already influenced mainstream accelerator design.

Memory-centric architectures attack the von Neumann bottleneck by moving computation toward storage. Biological tissue stores and processes information in the same substrate, avoiding the data movement that dominates the energy budget of conventional systems. Near-memory designs place processing elements adjacent to memory arrays or in the logic die of a stacked memory; in-memory designs perform computation within the array itself. IBM's NorthPole, described in 2023, illustrates the digital end of this spectrum: 22 billion transistors in a 12 nm process organized as 256 cores with 224 MB of on-chip SRAM and no external weight memory at all, an arrangement that yielded large gains in both energy efficiency and latency on image recognition benchmarks relative to a contemporary GPU.

Attention-based architectures echo the brain's ability to concentrate limited processing on relevant information. Hardware support for attention now matters greatly, because transformer models dominate language and increasingly vision workloads. The engineering problems are concrete: self-attention scales quadratically with sequence length, the key-value cache used during generation grows linearly with context and quickly exceeds on-chip memory, and effective implementations such as fused attention kernels depend on the accelerator's ability to keep tiles of the attention matrix in fast memory. Recent NPUs add dedicated softmax and transpose units, larger on-chip buffers, and support for the irregular sparsity patterns that make long-context attention tractable.

Continual learning systems aim to absorb new experience without catastrophically forgetting prior knowledge, as biological systems do. Hardware support includes mechanisms for selectively updating a subset of weights, protecting parameters identified as important, and storing and replaying past examples efficiently. Parameter-efficient adaptation methods, which train small auxiliary matrices while leaving the base model frozen, have made on-device personalization practical and are shaping the memory and update paths of newer accelerators.

Measuring NPU Performance

Comparing NPUs requires care, because the headline metric is easy to inflate. Tera-operations per second counts a multiply-accumulate as two operations and is quoted at peak, at the narrowest supported precision, and under the assumption of perfect utilization. A part rated at 50 TOPS in 4-bit mode may offer a quarter of that in the 8-bit mode an application actually uses.

The roofline model explains most of the gap between peak and observed performance. Every workload has an arithmetic intensity, the ratio of operations performed to bytes moved, and a machine has both a peak compute rate and a peak memory bandwidth. Large batched convolutions and matrix multiplications have high arithmetic intensity and can approach the compute roof. Single-stream token generation from a large language model has arithmetic intensity near one and is bound entirely by memory bandwidth, which is why memory capacity and bandwidth, rather than TOPS, govern that workload.

For defensible comparison, engineers should look to standardized benchmarks and to the metrics that match their deployment. MLPerf, maintained by MLCommons, provides audited inference and training suites with defined models, datasets, accuracy targets, and latency constraints, including a dedicated tiny inference suite for microcontroller-class devices. The metrics that matter in practice are latency at a stated batch size and percentile, sustained throughput per watt, memory capacity and bandwidth, and accuracy after quantization, since an accelerator that hits its throughput target only by dropping below the required accuracy has not solved the problem.

Software Stacks and Compilation

An NPU is only as useful as the toolchain that targets it. The path from a trained model to executing hardware runs through an exchange or intermediate representation, a graph-level optimizer that fuses operators and folds constants, a quantization stage, a memory planner that allocates on-chip buffers and schedules transfers, and finally a code generator or a library of hand-tuned kernels.

Several layers of this stack have become common infrastructure. ONNX serves as a portable model exchange format; MLIR provides a multi-level compiler framework that many vendors build on; and runtimes such as ONNX Runtime, LiteRT, which Google renamed from TensorFlow Lite in 2024, and ExecuTorch, the successor to the deprecated PyTorch Mobile, dispatch subgraphs to whichever accelerator delegate can execute them. Vendor stacks then supply the hardware-specific back end, as CUDA and TensorRT do for NVIDIA GPUs, XLA for Google TPUs, and the corresponding neural network SDKs for mobile and embedded NPUs.

Two practical failure modes recur. The first is operator coverage: when a delegate cannot execute a layer, the runtime partitions the graph and falls back to the CPU, and the transfers around each fallback can cost more than the layers they surround. The second is quantization mismatch, where a model quantized without regard to the target's supported schemes silently loses accuracy or forces a slower execution path. Evaluating an NPU therefore means compiling and profiling the actual target model, not reading a specification sheet.

Design Considerations for NPU Selection

Selecting an NPU begins with the workload, not the hardware. The relevant inputs are the model families to be executed and their expected evolution, the required throughput and the latency percentile that must be met, the power and thermal envelope, unit cost at the intended volume, memory capacity and bandwidth needs, and the integration path into the rest of the system. No single architecture is optimal across these axes.

Different deployments emphasize different factors. Training large models rewards memory capacity, memory bandwidth, and high-speed interconnect between accelerators, since model and optimizer state must be partitioned across many devices. Large-scale inference rewards throughput per watt and per dollar within a latency budget, which favors dedicated inference parts with narrow precision support. Edge deployment is governed by absolute power and by unit cost, and often accepts reduced flexibility in exchange. Applications whose models change frequently favor programmability and broad operator coverage over peak efficiency, because a model the accelerator cannot execute has no performance at all.

Ecosystem maturity and supply are decisive in practice. Capable silicon delivers little if the compiler cannot execute the required models efficiently, and the entrenched positions of CUDA for NVIDIA GPUs and XLA for Google TPUs illustrate how much accumulated software investment matters. Newer entrants must either offer compatible stacks or offer an advantage large enough to justify the porting effort. Long-lived products should also weigh availability commitments, since embedded designs frequently outlast the market life of the accelerator they were built around.

Future Directions

NPU development continues to accelerate, driven by growing model sizes and by the end of easy gains from transistor scaling. Near-term directions are largely about integration. Chiplet-based designs assemble accelerators from smaller dies connected over standardized die-to-die interfaces such as UCIe, improving yield and allowing compute, memory, and I/O to advance on independent schedules. Three-dimensional stacking places compute directly beneath or above memory, shortening the distance that dominates energy cost. Heterogeneous systems combine accelerator types so that each phase of a workload runs on the silicon best suited to it.

Algorithm-hardware co-design increasingly shapes both sides. Structured sparsity patterns exist because hardware can exploit them predictably; low-precision block formats exist because they fit multiplier arrays; mixture-of-experts models are attractive partly because they raise capacity without proportionally raising computation per token. Hardware-aware neural architecture search and workload-aware accelerator design now proceed together, and the boundary between algorithm and architecture continues to blur.

Longer-term possibilities remain genuinely uncertain. Analog in-memory computing could deliver order-of-magnitude efficiency gains if precision, drift, and converter overhead are brought under control. Photonic processors could serve very high-bandwidth inference if the electro-optical interfaces improve. Neuromorphic systems could dominate always-on sensing if their software ecosystem matures. Quantum processors may eventually accelerate specific learning tasks. None of these has a settled timeline, and the prudent expectation is continued rapid refinement of digital accelerators alongside sustained research into the alternatives.

Conclusion

Neural processing units earn their efficiency by narrowing what they can do. Dense arithmetic arrays, carefully engineered dataflows, reduced-precision number formats, and memory hierarchies designed to maximize operand reuse together deliver performance per watt that general-purpose processors cannot approach on neural network workloads. The same specialization makes the software stack, the numeric format, and the memory system as consequential as the arithmetic array itself.

For the engineer, the practical lesson is to evaluate an NPU against the model that will actually run on it. Peak throughput figures describe an upper bound that most real workloads never reach; sustained performance follows from arithmetic intensity, memory bandwidth, operator coverage, and post-quantization accuracy. Whether the eventual winners are refined digital accelerators, analog crossbars, photonic meshes, or neuromorphic arrays, those measurement principles will remain the ones that separate a useful accelerator from an impressive specification.

Related Topics