High-bandwidth flash GPU memory targets terabyte-scale AI accelerators

Sandisk and SK Hynix are developing high-bandwidth flash GPU memory, a storage-derived approach meant to push AI accelerator memory from hundreds of gigabytes into the terabytes. Sandisk says the technology, known as HBF, is aimed at inference systems that need far more model capacity without spreading weights across as many processors, although commercial details and customer commitments have not been disclosed.

The pitch is straightforward: use NAND flash in a stacked package that resembles high bandwidth memory, but deliver much higher capacity than DRAM-based HBM can economically provide. Sandisk says its first-generation HBF modules use 16 stacked layers and can reach read bandwidth of up to 1.6 TB/s. That would put the initial version above HBM3e on read bandwidth, but below HBM4, which is already cited at 2.5 TB/s per 12-high stack.

What is high-bandwidth flash?

High-bandwidth flash is a proposed accelerator memory technology that stacks NAND flash dies to deliver SSD-like capacity in a package designed for HBM-style bandwidth. Because it uses NAND rather than DRAM, Sandisk says HBF can reach up to 256 Gb per die, or 512 GB in a 16-high module.

That capacity is the main claim. Sandisk’s figures imply more than 14 times the capacity of the HBM4 used with the latest AMD and Nvidia accelerators. Future HBF generations are projected by Sandisk to move beyond 2 TB/s and later to 3.2 TB/s, but those figures remain roadmap claims.

HBF is not being positioned as a drop-in replacement for HBM. NAND has finite write endurance and access latency measured in microseconds, compared with tens of nanoseconds for DRAM. For expensive accelerators expected to stay in service for years, those limits matter.

Why HBF is aimed at inference, not general GPU memory

Sandisk and SK Hynix are proposing HBF as a complement to HBM. The idea is to keep write-heavy work in HBM while using HBF for read-heavy parts of large language model inference.

Inference can be split broadly into prefill and decode phases. Prefill processes the prompt, creates embeddings, runs the forward pass, builds key-value caches and produces the first token. Decode repeatedly reads model weights, or the active parameters in a mixture-of-experts model, for each generated token. Since decode is dominated by repeated reads, HBF is a better fit there than in workloads that constantly rewrite memory.

The non-volatile nature of HBF also changes system behavior. Model weights stored in HBF would not need to be loaded from separate storage into GPU memory each time, making it closer in concept to persistent memory than to a conventional SSD.

Sandisk has shown an example configuration with 3.12 TB of total memory using two HBM stacks and six HBF stacks. In theory, that kind of capacity could allow very large mixture-of-experts models, including multi-trillion-parameter models such as Kimi K3, to fit on one accelerator instead of being split across multiple GPUs and their interconnects. At rack scale, the same approach could support models with hundreds of trillions of parameters, although training and active-parameter bandwidth would remain limiting factors.

When could high-bandwidth flash reach data centers?

Sandisk has said first HBF samples are expected later this year, with the first AI inference devices using the technology targeted for early next year. That timeline depends on more than the memory modules being ready.

Standardization is still a gating item. Sandisk and SK Hynix began work under the Open Compute Project earlier this year, and SK Hynix president and chief development officer Ahn Hyun said the goal was to optimize the AI infrastructure ecosystem rather than compete only on individual technology performance.

Manufacturing and packaging are also open questions. HBF modules would require many more dies than typical flash storage products, and the NAND used is understood to differ from SSD NAND. The modules would also need to be co-packaged with accelerators using techniques such as TSMC’s CoWoS or Intel’s EMIB and Foveros, which means GPU and ASIC vendors have to design for them. That makes broad deployment more likely to follow accelerator design cycles than memory vendor roadmaps.

This story draws on original reporting from The Register.