DESIGN TOOLS
Loading...

Suggested Search

    Recent Search

    Invalid input. Special characters are not supported.

    Memory

    Bottlenecks to breakthroughs: The future of memory and storage

    Karthik Ganesan | September 2025

    Over three decades of system architecture evolution, memory has evolved from a quiet background player to the star of the show. In the early days, we worried about cache misses and DRAM latency. Today, we’re grappling with how to feed trillion-parameter AI models without choking on bandwidth or power. The memory wall hasn’t disappeared — it’s just moved. And the AI era has made it taller.

    The memory wall reimagined

    In 1994, Wulf and McKee warned of the “memory wall” — a bottleneck that caused CPU speeds to outpace memory access times. Their prediction catalyzed a wave of architectural responses: multi-level caches, speculative execution and out-of-order processing. But those tricks only went so far.

    Now, AI workloads have redefined the problem. It’s no longer just about latency — it’s about scale, bandwidth and energy. Training a large language model means streaming petabytes of data, storing hundreds of gigabytes of weights and doing it all in real time. Traditional memory architectures weren’t built for this.

    AI’s appetite for memory

    AI models are memory monsters. They demand:

    • High bandwidth to feed GPUs and accelerators at full throttle.
    • Large capacity to hold massive datasets and model parameters.
    • Low latency for real-time inference and responsiveness.
    • Energy efficiency to keep data centers sustainable.

    Increasing focus on inference creates new challenges:

    • Modern LLMs like GPT-3 (175 billion parameters) or GPT-4 require hundreds of gigabytes of memory just to store weights.
    • Memory usage scales dramatically when serving multiple concurrent requests. For example, a 66 billion parameter model with 128k token context across 10 requests can consume over 3TB of memory.
    • Longer context windows (e.g., 128k tokens) increase memory usage quadratically due to attention mechanisms.
    • Unlike training, inference is often real-time (e.g., chatbots, search engines) and memory latency directly impacts user experience. If memory access is slow, response times suffer.

    Flash storage: Feeding the pipeline

    AI isn’t just memory-bound — it’s storage-bound too. Feeding GPUs with data fast enough requires storage that can keep up. The Micron 9650 PCIe Gen6 SSDs offer up to 28GB/s read speeds and millions of IOPS, ensuring that data pipelines don’t stall. The Micron 6600 ION SSD, with capacities up to 245TB, allows entire datasets to reside close to compute, minimizing I/O bottlenecks.

    These aren’t just specs — they’re enablers. They allow AI systems to operate at scale, with minimal latency and maximum throughput.

    Chart pyramid

    Solving the AI memory wall challenge

    So how do we tackle the AI memory wall? It’s not one solution — it’s a layered strategy:

    1. Tiered memory and storage architectures

    AI systems need smart memory tiering — placing hot data in fast memory (HBM, DDR5), warm data in slower memory (LPDDR5, flash) and cold data in archival storage. This continues to evolve as we identify new use cases. Key value caching demands high-performance storage to supplement the memory tier. New inference development around RAG and vector embeddings demands more memory and fast access to small IO storage. Micron’s portfolio spans all these tiers, enabling seamless data movement and optimal performance.

    2. Processing-in-memory (PIM)

    Instead of moving data to compute, why not move compute to data? Micron is exploring PIM architectures, embedding logic into memory modules to perform operations like filtering or matrix multiplication directly in memory. This reduces data movement, cuts power and accelerates AI tasks.

    3. Energy-efficient memory and storage

    AI workloads are power-hungry. Memory can consume over 30% of a data center’s energy. Micron’s innovations in low-power DRAM and emerging non-volatile memories (like MRAM and ReRAM) aim to reduce this footprint. Similarly, high performance coupled with power efficiency and extreme reliability of storage solutions like the Micron 9550 and Micron 9650 SSDs help reduce total cost of ownership by saving rack space, energy and replacement cost in large AI deployments.

    4. Software-driven optimization

    Hardware is only half the battle. Smarter software — compilers, runtimes and orchestration layers — can optimize memory usage, compress data and manage buffers intelligently. Micron’s collaborations in this space help ensure that memory is used efficiently, not just abundantly.

    The road ahead

    We’re entering a new phase of computing where memory is no longer a bottleneck — it’s a strategic asset. The AI era demands memory and storage systems that are fast, scalable, persistent and power-aware. As we look forward, the next breakthroughs in AI won’t come from faster processors alone. They’ll come from smarter memory and storage systems to provide the best TCO for large-scale deployment. TCO is now measured by a combination of IOPS/W/$ and TB/$ to meet specific use cases. By reimagining memory and storage with AI-era requirements in mind, we aim to turn the Achilles’ heel of AI into its next great strength. In doing so, we will unlock computing capabilities that today we can only imagine — much like how far we’ve come since the floppy disks and 32 MB RAM of 30 years ago.

    And at Micron, we’re not just dreaming about that future — we’re building it.

    Technical Fellow, Storage Solutions Architecture

    Ramkarthik Ganesan

    Ramkarthik Ganesan is a Technical Fellow at Micron Technology, leading storage solutions architecture for the Core Datacenter Business Unit. He is a seasoned technologist with deep expertise in storage systems, performance optimization, and systems design. At Micron, he focuses on next-generation storage innovations that power AI, cloud, and enterprise workloads.