Frontier models are already outpacing what today's memory architecture can handle. The next generation will leave the industry with nowhere left to go.
For the past decade, the AI industry has chased a single solution to the unrelenting need for greater performance: more GPUs. Those who can afford it add more racks, bigger clusters, and pay for more power to run them all. The assumption baked into that solution is that compute is the constraint worth solving, which has driven hundreds of billions of dollars in infrastructure investment and reshaped the semiconductor industry and global economy. So we often find people are shocked when we tell them that compute is no longer the biggest problem.
AI’s real bottleneck is memory. Processors only generate output when they are fed data to process. But racks of GPUs, starved of data and sitting idle, still consume power at full draw. Moving tokens or weights between 10s-100s of GPUs is slow and consumes even more power. Just look to Kimi K3, the world’s largest open-weight model at 2.8 trillion parameters. Kimi K3 requires at least 64 GPUs just to run, and even that configuration struggles under real-world inference loads.
Compounding the parameter count are context windows that have already crossed one million tokens and are on a trajectory toward ten million and beyond. Every new generation of models will demand more memory, not less. As models continue to grow at this rate, the gap between what the most advanced architectures can support today and what AI Inference actually needs for frontier model workloads to run efficiently, is the defining constraint on the future of AI.
The industry's response has been brute force: buy far more GPUs than are actually necessary, merely to access more memory. This workaround has become the industry standard. Teams running large-scale inference know what this produces: systems that are overly complex, monstrously expensive, fragile, and error-prone, with dramatically low compute utilization, resulting in massive sunk costs and huge power draw. Memory has always been the binding constraint. Everything else has been a workaround.
We founded Majestic to address this exact problem: AI’s memory wall. This is the first in a three-part series on why that problem exists, how we set out to solve it, and what becomes possible when memory is no longer holding AI back.
The Memory Wall and AI's $1 Trillion Workaround
AI model sizes have gone from billions to trillions of parameters. Today’s frontier models exceed five trillion parameters, and the next generation will surpass ten trillion. Memory requirements have grown in lockstep, and the architecture underneath hasn’t kept pace.
When models were still relatively small, the industry found an elegant solution in high-bandwidth memory, or HBM. HBM stacks memory dies, aggregates bandwidth, and puts memory adjacent to the processor. It worked well when models were smaller, but the math has since broken down. A single HBM stack holds roughly 24-36 GB. Today's largest workloads need many orders of magnitude more memory capacity; DeepSeek v4 Pro, for example, is 1.6 TBs, Kimi K3 is 2.8 TB (at FP8), and accounting for KV Cache during run time, this memory footprint easily doubles and quadruples. In addition, all modern server architectures couple the ratio of compute to memory, meaning one cannot grow without the other. Data centers try to compensate for limited memory capacity by chaining together hundreds of processors – and attached HBM – across racks. But in doing so, HBM's core advantage, its extreme local bandwidth, gets strangled by the comparatively slow links between GPUs and machines.
A processor is only as good as the amount of data it can access fast!
There is also an economic dimension that extends well beyond the data center. The HBM supply squeeze has very real downstream effects across the entire technology ecosystem. Production lines that once made Dynamic Random-Access Memory (DRAM), or commodity memory for consumer electronics, are being reallocated to HBM production. The memory used in phones, laptops, cars, and appliances is caught in the crossfire. The result is the skyrocketing costs and memory shortage that the global economy is experiencing today. AI's infrastructure buildout is quietly taxing the whole ecosystem, straining relationships between memory vendors and the long-standing customers who built their businesses on standard DRAM.
Today’s Architecture is Out of Runway
The memory wall is not a temporary obstacle that can be solved by more investment in more of the same. It is the structural consequence of an architecture that was never meant for the scale of today, much less tomorrow’s, AI. Understanding why this is comes down to a question the industry never asked when LLMs exploded into the public consciousness in November 2022: where is AI and AI infrastructure headed? Asking that question is what convinced us “more of the same” was grossly insufficient, and led us to build something truly different: Prometheus.
Prometheus is our solution to AI’s memory wall. It’s a server and an evolution in architecture designed specifically to remove memory as a constraint from the future of AI.
This is part one on why we built Prometheus, the first AI server purpose-built to tear down the memory wall. Next: The Architecture the Industry Wasn’t Pursuing
