The AI chip sector is shifting toward a new performance benchmark as leading companies at the Hot Chips conference emphasized maximizing computing output within strict power constraints. Analysts note that "tokens per watt" has emerged as a critical metric, reflecting constrained data center capacity and surging demand for compute tokens that outstrips supply.
AMD outlined enhancements to its transcendental engine and introduced a tensor data mover to improve compute efficiency, while also partnering with Cerebras on disaggregated inference architectures. Microsoft debuted a new architecture called Software Defined Local Access Dataflow, leveraging explicit software orchestration to reduce total cost of ownership through improved energy efficiency.
Nvidia highlighted a speculative decode approach and unveiled its NVHBM technology, which is designed to deliver a 30% improvement in memory bandwidth, a 15% reduction in HBM power consumption, and a 25% increase in usable compute die area compared with standard HBM4E. The company also emphasized its focus on optimizing power efficiency amid rising data center power demands.
Cerebras is pursuing disaggregated inference through partnerships with AMD and AWS, a strategy Barclays analyst Tom O'Malley described as potentially offering "an order of magnitude improvement" in tokens-per-second-per-kilowatt. The company is also exploring stacked 3D DRAM configurations in collaboration with Samsung.
OpenAI took a different approach with its Jalapeño chip, consolidating prefill, draft and decode functions onto a single chip instead of distributing them across separate hardware. Google, meanwhile, is dual-sourcing custom chips, using MediaTek for one line and Marvell for an inference variant, while Marvell and Broadcom are leveraging their SerDes expertise and supply chain capabilities to maintain competitive positioning in the chip-on-tile model.
