The artificial intelligence chip sector is transitioning toward energy-efficient architectures as data center power and space constraints reshape priorities, according to discussions at the Hot Chips conference.
Industry executives highlighted "tokens per watt" as a critical performance metric amid surging demand for AI inference workloads and limited supply of computing capacity. Barclays analyst Tom O'Malley noted that full-system performance—measured at the rack level—has become the primary focus, with companies optimizing for power consumption and memory efficiency.
Nvidia unveiled its NVHBM memory interface, claiming up to a 30% improvement in bandwidth, a 15% reduction in power consumption compared to HBM4E, and a 25% increase in usable compute die area. The company also discussed a speculative decode approach to enhance inference efficiency.
AMD outlined enhancements to its transcendental engine and introduced a tensor data mover to improve compute performance while reducing power draw. The firm is collaborating with Cerebras on disaggregated inference solutions, which O'Malley described as potentially delivering an order-of-magnitude improvement in tokens-per-second per kilowatt.
Microsoft introduced the Software Defined Local Access Dataflow architecture, designed to lower total cost of ownership through software-driven energy optimization. OpenAI, meanwhile, took a contrasting approach with its Jalapeño chip, consolidating prefill, draft, and decode functions on a single die rather than distributing them across separate hardware components.
Google is diversifying its chip supply chain, utilizing MediaTek for one product line and Marvell for an inference variant. Broadcom and Marvell were cited for their SerDes expertise and supply chain capabilities, while Samsung is exploring 3D stacked DRAM configurations in early-stage development despite thermal and power delivery challenges.
