Issue 1, Tue 25 Aug 20265 stories

Hot Chips 2026 Day 2: AMD MI400 and NVIDIA Vera Rubin Take the Stage as llama.cpp and MoE Research Push Inference Forward

Hot Chips 2026's second day delivered architectural deep-dives on AMD's MI400 GPU for Helios racks and NVIDIA's Vera Rubin NVL72, plus Nvidia's unexpected CUDA-on-RISC-V initiative. Meanwhile, llama.cpp shipped per-device Metal flash-attention tuning for Apple Silicon, and a new arXiv framework promises up to 3.1x faster MoE inference on memory-constrained GPUs.

llama.cpp Ships Per‑Device Metal Flash‑Attention Tuning for Apple Silicon

llama.cpp releases b10615 and b10614 land on August 24 with a major Metal backend overhaul: per-device tuned flash-attention vector kernels for Q and NE widths across Apple Silicon SKUs, expanding from 80 to 133 f16 instantiations and extending tuning to quantized KV caches. The companion b10614 splits Metal kernels into 20 separately compiled libraries loaded in parallel, cutting first-load latency. For practitioners running local LLMs on Macs, these changes target the single most impactful kernel in the inference pipeline — flash-attention — with hardware-specific tuning tables that should yield measurable throughput gains on M-series GPUs.

github:ggml-org/llama.cppgithub.com

AMD MI400 GPU Architecture Revealed at Hot Chips 2026

At Hot Chips 2026, AMD presented the MI400 GPU architecture powering its Helios rack-scale AI infrastructure, detailing how the company scaled up a single die to compete with NVIDIA's largest accelerators. The MI400 (also referred to as MI455X in product context) is AMD's CDNA5-generation flagship, designed for the Helios rack ecosystem that AMD positions as its answer to NVIDIA's NVL72. For AI hardware practitioners, this is the first detailed architectural look at AMD's next-generation accelerator and its integration into a rack-scale compute fabric, which will shape the competitive landscape for large-model inference and training.

ServeTheHomeservethehome.com

NVIDIA Vera Rubin NVL72 Rack Design Detailed at Hot Chips 2026

NVIDIA used its Hot Chips 2026 session to go deep on the Vera Rubin NVL72 rack architecture, the next-generation AI factory platform succeeding the GB200 NVL72. The presentation covered the rack-level design as part of NVIDIA's integrated AI Factory strategy, with the Rubin GPU delivering what NVIDIA labels industry-leading performance metrics. For engineers tracking the infrastructure that will run the next wave of large-model training and inference, the NVL72 rack design sets the reference architecture that software and systems will need to target.

ServeTheHomeservethehome.com

SPICE: Speculative Prefetching Cuts MoE Inference Latency by Up to 3.1x

A new arXiv paper from the cs.AR listing introduces SPICE, a speculative prefetching framework for Mixture-of-Experts (MoE) inference offloading that combines lightweight expert prediction with confidence-aware CPU-GPU orchestration. When predictions hit, SPICE prefetches the right experts ahead of time; when they miss, it falls back to low-rank expert surrogates on the GPU while offloading exact residual computation to the CPU asynchronously. Evaluated on DeepSeek-V2-Lite and Qwen2-57B-A14B across diverse GPU platforms, SPICE achieves up to 3.12x speedup in Time Per Output Token with minimal quality loss. For practitioners running large MoE models on memory-constrained hardware, this offers a concrete strategy to reduce the PCIe transfer bottleneck that dominates expert loading.

arXiv cs.ARarxiv.org

NVIDIA Targets RISC‑V for CUDA at Hot Chips 2026

At Hot Chips 2026, NVIDIA revealed plans to support CUDA on RISC-V, a move that could reshape the GPU compute software ecosystem's platform reach. Chips and Cheese's technical analysis notes that CUDA remains the most important software framework in GPU compute, and bringing it to RISC-V opens a path for open-hardware architectures to participate in the CUDA software stack — though, as the analysis cautions, terms and conditions may apply. For practitioners watching the intersection of open silicon and GPU computing, this signals a potential shift in how portable CUDA-based workloads could become across non-NVIDIA accelerator architectures.