Ilmu Komputer & AI editorial
Open AccessOA2023
TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
TPU v4 is Google's optically reconfigurable supercomputer for machine learning, achieving 2.1x performance over TPU v3 and 2.7x better performance/Watt. It introduces SparseCores for embedding-heavy models and uses optical circuit switches for flexible, low-power interconnects.
Norman P. Jouppi; George Kurian; Sheng Li; Peter C. Ma; R. Nagarajan; Lifeng Nai; Nishant Patil; Suvinay Subramanian; Andy Swing; Brian Towles; Cliff Young; Xiaoping Zhou; Zong-Wei Zhou; David A. Pattersonยท International Symposium on Computer Architectureยท 2023ยท DOI 10.1145/3579371.3589350
The core problem
The rapid evolution of machine learning (ML) models has led to radical and rapid changes in production workloads. To address these demands, Google developed TPU v4, its fifth domain-specific architecture (DSA) and third supercomputer for ML models. The key innovation is the use of optical circuit switches (OCSes) to dynamically reconfigure interconnect topology, improving scale, availability, utilization, modularity, deployment, security, power, and performance. Users can also choose a twisted 3D torus topology. Additionally, each TPU v4 includes SparseCores, dataflow processors that accelerate models relying on embeddings by 5xโ7x while using only 5% of die area and power. Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. The TPU v4 supercomputer is 4x larger (4096 chips) and nearly 10x faster overall, enabling large language models to train at ~60% of peak FLOPS/second. Compared to contemporary systems, it is ~4.3xโ4.5x faster than Graphcore IPU Bow and 1.2xโ1.7x faster with 1.3xโ1.9x less power than Nvidia A100. Furthermore, TPU v4s in Google Cloud's energy-optimized warehouse scale computers use ~2โ6x less energy and produce ~20x less CO2e than on-premise DSAs.
Innovation
The TPU v4 system architecture centers on optical circuit switches (OCSes) that dynamically reconfigure the interconnect topology. This optical reconfigurability allows the supercomputer to adapt to varying workload demands, improving scale, availability, and utilization. The OCSes and underlying optical components are cheaper, lower power, and faster than Infiniband, constituting <5% of system cost and <3% of system power. Each TPU v4 chip integrates SparseCores, which are dataflow processors specifically designed to accelerate models that rely on embeddings. These SparseCores provide 5xโ7x acceleration for embedding-heavy workloads while consuming only 5% of die area and power. The TPU v4 supercomputer scales to 4096 chips, 4x larger than TPU v3, and achieves nearly 10x faster overall performance. The system supports a twisted 3D torus topology if desired, and the OCS flexibility and availability enable large language models to train at an average of ~60% of peak FLOPS/second.
Introduction
The rapid evolution of machine learning (ML) models has led to radical and rapid changes in production workloads. To address these demands, Google developed TPU v4, its fifth domain-specific architecture (DSA) and third supercomputer for ML models. The key innovation is the use of optical circuit switches (OCSes) to dynamically reconfigure interconnect topology, improving scale, availability, utilization, modularity, deployment, security, power, and performance. Users can also choose a twisted 3D torus topology. Additionally, each TPU v4 includes SparseCores, dataflow processors that accelerate models relying on embeddings by 5xโ7x while using only 5% of die area and power. Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. The TPU v4 supercomputer is 4x larger (4096 chips) and nearly 10x faster overall, enabling large language models to train at ~60% of peak FLOPS/second. Compared to contemporary systems, it is ~4.3xโ4.5x faster than Graphcore IPU Bow and 1.2xโ1.7x faster with 1.3xโ1.9x less power than Nvidia A100. Furthermore, TPU v4s in Google Cloud's energy-optimized warehouse scale computers use ~2โ6x less energy and produce ~20x less CO2e than on-premise DSAs.
Why it matters
The TPU v4 architecture addresses the evolving needs of ML workloads through optical reconfigurability and specialized hardware for embeddings. The use of OCSes provides flexibility and availability, allowing the system to adapt to different model requirements and improve overall utilization. The SparseCores demonstrate a targeted approach to accelerating embedding-heavy models, which are common in large language models and recommendation systems, without incurring significant area or power costs. The performance gains over TPU v3 and competitors like Graphcore IPU Bow and Nvidia A100 underscore the effectiveness of Google's DSA design. Moreover, the energy efficiency and reduced CO2e emissions of TPU v4 in Google Cloud's warehouse scale computers emphasize the importance of sustainable computing. The ability to train large language models at ~60% of peak FLOPS/second on average showcases the practical benefits of the system's scale and optical reconfigurability. Overall, TPU v4 represents a significant advancement in ML supercomputing, balancing performance, efficiency, and environmental considerations.
Who should read this
Pembaca Research Digest dan praktisi rumpun ini.
Opening member contentโฆ