NVIDIA Investor Presentation Deck

MEGATRON SCALING ON DGX SUPERPOD ► All of NVIDIA's HW and SW working together ► CUDA, CUDA-X AI, NVSwitch, DGX SuperPod, NCCL, CUBLAS, CUDNN ► Trained with PyTorch Language models useful for NVIDIA's own products 502 petaFLOP/s sustained at 3072 GPUs 52% of tensor-core peak 163 Tflops/GPU Case 1T 530B 145B 39B 7.5B PetaFLOP/s 600 500 400 300 200 100 0 Hidden Size 25600 20480 12288 8192 4096 500 1000 Number of Layers 128 105 80 48 36 1500 2000 GPU Count Model Parallel Size 512 280 64 16 4 2500 ■ Sustained performance ----- Linear scaling 3000 Number of GPUs 3072 2520 1536 512 128 3500 NVIDIA.

View entire presentation