PyTorch 2.15 Release Blog

We are excited to announce the release of PyTorch® 2.15 (release notes)!

The PyTorch 2.15 release features the following changes:

• Rewritten NCCL backend becomes the default, with a flight recorder that dumps on hangs and TORCH_DIST_USE_NCCL2=0 or nccl-legacy as rollback paths
• One-sided transports for torch.distributed, with a NIXL backend and CUDA-graph-capturable transfers
• FSDP2 and pipelining gain finer memory and communication control, with fp32 for precision-sensitive parameters, CUDA-graph-captured gradient accumulation, and gradient reduction overlapped with compute
• Faster 4-bit (NVFP4) LLM decoding on NVIDIA Blackwell GPUs through fewer, better-overlapped kernel launches in NVGEMM
• Apple Silicon gains FP8 and faster decode, with float8_e4m3fn and _scaled_mm, a new grouped_mm with a decode fast path (1.32–1.34x tokens/s on MoE decode), and public torch.mps.Stream APIs
• Broader platform support with FlyDSL kernels on AMD gfx950, Tensor Data Mover support for gfx1250 (MI450), ROCm 10.0 wheels, simpler SYCL kernel authoring and multi-GPU profiling on Intel GPUs, CUDA 13.2 as the default wheel with CUDA 13.4 alongside it, and torch.compile on Python 3.15

This release is composed of 3,117 commits from 583 contributors since PyTorch 2.14. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.15. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.

Bring any questions you may have about PyTorch 2.15 to our Q&A webinar on Thursday, November 5, 2026, at 11 AM PT. Tristan Rice, Natalia Gimelshein, Huy Do, and Jane Xu (all Meta) will share an overview of the 2.15 release and answer community questions about PyTorch and the new capabilities in this release. Chris Gottbrath (Gottbrath Tech) will moderate. Register today.

Throughout the 2.x series, PyTorch has been evolving from a research-first framework into a unified, hardware-agnostic platform for production training and inference at scale. PyTorch 2.13 landed FlexAttention on Apple Silicon, gave Inductor a CuTeDSL code path, and introduced torchcomms for large-cluster training. PyTorch 2.14 matured that path into NVGEMM, previewed the rewritten NCCL backend, and brought native linear algebra to Apple Silicon.

PyTorch 2.15 builds on that work. The rewritten NCCL backend now backs every nccl process group, NVGEMM is tuned for low-precision decode, Apple Silicon adds FP8 and decode-specific kernels, AMD support extends to new compiler DSLs and (preview support) for next Instinct generation, and Intel GPUs gain a simpler kernel programming model and deeper profiling.

Distributed Training

The Rewritten NCCL Backend Becomes the Default

When a model trains across many GPUs, those GPUs constantly exchange data through a communication backend, and on NVIDIA hardware that backend is NCCL. In PyTorch 2.15, every job that uses nccl runs on our rewritten backend by default.

PyTorch 2.14 previewed our rewritten NCCL backend, ported from torchcomms, as the opt-in nccl2 backend. In 2.15 it is the default for every nccl process group, with TORCH_DIST_USE_NCCL2=0 and the nccl-legacy backend name as rollback paths. Communicators are now created eagerly with the process group, and reconfiguration can opt into a shrink/grow fast path (NCCL 2.29 or newer) so surviving ranks skip a full bootstrap. Reconfigure for fault tolerance is also enabled on ROCm with RCCL 2.30.7.

To make the backend debuggable in production, a backend-agnostic flight recorder writes a post-mortem to disk when a collective hangs, and watchdog health is published to the debug server so a timeout on any rank triggers flight-recorder dumps from every rank. At cluster scale, a single stalled GPU can halt an entire training run, and finding the cause has often meant reproducing the failure by hand. The flight recorder captures what each rank was doing at the moment of the hang.

API Unstable

(PRs #192281, #192895, #199240, #198865 and #198339 by Tristan Rice, Meta; #194774 by Chinmay Kuchinad, AMD; #192233 and #199092 by Tushar Jain, Meta; #192075 by Atharva Kshirsagar, Red Hat)

One-Sided Transports with a NIXL Backend

torch.distributed gains a Python transport API for one-sided data movement, with NIXL as the first backend. Transports connect to a peer through a store, without needing a process group or collectives, and Transport.read_stream/write_stream enqueue transfers on a CUDA stream so they can be captured with torch.cuda.graph.

API Unstable

(PRs #195643, #197032, #198889 and #198918 by Tristan Rice, Meta)

FSDP2 and Pipeline Parallelism: Memory and Communication Control

FSDP2 now gives finer control over precision and capture. Precision-sensitive parameters, such as MoE router gates and norm weights, can stay in fp32 while the rest of the model trains in bf16, still with one all-gather per fully_shard (MixedPrecisionPolicy(param_dtype_override_fn=...)). A full gradient-accumulation step can now be captured in a single CUDA graph. Gradient precision can also be set per parameter. Gradients accumulate directly in .grad and follow each parameter’s grad_dtype, which changes the default behavior when reduce_dtype=None (see Deprecations and Backwards-Incompatible Changes).

Pipeline parallelism now allocates receive buffers just before each receive instead of for every microbatch up front, and releases forward outputs once their sends complete. Schedules accept a max_outstanding_sends budget, and FSDP unshards are staggered across each rank’s startup bubble instead of all being issued at the start of a step. Each unique edge between pipeline-parallel ranks, as determined by the schedule, now gets a dedicated P2P communicator. FSDP reduce grads can now be overlapped with subsequent compute. These improvements make PyTorch pipelining cudagraphable, deterministic, with an optimized memory profile, and expose performance tuning knobs for minimizing exposed communication.

API Unstable

(PRs #196640, #196724, #196727, #196641 and #196963 by Chien-Chin Huang, Meta; #196626 by Anshul Sinha, Meta; #198417 by Andrew Or, Apache Spark PMC); #194114 by Animesh Jain, Meta; #196463, #196464 and #196706 by Sanket Purandare, Meta)

Performance Improvements

Faster MPS Kernels for Decode and Linear Algebra

Building on the 2.14 migration, more MPS operators move from MPSGraph to native Metal kernels. A single-row softmax over a 128,256-entry vocabulary drops by 8x, upsample backward is up to 21.7x faster, and reductions now share one set of kernels, making norm full reductions more than 100x faster.

For decode, F.linear now uses the existing gemv kernels (1.23x geometric-mean speedup, up to 5x on small projections), and the new MPS grouped_mm has a gemv path that gives 1.32–1.34x higher MoE decode throughput on an M3 Ultra. Several linear algebra solvers were also rewritten: QR of a 512 x 512 matrix falls from about 7 seconds to 1.5 ms, and triangular solves are now blocked.

(PRs #196071, #199687, #198611, #198495, #194933, #198721, #198725, #198732, #195944 and #196900 by Irakli Salia, Hugging Face; #196922, #196241 and #195369 by Nikita Shulga, Thinking Machines Lab; #193153 by Sunghyun Choi, Tech University of Korea; #188473 by Shrivas Shankar, University of Wisconsin-Madison)

Core Features

CUDA 13.2 by Default and CUDA 13.4 Wheels

The default PyTorch wheel moves to CUDA 13.2 (13.2.2 toolkit), with CUDA 13.4 GA wheels shipping alongside it for Linux x86_64, Linux aarch64 and Windows. The CUDA 13.4 wheels add experimental support for NVIDIA Rubin GPUs.

(PRs #198913 and #196014 by Andrey Talman, Meta; #194996 and #198843 by Ashish Karale, NVIDIA; #196560, #199269 and #196351 by Ting Lu, NVIDIA; #192653 by Wei Wang, NVIDIA; #186245 by Lakshay Garg, NVIDIA)

ROCm 10.0 and TheRock Wheels

PyTorch 2.15 adds ROCm 10.0 wheels alongside the ROCm 7.14 wheels introduced in 2.14. All ROCm wheels are now built on AMD’s TheRock packages, and they install ROCm as a pip dependency (the matching rocm package) instead of bundling the ROCm libraries.

(PRs #194919, #198895 and #196610 by Ethan Wee, AMD; #195672 by Andrey Talman, Meta)

Python 3.11 to 3.15, and torch.compile on Python 3.15

PyTorch 2.15 supports Python 3.11 through 3.15, including the free-threaded 3.14t and 3.15t builds. In 2.15, Dynamo adds Python 3.15 support.

(PRs #178393, #194117 and #198018 by Rob Timpe, OpenTeams; #198233, #199398 and #199356 by Andrey Talman, Meta)

Platform Features and Updates

CUDA

NVGEMM for NVFP4 Decode

NVGEMM, the CuTeDSL GEMM backend added in 2.14, now folds the scalar dequantization factor of ModelOpt NVFP4 linear layers into the GEMM, removing a separate pointwise launch per layer during decode. A bounded, shape-aware candidate policy covers Blackwell NVFP4 decode shapes that nvMatmulHeuristics misses, and NVGEMM kernels can use programmatic dependent launch so a Triton consumer overlaps with the tail of the producer GEMM. NVGEMM also moves onto CUTLASS Operators.

API Unstable

(PRs #194651, #194652, #197485, #194792 and #194793 by Michael Lazos, Meta; #194364 by Masaki Kozuki, NVIDIA)

ROCm

FlyDSL Kernels on gfx950 and Tensor Data Mover on gfx1250

FlyDSL joins CuTeDSL, Triton and Helion as a kernel DSL that PyTorch can use. On gfx950, Inductor autotunes FlyDSL templates for GEMM, ragged F.grouped_mm and MXFP4/MXFP8 scaled_mm alongside Triton and ATen, and in eager mode FlyDSL can override RMSNorm and topk. FlyDSL is optional and requires FlyDSL 0.3.x.

On gfx1250 (MI450), Inductor uses the Tensor Data Mover for dense mm/addmm, pointwise and reduction codegen, and FlexAttention forward and decode. gfx1250 is added to the nightly wheel architecture list.

API Unstable

(PRs #192877, #190903, #194032, #191446 and #191447 by Xiaobing Zhang, AMD; #194981 by Yutao Xu, AMD; #196719 by Heng Yi, AMD; #191166, #196525 and #197329 by Glen Cao, AMD; #196610 by Ethan Wee, AMD)

MPS (Apple Silicon)

FP8, Streams and faster MoE inference on Apple Silicon

torch.float8_e4m3fn is now supported on MPS, along with _scaled_mm with tensorwise or rowwise scaling and a gemv fast path for decode-sized shapes. MPS also gains a public stream API modeled on CUDA’s: torch.mps.Stream, current_stream, set_stream, the torch.mps.stream context manager, and Tensor.record_stream.

API Unstable

(PRs #194074, #196916, #197862, #196945, #196918 and #196955 by Irakli Salia, Hugging Face; #191415, #193464 and #195867 by Kurt Mohler, OpenTeams; #196395 by Aditya Venkataraman, Meta)

XPU (Intel GPUs)

SYCL Kernel Programming Simplified

This release introduces SYCL free-function kernel support for the XPU backend, with most existing SYCL kernels already migrated to the new programming model. By replacing verbose functor-based implementations with explicit free-function kernels, developers can write device code in a more intuitive way, lowering the barrier to developing and porting custom GPU kernels for Intel XPUs.

(PRs #196387, #196932, #198200, #198720, and #200223 by Yifeng Cui, Intel)

Enhanced XPU Runtime Capabilities

This release expands XPU runtime capabilities with IPC support for torch.xpu.Event, enabling event sharing across processes and integration with torch.multiprocessing. In addition, torch.xpu.MemPool now supports use_on_oom and non_split allocation policies, providing greater flexibility for applications that need to handle out-of-memory conditions or leverage distributed symmetric-memory workflows.

API Unstable

(PRs #191182, #192495, #192514, #192038 and #192033 by Guangye Yu, Intel)

Improved XPU Profiling Experience

This release adds kernel-level visibility into XPU Graph replay, cross-rank correlation for XCCL collectives, and per-kernel hardware counter collection across all Intel GPUs in multi-GPU workloads. These enhancements enable deeper performance analysis for graph-based, multi-GPU, and distributed training and inference. 

(PR #195128 by Chuanqi Wang, Intel; PR #5486 and #1479 by Adrian Ostrowski, Intel) 

C++ ABI

Expanded Stable ABI and Header-Only Surface

torch::stable::sort joins the stable ABI, the Python interop shims gain dtype/device conversions and an is_tensor probe alongside the existing tensor conversions. More of c10/util (irange, load, math constants, type casting and overflow checks) moves into torch/headeronly, with the old paths kept as forwarding includes. Extensions now get a compile-time error if TORCH_TARGET_VERSION is newer than their libtorch headers, and load_inline can generate pybind11 modules that are safe to import on free-threaded Python.

(PRs #198038 by Adrian Abeyta, Red Hat; #194673, #194781 and #193962 by Chris Leonard, Red Hat; #192774 by Paweł Gadziński, NVIDIA; #196802 by Alex Malyshev, Meta)

Profiling and Debugging

CUDA Graph Kernel Attribution with Cuspy

Kernels replayed from a CUDA graph can now be attributed back to the code that captured them. Kernel annotations are recorded from CUPTI node-creation callbacks and written inline by export_chrome_trace, and with record_py_stacks CUDA graphs capture the Python launch stack for each node. The CUPTI monitor is renamed Cuspy and is enabled with a CuspyConfig.

API Unstable

(PRs #191999, #194854, #196528, #198419 and #196529 by Natalia Gimelshein, Meta; #196066 by Ryan Zhang, Meta)

Deprecations and Backwards-Incompatible Changes

• Python 3.10 is no longer supported. PyTorch 2.15 requires Python 3.11 or newer: 3.10 wheels are no longer built, and 3.10 is no longer tested in CI. Building from source on 3.10 is not supported, and upcoming releases will remove the remaining 3.10 code paths, including in torch.compile (Dynamo). See #198233 and #199417.
• nccl now selects the rewritten NCCL backend. Set TORCH_DIST_USE_NCCL2=0 or use the nccl-legacy backend to restore the legacy ProcessGroupNCCL. See #192281.
• C++ warnings from torch.ops and custom ops now propagate to Python. See #193451.
• macOS 14 is no longer supported. PyTorch 2.15 requires macOS 15 or newer for MPS (Apple Silicon GPU) acceleration. macOS 14 has reached end of life, and the MPS workarounds for it have been removed. PyTorch may still run on macOS 14, but it isn’t tested there and issues on it won’t be fixed. See #199261.
• FSDP2 separates sharded and unsharded gradient dtypes. See #194434.

The post PyTorch 2.15 Release Blog appeared first on PyTorch.