Skip to content

Prepend Windows DLL directories to PATH for cuDNN lazy loading - #3399

Open
akarale-nv wants to merge 1 commit into
NVIDIA:mainfrom
akarale-nv:cudnn-lazy-lib-resolution
Open

akarale-nv wants to merge 1 commit into
NVIDIA:mainfrom
akarale-nv:cudnn-lazy-lib-resolution

Conversation

@akarale-nv

Copy link
Copy Markdown

Description

Prepend the discovered DLL directory to the process PATH in the Windows add_dll_directory() helper. This gives it priority over existing entries during PATH-based dependent-DLL resolution, including cuDNN's lazy component loading. The existing handling of an unset PATH is unchanged; only the tuple order changes.

Issue

A PyTorch Windows pipeline failed with:

RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 27, 0) but found runtime version (9, 26, 0).

The reported environment used torch==2.16.0.dev20261105+cu132 (Python 3.12, Windows x64), nvidia-cudnn-cu13==9.27.0.42, and cuda-pathfinder==1.8.5. The reporter confirmed that placing cuDNN 9.26 on PATH reproduces the failure. Appending the discovered directory leaves those older entries ahead of it; prepending addresses that ordering for PATH-based resolution.

Validation

  • All applicable pre-commit checks passed, including Ruff, mypy-pathfinder, SPDX, and secret scanning.
  • In the supplied reproduction, a clean isolated environment loaded cuDNN 9.27 from site-packages/nvidia/cudnn/bin/cudnn64_9.dll and did not fail.
  • The equivalent one-line patch in that environment was verified to prepend the directory, and the clean-environment reproduction still passed.
  • The patched reproduction was not tested against the reporter's actual cuDNN 9.26 directory. The full smoke suite was not run.
  • Retests require a fresh Python process because DLLs already loaded remain resident.

AI Assistance Disclosure

This change and PR description were generated with AI assistance using OpenAI Codex. The reproduction details above were supplied by the contributor; the validation limits are stated explicitly.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

No documentation changes are needed for this internal PATH-ordering change. No new automated regression test was added.